As the first country to have an AI Strategy and the first to develop a policy on responsible use of AI for government, Canada has been on the leading edge of efforts to understand the important shift technology and automation are creating in on our society. At the cornerstone of all these efforts has been an open process with engagement from industry, academia, and civil society. As the co-founders of the Global Partnership on AI along with renowned experts constituting their AI Advisory Council Canada has demonstrated a strong commitment to understanding how AI is going to re-shape society, and what regulators need to do to help ensure it is not only a smooth transition, but that AI is used as a force for good.
Following with this approach, the Office of the Privacy Commissioner recently held a consultation to seek advice from experts on how to reform privacy legislation in the age of AI. The report and more information about the consultation can be found here. Since their questions were directly related to our work at AI Global, we wanted to take the time to participate in the consultation, but also share our thoughts with our community!
Warning, it’s a long post, we had lots to say! If you have any questions or comments, please feel free to reach out to us at any time.
Response to Questions
Proposal 1: Incorporate a definition of AI within the law that would serve to clarify which legal rules would apply only to it, while other rules would apply to all processing, including AI
- Should AI be governed by the same rules as other forms of processing, potentially enhanced as recommended in this paper (which means there would be no need for a definition and the principles of technological neutrality would be preserved) or should certain rules be limited to AI due to its specific risks to privacy and, consequently, to other human rights?
We don’t believe that it is necessary or even possible to have a comprehensive definition for AI as many technologies adhere to the broad definitions that are referenced in the report. From an implementation perspective, it would be incredibly difficult to build and maintain an authoritative inventory or comprehensive assessment to understand which systems/ solutions are in scope for any types of legal rules that it is being applied to.
This not only makes it difficult for the owner of the legal rule but it also makes it difficult for those building and implementing these tools who want to be respectful of the laws. Furthermore, it would create significant burden to identify the scope vs what the desired outcome of the legal rule is.
An alternative recommendation is to take a similar approach to what we did with the Directive on Automated Decision Making Systems. With this Directive, we focused on the outcome of the processing (ie. automation of decision making by a machine in part or whole) as opposed to the technology used to fulfill that action. This approach allows for both existing and emerging technologies to be covered by the legal rules.
That said, there is a significant difference between “AI” systems and traditional types of processing, especially when there are instances where the system is learning. This not only requires different types of rules and oversight, but does require existing legal rules be reviewed and adapted accordingly. This can still be done when looking at the intent not just the technology.
- If certain rules should apply to AI only, how should AI be defined in the law to help clarify the application of such rules?
As we do not believe that AI should have a specific definition, we will not comment on this, however, we believe that new legal rules are required to understand the increased use of automated processing independent of whether or not there is a clear definition of AI.
We would recommend that within the support material (glossary, guidance, etc) it would be ideal to highlight various categories of AI technologies so there is a clear understanding of which types of technologies and their potential use and outcomes could be included and differ from traditional processing. This could include image and object recognition, text to speech analysis, risk assessment, content generation, or process optimization.
Proposal 2: Adopt a rights-based approach in the law, whereby data protection principles are implemented as a means to protect a broader right to privacy—recognized as a fundamental human right and as foundational to the exercise of other human rights
- What challenges, if any, would be created for organizations if the law were amended to more clearly require that any development of AI systems must first be checked against privacy, human rights and the basic tenets of constitutional democracy?
While this is an incredible concept that we fully support, however, it will be very important to think through how such an evaluation would work in practice.
First and foremost, it will be imperative to provide concrete and clear definitions for privacy, human rights, and other protections afforded to individuals in constitutional democracies.
Second, it will be important to understand to what extent these rights need to be protected. While this will be reliant on the first point of having clear and concreted definitions, there will also need to be standards or thresholds of acceptability to be set for potential violations. Contextual considerations like domain and jurisdiction will need to be taken into account as relevant legal rules at the provincial, territorial, and municipal level will be implicated as well.
Third, it will be important to understand who is being impacted, is it an individual, group, or organization or a multiple of the above.
When all of these considerations are understood there will be a need to harmonize and prioritize for the particular desired outcome of the legal rule and/ or for the particular use case. As currently is the case with non-automated processing, certain trade-offs need to be made. Sometimes these trade-offs are left to the human operator of the system, however, when there is end-to-end automation, these implications and potential unintended consequences need to be thought through from the onset of the project and then coded in to the function of the system. As such, legal rules will have to set out clear guardrails and acceptable practices for these inevitable trade-offs.
We would recommend that the OPC makes it mandatory to consult with companies, civil society organizations, other governments, and the public when assessing the development of new legal rules or the revision of existing legal rules. It is too difficult to provide all of the different scenarios in a high-level consultation related to the various trade-off decisions which would need to be made.
Lastly, there should always be an aim to be as practical as possible when implementing these rules, meaning that there is a general desire to use these tools to enhance how the public accesses services or experiences day to day activities, this may come at the expense of sharing more information than they previously have with service providers.
We are of the opinion that these different uses should be contextualized based on potential risk. For example, recommending a song through a music app is not the same degree of risk that profiling someone for a potential crime is. However, both of these examples could use an AI system to achieve that outcome. Therefore, blanket legal rules can become burdensome and overly stringent in implementation.
Proposal 3: Create a right in the law to object to automated decision-making and not to be subject to decisions based solely on automated processing, subject to certain exceptions
- Should PIPEDA include a right to object as framed in this proposal?
Yes, this will be important to ensure the individual’s rights are being protected throughout the entire lifecycle of a decision or process.
There are certain exceptions that need to be made, Article 22 of GDPR highlights some of these exceptions, however, additional nuance may be required. For example, when a person is involved in an insurance claim that they deem unfair, is this individual considered as engaged in a contract and has no recourse? Especially where an individual’s rights are in question, there should be the option to have an non-automated review process to compliment the automated review process if requested.
- If so, what should be the relevant parameters and conditions for its application?
Understanding what the parameters should be will depend on the context of the situation. It will be even more crucial to have a recourse mechanism and a human-in-the loop as part of these automated decision processes where there are more potential for bias including, but not limited to the following areas: sexual orientation, ethnicity, cultural associations, religious affiliations, age, mental health, gender, socio-economic capacity, physical attributes, level of education, political affiliation, degree of ability.
Proposal 4: Provide individuals with a right to explanation and increased transparency when they interact with, or are subject to, automated processing
- What should the right to an explanation entail?
Once again, it is difficult to assert broad-based rules without being context specific. Lots of research has been done on different approaches to explainability for various different types of systems. In general, we believe that understanding how the system operates is important and the creator and operator is able to defend how an outcome was arrived at with a degree of confidence. Interpretability, a post-facto review of the system can also be acceptable in many situations and contexts.
Based on the potential risk of the system, different degrees of explanation will be dependent on the model, the use, the users, and the jurisdiction the AI system is operating in. Full explainability or interpretability of will be difficult and not necessarily needed for all circumstances. However, for higher risk AI systems where human rights could be at risk it will be essential.
In addition to requiring those developing these systems to understand in most circumstances how the system operates, it will also be important to be in a position to share this information with the user and regulators on an as needed basis. This information could include an understanding of how the system renders a decision or performs a function, the inputs of the model, the data sources, the development process, and stakeholders who were consulted in the development.
With this, it is important to note that there are circumstances where this information might be proprietary for the creator, given Canada’s desire to promote commercialization of AI and be a leader in the global AI community, it will be important to continue to balance these aspects as well.
One of the key reasons for requiring explanation is to enhance and improve users trust of these systems. Other important things to consider that can be included in new legal rules includes, having easy to understand terms and conditions for the operation of the system, evidence that there was an appropriate review of the training data used (both for quality and diversity), and what variances in the model exist or are able to be changed with user input.
Overall, we feel that Citizens should have a right to know all of the data sources and criteria being used by autonomous processes. This gets at the heart of the “black box algorithm” problem.
It is important that within each of these contexts, things like the “right to explanation” should identify who within an organization’s governance structure is responsible for providing this explanation. Based on the context, thresholds of acceptability should also be put in place looking at things like the time it takes for the operator to respond, (eg. will an explanation be provided within 5,10, 30 days?)
- Would enhanced transparency measures significantly improve privacy protection, or would more traditional measures suffice, such as audits and other enforcement actions of regulators?
Transparency is key for improving privacy protection and promoting trust overall. However, transparency is only one aspect that enables this trust to grow. Furthermore, it is important that the transparent activity also has meaning to the intended audience, whether it is a user of a service or a regulator providing oversight of a legal rule. For example, the machine readable data with an individual’s location could be released by a telecom company, but if the user doesn’t have the knowledge to interpret the machine readable data, then this isn’t a useful transaction.
We believe that aspects of transparency need to be included in traditional and non-traditional oversight mechanisms. Audits combined with enforcement actions can be an excellent model, however, the audit needs to be designed in a way that factors in not just transparency, but the evaluation of one’s rights, the efficacy of the data, and the governance controls to ensure the appropriate operation of the system.
This is why it is important to understand what the desired intent is in a given scenario, and oversight measures, traditional and emerging can be leveraged effectively.
Proposal 5: Require the application of Privacy by Design and Human Rights by Design in all phases of processing, including data collection
- Should Privacy by Design be a legal requirement under PIPEDA?
Yes, we believe that Privacy by Design and in a broader sense, Responsibility by Design is imperative. How audits and most regulations are built are only evaluating a system, tool, or activity after development, or after a harm has been incurred. There is a lot of great work that has been done to support those developing, designing, and deploying these systems to mitigate any potential issues at the onset of a project. Being proactive from the start can mitigate a lot of potential unintended consequences.
This type of activity can be found in other sectors, like manufacturing for medical devices where testing is required before there is an acceptable entrance into the market.
At AI Global, we have built a Responsible AI Trust Index incorporating the most referenced best practices and principles in this domain culminating in easy to use evaluation. Supporting a Responsible by Design Approach, the Responsible AI Trust Index is available in an open source tool as an open virtual assessment to help those designing, developing, and deploying AI systems anticipate problems and future-proof their AI system. This tool was built by subject matter experts in industry, academia, civil society, and government to translate best practices in a way that translates responsible AI to practitioners. The Responsible AI Trust Index looks at data rights and use, privacy, security, explainability, fairness, bias, and robustness.
AI Global is working to test these different practices with businesses and governments who are actually using these tools. Taking an evidence based approach to continue to develop the Trust Index. We envision that this work will ultimately support a regulatory sandbox.
- Would it be feasible or desirable to create an obligation for manufacturers to test AI products and procedures for privacy and human rights impacts as a precondition of access to the market?
Yes, but it would have to be fairly straightforward to use. One of the things that we found when developing the Directive on Automated Decision Making is that the Algorithmic Impact Assessment (AIA) was an easy to use tool that often prompted them to think about requirements that they hadn’t previously thought about. Many of the questions were built from existing legal rules or policies that they weren’t always aware existed or they thought to need to adhere to in their current context.
This said, there were still gaps in this process, which is why through AI Global I am building off of this work, and we are developing a private certification program to address these challenges. This activity will be nuanced by industry and jurisdiction as will include traditional and non-traditional evaluation mechanisms. This work is being developed in a multi-stakeholder forum to ensure that the appropriate subject matter experts are engaged as the proper assessment of AI requires a multi-disciplinary approach given the amount of industries and applications AI can be used for.
This work is broken down into an open source evaluation, the Design Assistant to help those building these systems to follow a Responsible by Design approach.
Additional comments: Since the AIA was raised in this question, it’s important to note that the AIA only assesses the potential risk of a project, as such, scaling the requirements within the Directive. This is an important distinction as it is unique to other policies that provide oversight over “AI” systems. The scaling allows for there to be a lowered barrier to entry (less stringent oversight and compliance requirements) so that innovation for lower-risk projects doesn’t have the same treatment as higher-risk projects. This is incredibly important to balance to support the advanced use of technology in the Canadian ecosystem with the protection of the public for the projects that really matter. Audits to ensure companies are truthful on an assessment like the AIA will be important as part of a fair ecosystem.
Proposal 6: Make compliance with purpose specification and data minimization principles in the AI context both realistic and effective
- Can the legal principles of purpose specification and data minimization work in an AI context and be designed for at the outset?
Yes, this is a common concept used in Data Architecture as well. That data should be fit for the intended purpose and that only the data that is absolutely necessary to perform the function or processing should be collected, managed, and stored. Additionally, data governance looks at the necessary life cycle of the data and determines how long data should be retained for and when it should be disposed of. Given that AI is powered by data, good data architecture and governance practices must be followed throughout the lifecycle of an AI system.
It’s important to recognise that many companies who have developed business models that are reliant on the collection and re-selling of data, so they could argue that data minimization practices would hinder their business. While this is true, the developers of any legal rules should anticipate these challenges, which could force the market to shift their business practices. There may be a transition period required for the market to adjust. This is similar to why the Directive on Automated Decision Making went into force on April 1, 2019, but compliance isn’t required until April 1, 2020.
- If yes, would doing so limit potential societal benefits to be gained from use of AI?
This is a possibility, if the same definition for Private Information that CCPA has used “information that identifies, relates to, describes, is capable of being associated with, or could reasonably be linked, directly or indirectly, with a particular consumer or household. 1798.140(o)(1-2)” This could create significant challenges as many systems are not developed to track a user’s data once it is aggregated and then used for different purposes within the business. This is why understanding the intended use and the corresponding potential harms throughout the lifecycle of the project should be factored in before broad definitions like these are used.
The societal benefits could not only impact access to services, but could also hinder the advancement of these technologies. We believe that it is important to develop a positive environment where technologies are used, because if they are prohibited from being used in Canada due to strict legal rules, they will still be used in other jurisdictions and potentially have an impact on Canadians. These technologies are eroding conventional legislative boundaries, therefore, Canada needs to think about the oversight of these tools in a global context, while upholding Canadian values to the greatest extent possible.
Another potential societal impact that could be created due to data minimization is related to training data. In the same way that there is the potential for over collection of data, there could also be an under collection when considering training data. Data collected for one region might not always translate appropriate to another region, let alone use. Designers and developers need to be cognisant of what the appropriate thresholds of data use are. For example, data used for automated vehicles, might not easily translate if training data collected in a European context was applied to automated vehicles being operated in Canada. This example could be extended to other situations like the use of facial recognition in facial management, if the training data used to train the algorithm was developed in a homogeneous country and applied to Canada which has a more diverse population, there could be potential restrictions for faces that are not as well recognized due to the sample data. If used in other contexts, this could have safety, security, and potential health risks.
- If no, what are the alternatives or safeguards to consider?
Important safeguards to consider generally include, but are not limited to:
- The collection of private information only takes place if a user consents or for authorized surveillance purposes.
- Data related to personal thoughts and emotions are not used in situations where the system could cause harm, especially in circumstances where moral judgements (eg. lifestyle choices could be made).
- Users are able to disconnect or stop sharing information with the system at any point in time. If a disruption in service occurs, data collection is a direct dependency vs an indirect dependency.
- Only users have the ability to set profile preferences, changes to these preferences can easily be done at any time.
- Access to personal information is limited to only individuals who require it for the direct functioning of the system.
- Individuals have the ability to access their personal data including, but not limited to, the collection, use, and sharing of this data at any time.
- Individuals have the ability to donate their personal data to research organizations.
- Data integrity is assured. The system does not use private data to imitate or alter a person’s appearance, voice, or other individual characteristics in order to damage one’s reputation or manipulate other people.
- The system does not curtail people’s real or perceived liberty.
- If data can’t be minimized sensitive data is not able to be viewed by the user (or authorized users)
Proposal 7: Include in the law alternative grounds for processing and solutions to protect privacy when obtaining meaningful consent is not practicable
- If a new law were to add grounds for processing beyond consent, with privacy protective conditions, should it require organizations to seek to obtain consent in the first place, including through innovative models, before turning to other grounds?
- Is it fair to consumers to create a system where, through the consent model, they would share the burden of authorizing AI versus one where the law would accept that consent is often not practical and other forms of protection must be found?
- Requiring consent implies organizations are able to define purposes for which they intend to use data with sufficient precision for the consent to be meaningful. Are the various purposes inherent in AI processing sufficiently knowable so that they can be clearly explained to an individual at the time of collection in order for meaningful consent to be obtained?
- Should consent be reserved for situations where purposes are clear and directly relevant to a service, leaving certain situations to be governed by other grounds? In your view, what are the situations that should be governed by other grounds?
- How should any new grounds for processing in PIPEDA be framed: as socially beneficial purposes (where the public interest clearly outweighs privacy incursions) or more broadly, such as the GDPR’s legitimate interests (which includes legitimate commercial interests)?
- What are your views on adopting incentives that would encourage meaningful consent models for use of personal information for business innovation?
Combined response: We believe that new ways of obtaining consent will be required as meaningful consent provided through the reading of long and difficult to understand terms of conditions is not currently protecting individuals. It is also important to think about an individual’s consent throughout the lifecycle of an initiative. If data that is being collected for one purpose is then being used for another service, either sold, or used by the existing company we believe that this is something that should be made clear to the individual at initial use. If terms change throughout the use of the service or product, then the user should be notified in an easy to understand way. In circumstances where anonymized data is being collected in an aggregated way (eg. monitors in smart cities) there should still be awareness provided to the user that their data is being collected, and what the potential implications are to them.
Most importantly, in all situations possible, there should be an appropriate alternative if the individual does not want to consent to their data being collected and used. If the collection of their data is non-essential for the use of the product, then there shouldn’t be a disruption in service, or there is a limited disruption of the service if their personal data is required for a specific aspect of that service. However, as much as possible, these tools should be developed in a way to mitigate the dependency on an individual’s personal data. Once again, we recognize that this is highly use case specific.
Consent is where citizens need the most protection. In most cases there is an information asymmetry in favor of the organization over the individual. Only the organization has full visibility (and understanding) of its present and future uses and therefore has the affirmative burden of explaining their intentions to citizens to ensure understanding before gaining/accepting consent from citizens.
Proposal 8: Establish rules that allow for flexibility in using information that has been rendered non-identifiable, while ensuring there are enhanced measures to protect against re-identification
- What could be the role of de-identification or other comparable state of the art techniques (synthetic data, differential privacy, etc.) in achieving both legitimate commercial interests and protection of privacy?
We support this approach as it balances advanced use of innovation with protection of the public.
- Which PIPEDA principles would be subject to exceptions or relaxation?
As mentioned above, depending on the context including the intended users of a system as well as the potential risk of unintended consequences, different principles like consent could be relaxed, or even identifying purposes if there are multiple uses and good rationale for why the same data or consent works in these different scenarios.
- What could be enhanced measures under a reformed Act to prevent re-identification?
Oversight will be required as these techniques have not yet been mastered in all contexts. Increased data governance practices need to be implemented by those who are overseeing these legal rules, as re-identification is often possible when data from various systems is combined. This was a finding in the ISO/IEC Technical Report 24028, clause 10.6 which recognizes that syntactic methods (such as k-anonymity) or semantic methods (such as differential privacy), used to de-identify personal data, present a risk of re-identification when personal data is available from multiple sources. However, TR 24028 asserts that “regardless of the initial de-identification approach, it is possible to manage the residual risk of re-identification with data-usage agreements between the parties receiving the data.”
This can be mitigated with good data governance practices like the establishment of a data trust. While they can range in their effectiveness depending on how they are established, data trusts are a good approach to have data be stored and managed and only used interacted with for validation when required.
Proposal 9: Require organizations to ensure data and algorithmic traceability, including in relation to datasets, processes and decisions made during the AI system lifecycle
- Is data traceability necessary, in an AI context, to ensure compliance with principles of data accuracy, transparency, access and correction and accountability, or are there other effective ways to achieve meaningful compliance with these principles?
Related to the response for proposal 4, explainability is required through all aspects of the life cycle, especially as it relates to the data used for training and operating an AI system. Understanding how data is collected, stored, managed, moved, sold, and disposed is essential to both the developers and operators of the system, however, it can be equally important to the individuals using a system, especially for high-risk applications. Likewise, understanding the origin of how the algorithm was built, trained, and is being managed, including the ongoing monitoring of a system is equally as important.
For proposal 9, the report cites several aspects of data and algorithmic traceability including, accuracy, transparency, access, and correction. In order for all of these components to be possible, it is essential for the operator of the system to understand the origin and/ or lineage of the data being used. To ensure accuracy, whether the data is being collected by the operator, or purchased from a third party, as per the examples above related to differences in use of training data vs operation of the system it is important that the data is fit for purpose as well as of high quality in a way that is relevant for the context of the processing.
When data is in transit, some things to consider include, standards for cryptography or security are followed, data is encrypted when in transit or rest, data is anonymized and aggregated through incoming channels, users are notified when personal information is being collected, used, or shared, and where possible systems leverage on-device processing.
Access and correction are only possible if the systems is aware when a particular users data is being used and there is a notification for that user to make those changes, either during the processing or after the processing occurs. In order to do this, a system needs to be built in a way that allows for that connection and ultimate notification to be made, as well as a plan for the ongoing monitoring of the system to ensure that the system is continuing to operate as expected.
In the eventuality of a system not operating as expected, there is a need for contingency planning for the operation. All of this work is dependent on having a robust governance and risk management practice in place.
Proposal 10: Mandate demonstrable accountability for the development and implementation of AI processing
- Would enhanced measures such as those as we propose (record-keeping, third party audits, proactive inspections by the OPC) be effective means to ensure demonstrable accountability on the part of organizations?
- What are the implementation considerations for the various measures identified?
- What additional measures should be put in place to ensure that humans remain accountable for AI decisions?
Yes, we believe that to properly address these concerns enhanced measures need to be taken. In Annex A we’ve included an initial whitepaper on the work required to develop an independent certification process which could be used to audit this work.
Given the scope, scale, and nature of technology that crosses traditional geographic borders we see how an independent governance scheme can support the evaluation of these proposed legal rules.
We recommend that each organization should have an individual (or department) in their organization responsible for certifying human accountability for AI led decisions. A good analog for this function would be the role that Transfer Agents play in the financial services industry. Transfers agents are regulated by the Securities and Exchange Commission (SEC) and provide a crucial recording and regulatory role and enhance trust in the financial system.
Proposal 11: Empower the OPC to issue binding orders and financial penalties to organizations for non-compliance with the law
- Do you agree that in order for AI to be implemented in respect of privacy and human rights, organizations need to be subject to enforceable penalties for non-compliance with the law?
Yes, these legal rules need to have appropriate measures of enforcement including fines for infractions.
- Are there additional or alternative measures that could achieve the same objectives?
We believe that the majority of organizations want to develop AI systems and tools in a way that protects the public while maintaining a viable business. Many of these systems have been developed in a way that is heavily depended on the collection, re-use, and re-selling of data. Providing incentives for different types of business models will be essential to the evolution of these tools so that the protection of the public is balanced with innovation.