Introduction to AI Bias

By Nidhi Singh, CCG

Note: This article is adapted from an op-ed published in the Hindu Business Line which can be accessed here

A recent report by Nasscom talks about the integrated adoption of artificial intelligence (AI) and data utilisation strategy, which can add an estimated USD 500 billion to the Indian economy. In June 2022, Meity published the Draft National Data Governance Framework Policy, which aims to enhance the access, quality, and use of non-personal data in ‘line with the current emerging technology needs of the decade.’ This is another step, in the world-wide push by governments to adopt machine learning and AI models, which are trained on individuals’ data, into the sphere of governance. 

While India is currently considering the legislative and regulatory safeguards which must be implemented for the use of this data and its use in AI systems, many countries have begun implementing these AI systems. For example, in January 2021, the Dutch government resigned en masse in response to a child welfare fraud scandal that involved the alleged misuse of benefit schemes. 

The Dutch tax authorities used a ‘self-learning’ algorithm to assess benefit claims and classify them according to the potential risk for fraud. The algorithm flagged certain applications as being at a higher risk for fraud, and these applications were then forwarded to an official for manual scrutiny. While the officials would receive applications from the system stating that they had a higher likelihood of containing false claims, they were not told why the system flagged these applications as being high-risk. 

Following the adoption of an overly strict interpretation of the government policy on identifying fraudulent claims, the AI system being used by the tax authorities began to flag every data inconsistency — including actions like failure to sign a page of the form — as an act of fraud. Additionally, the Dutch government’s zero tolerance for tax fraud policy meant that the erroneously flagged families would have to return benefits not only from the time period in which the fraud was alleged to be committed but up to 5 years before that as well. Finally, the algorithm also learnt to systematically identify claims which were filed by parents with dual citizenship — as being high-risk. These were subsequently marked as potentially fraudulent. This meant that out of the people who were labelled as fraudsters by the algorithm, a disproportionately high number of them had an immigrant background. 

What makes the situation more complicated is that it is difficult to narrow down to a single factor that caused the ‘self-learning algorithm’ to arrive at the biassed output due to the ‘black box effect’ and the lack of transparency about how an AI system makes its decisions. This biassed output delivered by the AI system is an example of AI bias.

The problems of AI Bias

AI bias is said to occur when there is an anomaly in the output produced by a machine learning algorithm. This may be caused due to prejudiced assumptions made during the algorithm’s development process or prejudices in the training data. The concerns surrounding potential AI bias in the deployment of algorithms are not new. For almost a decade, researchers, journalists, activists, and even tech workers have repeatedly warned about the consequences of bias in AI. The process of creating a machine learning algorithm is based upon the concept of ‘training’. In a machine learning process, the computer is exposed to vast amounts of data, which it uses as a sample to study how to make judgements or predictions. For example, an algorithm designed to judge a beauty contest would be trained upon pictures and data relating to beauty pageants from the past. AI systems use algorithms made by human researchers, and if they are trained on flawed data sets, they may end up hardcoding bias into the system. In the example of the algorithm used for the beauty contest, the algorithm failed its desired objective as it eventually made its choice of winners based solely on skin colour, thereby excluding contestants who were not light-skinned.

This brings us to one of the most fundamental problems in AI systems – ‘Garbage in – Garbage out’. AI systems are heavily dependent on the use of accurate, clean, and well-labeled training data to learn from, which will, in turn, produce accurate and functional results. A vast majority of the time in the deployment of AI systems is spent in the process of preparing the data through processes like data collection, cleaning, preparation, and labeling, some of which tend to be very human-intensive. Additionally, AI systems are usually designed and operationalised by teams that tend to be more homogenous in their composition, that is to say, they are generally composed of white men. 

There are several factors that make AI bias hard to oppose. One of the main problems of AI systems is that the very foundations of these systems are often riddled with errors. Recent research has shown that ten key data sets, which are often used for machine learning and data science, including ImageNet (a large dataset of annotated photographs intended to be used as training data) are in fact riddled with errors. These errors can be traced to the quality of data the system was trained on or, for instance, biases being introduced by the labelers themselves, such as labelling more men as doctors and more women as nurses in pictures. 

How do we fix bias in AI systems?

This is a question that many researchers, technologists, and activists are trying to answer. Some of the more common approaches to this question include inclusivity – both in the context of data collection as well as the design of the system. There have also been calls about the need for increased transparency and explainability, which would allow people to understand how AI systems make their decisions. For example, in the case of the Dutch algorithm, while the officials received an assessment from the algorithm stating that the application was likely to be fraudulent, it did not provide any reasons as to why the algorithm detected fraud. If the officials in charge of the second round of review had more transparency about what the system would flag as an error, including missed signatures or dual citizenship, it is possible that they may have been able to mitigate the damage.

One possible mechanism to address the problem of bias is — the blind taste test mechanism – The mechanism works to check if the results produced by an AI system are dependent upon a specific variable such as sex, race, economic status or sexual orientation. Simply put, the mechanism tries to ensure that protected characteristics like gender, skin colour, or race should not play a role in decision-making processes.

The mechanism includes testing the algorithm twice, the first time with the variable, such as race, and the second time without it. Therefore in the first set, the model is trained on all the variables including race, and the second time the model is trained on all variables, excluding race.If the model returns the same results, then the AI system can be said to make predictions that are blind to the factor, but if the predictions change with the inclusion of a variable, such as by inclusion of dual citizenship status in the case of the Dutch algorithm, or the inclusion of skin colour in the beauty contest the AI system would have to be investigated for bias. This is just one of the potential mitigation tests. States are also experimenting with other technical interventions such as the use of synthetic data, which can be used to create less biased data sets. 

Where do we go from here 

The Dutch case is merely one of the examples in a long line of instances that warrant higher transparency and accountability requirements for the deployment of AI systems. There are many approaches that have been, and are still being developed and considered to counter bias in AI systems. However, the crux remains that it may be impossible to fully eradicate bias from AI systems due to the biased nature of human developers and engineers, which is bound to be reflected within technological systems. The effects of these biases can be devastating depending upon the context and the scale at which they are implemented. 

While new and emerging technical measures can be used as stopgaps, in order to comprehensively deal with bias in AI systems, we must address the issues of bias in those who design and operationalise the system. In the interim, regulators and states must step up to carefully scrutinise, regulate or in some cases halt the use of AI systems which are being used to provide essential services to people. An example of such regulation could include the framing and adoption of risk based assessment frameworks for the adoption of AI systems, wherein the regulatory requirements for AI systems are dependent upon the level of risk they pose to individuals. This could include permanently banning the deployment of AI systems in areas where AI systems may pose a threat to people’s safety, livelihood, or rights, such as credit scoring systems, or other systems which could manipulate human behaviour. For AI systems which are scored to be lower risk, such as AI chatbots being used for customer service, there may be a lower threshold for the prescribed safeguards. The debate on whether or not AI systems can ever truly be free from bias may never be fully answered; however, we can say that the harms that these biases cause can be mitigated with proper regulatory and technical measures. 

AI Law and Policy Diploma Course

The Centre for Communication Governance at the National Law University, Delhi is excited to announce the first edition of the AI Law and Policy Diploma Course – an 8 month online diploma course curated and delivered by expert academics and researchers at CCG and NLU Delhi. The Course is an exciting opportunity to learn the legal, public policy, socio-political and economic contours of AI systems and their implications on our society and its governance. The course provides students the opportunity to interact with and learn from renowned policy practitioners and experienced professionals in the domain of technology law and policy. The course will commence in October 2022 and end in May 2022. Registration for the course is now open and will close by 3rd October 2022 11:59 PM IST. 

About the Centre 

The Centre for Communication Governance at National Law University Delhi (CCG) was established in 2013 to ensure that Indian legal education establishments engage more meaningfully with information technology law and policy and contribute to improved governance and policy making. CCG is the only academic research centre dedicated to undertaking rigorous academic research on information law and policy in India and in a short span of time has become a leading institution in Asia. 

CCG has built an extensive network and works with a range of international academic institutions and policy organisations. These include the United Nations Development Programme, Law Commission of India, NITI Aayog, various Indian government ministries and regulators, International Telecommunications Union, UNGA WSIS, Paris Call, Berkman Klein Center for Internet and Society at Harvard University, the Center for Internet and Society at Stanford University, Columbia University’s Global Freedom of Expression and Information Jurisprudence Project, the Hans Bredow Institute at the University of Hamburg, the Programme in Comparative Media Law and Policy at the University of Oxford, the Annenberg School for Communication at the University of Pennsylvania, and the Singapore Management University’s Centre for AI and Data Governance.

About the Course 

The Course is designed to ensure the nuanced engagement of the students with the legal, public policy, socio-political and economic contours of AI systems and their implications on our society and its governance. 

The course will engage with key themes in the interaction of artificial intelligence with law and policy including implications of AI on our society, emerging use cases of AI and related opportunities and challenges, domestic and global approaches to AI governance, ethics in AI, the application of data protection principles to AI systems, and AI discrimination and bias. Students will be exposed to proposed legislation and policy frameworks on artificial intelligence in India and globally, international policy developments, current uses of AI technology and emerging challenges.

This course will equip students with the necessary understanding and knowledge required to effectively navigate the rapidly evolving space of AI law and policy, and assess the contemporary developments.

Course objectives and learning outcomes 

The course aims to ensure that students are:

  1. The students of the course will be introduced to AI technology and will become cognisant of its opportunities and challenges, and its potential impacts on society, individuals and the law.
  2. The course will provide an overview of the interactions between AI and Law and delve into the current domestic and international frameworks which seek to govern AI technology.
  3. The students will be equipped to navigate the interaction between AI and ethics, and consider the ethical principles within which the use of AI technologies are being situated. They will be provided with a breakdown of the ethical principles which have emerged surrounding the use of AI.  
  4. Students will become familiar with the regional and international policy processes which surround  AI technology and the role of intergovernmental organisations in AI governance.
  5. Students will be equipped with knowledge of data protection principles and their interaction with AI systems. 
  6. Students will delve into problems surrounding AI discrimination and explore how bias creeps into AI systems at various stages, and the implications that this may have upon individuals and our society. 
  7. The students will become conversant with global practices, and governance and regulatory frameworks around AI, focusing on multilateral processes which are currently underway as well as specific domestic approaches. 
  8. The course also has a specialised module on AI in India, focusing upon the regulatory and governance framework around the deployment of AI systems.
  9. Students will also become familiar with the novel use of AI in India, including the use of AI systems for FRT as well as its use in judicial systems.
  10. The students will explore the emerging application and use cases of AI technologies. Students will familiarise themselves with the new uses of AI technologies such as facial recognition, emotional recognition, predictive policing, AI use in workplaces, AI use in healthcare, etc. and consider how this may impact individuals and society. 

For the detailed course outline please visit here

Eligibility 

  • Lawyers/advocates, professionals involved in information technology, professionals in the corporate, industry, government, media, and civil society sector, technology policy professionals, academicians, and research scholars interested in the field of technology and information technology law and policy and under graduates from any discipline are well positioned to apply for the course.
  • Candidates having a 10+2 degree from any recognized board of education, with a minimum of 55% marks, are eligible to apply for this course.
  • There shall be no restriction as to age, nationality, gender, employment status in the admission process

Time Commitment

We recommend students set aside an average of 4-8 hours per week for attending the scheduled monthly live online sessions on weekends and for completing the mandatory coursework (including viewing recorded lectures, any assessment exercises) and prescribed readings.

Seats Available 

A total of 50 seats are available for the course. 

Registration 

Interested candidates may register for the course through the online link provided here

Deadline

Last date to apply: 3rd October 2022 (11:59pm IST)

Course Fee 

INR 90,000/- (all inclusive and non-refundable) to be paid at the time of registration. 

Contact us: For inquiries please contact us at ccgcourse@nludelhi.ac.in with the subject line ‘CCG NLUD Diploma Course on AI Law and Policy’. Emails sent without the subject line ‘CCG NLUD  Diploma Course on AI Law and Policy’ may go unnoticed.

Technology Regulation: Risk-based approaches to Artificial Intelligence governance, Part II

Post authored by Prateek Sibal

The previous post on “Technology Regulation: Risk-based approaches to Artificial Intelligence governance, Part I” discussed recent advancements in AI technologies that have led to new commercial applications with potentially adverse social implications. We also considered the challenges of AI governance and discussed the role of technical benchmarks for evaluating AI systems.

In this post, we will explore the different AI risk assessment approaches that can underpin AI regulation. This post will conclude with a discussion on the next steps for national AI governance initiatives.

Artificial Intelligence Risk Assessment Frameworks

Risk assessments can help identify the AI systems that need to be regulated.  Risk is determined by the severity of the impact of a problem and the probability of its occurrence. For example, the risk profile of a facial recognition system used to unlock a mobile phone would differ from a facial recognition system used by law enforcement in the public arena. The former may be beneficial as it adds a privacy-protecting security feature on the mobile phone. In contrast, the latter will have a chilling effect on free expression and privacy due to its mass surveillance capability. Therefore, the risk score for facial recognition systems will depend on their use and deployment context. This section will discuss some of the approaches followed by various bodies in developing risk assessment frameworks for AI systems.

European Commission’s approach

The European Commission’s legislative proposal on Artificial Intelligence classifies AI systems by four levels of risk and outline risk proportionate regulatory requirements. The categories proposed by the EU include:

  1. Unacceptable Risk: AI systems that pose a clear threat to people’s safety, livelihood, and rights fall under the category of unacceptable risk. The EU Commission has stated that applications that include social credit scoring systems and AI systems that can manipulate human behaviour will be banned.
  2. High Risk: AI systems that harm the safety or fundamental rights of people are categorised as high-risk. There are mandatory requirements for such systems, including the “quality of data sets used; technical documentation and record-keeping; transparency and the provision of information to users; human oversight; and robustness, accuracy and cybersecurity”. The EU will maintain an updated list of high-risk AI systems to respond to emerging challenges. At present, high-risk AI systems include AI algorithms used in transport systems, job hiring processes, border control and management, law enforcement, education systems, and democratic processes.
  3. Limited Risk: When the risks associated with the AI systems are limited, only transparency requirements are prescribed. For example, in the case of a customer engaging with an AI-based chatbot, the customer should be informed that they are interacting with an AI system.
  4. Minimal Risk: When the risk level is identified as minimal, there are no mandatory requirements, but the developers of such AI systems may voluntarily choose to follow industry standards. Examples of such applications include AI-enabled video games or spam filters.

The EU proposal bans real-time remote biometric identification like facial recognition systems installed in public spaces due to their adverse impact on fundamental rights like freedom of expression and privacy.

German approach

In Germany, the Data Ethics Commission has proposed a five-layer criticality pyramid that requires no regulation at a low-risk level to a complete ban at high-risk levels. Figure 2 presents the criticality pyramid and risk-adapted regulation framework for AI systems. The EU approach is similar to the German approach but differs in the number of levels.

Figure 2: Criticality pyramid and risk-adapted regulatory system for the use of algorithmic systems (Source: Opinion of the Data Ethics Commission)

UK approach

The AI Barometer Report of the Centre for Data Ethics and Innovation, tasked by the UK government to facilitate multistakeholder cooperation for developing the governance regime for data-driven technologies, identifies some common risks associated with AI systems and some sector-specific risks. The common risks include:

  1. Bias: Algorithmic bias and discrimination
  2. Explainability: Lack of explainability of AI systems
  3. Regulatory capacity: Regulatory capacities of the state, i.e. their capacity to develop and enforce regulation
  4. Data privacy: Breach in data privacy due to failure in user consent
  5. Public trust: Loss of public trust in institutions due to problematic AI and data use

The researchers identified that the severity of common risks varies across different sectors like criminal justice, financial services, health & social care, digital & social media and energy and utilities. For example, algorithmic bias leading to discrimination is considered high-risk in criminal justice, financial services, health and social media but medium risk in energy and utilities. The risk assignment, in this case, was done through expert discussions.

Organisation of Economic Cooperation and Development (OECD) approach

The OECD’s work on AI classification presents a model for classifying an AI system that can inform risk assessment under each class. The preliminary classification of AI systems developed by the OECD Network of Experts’ working group on AI classification has four dimensions:

  1. Context: The context in which an AI system is developed and deployed. Context includes stakeholders that deploy an AI system, the stakeholders impacted by its use and the sector in which an AI system is deployed.
  2. Data: Data and inputs to an AI system play a vital role in determining the system’s outputs based on the data classifiers used, the source of the data, its structure, scale, and how it was collected.
  3. Type of algorithm: The type of algorithms used in AI systems has implications for transparency, explainability, autonomy and privacy, among other principles. For example, an AI system can use a rules-based algorithm, which executes a series of pre-defined steps. Manufacturing robots used in assembly lines are an example of such a rules-based AI. In contrast, AI systems based on artificial neural networks (ANN) are inspired by the human brain’s structure and functions. These neural networks learn to solve problems by performing many iterations until they get the correct outcomes. In ANNs, the rules to reach a decision are developed by the AI model, and the decision-making process is opaque to humans.
  4. Task: The kind of task to be performed and the type of output expected vary across AI systems. AI systems can perform various tasks from forecasting, content personalisation to detection and recognition of voice or images.

Applying this classification framework to different cases, from facial recognition systems and medical devices to autonomous vehicles, allows us to understand the risks under each dimension and design appropriate regulation. In autonomous vehicles, the context of transportation and its significant risk of accidents increase the risk associated with AI systems. Such vehicles dynamically collect data and other inputs through sensors. They can suffer from security risks due to adversarial attacks where input data fed to the AI models can be tampered with, leading to accidents. The AI algorithms used in autonomous vehicles perform tasks like detecting road signs, deciding vehicle parameters like speed and direction, and responding to road conditions. If such decision-making happens without human control or oversight, it can pose significant risks to passengers and pedestrians’ lives. This example illustrates that autonomous vehicles can be considered a high-risk category requiring robust regulatory oversight to ensure public safety.

The four approaches to risk assessment discussed above are systematic attempts to understand AI-related risks and develop a foundation for downstream regulation that could address risks without being overly prescriptive.

Next Steps in Strengthening Risk-Adaptive Regulation for AI

This two-part blog series has framed the challenges of AI governance in terms of the Collingridge Dilemma concerning the social control of technology. Then it discussed the role of technical benchmarks in assessing the performance of AI systems vis. a vis. AI ethics principles. The section on AI risks assessment presents different approaches to identify AI applications and contexts that require regulation.

As the next step, national-level AI governance initiatives could work towards strengthening AI governance through:

  1. AI Benchmarking: Continuous development and updating of technical benchmarks for AI systems to assess their performance under different contexts with respect to AI ethics principles.
  2. Risk Assessments at the level of individual AI applications: Development of use cases and risk-assessment of different AI applications under different combinations of contexts, data and inputs, AI models and outputs.
  3. Systemic Risk Assessments: Analysis of risks at a systemic level, primarily when different AI systems interact. For example, in financial markets, AI algorithms interact with each other, and in certain situations, their interactions can cascade into a market crash.

Once AI risks are better understood, proportional regulatory approaches should be developed and subjected to Regulatory Impact Analysis (RIA). The OECD defines Regulatory Impact Analysis as a “systemic approach to critically assessing the positive and negative effects of proposed and existing regulations and non-regulatory alternatives”. RIAs can guide governments in understanding if the proposed regulations are effective and efficient in achieving the desired objective. As a complement to its legislative proposal for AI, the European Commission conducted an impact assessment of the proposed legislation and reported an aggregate compliance cost of between 100 and 500 million euros by 2025, mainly for high-risk AI applications that account for 5-15 per cent of all AI applications. The assessment analyses other factors like the impact of the legislation on the competitiveness of Small and Medium Enterprises (SMEs), additional budgetary responsibility on national governments and whether the measures proposed are proportionate to the objectives of the legislation. Such impact assessments are good regulatory practice and will be important as more countries work towards national AI legislations.

Finally, given the globalised nature of different AI services and products, countries should develop national-level regulatory approaches to AI in conversation with each other. Importantly, these dialogues at the global and national level should be multistakeholder driven to ensure that different perspectives inform any ensuing regulation. The pooling of knowledge and coordination on governing AI risks will lead to overall benefits by ensuring AI development in a manner that is ethically aligned while providing a stable environment for innovation and interoperability due to policy coherence.

The author would like to thank Jhalak Kakkar and Nidhi Singh for their helpful feedback.

This blog was written with the support of the Friedrich Naumann Foundation for Freedom.

Technology Regulation: Risk-based approaches to Artificial Intelligence governance, Part 1

Post authored by Prateek Sibal

In five years, between 2015 and 2020, 117 initiatives have published AI ethics principles worldwide. Despite a skewed geographical scope, with 91 of these initiatives emerging in Europe and North America, the proliferation of such initiatives on AI ethics principles paves the way for building global consensus on AI governance. Notably, the 37 OECD Member States have adopted the OECD AI Recommendation, the G20 has endorsed these principles, and the Global Partnership on AI is operationalising them. In the UN system, the United Nations Educational, Scientific and Cultural Organization (UNESCO) is developing a Recommendation on the Ethics of AI that 193 countries may adopt in 2021.

An analysis of different principles reveals a high-level consensus around eight themes: privacy, accountability, safety and security, transparency and explainability, fairness and non-discrimination, human control of technology, professional responsibility, and promotion of human values. At the same time, ethical principles are criticised for lacking enforcement mechanisms. Companies often commit to AI ethics principles to improve their public image with little follow-up on implementing them; an exercise termed as “ethics washing”. Evidence also suggests that knowledge of the ethical tenets has little or no effect on whether software engineers factor in ethical principles in developing products or services.

Defining principles is essential, but it is only the first step for ethical AI governance. There is a need for mid-level norms, standards and guidelines at the international level that may inform regional or national regulation to translate principles into practice. This two-part blog will discuss the need for AI governance to evolve past the ‘ethics formation stage’ into concrete and tangible steps such as developing technical benchmarks and adopting risk-based regulation for AI systems.

Part one of the blog has three sections. The first section discusses some of the technical advances in AI technologies in recent years. These advances have led to new commercial applications with some potentially adverse social implications. Section two discusses the challenges of AI governance and presents a framework for mitigating the adverse implications of technology on society. Finally, section three discusses the role of technical benchmarks for evaluating AI systems. Part two of the blog will contain further discussion on risk assessment approaches to help identify the AI applications and contexts that need to be regulated.  It will also discuss the next steps for national initiatives for AI governance.

The blog follows the definition of an AI system proposed by the OECD’s AI Experts Group. They describe an AI system as a “machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments. It uses machine or human-based inputs to perceive real or virtual environments, abstract such perceptions into models (in an automated manner, e.g. with ML or manually), and use model inference to formulate options for information or action. AI systems are designed to operate with varying levels of autonomy.”

Recent Advances in AI Technologies

Artificial Intelligence is developing rapidly. It is important to lay down a broad overview of AI developments, which may have profound and potentially adverse impacts on individuals and society. The 2021 AI Index report notes four crucial technical advances that hastened the commercialisation of AI technologies:

  • AI-Generated Content: AI systems can generate high-quality text, audio and visual content to a level that it is difficult for humans to distinguish between synthetic and non-synthetic content.
  • Image Processing: Computer vision, a branch of computer science that “works on enabling computers to see, identify and process images in the same way that human vision does, and then provide appropriate output”, has seen immense progress in the past decade and is fast industrialising in applications that include autonomous vehicles.
  • Language Processing: Natural Language Processing (NLP) is a branch of computer science “concerned with giving computers the ability to understand the text and spoken words in much the same way human beings can”. NLP has advanced such that AI systems with language capabilities now have meaningful economic impact through live translations, captioning, and virtual voice assistants.
  • Healthcare and biology:DeepMind’s AlphaFold solved the decades-old protein folding problem using machine learning techniques. This breakthrough will allow the study of protein structure and will contribute to drug discovery.

These technological advances have social implications. For instance, the technology generating synthetic faces has rapidly improved. As shown in Figure 1, in 2014, AI systems produced grainy faces, but by 2017, they were generating realistic synthetic faces. Such AI systems have led to the proliferation of ‘deepfake’ pornography that overwhelmingly targets women and has the potential to erode people’s trust in information and videos they encounter online. Some actors misuse the deepfake technology to spread online disinformation, resulting in adverse implications for democracy and political stability. Such developments have made AI governance a pressing matter.


Figure 1: Improvement in AI-generated images. Source: https://arxiv.org/pdf/1802.07228.pdf

Challenges of AI Governance

In this blog, AI governance is understood as the development and application by governments, the private sector, and civil society, in their respective roles, of shared principles, norms, rules, decision-making procedures, and programmes that shape AI’s evolution and use. As highlighted in the previous section, the rapid advancements in the field of AI technologies have brought the need for better AI governance to the forefront.

In thinking about AI governance, a conundrum that preoccupies many governments worldwide concerns enactment of regulation that does not stifle innovation while also providing adequate safeguards to protect human rights and fundamental freedoms.

Technology regulation is complicated because until a technology has been extensively developed and widely used, its impact on society is difficult to predict. However, once it is deeply entrenched and its effect on society is understood better, it becomes more challenging to regulate the technology. This tension between free and unimpeded technology development and regulating adverse implications is termed the Collingridge dilemma.

David Collingridge, the author of the Social Control of Technologies, noted that when regulatory decisions have to be made under ignorance of technologies’ social impact, continuous monitoring of the impact of technology on society can help correct unexpected consequences early. Collingridge’s guidelines for decision-making under ignorance can inform AI governance as well. These include choosing technology options with:

  • Low failure costs: Selecting options with low error costs, i.e. if a policy or regulation fails to achieve its intended objective, the costs associated with failure are limited.
  • Quicker to correct: Selecting technologies with low response time for correction after the discovery of unanticipated problems.
  • Low cost of applying remedy: Selecting solutions with low cost of applying the remedy, i.e. options with a low fixed cost and a higher variable cost, should be given preference over the ones with a higher fixed cost, and
  • Continuous monitoring: Cost-effective and efficient monitoring can ensure the discovery of unpredicted consequences quickly.

For instance, the requirements around transparency in AI systems provide information for monitoring the impact of AI systems on society. Similarly, risk assessments of AI systems offer a pre-emptive form of oversight over technology development and use, which can help minimise potential social harms.  

Technical benchmarks for evaluating AI systems

To address ethical problems related to bias, discrimination, lack of transparency, and accountability in algorithmic decision-making,  quantitative benchmarks to assess AI systems’ performance against these ethical principles are needed.

The Institute of Electrical and Electronics Engineers (IEEE), through its Global Initiative on Ethics of Autonomous and Intelligent Systems, is developing technical standards, including on bias in AI systems. They describe “specific methodologies to help users certify how they worked to address and eliminate issues of negative bias in the creation of their algorithms”. Similarly, in the United States, the National Institute of Standards and Technology (NIST) is developing standards for explainable AI based on principles that call for AI systems to provide reasons for their outputs in a manner that is understandable to individual users, explain the process used for generating the output, and deliver their decision only when the AI system is fully confident.

For example, there is significant progress in introducing benchmarks for the regulation of facial recognition technology. Facial recognition systems have a large commercial market. They and used for various tasks, including law enforcement and border controls. These tasks involve detecting visa photos, matching photos in criminal databases, and child abuse images. Such facial recognition systems have been the cause of significant concern due to high error rates in detecting faces and impinging on human rights. Biases in such systems have adverse consequences for individuals denied entry at borders or wrongfully incarcerated. In the United States, the National Institute of Standards and Technology’s Face Recognition Vendor Test provides a benchmark to compare different commercially available facial recognition systems’ performance by operating their algorithms on different image datasets.

The progress in defining benchmarks for ethical principles needs to be complemented by risk assessments of AI systems to pre-empt potentially adverse social impact in line with the Collingridge Dilemma discussed in the previous section. Risk assessments allow the categorisation of AI applications by their risk ratings. They can help develop risk-proportionate regulation for AI systems instead of blanket rules that may place an unnecessary compliance burden on technology development. The next blog in this two-part series will engage with potential risk-based approaches to AI regulation.

The author would like to thank Jhalak Kakkar and Nidhi Singh for their helpful feedback.

This blog was written with the support of the Friedrich Naumann Foundation for Freedom.

The Personal Data Protection Bill, 2019 vs. GDPR: Provisions for the rights of the child and its implications

This post is authored by Puja Nair

The debate on privacy rose to the forefront after the Supreme Court passed a judgement in the case of Justice K.S Puttaswamy (Retd.) v. Union of India, where the Court held that the right to privacy was an intrinsic part of the right to life and personal liberty under Article 21 of the Constitution of India. In arriving at this conclusion, the Court examined a wide range of privacy-related issues and held that the right to privacy included the right to personal autonomy over a wide range of domains in a person’s life.

While the above decision seems obvious in its simplicity, complications arise when one considers that a child or adolescent may not understand the consequences of their individual choices. When taken in the context of online data privacy, it is safe to say that children may be unaware of the exact manner in which any data that they share online is put to use. The report submitted by the committee of experts under the chairmanship of Justice B.N Srikrishna clearly endorses this belief.

Clause 16 of the Indian Personal Data Protection Bill, 2019 (‘PDPB 2019’), which was tabled in parliament on December 11, 2019, deals with the processing of personal and sensitive personal data of children. It states categorically that every data fiduciary shall “process the personal data of a child in a manner that protects the rights of, and is in the best interests of, the child.” It further states that a data fiduciary shall only process the personal data of a child, after verifying their age and obtaining the consent of their parent or guardian, in the manner specified by future regulations.

Based on this provision, the primary question that arises is, who is a child as per the PDPB 2019? According to the provisions of the bill, a child is someone who “has not completed the age of 18 years.” This is distinct from the data protection statutes passed in other jurisdictions. The EU General Data Protection Rules (‘GDPR’) specifies that the age limit on the definition of ‘child’ may be up to the discretion of individual member states and can be anywhere between 13-16 years. The US Children’s Online Privacy Protection Act, 1998 on the other hand, puts the age limit at a firm 13 years. Notwithstanding the above, the PDPB 2019 specifies 18 as the age of majority. This was done to ensure that the provisions of the bill would be in conformity with the prevailing laws of the country.

The adoption of a singular age of majority serves to prevent confusion and conflict between the laws in the country, however, it also serves to underestimate the awareness and advancement of today’s youth. An example of this understanding was espoused by the Madras High Court in the case of Sabari Sabarinathan Sabarivasan v. State Commission for Protection of Child Rights and Ors. This judgment examines existing flaws in the Protection of Children from Sexual Offences (POCSO) Act, 2012 and recommends a change in the definition of the term ‘child,’ so that a consensual relationship between a girl above 16 years of age and a boy between 16 to 21 years of age, would not attract the draconian provisions of the law. The drafters of the PDPB 2019 could have taken a similar view, rather than conforming with the provisions of a statute like the Indian Contract Act or the Indian Majority Act, both of which were enacted in the late-1800’s. Furthermore, a 2019 study conducted among 630 adolescents across 8 schools in the nation’s capital, revealed that 60 per cent of the boys and 40 per cent of the girls, owned their own device while almost half reportedly used two or more devices to access the Internet. The numbers have no doubt increased since then and the COVID-19 crises has further accelerated the adoption of online services for both education and entertainment. This means that mandating a guardian’s consent for anyone below the age of 18 years could very well result in some data fiduciaries inadvertently being on the wrong side of the law.

Another question raised by Clause 16 of the PDPB 2019, is the determination of what constitutes the best interests of the child. The bill does not specify how this is to be determined; however, subclause 5 of Clause 16 categorizes certain types of data processing like behavioural monitoring, tracking, and targeted advertising as harmful for children.

We then come to the requirement for age verification and parental consent. The provisions of the bill do not explore this in detail. It merely states that the process of acquiring such consent and/or verification will be specified in further rules, after taking into account factors like the volume of personal data processed, the proportion of such personal data likely to be that of a child, the potential of harm that may occur to said child as a result of the processing of his/her personal data etc.

Regardless, one issue that may arise when it comes to consent is the question of capacity. Clause 11 of the PDPB 2019 states that among other things, consent must be free and informed. However, parents cannot provide such free and informed consent on behalf of their children, if they do not understand the terms and conditions provided in the policies of these websites. In many instances, we find that children possess a much greater awareness of current technology trends and their implications. Additional issues arise when we consider the concept of free choice. However, the fact of the matter is that if one wants to register with any of the popular online apps and services available, one inevitably has to agree with their terms and conditions, regardless of any reservations one might have. Therefore, the concept of consent being “freely given” is rendered pointless.

GDPR and the European Union

Article 8 of the GDPR states that where there is an offer of “information society service directly to a child” the processing of personal data of said child shall be lawful, where the child is at least 16 years old. If the child is below the age of 16 years, such processing shall be lawful only if consent has been obtained by the “holder of parental responsibility over the child.”Member States can provide for a lower age limit, provided it is not below 13 years of age. The provision further provides that “reasonable efforts” must be made to verify that consent is given or authorised by the holder of parental responsibility over the child, taking into consideration available technology.

Article 8 is the principal provision relating to the protection of children’s personal data in the GDPR. There are other provisions that mandate the type of measures that must be taken for the protection of the personal data of a child. For example, when obtaining data from a child, data controllers must ensure that any information on the processing of such data, should be in clear and plain terms for a child to easily understand. The GDPR also provides for the ‘right of erasure’ for children’s personal data. This is particularly relevant in cases where the data subject may have provided their consent as a child, without being fully aware of the risks involved and now seek the erasure of such personal data. Clause 16 of the PDPB, which relates to the processing of personal data of children, closely mirrors Article 8 of the GDPR. To that end, this post will be limited to an examination of Article 8 of the GDPR to examine the potential pitfalls that await in the implementation of Clause 16 of PDPB 2019.

Article 8 applies only to information society services offered directly to a child. Information society services or ISS is any service that is provided at a distance, by electronic means, and at the individual request of a recipient of the services. The definition also includes the requirement that the service be one that is provided in exchange for “remuneration”. However, the majority of online services that teenagers have access to do not directly require remuneration from the users. Common examples of this include popular social media sites like Facebook, Instagram etc. For this reason, the phrase “remuneration” is interpreted broadly by the European Court of Justice (‘ECJ’). The Court has held that “the essential characteristic of remuneration […] lies in the fact that it constitutes consideration for the service in question and is normally agreed upon between the provider and the recipient of the service’’. It is not essential that the recipient of the services provide the consideration. It is only essential for the consideration to have been received by the service provider. Subsequent rulings specified that such services may also include services provided by a non-profit organization, services involving an element of chance, and services that are of a recreational or sporting nature.

Some confusion may arise in situations where the ISS has both online and offline components. In such cases one must determine whether or not the online component is integral to the nature of the service provided. If it is not integral, then such services cannot be categorized as an ISS. While these cases provide some clarity, it is clear that the definition and scope of what constitutes an ISS will continue to evolve with the evolution of technology. This is in direct contrast to the definition of a data fiduciary in the PDPB 2019, which is much more straightforward. The bill defines a data fiduciary as “any person, including the State, a company, any juristic entity or any individual who alone or in conjunction with others determines the purpose and means of processing of personal data.”

Further, much like Clause 16 of the PDPB 2019, the drafting of Article 8 raises questions on what constitutes proper consent and how such consent can be appropriately verified. Some of these questions have been delineated above in the Indian context and are also applicable here. The European Data Protection Board (‘EDPB’) have addressed these issues in its guidelines on consent under issued under the GDPR. The guidelines state that if a data subject consents because they feel they have no real choice, then the consent is not valid. The guidelines also specify certain situations where the existence of an imbalance of power between the data subject and the controller, would render consent invalid. It further provides that consent would not be considered to be “freely given” if the consent was bundled with the acceptance of the terms and conditions of a website. Additionally, when it comes to the issue of capacity, the guidelines provide that for the consent to be informed, the data subject, or the individual having parental responsibility over the data subject, must have knowledge of the controller’s identity, knowledge of the purpose of each of the processing operations for which consent is sought, knowledge of the type of data collected and used, and knowledge of the existence of the right to withdraw consent.

Finally, even if the validity of consent is established, there is no provision to determine whether the person providing such consent is qualified to do so. According to the provisions of Article 8, consent must be given by a holder of parental responsibility. Does this include even individuals who are acting in loco parenti? For example, in the US, schools may act on the parents’ behalf in an educational context, when personal data is collected from the students for the use and benefit of the school. Further, once this consent is obtained, how is it to be verified? The GDPR has merely required that the controller take “reasonable efforts” to verify said consent. This means that in situations where consent was not verifiable, the controller could still rely on the un-verified consent so long as they prove that “reasonable” efforts were made to verify the same. Fortunately, the EDPB Guidelines on consent fills this gap in Article 8 by recommending two types of verification mechanisms for high-risk and low-risk categories respectively. In the low-risk category, verification of parental consent via email was held to be sufficient. In the high-risk category, it was recommended that further proof of consent would need to be acquired. Trusted third-party verification services were also recommended, to minimise the amount of personal data the controller had to process itself.

Conclusion

The examination of the GDPR provisions clearly shows that numerous issues have arisen in the course of its implementation. These issues have been resolved on a case-by-case basis by courts and other authorities. However, these solutions are remedial and not preventative. One preventative approach is the implementation of principles like data protection by design and default as specified in Article 25 of the GDPR. Data protection by design ensures that privacy and data protection issues are considered at the design phase of any system, service or product and then implemented throughout the lifecycle of the same. Data protection by default limits the type of data collected. It requires controllers to collect and process only such data as is necessary to achieve their specific purpose.

Data protection by design is a principle that is already enshrined in Clause 22 of the PDPB, which provides that every data fiduciary shall submit a privacy by design policy to the proposed Data Protection Authority (DPA) for approval and certification. The manner in which this is to be implemented and the standards of protection required for certification would be subject to future regulations. However, by requiring data fiduciaries engaged in the collection and processing of children’s data to adhere to a higher standard of data protection, the DPA could probably ensure the protection of children’s data regardless of any pitfalls in the practical implementation of Clause 16.

The above measure might not effectively solve the issues specified with the implementation of Clause 16. Notwithstanding these drawbacks, the provisions of this Bill might be the very first step in bringing India’s data protection thresholds at par with the rest of the world.


CJEU sets limits on Mass Communications Surveillance – A Win for Privacy in the EU and Possibly Across the World

This post has been authored by Swati Punia

On 6th October, the European Court of Justice (ECJ/ Court) delivered its much anticipated judgments in the consolidated matter of C-623/17, Privacy International from the UK and joined cases from France, C-511/18, La Quadrature du Net and others, C-512/18, French Data Network and others, and Belgium, C-520/18, Ordre des barreaux francophones et germanophone and others (Collectively “Bulk Communications Surveillance Judgments”). 

In this post, I briefly discuss the Bulk Communication Surveillance Judgments, their significance for other countries and for India. 

Through these cases, the Court invalidated the disproportionate interference by Member States with the rights of their citizens, as provided by EU law, in particular the Directive on privacy and electronic communications (e-Privacy Directive) and European Union’s Charter of Fundamental Rights (EU Charter). The Court assessed the Member States’ bulk communications surveillance laws and practices relating to their access and use of telecommunications data. 

The Court recognised the importance of the State’s positive obligations towards conducting surveillance, although it noted that it was essential for surveillance systems to conform with the general principles of EU law and the rights guaranteed under the EU Charter. It laid down clear principles and measures as to when and how the national authorities could access and use telecommunications data (further discussed in the sections ‘The UK Judgment’ and ‘The French and Belgian Judgment’). It carved a few exceptions as well (in the joined cases of France and Belgium) for emergency situations, but held that such measures would have to pass the threshold of being serious and genuine (further discussed in the section ‘The French and Belgian Judgment’). 

The Cases in Brief 

The Court delivered two separate judgments, one in the UK case and one in the joined cases of France and Belgium. Since these cases had similar sets of issues, the proceedings were adjoined. The UK application challenged the bulk acquisition and use of telecommunications data by its Security and Intelligence Agencies (SIAs) in the interest of national security (as per the UK’s Telecommunication Act of 1984). The French and Belgian applications challenged the indiscriminate data retention and access by SIAs for combating crime. 

The French and Belgian applications questioned the legality of their respective data retention laws (numerous domestic surveillance laws which permitted bulk collection of telecommunication data) that imposed blanket obligations on Electronic Communications Service Providers (ECSP) to provide relevant data. The Belgian law required ECSPs to retain various kinds of traffic and location data for a period of 12 months. Whereas, the French law provided for automated analysis and real time data collection measures for preventing terrorism. The French application also raised the issue of providing a notification to the person under the surveillance. 

The Member States contended that such surveillance measures enabled them to inter alia, safeguard national security, prevent terrorism, and combat serious crimes. Hence, they claimed inapplicability of the e-Privacy Directive on their surveillance laws/ activities.

The UK Judgment

The ECJ found the UK surveillance regime unlawful and inconsistent with EU law, and specifically the e-Privacy Directive. The Court analysed the scope and scheme of the e-Privacy Directive with regard to exclusion of certain State purposes such as national and public security, defence, and criminal investigation. Noting the importance of such State purposes, it held that EU Member States could adopt legislative measures that restricted the scope of rights and obligations (Article 5, 6 and 9) provided in the e-Privacy Directive. However, this was allowed only if the Member States complied with the requirements laid down by the Court in Tele2 Sverige and Watson and Others (C-203/15 and C-698/15) (Tele2) and the e-Privacy Directive. In addition to these, the Court held that the EU Charter must be respected too. In Tele2, the ECJ held that legislative measures obligating ECSPs to retain data must be targeted and limited to what was strictly necessary. Such targeted retention had to be with regard to specific categories of persons and data for a limited time period. Also, the access to data must be subject to a prior review by an independent body.

The e-Privacy Directive ensures the confidentiality of electronic communications and the data relating to it (Article 5(1)). It allows ECSPs to retain metadata (context specific data relating to the users and subscribers, location and traffic) for various purposes such as billing, valued added services and security purposes. However, this data must be deleted or made anonymous, once the purpose is fulfilled unless a law allows for a derogation for State purposes. The e-Privacy Directive allows the Member States to derogate (Article 15(1)) from the principle of confidentiality and corresponding obligations (contained in Article 6 (traffic data) and 9 (location data other than traffic data)) for certain State purposes when it is appropriate, necessary and proportionate. 

The Court clarified that measures undertaken for the purpose of national security would not make EU law inapplicable and exempt the Member States from their obligation to ensure confidentiality of communications under the e-Privacy Directive. Hence, an independent review of surveillance activities such as data retention for indefinite time periods, or further processing or sharing, must be conducted for authorising such activities. It was noted that the domestic law at present did not provide for prior review, as a limit on the above mentioned surveillance activities. 

The French and Belgian Judgment

While assessing the joined cases, the Court arrived at a determination in similar terms as the UK case. It reiterated that the exception (Article 15(1) of the e-Privacy Directive) to the principle of confidentiality of communications (Article 5(1) of the e-Privacy Directive) should not become the norm. Hence, national measures that provided for general and indiscriminate data retention and access for State purposes were held to be incompatible with EU law, specifically the e-Privacy Directive.

The Court in the joined cases, unlike the UK case, allowed for specific derogations for State purposes such as safeguarding national security, combating serious crimes and preventing serious threats. It laid down certain requirements that the Member States had to comply with in case of derogations. The derogations should (1) be clear and precise to the stated objective (2) be limited to what is strictly necessary and for a limited time period (3) have a safeguards framework including substantive and procedural conditions to regulate such instances (4) include guarantees to protect the concerned individuals against abuse. They should also be subjected to an ‘effective review’ by a court or an independent body and must be in compliance of general rules and proportionality principles of EU law and the rights provided in the EU Charter. 

The Court held that in establishing a minimum threshold for a safeguards framework, the EU Charter must be interpreted along with the European Convention on Human Rights (ECHR). This would ensure consistency between the rights guaranteed under the EU Charter and the corresponding rights guaranteed in the ECHR (as per Article 52(3) of the EU Charter).

The Court, in particular, allowed for general and indiscriminate data retention in cases of serious threat to national security. Such a threat should be genuine, and present or foreseeable. Real-time data collection and automated analysis were allowed in such circumstances. But the real-time data collection of persons should be limited to those suspected of terrorist activities. Moreover, it should be limited to what was strictly necessary and subject to prior review. It even allowed for general and indiscriminate data retention of IP addresses for the purpose of national security, combating serious crimes and preventing serious threats to public security. Such retention must be for a limited time period to what was strictly necessary. For such purposes, the Court also permitted ECSPs to retain data relating to the identity particulars of their customers (such as name, postal and email/account addresses and payment details) in a general and indiscriminate manner, without specifying any time limitations. 

The Court allowed targeted data retention for the purpose of safeguarding national security and preventing crime, provided that it was for a limited time period and strictly necessary and was done on the basis of objective and non-discriminatory factors. It was held that such retention should be specific to certain categories of persons or geographical areas. The Court also allowed, subject to effective judicial review, expedited data retention after the initial retention period ended, to shed light on serious criminal offences or acts affecting national security. Lastly, in the context of criminal proceedings, the Court held that it was for the Member States to assess the admissibility of evidence resulting from general and indiscriminate data retention. However, the information and evidence must be excluded where it infringes on the right to a fair trial. 

Significance of the Bulk Communication Surveillance Judgments

With these cases, the ECJ decisively resolved a long-standing discord between the Member States and privacy activists in the EU. For a while now, the Court has been dealing with questions relating to surveillance programs for national security and law enforcement purposes. Though the Member States have largely considered these programs outside the ambit of EU privacy law, the Court has been expanding the scope of privacy rights. 

Placing limitations and controls on State powers in democratic societies was considered necessary by the Court in its ruling in Privacy International. This decision may act as a trigger for considering surveillance reforms in many parts of the world, and more specifically for those aspiring to attain an EU adequacy status. India could benefit immensely should it choose to pay heed. 

As of date, India does not have a comprehensive surveillance framework. Various provisions of the Personal Data Protection Bill, 2019 (Bill), Information Technology Act, 2000, Telegraph Act, 1885, and the Code of Criminal Procedure, 1973 provide for targeted surveillance measures. The Bill provides for wide powers to the executive (under Clause 35, 36 and 91 of the Bill) to access personal and non-personal data in the absence of proper and necessary safeguards. This may cause problems for achieving the EU adequacy status as per Article 45 of the EU General Data Protection Regulation (GDPR) that assesses the personal data management rules of third-party countries. 

Recent news reports suggest that the Bill, which is under legislative consideration, is likely to undergo a significant overhaul. India could use this as an opportunity to introduce meaningful changes in the Bill as well as its surveillance regime. India’s privacy framework could be strengthened by adhering to the principles outlined in the Justice K.S. Puttaswamy v. Union of Indiajudgment and the Bulk Communications Surveillance Judgments.

Building an AI Governance Framework for India, Part III

Embedding Principles of Privacy, Transparency and Accountability

This post has been authored by Jhalak M. Kakkar and Nidhi Singh

In July 2020, the NITI Aayog released a draft Working Document entitled “Towards Responsible AI for All” (hereafter ‘NITI Aayog Working Document’ or ‘Working Document’). This Working Document was initially prepared for an expert consultation that was held on 21 July 2020. It was later released for comments by stakeholders on the development of a ‘Responsible AI’ policy in India. CCG’s comments and analysis  on the Working Document can be accessed here.

In our first post in the series, ‘Building an AI governance framework for India’, we discussed the legal and regulatory implications of the Working Document and argued that India’s approach to regulating AI should be (1) firmly grounded in its constitutional framework, and (2) based on clearly articulated overarching ‘Principles for Responsible AI’. Part II of the series discussed specific Principles for Responsible AI – Safety and Reliability, Equality, and Inclusivity and Non-Discrimination. We explored the constituent elements of these principles and the avenues for incorporating them into the Indian regulatory framework. 

In this final post of the series, we will discuss the remaining principles of Privacy, Transparency and Accountability. 

Principle of Privacy 

Given the diversity of AI systems, the privacy risks which they pose to the individuals, and society as a whole are also varied. These may be be broadly related to : 

(i) Data protection and privacy: This relates to privacy implications of the use of data by AI systems and subsequent data protection considerations which arise from this use. There are two broad aspects to think about in terms of the privacy implications from the use of data by AI systems. Firstly, AI systems must be tailored to the legal frameworks for data protection. Secondly, given that AI systems can be used to re-identify anonymised data, the mere anonymisation of data for the training of AI systems may not provide adequate levels of protection for the privacy of an individual.

a) Data protection legal frameworks: Machine learning and AI technologies have existed for decades, however, it was the explosion in the availability of data, which accounts for the advancement of AI technologies in recent years. Machine Learning and AI systems depend upon data for their training. Generally, the more data the system is given, the more it learns and ultimately the more accurate it becomes. The application of existing data protection frameworks to the use of data by AI systems may raise challenges. 

In the Indian context, the Personal Data Protection Bill, 2019 (PDP Bill), currently being considered by Parliament, contains some provisions that may apply to some aspects of the use of data by AI systems. One such provision is Clause 22 of the PDP Bill, which requires data fiduciaries to incorporate the seven ‘privacy by design’ principles and embed privacy and security into the design and operation of their product and/or network. However, given that AI systems rely significantly on anonymised personal data, their use of data may not fall squarely within the regulatory domain of the PDP Bill. The PDP Bill does not apply to the regulation of anonymised data at large but the Data Protection Authority has the power to specify a code of practice for methods of de-identification and anonymisation, which will necessarily impact AI technologies’ use of data.

b) Use of AI to re-identify anonymised data: AI applications can be used to re-identify anonymised personal data. To safeguard the privacy of individuals, datasets composed of the personal data of individuals are often anonymised through a de-identification and sampling process, before they are shared for the purposes of training AI systems to address privacy concerns. However, current technology makes it possible for AI systems to reverse this process of anonymisation to re-identify people, having significant privacy implications for an individual’s personal data. 

(ii) Impact on society: The impact of the use of AI systems on society essentially relates to broader privacy considerations that arise at a societal level due to the deployment and use of AI, including mass surveillance, psychological profiling, and the use of data to manipulate public opinion. The use of AI in facial recognition surveillance technology is one such AI system that has significant privacy implications for society as a whole. Such AI technology enables individuals to be easily tracked and identified and has the potential to significantly transform expectations of privacy and anonymity in public spaces. 

Due to the varying nature of privacy risks and implications caused by AI systems, we will have to design various regulatory mechanisms to address these concerns. It is important to put in place a reporting and investigation mechanism that collects and analyses information on privacy impacts caused by the deployment of AI systems, and privacy incidents that occur in different contexts. The collection of this data would allow actors across the globe to identify common threads of failure and mitigate against potential privacy failures arising from the deployment of AI systems. 

To this end, we can draw on a mechanism that is currently in place in the context of reporting and investigating aircraft incidents, as detailed under Annexure 13 of the Convention on International Civil Aviation (Chicago Convention). It lays down the procedure for investigating aviation incidents and a reporting mechanism to share information between countries. The aim of this accident investigation report is not to apportion blame or liability from the investigation, but rather to extensively study the cause of the accident and prevent future incidents. 

A similar incident investigation mechanism may be employed for AI incidents involving privacy breaches. With many countries now widely developing and deploying AI systems, such a model of incident investigation would ensure that countries can learn from each other’s experiences and deploy more privacy-secure AI systems.

Principle of Transparency

The concept of transparency is a recognised prerequisite for the realisation of ‘trustworthy AI’. The goal of transparency in ethical AI is to make sure that the functioning of the AI system and resultant outcomes are non-discriminatory, fair, and bias mitigating, and that the AI system inspires public confidence in the delivery of safe and reliable AI innovation and development. Additionally, transparency is also important in ensuring better adoption of AI technology—the more users feel that they understand the overall AI system, the more inclined and better equipped they are to use it.

The level of transparency must be tailored to its intended audience. Information about the working of an AI system should be contextualised to the various stakeholder groups interacting and using the AI system. The Institute of Electrical and Electronics Engineers, a global professional organisation of electronic and electrical engineers,  suggested that different stakeholder groups may require varying levels of transparency in accordance with the target group. This means that groups such as users, incident investigators, and the general public would require different standards of transparency depending upon the nature of information relevant for their use of the AI system.

Presently, many AI algorithms are black boxes where automated decisions are taken, based on machine learning over training datasets, and the decision making process is not explainable. When such AI systems produce a decision, human end users don’t know how it arrived at its conclusions. This brings us to two major transparency problems, the public perception and understanding of how AI works, and how much developers actually understand about their own AI system’s decision making process. In many cases, developers may not know, or be able to explain how an AI system makes conclusions or how it has arrived at certain solutions.

This results in a lack of transparency. Some organisations have suggested opening up AI algorithms for scrutiny and ending reliance on opaque algorithms. On the other hand, the NITI Working Document is of the view that disclosing the algorithm is not the solution and instead, the focus should be on explaining how the decisions are taken by AI systems. Given the challenges around explainability discussed above, it will be important for NITI Aayog to discuss how such an approach will be operationalised in practice.

While many countries and organisations are researching different techniques which may be useful in increasing the transparency of an AI system, one of the common suggestions which have gained traction in the last few years is the introduction of labelling mechanisms in AI systems. An example of this is Google’s proposal to use ‘Model Cards’, which are intended to clarify the scope of the AI systems deployment and minimise their usage in contexts for which they may not be well suited. 

Model cards are short documents which accompany a trained machine learning model. They enumerate the benchmarked evaluation of the working of an AI system in a variety of conditions, across different cultural, demographic, and intersectional groups which may be relevant to the intended application of the AI system. They also contain clear information on an AI system’s capabilities including the intended purpose for which it is being deployed, conditions under which it has been designed to function, expected accuracy and limitations. Adopting model cards and other similar labelling requirements in the Indian context may be a useful step towards introducing transparency into AI systems. 

Principle of Accountability

The Principle of Accountability aims to recognise the responsibility of different organisations and individuals that develop, deploy and use the AI systems. Accountability is about responsibility, answerability and trust. There is no one standard form of accountability, rather this is dependent upon the context of the AI and the circumstances of its deployment.

Holding individuals and entities accountable for harm caused by AI systems has significant challenges as AI systems generally involve multiple parties at various stages of the development process. The regulation of the adverse impacts caused by AI systems often goes beyond the existing regimes of tort law, privacy law or consumer protection law. Some degree of accountability can be achieved by enabling greater human oversight. In order to foster trust in AI and appropriately determine the party who is accountable, it is necessary to build a set of shared principles that clarify responsibilities of each stakeholder involved with the research, development and implementation of an AI system ranging from the developers, service providers and end users.

Accountability has to be ensured at the following stages of an AI system: 

(i) Pre-deployment: It would be useful to implement an audit process before the AI system is deployed. A potential mechanism for implementing this could be a multi-stage audit process which is undertaken post design, but before the deployment of the AI system by the developer. This would involve scoping, mapping and testing a potential AI system before it is released to the public. This can include ensuring risk mitigation strategies for changing development environments and ensuring documentation of policies, processes and technologies used in the AI system.

Depending on the nature of the AI system and the potential for risk, regulatory guidelines can be developed prescribing the involvement of various categories of auditors such as internal, expert third party and from the relevant regulatory agency, at various stages of the audit. Such audits which are conducted pre-deployment are aimed at closing the accountability gap which exists currently.

(ii) During deployment: Once the AI system has been deployed, it is important to keep auditing the AI system to note the changes being made/evolution happening in the AI system in the course of its deployment. AI systems constantly learn from the data and evolve to become better and more accurate. It is important that the development team is continuously monitoring the system to capture any errors that may arise, including inconsistencies arising from input data or design features, and address them promptly.

(iii) Post-deployment: Ensuring accountability post-deployment in an AI system can be challenging. The NITI Working Document also recognised that assigning accountability for specific decisions becomes difficult in a scenario with multiple players in the development and deployment of an AI system. In the absence of any consequences for decisions harming others, no one party would feel obligated to take responsibility or take actions to mitigate the effect of the AI systems. Additionally, the lack of accountability also leads to difficulties in grievance redressal mechanisms which can be used to address scenarios where harm has arisen from the use of AI systems. 

The Council of Europe, in its guidelines on the human rights impacts of algorithmic systems, highlighted the need for effective remedies to ensure responsibility and accountability for the protection of human rights in the context of the deployment of AI systems. A potential model for grievance redressal is the redressal mechanism suggested in the AI4People’s Ethical Framework for a Good Society report by the Atomium – European Institute for Science, Media and Democracy. The report suggests that any grievance redressal mechanism for AI systems would have to be widely accessible and include redress for harms inflicted, costs incurred, and other grievances caused by the AI system. It must demarcate a clear system of accountability for both organisations and individuals. Of the various redressal mechanisms they have suggested, two significant mechanisms are: 

(a) AI ombudsperson: This would ensure the auditing of allegedly unfair or inequitable uses of AI reported by users of the public at large through an accessible judicial process. 

(b) Guided process for registering a complaint: This envisions laying down a simple process, similar to filing a Right to Information request, which can be used to bring discrepancies, or faults in an AI system to the notice of the authorities.

Such mechanisms can be evolved to address the human rights concerns and harms arising from the use of AI systems in India. 

Conclusion

In early October, the Government of India hosted the Responsible AI for Social Empowerment (RAISE) Summit which has involved discussions around India’s vision and a roadmap for social transformation, inclusion and empowerment through Responsible AI. At the RAISE Summit, speakers underlined the need for adopting AI ethics and a human centred approach to the deployment of AI systems. However, this conversation is still at a nascent stage and several rounds of consultations may be required to build these principles into an Indian AI governance and regulatory framework. 

As India enters into the next stage of developing and deploying AI systems, it is important to have multi-stakeholder consultations to discuss mechanisms for the adoption of principles for Responsible AI. This will enable the framing of an effective governance framework for AI in India that is firmly grounded in India’s constitutional framework. While the NITI Aayog Working Document has introduced the concept of ‘Responsible AI’ and the ethics around which AI systems may be designed, it lacks substantive discussion on these principles. Hence, in our analysis, we have explored global views and practices around these principles and suggested mechanisms appropriate for adoption in India’s governance framework for AI. Our detailed analysis of these principles can be accessed in our comments to the NITI Aayog’s Working Document Towards Responsible AI for All.

Building an AI Governance Framework for India, Part II

Embedding Principles of Safety, Equality and Non-Discrimination

This post has been authored by Jhalak M. Kakkar and Nidhi Singh

In July 2020, the NITI Aayog released a draft Working Document entitled “Towards Responsible AI for All” (hereafter ‘NITI Working Document’ or ‘Working Document’). This Working Document was initially prepared for an expert consultation held on 21 July 2020. It was later released for comments by stakeholders on the development of a ‘Responsible AI’ policy in India. CCG responded with comments to the Working Document, and our analysis can be accessed here.

In our previous post on building an AI governance framework for India, we discussed the legal and regulatory implications of the proposed Working Document and argued that India’s approach to regulating AI should be (1) firmly grounded in its Constitutional framework and (2) based on clearly articulated overarching principles. While the NITI Working Document introduces certain principles, it does not go into any substantive details on what the adoption of these principles into India’s regulatory framework would entail.

We will now examine these ‘Principles for Responsible AI’, their constituent elements and avenues for incorporating them into the Indian regulatory framework. The NITI Working Document proposed the following seven ‘Principles for Responsible AI’ to guide India’s regulatory framework for AI systems: 

  1. Safety and reliability
  2. Equality
  3. Inclusivity and Non-Discrimination
  4. Privacy and Security 
  5. Transparency
  6. Accountability
  7. Protection and Reinforcement of Positive Human Values. 

This post explores the principles of Safety and Reliability, Equality, and Inclusivity and Non-Discrimination. A subsequent post will discuss the principles of Privacy and Security, Transparency, Accountability and the Protection and Reinforcement of Positive Human Values.

Principle of Safety and Reliability

The Principle of Reliability and Safety aims to ensure that AI systems operate reliably in accordance with their intended purpose throughout their lifecycle and ensures the security, safety and robustness of an AI system. It requires that AI systems should not pose unreasonable safety risks, should adopt safety measures which are proportionate to the potential risks, should be continuously monitored and tested to ensure compliance with their intended purpose, and should have a continuous risk management system to address any identified problems. 

Here, it is important to note the distinction between safety and reliability. The reliability of a system relates to the ability of an AI system to behave exactly as its designers have intended and anticipated. A reliable system would adhere to the specifications it was programmed to carry out. Reliability is therefore, a measure of consistency and establishes confidence in the safety of a system. Whereas, safety refers to an AI system’s ability to do what it is supposed to do without harming users (human physical integrity), resources or the environment.

Human oversight: An important aspect of ensuring the safety and reliability of AI systems is the presence of human oversight over the system. Any regulatory framework that is developed in India to govern AI systems must incorporate norms that specify the circumstances and degree to which human oversight is required over various AI systems. 

The level of involvement of human oversight would depend upon the sensitivity of the function and potential for significant impact on an individual’s life which the AI system may have. For example, AI systems deployed in the context of the provision of government benefits should have a high level of human oversight. Decisions made by the AI system in this context should be reviewed by a human before being implemented. Other AI systems may be deployed in contexts that do not need constant human involvement. However, these systems should have a mechanism in place for human review if a question is subsequently raised for review by, say a user. An example of this may be vending machines which have simple algorithms. Hence, the purpose for which the system is deployed and the impact it could have on individuals would be relevant factors in determining if ‘human in the loop’, ‘human on the loop’, or any other oversight mechanism is appropriate. 

Principle of Equality

The principle of equality holds that everyone, irrespective of their status in the society, should get the same opportunities and protections with the development of AI systems. 

Implementing equality in the context of AI systems essentially requires three components: 

(i) Protection of human rights: AI instruments developed across the globe have highlighted that the implementation of AI would pose risks to the right to equality, and countries would have to take steps to mitigate such risks proactively. 

(ii) Access to technology: The AI systems should be designed in a way to ensure widespread access to technology, so that people may derive benefits from AI technology.

(iii) Guarantees of equal opportunities through technology: The guarantee of equal opportunity relies upon the transformative power of AI systems to “help eliminate relationships of domination between groups and people based on differences of power, wealth, or knowledge” and “produce social and economic benefits for all by reducing social inequalities and vulnerabilities.” AI systems will have to be designed and deployed such that they further the guarantees of equal opportunity and do not exacerbate and further entrench existing inequality.

The development, use and deployment of AI systems in society would pose the above-mentioned risks to the right to equality, and India’s regulatory framework for AI must take steps to mitigate such risks proactively.

Principle of Inclusivity and Non-Discrimination

The idea of non-discrimination mostly arises out of technical considerations in the context of AI. It holds that non-discrimination and the prevention of bias in AI should be mitigated in the training data, technical design choices, or the technology’s deployment to prevent discriminatory impacts. 

Examples of this can be seen in data collection in policing, where the disproportionate attention paid to neighbourhoods with minorities, would show higher incidences of crime in minority neighbourhoods, thereby skewing AI results. Use of AI systems becomes safer when they are trained on datasets that are sufficiently broad, and the datasets encompass the various scenarios in which the system is envisaged to be deployed. Additionally, datasets should be developed to be representative and hence avoid discriminatory outcomes from the use of the AI system. 

Another example of this can be semi-autonomous vehicles which experience higher accident rates among dark-skinned pedestrians due to the software’s poorer performance in recognising darker-skinned individuals. This can be traced back to training datasets, which contained mostly light-skinned people. The lack of diversity in the data set can lead to discrimination against specific groups in society. To ensure effective non-discrimination, AI policies must be truly representative of the society in its training data and ensure that no section of the populace is either over-represented or under-represented, which may skew the data sets. While designing the AI systems for deployment in India, the constitutional rights of individuals should be used as central values around which the AI systems are designed. 

In order to implement inclusivity in AI, the diversity of the team involved in design as well as the diversity of the training data set would have to be assessed. This would involve the creation of guidelines under India’s regulatory framework for AI to help researchers and programmers in designing inclusive data sets, measuring product performance on the parameter of inclusivity, selecting features to avoid exclusion and testing new systems through the lens of inclusivity.

Checklist Model: To address the challenges of non-discrimination and inclusivity a potential model which can be adopted in India’s regulatory framework for AI would be the ‘Checklist’. The European Network of Equality Bodies (EQUINET), in its recent report on ‘Meeting the new challenges to equality and non-discrimination from increased digitisation and the use of Artificial Intelligence’ provides a checklist to assess whether an AI system is complying with the principles of equality and non-discrimination. The checklist consists of several broad categories, with a focus on the deployment of AI technology in Europe. This includes heads such as direct discrimination, indirect discrimination, transparency, other types of equity claims, data protection, liability issues, and identification of the liable party. 

The list contains a series of questions which judges whether an AI system meets standards of equality, and identifies any potential biases it may have. For example, the question “Does the artificial intelligence system treat people differently because of a protected characteristic?” includes the parameters of both direct data and proxies. If the answer to the question is yes, the system would be identified as indulging in indirect bias. A similar checklist system, which has been contextualised for India, can be developed and employed in India’s regulatory framework for AI. 

Way forward

This post highlights some of the key aspects of the principles of Safety and Reliability, Equality, and Inclusivity and Non-Discrimination. Integration of these principles which have been identified in the NITI Working Document into India’s regulatory framework requires that we first clearly define their content, scope and ambit to identify the right mechanisms to operationalise them. Given the absence of any exploration of the content of these AI principles or the mechanism for their implementation in India in the NITI Working Document, we have examined the relevant international literature surrounding the adoption of AI ethics and suggested mechanisms for their adoption. The NITI Working Document has spurred discussion around designing an effective regulatory framework for AI. However, these discussions are at a preliminary stage and there is a need to develop a far more nuanced proposal for a regulatory framework for AI.

Over the last week, India has hosted the Responsible AI for Social Empowerment (RAISE) Summit which has involved discussions around India’s vision and roadmap for social transformation, inclusion and empowerment through Responsible AI. As we discuss mechanisms for India to effectively harness the economic potential of AI, we also need to design an effective framework to address the massive regulatory challenges emerging from the deployment of AI—simultaneously, and not as an afterthought post-deployment. While a few of the RAISE sessions engaged with certain aspects of regulating AI, there still remains a need for extensive, continued public consultations with a cross section of stakeholders to embed principles for Responsible AI in the design of an effective AI regulatory framework for India. 

For a more detailed discussion on these principles and their integration into the Indian context, refer to our comments to the NITI Aayog here. 

Building an AI governance framework for India

This post has been authored by Jhalak M. Kakkar and Nidhi Singh

In July 2020, the NITI Aayog released a “Working Document: Towards Responsible AI for All” (“NITI Working Document/Working Document”). The Working Document was initially prepared for an expert consultation held on 21 July 2020. It was later released for comments by stakeholders on the development of a ‘Responsible AI’ policy in India. CCG responded with comments to the Working Document, and our analysis can be accessed here.

The Working Document highlights the potential of Artificial Intelligence (“AI”) in the Indian context. It attempts to identify the challenges that will be faced in the adoption of AI and makes some recommendations on how to address these challenges. The Working Document emphasises the economic potential of the adoption of AI in boosting India’s annual growth rate, its potential for use in the social sector (‘AI for All’) and the potential for India to export relevant social sector products to other emerging economies (‘AI Garage’). 

However, this is not the first time that the NITI Aayog has discussed the large-scale adoption of AI in India. In 2018, the NITI Aayog released a discussion paper on the “National Strategy for Artificial Intelligence” (“National Strategy”). Building upon the National Strategy, the Working Document attempts to delineate ‘Principles for Responsible AI’ and identify relevant policy and governance recommendations. 

Any framework for the regulation of AI systems needs to be based on clear principles. The ‘Principles for Responsible AI’ identified by the Working Document include the principles of safety and reliability, equality, inclusivity and non-discrimination, privacy and security, transparency, accountability, and the protection and reinforcement of positive human values. While the NITI Working Document introduces these principles, it does not go into any substantive details on the regulatory approach that India should adopt and what the adoption of these principles into India’s regulatory framework would entail. 

In a series of posts, we will discuss the legal and regulatory implications of the proposed Working Document and more broadly discuss the regulatory approach India should adopt to AI and the principles India should embed in it. In this first post, we map out key considerations that should be kept in mind in order to develop a comprehensive regulatory regime to govern the adoption and deployment of AI systems in India. Subsequent posts will discuss the various ‘Principles for Responsible AI’, their constituent elements and how we should think of incorporating them into the Indian regulatory framework.

Approach to building an AI regulatory framework 

While the adoption of AI has several benefits, there are several potential harms and unintended risks if the technology is not assessed adequately for its alignment with India’s constitutional principles and its impact on the safety of individuals. Depending upon the nature and scope of the deployment of an AI system, its potential risks can include the discriminatory impact on vulnerable and marginalised communities, and material harms such as the negative impact on the health and safety of individuals. In the case of deployments by the State, risks include violation of the fundamental rights to equality, privacy, freedom of assembly and association, and freedom of speech and expression. 

We highlight some of the regulatory considerations that should be considered below:

Anchoring AI regulatory principles within the constitutional framework of India

The use of AI systems has raised concerns about their potential to violate multiple rights protected under the Indian Constitution such as the right against discrimination, the right to privacy, the right to freedom of speech and expression, the right to assemble peaceably and the right to freedom of association. Any regulatory framework put in place to govern the adoption and deployment of AI technology in India will have to be in consonance with its constitutional framework. While the NITI Working Document does refer to the idea of the prevailing morality of India and its relation to constitutional morality, it does not comprehensively address the idea of framing AI principles in compliance with India’s constitutional principles.

For instance, the government is seeking to acquire facial surveillance technology, and the National Strategy discusses the use of AI-powered surveillance applications by the government to predict crowd behaviour and for crowd management. The use of AI powered surveillance systems such as these needs to be balanced with their impact on an individual’s right to freedom of speech and expression, privacy and equality. Operational challenges surrounding accuracy and fairness in these systems raise further concerns. Considering the risks posed to the privacy of individuals, the deployment of these systems by the government, if at all, should only be done in specific contexts for a particular purpose and in compliance with the principles laid down by the Supreme Court in the Puttaswamy case.

In the context of AI’s potential to exacerbate discrimination, it would be relevant to discuss the State’s use of AI systems for the sentencing of criminals and assessing recidivism. AI systems are trained on existing datasets. These datasets tend to contain historically biased, unequal and discriminatory data. We have to be cognizant of the propensity for historical bias’ and discrimination getting imported into AI systems and their decision making. This could further reinforce and exacerbate the existing discrimination in the criminal justice system towards marginalised and vulnerable communities, and result in a potential violation of their fundamental rights.

The National Strategy acknowledges the presence of such biases and proposes a technical approach to reduce bias. While such attempts are appreciable in their efforts to rectify the situation and yield fairer outcomes, such an approach disregards the fact that these datasets are biased because they arise from a biased, unequal and discriminatory world. As we seek to build effective regulation to govern the use and deployment of AI systems, we have to remember that these are socio-technical systems that reflect the world around us and embed the biases, inequality and discrimination inherent in the Indian society. We have to keep this broader Indian social context in mind as we design AI systems and create regulatory frameworks to govern their deployment. 

While, the Working Document introduces the principles for responsible AI such as equality, inclusivity and non-discrimination, and privacy and security, there needs to be substantive discussion around incorporating these principles into India’s regulatory framework in consonance with constitutional guaranteed rights.

Regulatory Challenges in the adoption of AI in India

As India designs a regulatory framework to govern the adoption and deployment of AI systems, it is important that we keep the following in focus: 

  • Heightened threshold of responsibility for government or public sector deployment of AI systems

The EU is considering adopting a risk-based approach for regulation of AI, with heavier regulation for high-risk AI systems. The extent of risk factors such as safety, consumer rights and fundamental rights are assessed by looking at the sector of deployment and the intended use of the AI system. Similarly, India must consider the adoption of a higher regulatory threshold for the use of AI by at least government institutions, given their potential for impacting citizen’s rights. Government use of AI systems that have the potential of severely impacting citizens’ fundamental rights include the use of AI in the disbursal of government benefits, surveillance, law enforcement and judicial sentencing

  • Need for overarching principles based AI regulatory framework

Different sectoral regulators are currently evolving regulations to address the specific challenges posed by AI in their sector. While it is vital to harness the domain expertise of a sectoral regulator and encourage the development of sector-specific AI regulations, such piecemeal development of AI principles can lead to fragmentation in the overall approach to regulating AI in India. Therefore, to ensure uniformity in the approach to regulating AI systems across sectors, it is crucial to put in place a horizontal overarching principles-based framework. 

  • Adaptation of sectoral regulation to effectively regulate AI

In addition to an overarching regulatory framework which forms the basis for the regulation of AI, it is equally important to envisage how this framework would work with horizontal or sector-specific laws such as consumer protection law and the applicability of product liability to various AI systems. Traditionally consumer protection and product liability regulatory frameworks have been structured around fault-based claims. However, given the challenges concerning explainability and transparency of decision making by AI systems, it may be difficult to establish the presence of defects in products and, for an individual who has suffered harm, to provide the necessary evidence in court. Hence, consumer protection laws may have to be adapted to stay relevant in the context of AI systems. Even sectoral legislation regulating the use of motor vehicles, such as the Motor Vehicles Act, 1988 would have to be modified to enable and regulate the use of autonomous vehicles and other AI transport systems. 

  • Contextualising AI systems for both their safe development and use

To ensure the effective and safe use of AI systems, they have to be designed, adapted and trained on relevant datasets depending on the context in which they will be deployed. The Working Document envisages India being the AI Garage for 40% of the world – developing AI solutions in India which can then be deployed in other emerging economies. Additionally, India will likely import AI systems developed in countries such as the US, EU and China to be deployed within the Indian context. Both scenarios involve the use of AI systems in a context distinct from the one in which they have been developed. Without effectively contextualising socio-technical systems like AI systems to the environment they are to be deployed in, there are enhanced safety, accuracy and reliability concerns. Regulatory standards and processes need to be developed in India to ascertain the safe use and deployment of AI systems that have been developed in contexts that are distinct from the ones in which they will be deployed. 

The NITI Working Document is the first step towards an informed discussion on the adoption of a regulatory framework to govern AI technology in India. However, there is a great deal of work to be done. Any regulatory framework developed by India to govern AI must balance the benefits and risks of deploying AI, diminish the risk of any harm and have a consumer protection framework in place to adequately address any harm that may arise. Besides this, the regulatory framework must ensure that the deployment and use of AI systems are in consonance with India’s constitutional scheme.

CCG’s Comments on the NODE Whitepaper

By Shashank Mohan and Nidhi Singh

In late March, the Ministry of Electronics and Information Technology (MeitY) released its consultation whitepaper on the National Open Digital Ecosystems (NODE). The NODE strategy was developed by MeitY in consultation with other departments and stakeholders, as a part of its efforts to build an enabling ecosystem to leverage digital platforms for transformative social, economic and governance impact, through a citizen-centric approach. The Whitepaper highlights key elements of NODE, and also its distinction from the previous models of GovTech. The Centre submitted its comments on the NODE Whitepaper on 31 May 2020, highlighting some of our key concerns with the proposed strategy.

The NODE Whitepaper proposes a complex network of digital platforms with the aim of providing efficient public services to the citizens of India. It defines NODE as open and secure delivery platforms anchored by transparent governance mechanisms, which enable a community of partners to unlock innovative solutions, to transform societal outcomes.

Our comments on the NODE strategy revolve around four key challenges: open standards, privacy and security, transparency and accountability, and community engagement. We have provided recommendations at each stage and have relied upon our previous work around privacy, cyber security and technology policy for our analysis.

Firstly, we believe that the NODE Whitepaper stops short of providing a robust definition of openness, and does not comprehensively address existing Government policies on open source software and open APIs. We recommend that existing policies are adopted by MeitY where relevant, and are revised and updated at least in the context of NODEs where required.

Secondly, one of the key concerns with the NODE Whitepaper is the lack of detailed discussion on the aspects of data privacy and security. The Whitepaper does not consider the principles of data protection established in the Personal Data Protection Bill, 2019 (PDPB 2019) or take into account other internationally recognised principles. Without adequately addressing the data privacy concerns which arise from NODEs, any policy framework on the subject runs the risk of being devoid of context. The existence of a robust privacy framework is essential before instituting a NODE like architecture. As the PDPB 2019 is considered by Parliament, MeitY should, as a minimum, incorporate the data protection principles as laid down in the PDPB 2019 in any policy framework for NODEs. We also recommend that in order to fully protect the right to privacy and autonomy of citizens, participation in or the use of NODEs must be strictly voluntary.

Thirdly, a NODE framework built with the aim of public service delivery should also incorporate principles of transparency and accountability at each level of the ecosystem. In a network involving numerous stakeholders including private entities, it is essential that the NODE architecture operates on sound principles of transparency and accountability and sets up independent institutions for regulatory and grievance redressal purposes. Public private relationships within the ecosystem must remain transparent in line with the Supreme Court jurisprudence on the subject. To this end, we recommend that each NODE platform should be supported and governed by accountable institutions, in a transparent manner. These institutions must be independent and not disproportionately controlled by the Executive arm of the Government.

Lastly, we focus on the importance of inclusion in a digital first solution like the NODE. Despite steady growth in Internet penetration in India, more than half of its population does not enjoy access to the Internet and there is a crucial gender gap in the access to Internet amongst Indians, with men forming a majority of the user base. Learning from studies on the challenges of exclusion from the Aadhaar project, we recommend that the NODE architecture must be built keeping in mind India’s digital infrastructure. Global best practices suggest that designing frameworks which are based on inclusion is a pre-condition for building successful models of e-governance. Similarly, NODEs should be built with the aim of inclusion, and must not become a roadblock for accessing public services by citizens.

Public consultations like these will go a long way in building a robust strategy on open data systems as numerous stakeholders with varied skills must be consulted to ensure quality and efficacy in e-governance models. We thank MeitY for this opportunity and hope that future developments would also follow a similar process of public consultations to foster transparency, openness and public participation in the process of policy making.

Our full comments submitted to the Ministry can be found here.