As companies increase their use of artificial intelligence (AI), people are questioning the extent to which human biases have made their way into AI systems. Examples of AI bias in the real world show us that when discriminatory data and algorithms are baked into AI models, the models deploy biases at scale and amplify the resulting negative effects.
Companies are motivated to tackle the challenge of bias in AI not only to achieve fairness, but also to ensure better results. However, just as systemic racial and gender bias have proven difficult to eliminate in the real world, eliminating bias in AI is no easy task.
In the article, What AI can and can’t do (yet) for your business, authors Michael Chui, James Manyika, and Mehdi Miremadi of McKinsey noted, “Such biases have a tendency to stay embedded because recognizing them, and taking steps to address them, requires a deep mastery of data-science techniques, as well as a more meta-understanding of existing social forces, including data collection. In all, debiasing is proving to be among the most daunting obstacles, and certainly the most socially fraught, to date.”
Examples of AI bias from real life provide organizations with useful insights on how to identify and address bias. By looking critically at these examples, and at successes in overcoming bias, data scientists can begin to build a roadmap for identifying and preventing bias in their machine learning models.
AI bias, also referred to as machine learning bias or algorithm bias, refers to AI systems that produce biased results that reflect and perpetuate human biases within a society, including historical and current social inequality. Bias can be found in the initial training data, the algorithm, or the predictions the algorithm produces.
When bias goes unaddressed, it hinders people’s ability to participate in the economy and society. It also reduces AI’s potential. Businesses cannot benefit from systems that produce distorted results and foster mistrust among people of color, women, people with disabilities, the LGBTQ community, or other marginalized groups of people.
Eliminating AI bias requires drilling down into datasets, machine learning algorithms and other elements of AI systems to identify sources of potential bias.
AI systems learn to make decisions based on training data, so it is essential to assess datasets for the presence of bias. One method is to review data sampling for over- or underrepresented groups within the training data. For example, training data for a facial recognition algorithm that over-represents white people may create errors when attempting facial recognition for people of color. Similarly, security data that includes information gathered in geographic areas that are predominantly black could create racial bias in AI tools used by police.
Bias can also result from how the training data is labeled. For example, AI recruiting tools that use inconsistent labeling or exclude or over-represent certain characteristics could eliminate qualified job applicants from consideration.
Using flawed training data can result in algorithms that repeatedly produce errors, unfair outcomes, or even amplify the bias inherent in the flawed data. Algorithmic bias can also be caused by programming errors, such as a developer unfairly weighting factors in algorithm decision-making based on their own conscious or unconscious biases. For example, indicators like income or vocabulary might be used by the algorithm to unintentionally discriminate against people of a certain race or gender.
When people process information and make judgments, we are inevitably influenced by our experiences and our preferences. As a result, people may build these biases into AI systems through the selection of data or how the data is weighted. For example, cognitive bias could lead to favoring datasets gathered from Americans rather than sampling from a range of populations around the globe.
According to NIST, this source of bias is more common than you might think. In its report Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (NIST Special Publication 1270), NIST noted that “human and systemic institutional and societal factors are significant sources of AI bias as well, and are currently overlooked. Successfully meeting this challenge will require taking all forms of bias into account. This means expanding our perspective beyond the machine learning pipeline to recognize and investigate how this technology is both created within and impacts our society.”
As society becomes more aware of how AI works and the possibility for bias, organizations have uncovered numerous high-profile examples of bias in AI in a wide range of use cases.
Identifying and addressing bias in AI begins with AI governance, or the ability to direct, manage and monitor the AI activities of an organization. In practice, AI governance creates a set of policies, practices and frameworks to guide the responsible development and use of AI technologies. When done well, AI governance ensures that there is a balance of benefits bestowed upon businesses, customers, employees and society as a whole.
Through AI governance policies, companies can build the following practices:
A proper technology mix can be crucial to an effective data and AI governance strategy, with a modern data architecture and trustworthy AI platform being key components. Policy orchestration within a data fabric architecture is an excellent tool that can simplify the complex AI audit processes. By incorporating AI audit and related processes into the governance policies of your data architecture, your organization can help gain an understanding of areas that require ongoing inspection.
At IBM Consulting, we have been helping clients set up an evaluation process for bias and other areas. As AI adoption scales and innovations evolve, so will the security guidance mature, as is the case with every technology that’s been embedded into the fabric of an enterprise across the years. Below, we share some best practices from IBM to help organizations prepare for the secure deployment of AI across their environments:
Explore what watson.governance can do for you