What is AI bias?

When AI bias goes unaddressed, it can impact an organization’s success and hinder people’s ability to participate in the economy and society. Bias reduces AI’s accuracy, and therefore its potential.

Businesses are less likely to benefit from systems that produce distorted results. And scandals resulting from AI bias could foster mistrust among existing and potential customers.

The models upon which AI efforts are based absorb the biases of society that can be quietly embedded in the mountains of data they’re trained on. Historically biased data collection that reflects societal inequity can result in harm to historically marginalized groups in use cases including hiring, policing, credit scoring and many others. According to The Wall Street Journal, “As use of artificial intelligence becomes more widespread, businesses are still struggling to address pervasive bias.”¹

Distorted results can harm organizations and society at large. Here are a few of the more common types of AI bias⁷.

• Algorithm bias: Misinformation can result if the problem or question asked is not fully correct or specific, or if the feedback to the machine learning algorithm does not help guide the search for a solution.
  
  
• Cognitive bias: AI technology requires human input, and humans are fallible. Personal bias can seep in without practitioners even realizing it. This can impact either the dataset or model behavior.
  
  
• Confirmation bias: Closely related to cognitive bias, this happens when AI relies too much on pre-existing beliefs or trends in the data—doubling-down on existing biases, and unable to identify new patterns or trends.

• Exclusion bias: This type of bias occurs when important data is left out of the data being used, often because the developer has failed to see new and important factors.
  
  
• Measurement bias: Measurement bias is caused by incomplete data. This is most often an oversight or lack of preparation that results in the dataset not including the whole population that should be considered. For example, if a college wanted to predict the factors to successful graduation, but included only graduates, the answers would completely miss the factors that cause some to drop out.

• Out-group homogeneity bias: This is a case of not knowing what one doesn’t know. There is a tendency for people to have a better understanding of ingroup members—the group one belongs to—and to think they are more diverse than outgroup members. The result can be developers creating algorithms that are less capable of distinguishing between individuals who are not part of the majority group in the training data, leading to racial bias, misclassification and incorrect answers.

• Prejudice bias: Occurs when stereotypes and faulty societal assumptions find their way into the algorithm’s dataset, which inevitably leads to biased results. For example, AI could return results showing that only males are doctors and all nurses are female.
  
  
• Recall bias: This develops during data labeling, where labels are inconsistently applied by subjective observations.
  
  
• Sample/Selection bias: This is a problem when the data used to train the machine learning model isn’t large enough, not representative enough or is too incomplete to sufficiently train the system. If all school teachers consulted to train an AI model have the same academic qualifications, then any future teachers considered would need to have identical academic qualifications.
  
  
• Stereotyping bias: This happens when an AI system—usually inadvertently—reinforces harmful stereotypes. For example, a language translation system could associate some languages with certain genders or ethnic stereotypes. McKinsey gives a word of warning about trying to remove prejudice from datasets: “A naive approach is removing protected classes (such as sex or race) from data and deleting the labels that make the algorithm biased. Yet, this approach may not work because removed labels may affect the understanding of the model and your results’ accuracy may get worse.”²

The first step in avoiding the bias trap is just to step back at the beginning and give an AI effort some thought. As is true with almost any business challenge, problems are much easier to fix up-front rather than waiting for the train wreck and then sorting through the damaged result. But many organizations are in a rush: penny-wise-and-pound-foolish, and it costs them.

Identifying and addressing bias in AI requires AI governance, or the ability to direct, manage and monitor the AI activities of an organization. In practice, AI governance creates a set of policies, practices and frameworks to guide the responsible development and use of AI technologies. When done well, AI governance helps to ensure that there is a balance of benefits bestowed upon businesses, customers, employees and society as a whole.

AI governance often includes methods that aim to assess fairness, equity and inclusion. Approaches such as counterfactual fairness identifies bias in a model’s decision making and ensures equitable results, even when sensitive attributes, such as gender, race or sexual orientation are included.

Because of the complexity of AI, an algorithm can be a black box system with little insight into the data used to create it. Transparency practices and technologies help ensure that unbiased data is used to build the system and that results will be fair. Companies that work to protect customers’ information build brand trust and are more likely to create trustworthy AI systems.

To provide another layer of quality assurance, institute a “human-in-the-loop” system to offer options or make recommendations that can then be approved by human decisions.

Here’s a checklist of six process steps that can keep AI programs free of bias.

1. Select the correct learning model: When using a supervised model, stakeholders select the training data. It’s critical that the stakeholder team be diverse—not just data scientists— and that they have had training to help prevent unconscious bias. Unsupervised models use AI alone to identify bias. Bias prevention tools need to be built into the neural network so that it learns to recognize what is biased.

2. Train with the right data: Machine learning trained on the wrong data will produce wrong results. Whatever data is fed into the AI should be complete and balanced to replicate the actual demographics of the group being considered.

3. Choose a balanced team: The more varied the AI team—racially, economically, by educational level, by gender and by job description—the more likely it will recognize bias. The talents and viewpoints on a well-rounded AI team should include AI business innovators, AI creators, AI implementers, and a representation of the consumers of this particular AI effort.³

4. Perform data processing mindfully: Businesses need to be aware of bias at each step when processing data. The risk is not just in data selection: whether during pre-processing, in-processing or post-processing, bias can creep in at any point and be fed into the AI.

5. Continually monitor: No model is ever complete or permanent. Ongoing monitoring and testing with real-world data from across an organization can help detect and correct bias before it causes harm. To further avoid bias, organizations should consider assessments by an independent team from within the organization or a trusted third-party.

6. Avoid infrastructural issues: Aside from human and data influences, sometimes infrastructure itself can cause bias. For example, using data collected from mechanical sensors, the equipment itself could inject bias if the sensors are malfunctioning. This kind of bias can be difficult to detect and requires investment in the latest digital and technological infrastructures.