Adverse Impact

IBM Watson Recruitment offers you an Adverse Impact Analysis (AIA) tool that uses Artificial Intelligence (AI) to observe and report on areas of possible Adverse Impact within hiring models produced by Watson Recruitment. The AIA tool indicates the extent to which particular groups score differently from other groups. IBM recommends AIA testing prior to implementing success or match score and periodically testing over time. AIA uses demographic information about candidates to identify the different people groups for analysis together with candidate scoring data in Watson Recruitment. The demographic information is NOT used as part of the scoring process so you need to provide this demographic information separately in an uploaded file. The IWR Adverse Impact feature uses your uploaded demographic information and either the Four Fifth's Rule or Fisher's Exact test to measure Adverse Impact. When AIA process completes, the uploaded demographic file is deleted for data protection reasons.

Note: IWR users who want to use the Adverse Impact tool must have the IWR_ADVERSE_IMPACT user designation. Contact your Technical Contact to request the user designation for Adverse Impact.
Relevant eLearning
What does Adverse Impact Analysis Do?
Adverse impact is a term that is used in federal guidelines and court proceedings that are related to employment that refers to a standard of fairness in the selection of protected demographic groups that are protected by the Civil Rights Act of 1964 (for example, Uniform Guidelines; US Equal Employment Opportunity Commission, 1978). In accordance with the standards in the industry, adverse impact is evaluated using these methods in this report.
Adverse Impact Reports allow you to test various types of candidate data in your Applicant Tracking Systems (ATS) such as historical candidate data, scored output data from IBM Watson Recruitment for match score, success score, and success metrics that are used to create models for IWR success scores.
Adverse Impact uses one of two standard tests to determine if a company's hiring practices have any adverse impact on a protected group. The Four Fifth's Rule and the Fisher's Exact test are standard tests that use both demographic information and employment data in your calculations.
  • The Four Fifth's Rule uses demographic information and a company's employment data to determine if the selection rate for any group is substantially less (less than 4/5ths or 80 percent) than the selection rate for the highest group.
  • For example, if 90 percent of men who apply for a job for a particular position are selected (highest group), and the women who are selected for the same position are calculated as being at a rate lower than 72 percent (80 percent of the highest selection rate, 90 percent), this calculation would indicate evidence of adverse impact.
  • The Fisher's Exact test uses statistical analysis to analyze contingency tables and the interaction of the variables within the contingency tables to see if there is a statistically significant difference in outcomes.
  • The Four Fifth's rule requires that the groups being tested are larger than 200 but reports are run for group sizes below 200 even though the Four Fifth's rule might not be the appropriate test. The Fisher's Exact test is considered a better measurement tool for smaller groups.
Notes About Adverse Impact Reports
Protected groups are identified from the values in the uploaded demographics file and totaled at the job role level. If a group does not meet the 2 percent of the applicant pool, it is dropped from the analysis and does not appear in the report. The 2 percent threshold comes from EEOC recommendations. The 2 percent threshold is calculated as a percentage of the total number of candidates who provided information for the demographic.
When performing Adverse Impact on Race, a multi-value demographic, two rolled up groups might be created. One is a rolled up group representing all groups apart from the maximum group and if this maximum group is not White, a Non-white group will also be created. For both cases, candidates excluded under the 2 percent test will be included.
Required Data for Adverse Impact Reports
Running Adverse Impact reports requires that a number of data points are collected by your ATS application.
  • The country of the requisitions being tested (reports are collated at the country level)
  • Job Roles for requisitions (Adverse Impact reports at the job level for each test)
  • Demographic Data for candidates (Tests for Age, Race, and Gender are supported)
    • The IWR application does not require that this data is collected but it is needed in order to run Adverse Impact analysis. If the country field in a req is empty or not recognizable as a country, the corresponding data and associated candidates are not included in the report.
Note: The information that is collected and used in Adverse Impact Reporting is not used as part of the hiring process or for any adverse decision.
When running the Scoring Candidate Data Analysis or either of the Adverse Impact for Match and Success Score users can input a percentile value for the test. The percentile value applies to continuous values like Match and Success scores and defaults to the 30th percentile for the scores in the data. For example, if the 30th percentile corresponded to a score of 38.6 for a Success Score, the selection groups for the analysis would be less than 38.6 and more than 38.6.
Demographic Reports - What do I need?
Running Adverse Impact requires that you upload a demographic file in .csv format to be used in Adverse Impact report calculations. IWR does not require that this demographic data is provided, but it needs to be collected and provided through the Adverse Impact user interface to run analysis. To fully support the ability to test for Adverse Impact with IWR, your ATS application should be able to optionally collect the required demographic information from applicants.
Note: IWR does not collect or persist sensitive personal information (SPI) at a candidate level. Uploaded demographic files are deleted immediately after an adverse impact report runs.
Your demographic file MUST contain one of the following columns of data AND all columns headings are case sensitive.
Sample Demographic File
Column Description
personId The Candidate personId from the your application. This is used to join the demographic data in this file with the IWR data for the candidate. Each candidate personId can only be in the file once. personIds that are not found for candidates stored in IWR are NOTincluded in the analysis. Similarly, personIds that are supplied but will not join with a requisition for the country that is specified in the Adverse Impact report, are not included in the Adverse Impact analysis. Note, Duplicate personIds are not allowed and will fail validation. Blank values are not allowed.
Age Values must be an integer. Values are categorized into two categories above and below a user configured threshold in the Adverse Impact UI. For example, over/under 40.
Race Raw values display in the Adverse Impact UI and no cleansing or standardization of values occurs. Recommended values should align with EEOC (US) or equivalent country-specific standard definitions.
Gender Raw values display in the Adverse Impact UI and no cleansing or standardization of values occurs. Recommended consistent values are provided, for example, Male and Female or M and F.
Demographic File Creation Rules
Rules for completing the demographic file MUST be followed in order to run an Adverse Impact Report. When the demographic file is dropped into the Adverse Impact application, error validation occurs. If there are errors in your demographic file, the Adverse Impact Reports do not run.
The following rules for completing the demographic file MUST be followed. Use the following rules when completing your demographic file.
  • Must contain personId and optionally one or more of the following for each candidate: Age, Race, or Gender.
  • There must be an personId for each row. Blank values are not allowed.
  • Include only one row for each candidate.
  • Each candidate's personId value must match the candidate identifier that is used in IBM Watson Recruitment.
  • personId column must not contain duplicates.
  • Be in .csv format.
  • Be encoded using UTF-8.
  • Age column value must an integer and blank values are allowed.
  • The Gender column must have two values, either M and F or Male and Female. Blank values are allowed.
  • The Race column is free format text but should ideally have a few discrete values.
  • Each row must have at least ONE demographic specified.
  • You can also download a sample to use as a reference.
    Demographic file sample image.
The demographic file must contain candidate demographic data appropriate to the test being performed. For example, if an Adverse Impact report is being created for training data, the demographic files should contain the candidate data that is used to train IWR.
At least one value for Age, Race, or Gender must be provided. Sparse data is acceptable in this file, and candidates who do not provide a specific data point, for example, for Gender, are not included in the Adverse Impact report for gender. Simply omit data and leave blank values rather than specifying a category of missing data, for example, do not include a value like "No Age Provided."