Algorithmic Bias & Invisible Discrimination
An AI system can discriminate against you without ever asking your name, your gender, or where you are from. It only needs data that correlates with those things.
The Rejection Nobody Explained
David and his university friend Anjali applied to the same batch of technology companies over three months. Same degree, same grades, same years of experience.
David got two callbacks. Anjali got eleven.
They compared notes. Different companies, different roles, different outcomes. No pattern they could identify. No feedback they could act on.
Neither knew that several of the companies they applied to used applicant tracking systems that scored CVs before human review. Neither knew that the scoring models weighted certain university names differently, flagged employment gaps as risk signals, or used factors neither had been asked to provide.
David could not contest a decision he did not know had been made. He could not correct a variable he did not know had been used.
What Is Actually Happening
Bias in AI is not always intentional. It is often inherited from the data the model was trained on.
72%
of hiring managers now use AI screening tools that filter CVs before human review.
The candidate usually does not know. The company often does not know what variables the model is using.
Source: LinkedIn Future of Recruiting Report, 2024Amazon scrapped its model
Amazon built and then scrapped an AI hiring tool in 2018 after discovering it systematically downgraded CVs containing the word "women's" and penalised graduates of all-women's colleges - trained on 10 years of male-dominated hiring history.
Zip code as a proxy
A UC Berkeley study found that algorithmic lenders charged Black and Latino borrowers 5.3% more in interest on identical risk profiles - attributed to geographic proxy variables in the model.
Lower care scores for same illness
A widely-used US healthcare algorithm was found to assign lower risk scores to Black patients than equally sick white patients, directing less care toward them. It used healthcare spending as a proxy for health need.
Who gets shown which jobs
Ad delivery algorithms show job ads to different demographic groups without targeting instructions. A 2019 study found delivery algorithms showed janitorial ads to Black users and taxi driver ads to men at higher rates with no explicit instruction given.
How Training Data Bias Becomes Model Bias
A model trained on historical hiring decisions inherits historical hiring biases. If the company historically hired fewer women into technical roles, the model learns that fewer women is the correct output.
The model does not decide to discriminate. It identifies patterns in past approved candidates and replicates them. The discrimination is structural, not intentional. The outcome is the same.
Training data reflects the world as it was. The model reproduces the world as it was.
Opacity: The Part You Cannot See
In most algorithmic decisions, you receive a result but not a reason. "Does not meet our current criteria" is legally sufficient in most jurisdictions.
You do not know whether an AI was involved, which variables were used, what weight each variable carried, or what score would have produced a different outcome. You cannot contest what you cannot see. This is not a technical limitation. In many cases, it is a design choice.
See It: The Decision You Didn't Make
Two identical loan applications. Same income, same credit score, same employment. The AI approves one and rejects the other - using variables neither applicant was told existed.
What That Just Showed You
1. Proxy discrimination does not require discriminatory intent. A model that uses ZIP code, device type, and application time can produce outcomes that track demographic lines - without encoding race, gender, or religion explicitly.
2. The feedback loop compounds the problem. If biased data trains a biased model, and the model's decisions generate new data that reflects its biases, the next model is trained on compounded bias. The cycle does not self-correct.
3. Opacity protects the system, not the individual. When you cannot see the variables used, you cannot challenge them. Opacity is frequently a design preference, not a technical necessity.
4. Fighting algorithmic discrimination requires knowing it happened. The first step is asking whether an automated decision was involved in any outcome that significantly affects you.
Three Things Worth Doing
1. Ask directly whether an automated system was involved. When rejected for a job, loan, or service, ask. In the EU under GDPR, you have a right to know when a solely automated decision was made about you and to request human review.
2. Document your application details before submitting. Keep copies of exactly what you submitted, when, and to whom. If a pattern of rejections emerges, documentation is the basis for a complaint.
3. Report suspected algorithmic discrimination. In India, complaints can be filed with sector regulators - RBI for lending, SEBI for financial services. In the EU, data protection authorities handle GDPR-related algorithmic decision complaints.
One Question Before You Continue
The Amazon hiring tool discriminated against women without being explicitly instructed to do so. How did this happen?