The Algorithm That Learned Bias — And The People Who Taught It

In 2018, Amazon abandoned an experimental artificial intelligence recruitment system that had been developed to help automate the process of reviewing job applications. The idea sounded straightforward: use machine learning to analyse large numbers of resumes, identify patterns associated with successful employees and help recruiters find promising candidates more quickly.

Instead, the project became one of the best-known examples of how AI can reproduce the biases hidden inside historical data. The system began penalising certain references associated with women, effectively learning that male candidates were more desirable for technical roles. Nobody had explicitly programmed the algorithm to discriminate. It arrived at that conclusion because of the hiring history it had been given to learn from.

The Idea Behind Amazon's Recruiting AI

Amazon's engineers wanted to build a system capable of doing something machine learning is particularly good at: finding patterns across enormous amounts of information.

The recruiting engine was trained using roughly a decade of resumes submitted to Amazon. It analysed information such as employment history, education, terminology and other characteristics appearing throughout those applications.

Its job was essentially to identify which applicants resembled people Amazon had previously hired successfully.

Candidates would then receive ratings ranging from one to five stars, somewhat resembling the scoring process a human recruiter might use when comparing applicants.

In theory, automation could make recruitment faster and potentially more consistent.

The problem was that the algorithm was learning from a hiring history that was anything but neutral.

The Bias Was Hidden Inside The Training Data

When the system was examined more closely, engineers discovered that it had developed a significant gender bias.

Resumes containing the word "women's" could be penalised. Someone mentioning that they had been captain of a women's chess club, for example, could receive a less favourable result.

The system reportedly also downgraded applicants associated with two women's colleges.

It had effectively discovered a pattern in which indicators associated with women correlated negatively with the profiles it had learned to prefer.

That pattern did not appear because someone deliberately wrote a rule saying that women should receive lower scores.

It emerged from the data.

The Algorithm Learned What Amazon Had Previously Rewarded

The recruiting system had been trained on applications submitted during a period when technical roles at Amazon, like much of the technology industry, were heavily male-dominated.

That meant the resumes associated with previous successful hires were disproportionately from men.

Machine learning systems do not inherently understand social inequality, historical discrimination or why one demographic may be underrepresented in a particular profession.

They identify statistical relationships.

If the historical data repeatedly tells the model that successful technical employees tend to have certain characteristics, the system can learn to prefer those characteristics — even when some of them indirectly reflect gender.

From the algorithm's perspective, it was simply doing what it had been asked to do.

Find people who looked like previous successful hires.

The Machine Was A Mirror Of The Past

This is what made Amazon's experiment so important.

The algorithm did not invent the underlying inequality.

It reflected it.

A decade of hiring decisions had already produced a workforce where men dominated many technical roles. When that history became training data, the AI treated those outcomes as examples of what successful hiring looked like.

The model effectively turned historical patterns into predictions about future candidates.

That creates a dangerous feedback loop.

If yesterday's workforce was shaped by unequal opportunities, an algorithm trained to reproduce yesterday's hiring patterns can quietly reinforce those same inequalities tomorrow.

So Who Actually Taught The Algorithm To Be Biased?

The uncomfortable answer is that no single individual did.

There was no engineer sitting at a desk explicitly programming a preference for male candidates.

The bias emerged from the institution's accumulated decisions.

Years of recruitment outcomes, workforce composition and industry patterns became the educational material from which the model learned.

In that sense, Amazon itself — or more accurately, its historical hiring data — became the teacher.

The algorithm simply identified what the data suggested had previously been successful.

That is precisely why algorithmic bias can be so difficult to detect. The system may appear objective because it is driven by mathematics, yet the patterns behind those calculations can originate from very human decisions.

Historical Data Is Not Automatically Neutral Data

A common mistake when designing machine-learning systems is assuming that large historical datasets represent objective reality.

They do not.

Historical data represents what actually happened, but what happened may have been influenced by inequality, cultural assumptions, unequal access to opportunities or previous human bias.

A hiring dataset may therefore contain much more than information about skills and qualifications.

It can encode decades of organisational behaviour.

When engineers feed that information into a model without examining the context behind it, the AI may interpret those patterns as signals worth preserving.

The computer cannot automatically distinguish between a meaningful predictor and evidence of historical discrimination.

Removing The Word "Women's" Was Not Enough

Once engineers discovered the problem, they attempted to correct it.

One obvious approach was to remove or neutralise terms that had become associated with gender.

But eliminating a handful of words did not solve the underlying problem.

Machine-learning models can identify indirect signals.

Even if the word "women's" disappears from the dataset, other information may correlate with gender. Educational history, membership organisations, career patterns and countless other details could potentially act as proxies.

The system does not need to know that a feature represents gender.

It only needs to learn that the feature correlates with previous hiring outcomes.

Bias Can Move Rather Than Disappear

This is one of the most challenging aspects of algorithmic fairness.

Removing an explicitly sensitive attribute such as gender does not automatically produce a neutral model.

Other variables may contain similar information indirectly.

A postcode might correlate with income or ethnicity. Employment gaps could correlate with caregiving responsibilities. Particular schools or organisations may correlate with gender or socioeconomic background.

An algorithm can therefore recreate discriminatory patterns even when protected characteristics are never explicitly included.

This is why bias mitigation requires much more than simply deleting a few obvious fields from the training data.

Amazon Eventually Abandoned The Project

By 2018, confidence in the recruiting system had deteriorated enough that Amazon abandoned the project.

Engineers had attempted to address the gender-related patterns, but there was no guarantee that the system would not discover other ways of reproducing similar preferences.

For a hiring tool, that uncertainty was particularly serious.

Recruitment decisions affect people's careers, incomes and opportunities. A system that quietly disadvantages certain candidates can create real consequences even if the discrimination was never intentional.

Automation was supposed to help make recruitment more efficient.

Instead, the experiment demonstrated how automation can scale existing problems just as easily as it can solve them.

Automation Does Not Automatically Create Fairness

There is a tempting assumption that replacing human judgement with algorithms will remove subjective bias.

That is only partly true.

Algorithms can eliminate certain forms of inconsistency. They do not become tired, develop personal grudges or favour someone simply because they had a friendly conversation during an interview.

But machine-learning systems depend heavily on the data, objectives and assumptions chosen by humans.

If those ingredients contain bias, automation can simply reproduce it at much greater scale.

A biased recruiter might affect dozens of candidates.

A biased automated screening system could influence thousands.

The Objective Function Matters Too

Another lesson from the Amazon example involves the question the system was asked to answer.

The recruiting tool was designed to identify candidates resembling people the company had successfully hired in the past.

That sounds logical until you examine what the objective actually means.

If the goal is to reproduce the characteristics of previous hires, then historical workforce composition becomes part of the definition of success.

A better question might have been whether a candidate possessed the skills required to perform the role successfully, independent of demographic patterns among previous employees.

How a machine-learning problem is framed can therefore be just as important as the data used to train it.

AI Systems Learn Correlations, Not Social Context

Humans can understand that a technical workforce being predominantly male may reflect decades of structural factors rather than evidence that men are naturally better engineers.

A machine-learning model does not automatically possess that context.

It sees relationships between variables.

If one set of characteristics appears frequently among successful examples, the system learns that those characteristics are useful predictors.

That is mathematically reasonable.

Socially, however, it can be completely wrong.

This gap between statistical correlation and human context is one of the central challenges of deploying AI in areas such as hiring, lending, healthcare and education.

The Same Problem Exists Far Beyond Recruitment

Amazon's abandoned recruiting tool became famous because the bias was relatively easy to understand.

But similar risks exist anywhere algorithms learn from human history.

A lending model trained on historical loan approvals could reproduce financial discrimination.

A healthcare system trained on unequal medical access could underestimate the needs of populations historically underserved by hospitals.

A predictive policing system trained on historical arrest records could reinforce patterns created by where policing resources were previously concentrated.

The specific dataset changes, but the underlying problem remains similar.

Algorithms learn from what happened before.

Whether that history should determine what happens next is a human decision.

Better Engineering Still Matters

It would be inaccurate to conclude that technical solutions cannot help.

Better dataset auditing, fairness testing, bias detection, representative sampling and model evaluation can all reduce risk.

Engineers can examine whether model outcomes differ significantly between demographic groups and investigate which features are influencing those decisions.

Independent testing can also reveal problems that internal development teams may overlook.

The point is not that engineering is powerless.

It is that engineering alone cannot completely solve social problems encoded in the underlying data.

Human Oversight Remains Essential

High-impact AI systems should not operate without meaningful human oversight.

That does not simply mean placing a human at the end of the process and asking them to approve whatever the algorithm recommends.

The people supervising the system need enough authority, information and understanding to challenge its decisions.

If recruiters automatically trust an AI ranking because the computer supposedly analysed more information than they could, human review becomes little more than a rubber stamp.

Effective oversight requires understanding where the model can fail and actively checking for those failures.

Transparency Matters When Algorithms Affect People

People affected by automated decisions should also have some understanding of how those systems are being used.

A job applicant may never know that an algorithm rejected their resume before a recruiter even saw it.

That raises difficult questions about accountability.

If an automated system systematically disadvantages a group of people, who is responsible?

The software developer?

The company deploying it?

The managers who approved the training data?

Ultimately, responsibility cannot be transferred to the algorithm.

AI has no accountability of its own.

The organisation choosing to use it does.

Training Data Needs To Be Questioned, Not Just Collected

The Amazon case demonstrates why developers should ask where training data came from before celebrating how much of it they have.

More data is not automatically better data.

Teams need to understand:

These questions belong at the beginning of an AI project, not after a problem reaches production.

Designers And Product Managers Share Responsibility

Algorithmic fairness is not solely the responsibility of data scientists.

Product managers decide what problems AI should solve.

Designers determine how algorithmic recommendations appear to users and how much authority those recommendations seem to carry.

Executives decide whether systems are deployed at all.

Developers determine how safeguards are implemented.

Legal and policy teams decide what kinds of automated decisions are acceptable.

When an AI system causes harm, responsibility therefore exists across the entire product lifecycle.

AI Reflects The Values Embedded In Its Environment

Machine-learning systems are often described as if they independently discover truth from data.

A more accurate description is that they discover patterns.

Whether those patterns should become decisions is another question entirely.

Historical data contains the values, inequalities and structures of the environment that produced it.

When algorithms learn from that data, some of those characteristics inevitably influence the model.

The responsibility of AI development is therefore not simply to ask whether a model predicts accurately.

We also need to ask what exactly it has learned to predict, and why.

The Real Lesson Was Not That AI Became Sexist

The simplified version of this story is that Amazon built an AI recruiter and the AI became sexist.

The deeper lesson is more important.

The system behaved consistently with the incentives and examples it was given.

It learned that previous successful hires disproportionately looked a certain way, then used that information when judging new candidates.

The failure happened when historical outcomes were treated as an appropriate blueprint for future decisions.

The algorithm exposed the bias.

It did not create the history behind it.

Final Thoughts

Amazon's abandoned recruiting system remains one of the clearest examples of why AI cannot simply be treated as an objective decision-maker. The machine-learning model was trained to recognise patterns among people the company had previously hired, and those historical patterns reflected a workforce where men overwhelmingly dominated technical positions.

The algorithm did exactly what machine learning systems are designed to do: it learned from the data.

Unfortunately, the data contained lessons that should not have been repeated.

Attempts to remove explicitly gendered terms could not guarantee fairness because bias can reappear through indirect signals and proxy variables. That ultimately made the system too difficult to trust for something as consequential as employment.

The responsibility was never really with the algorithm. An algorithm cannot decide whether the history contained in its training data represents a fair world.

That responsibility belongs to the people designing, training and deploying the system.

And that may be the most important lesson for everyone working with AI today: before asking what an algorithm has learned, we should first ask what we taught it.