Why 'Fair' AI Is Mathematically Impossible
We are increasingly turning to algorithms to make critical decisions about our lives. From who gets a loan to who gets a job, and even who might be a high risk for reoffending, automated systems are becoming the new arbiters of opportunity and justice. This shift is often fueled by a core belief: that algorithms, unlike humans, are objective, impartial, and fundamentally fairer. We trust the code to be free of the messy, irrational biases that plague human judgment.
But this belief, while comforting, is built on a fragile foundation. The concept of “algorithmic fairness” is not a simple technical problem to be solved with better code. It is a minefield of mathematical paradoxes, unavoidable trade-offs, and deep ethical questions. The pursuit of a perfectly fair algorithm often leads us to uncomfortable conclusions about our technology and ourselves.
This article will explore four surprising truths about AI fairness that challenge our common assumptions. These are not edge cases or theoretical worries; they are fundamental constraints and realities that anyone building, deploying, or being judged by an algorithm needs to understand.
Perfect Fairness Is Mathematically Impossible
The single most startling revelation from research into algorithmic fairness is this: under real-world conditions, a model cannot satisfy all desirable fairness metrics at the same time. This isn’t a failure of engineering; it is a mathematical certainty known as an “impossibility theorem.”
To understand why, we need to look at what we mean by “fair.” There are many different mathematical standards. For example, we might want a model to satisfy:
Statistical Parity: This standard requires that the proportion of individuals receiving a positive outcome (like a loan approval) is the same across all groups. It focuses purely on equalizing outcomes.
Predictive Parity: This standard requires the model to be equally accurate for all groups. If the algorithm predicts someone will succeed, the probability of them actually succeeding should be the same regardless of their race or gender.
Equalized Odds: This standard requires the model to make the same types of errors for all groups. It ensures that the rate of false positives (incorrectly flagging someone as high-risk) and false negatives (incorrectly flagging someone as low-risk) is equal across different demographic groups.
Each of these sounds like a reasonable goal. The problem is that when two common conditions are met—(1) the underlying “base rates” (the actual prevalence of an outcome) differ between groups, and (2) the model is not 100% accurate—it is mathematically impossible to satisfy Predictive Parity and Equalized Odds at the same time. You are forced to choose. This is profoundly impactful because it means there is no single, technically “correct” fair solution. The pursuit of fairness requires choosing which type of fairness to prioritize, and by extension, which type of unfairness we are willing to accept.
The COMPAS Paradox: Biased Against Everyone, Differently
The mathematical trade-offs of fairness are not just theoretical. A famous 2016 ProPublica analysis of the COMPAS algorithm, a tool used to predict the likelihood of a defendant reoffending, provides a stark real-world example. The investigation found significant racial disparities in the algorithm’s errors, but in a surprisingly complex way.
The analysis revealed two different kinds of bias working in opposite directions:
False Positives: The algorithm was twice as likely to incorrectly classify Black defendants who would not reoffend as “high risk.” This error occurred for 45% of these Black defendants, compared to only 23% of White defendants.
False Negatives: The algorithm was more likely to incorrectly classify White defendants who would go on to reoffend as “low risk.” This error occurred for 48% of these White defendants, compared to 23% for Black defendants.
This finding is crucial. The algorithm wasn’t just “biased against Black defendants” in a simple sense. It was also biased against White defendants, but by making the opposite kind of mistake. It disproportionately harmed one group by over-predicting risk and harmed another by under-predicting it. This powerfully illustrates the trade-off: a system can be biased against different groups in different, and sometimes conflicting, ways.
The Mirror Problem: When an Algorithm Is Unfair Because It’s Too Accurate
We often assume that algorithmic bias is a technical glitch—a result of “bad data” or a poorly designed model that needs fixing. While technical flaws like Measurement Bias can certainly exist (for example, using healthcare spending as a proxy for health needs, which systematically underrates the needs of Black patients), a deeper and more challenging problem is Societal Bias. This occurs when an AI model is technically accurate and makes correct predictions based on historical data, but its use still perpetuates and amplifies existing social inequalities.
A chilling example of this is the 2020 Ofqual algorithm in the UK, designed to assign A-level grades after exams were canceled due to the pandemic. The algorithm was built to match historical grade distributions, accurately reflecting that certain schools consistently produced lower grades. However, this “accuracy” meant it unfairly capped the marks of high-performing students at historically disadvantaged schools, effectively punishing them for their school’s past performance. The algorithm held up a perfect mirror to historical inequality, and in doing so, perpetuated it.
If the data used to train a model reflects a world of discrimination, an “accurate” algorithm will learn to reproduce those same patterns, imposing a relative disadvantage based on group membership.
“[D]iscrimination consists of acts, practices, or policies that impose a relative disadvantage on persons based on their membership in a salient social group.”
The implication is profound. In these cases, the primary challenge is not debugging the algorithm. It is confronting the fact that our algorithms are showing us an uncomfortable, but accurate, picture of a biased world. Fixing the problem isn’t about correcting the mirror; it’s about addressing the reality it reflects.
The Real Choice: Preserve the Past or Transform the Future?
When we decide to mitigate unfairness in an algorithm, we face a fundamental choice not between technical methods, but between two competing philosophies of justice: Formal Equality and Substantive Equality.
Bias Preserving measures are the algorithmic expression of Formal Equality. This philosophy, rooted in the idea of “treating like cases alike,” aims to ensure a process is applied consistently. An algorithm designed with this goal in mind seeks to match error rates across groups, essentially accepting the existing distribution of outcomes in society as a neutral starting point. It works to prevent the algorithm from introducing new bias, thereby preserving the societal status quo, inequalities and all.
Bias Transforming measures are the expression of Substantive Equality. This philosophy argues that true fairness requires proactively accounting for historical disadvantage to “level the playing field.” An algorithm built on this principle actively works to counteract socially unacceptable disparities. For example, it might enforce Statistical Parity, ensuring different groups receive positive outcomes (like a loan approval) at the same rate. This approach is designed to transform, not just reflect, the existing social order.
This is not a technical choice between two competing algorithms; it is a normative choice about the very purpose of our technology. Adopting a bias transforming approach requires developers and policymakers to make an explicit moral judgment about which societal biases are unacceptable and must be corrected. It forces a difficult but necessary conversation about what kind of society we want to build.
What Are We Asking Our Algorithms to Be?
The journey into algorithmic fairness quickly reveals that it is not a simple problem of debugging code. It is a complex, socio-technical challenge defined by unavoidable trade-offs, mirrored social injustices, and deep moral questions. There is no easy answer, no magic algorithm that can satisfy every definition of fairness for everyone, all at once.
This forces us to confront a more fundamental question. The central issue is not just whether our algorithms should reflect the world as it is or as it should be. It is a choice between competing visions of justice: Do we encode a formal equality that preserves the past, or a substantive equality that aims to build a different future?