AI systems are increasingly embedded in the everyday work of government. They screen applications for public benefits, support decisions in justice and policing, and shape the allocation of healthcare resources. These tools now operate as quiet administrative actors, influencing outcomes that carry real consequences for individuals and communities.

This raises a central question: when these systems make errors or produce harmful outcomes, who is responsible? Accountability in automated decision-making is often presented as a matter of technical transparency, but the more persistent barriers are legal, institutional, and structural. The sections below outline four problems that shape the accountability landscape and explain why traditional oversight mechanisms struggle to address them.

Opacity is often a legal barrier rather than a technical one

The difficulty of understanding complex models is well recognised, yet a significant portion of governmental opacity stems from deliberate legal protections rather than inherent technical limits. Many systems used in public administration are supplied by private vendors who rely on intellectual-property and trade-secret protections to prevent disclosure of how their models function.

The COMPAS recidivism tool is a clear example. Courts and defendants sought to examine how the model reached its conclusions, yet the vendor argued that the model’s internals were proprietary. This form of commercial secrecy is reinforced by business models that reward the production of complex, closed systems and penalise simplicity or interpretability.

For governance, the consequence is practical rather than abstract. Decisions that shape a person’s freedom, access to services, or financial security can be based on systems whose logic cannot be examined by the affected individual, by oversight bodies, or in some cases even by the deploying agency. The “black box” becomes a legal construct that limits scrutiny far more effectively than technical complexity.

Responsibility is fragmented across an algorithmic supply chain

Government AI systems rarely originate from a single actor. They are assembled through layered supply chains involving data providers, model developers, platform infrastructure, and the public agencies that integrate the tool into their operations. In such chains, no single actor has full visibility, leading to what accountability scholars describe as the “problem of many hands.”

This fragmentation creates several issues:

• Agencies often cannot trace the provenance or quality of the training data.

• Vendors may simultaneously hold substantial influence over how systems operate while claiming the limited responsibilities of a subordinate “data processor.”

• Decision-making authority and technical control are unevenly distributed, yet lines of accountability remain anchored in traditional administrative structures that assume singular responsibility.

This produces what some analysts call an “accountability horizon”: each actor can only see a limited distance along the chain, and beyond that point responsibility becomes indeterminate. A 2021 report by the Netherlands Court of Audit illustrates the effect. Agencies deploying algorithmic systems could not access documentation needed to evaluate their functioning and were instructed to defer to external suppliers. In such conditions, responsibility drifts rather than concentrates.

Oversight bodies lack the capacity to scrutinise algorithmic systems

Democratic accountability depends on institutional forums capable of examining and questioning state action. For algorithmic systems, these forums often include municipal councils, courts of audit, ombuds institutions, and national supervisory bodies. Yet many of these institutions are chronically under-resourced and lack the technical expertise needed to evaluate automated systems.

Several recurring constraints appear:

• Financial limits: Some local audit offices operate on budgets that cannot support even a single in-depth technical investigation.

• Expertise gaps: Many oversight bodies struggle to interpret model documentation, data flows, or system behaviour.

• Misplaced boundaries: Algorithmic design choices are frequently framed as purely technical matters, which leads forums to underestimate their relevance to democratic oversight.

When oversight organisations cannot interrogate or contest a system’s functioning, the effect is a widening accountability deficit. Government agencies may use systems that shape significant administrative decisions without independent verification of accuracy, fairness, or compliance with legal standards.

The “human in the loop” does not guarantee meaningful control

A common safeguard proposed for automated decision-making is the inclusion of a human reviewer who must validate or override algorithmic outputs. While attractive in principle, this safeguard often falters in practice. The core issue is automation bias: the well-documented tendency for people to defer to algorithmic recommendations even when they have the authority to reject them.

In high-volume or highly technical contexts, human reviewers may lack the information needed to critically assess an output. When the model’s reasoning is inaccessible, the reviewer’s role can degrade into a procedural formality. The presence of a human becomes a symbolic assurance of oversight rather than a substantive one.

This creates a dual problem. The appearance of human judgement is maintained, which can shield the system from scrutiny, but the substantive capacity for intervention is weakened. The final decision is nominally human yet materially governed by the system’s recommendations.

Conclusion: accountability requires institutional redesign, not just technical fixes

The difficulties outlined above reveal that challenges in algorithmic governance arise less from exotic technical problems and more from familiar administrative dynamics: commercial secrecy, diffusion of responsibility, resource constraints, and cognitive biases that affect decision-making. Addressing these problems requires institutional change rather than simply improving model transparency.

Governments will need procurement rules that prevent secrecy from undermining public oversight, clearer allocations of responsibility across supply chains, stronger investment in supervisory bodies, and decision-making environments where human review is feasible and informed rather than nominal. As automated systems become part of the underlying structure of public administration, accountability will depend on how institutions adapt, not only on how models are built.

What is emerging in practice is more complicated. Rather than solving entrenched social problems, many deployments of AI are entrenching them in new, less visible ways. Systems presented as neutral are amplifying longstanding inequalities; models sold as accurate are built on unreliable data; tools marketed as “smart” are deployed without basic evaluation. The issue is not a handful of technical glitches. It is a pattern of design and governance choices that undermines fairness and accountability.

The sections below outline five recurring problems in how AI is used in public administration and why they matter for anyone interested in digital governance.

Automation scales bias instead of removing it

A central claim about AI in government is that mathematical models can remove prejudices from decision-making. Algorithms are presented as neutral mechanisms that apply the same rules to everyone.

In practice, models trained on historical data learn the patterns and biases embedded in that data. The well-known Amazon recruitment case illustrates the mechanism: a hiring model trained on CVs from a male-dominated workforce learned to penalise terms associated with women and downgraded applicants from women’s colleges. Nothing in the code explicitly instructed it to do so; the pattern was inherited from the past.

Public-sector systems face the same dynamic. When historical records reflect discriminatory policing, unequal access to services, or biased enforcement, models built on that data will reproduce those patterns. The result is not the removal of bias but its codification into rules that appear objective and are applied consistently and at scale. Where individual bias is at least visible and contestable, algorithmic bias can be harder to detect and easier to defend as “just what the data says.”

Government models are trained on “dirty data”

AI systems depend on the quality of the data they are trained on. In many public-sector contexts, that data is compromised in ways that go far beyond occasional errors. Researchers describe this as “dirty data”: datasets that embed the effects of unlawful practices, organisational incentives, and systematic recording problems.

Examples include:

• Manipulated statistics: crime numbers altered to meet performance targets or political expectations.

• Unlawful enforcement practices: records generated by unconstitutional stops, discriminatory checks, or other illegal actions.

• Large-scale recording errors: serious offences misclassified as minor ones, missing entries, and inconsistent coding.

If these records form the basis for predictive policing, risk scoring, or resource allocation, the model learns a distorted picture of reality. Instead of forecasting future harm or need, it learns where institutions have chosen to act in the past. The system then presents those inherited distortions as neutral predictions.

For governance, this is a basic epistemic problem: when the input is structured by institutional failure, there is no technical fix that can make the output objective.

Predictive systems can lock institutions into self-fulfilling loops

Dirty data becomes more damaging when it is fed back into the system through repeated use. Predictive tools in areas like policing or welfare risk scoring can create self-reinforcing cycles.

A standard pattern in predictive policing looks like this:

1. A model is trained on historical arrest data that reflects over-policing in particular neighbourhoods.

2. The model flags those neighbourhoods as high risk.

3. Police are deployed there more frequently.

4. Increased presence produces more recorded offences, especially for minor infractions.

5. New data confirms the model’s original assessment, justifying further deployments.

Over time, the system becomes very good at predicting where police will make arrests, not where crime is most prevalent. The model’s apparent accuracy is a product of its own influence on institutional behaviour.

This matters for AI governance because it shifts the role of models from tools that inform decisions to mechanisms that stabilise particular practices. It becomes harder for agencies to change course, even when communities bear the cost of intensified scrutiny.

People are judged through weak proxies and arbitrary signals

Many algorithmic systems in government assess concepts that are hard to measure directly, such as “risk of fraud,” “likelihood of recidivism,” or “need for care.” Developers therefore reach for proxies: data points that are easier to record and quantify.

When proxies are poorly chosen, they encode arbitrary or unjust distinctions. The example of a system using possession of a trailer, a diesel car, or a wheelie bin as indicators of suspicious behaviour shows how far this can drift from any meaningful concept of risk.

In health, one widely discussed case involved a tool that used healthcare expenditure as a proxy for healthcare need. Because marginalised groups have historically received less care and funding, their costs were lower. The model inferred that they had lower need and recommended fewer resources for them. Underinvestment in care was translated into an apparently neutral signal of reduced risk.

These design choices matter because they determine who receives attention, scrutiny, or support. People are judged not on their circumstances or actions but on correlations that may have little to do with the underlying policy goals. Once embedded in software, these proxies are hard to contest and are often shielded by claims of commercial confidentiality.

Procurement is driving adoption faster than oversight can keep up

The rapid spread of AI in public administration is not primarily driven by internal technical capacity. It is driven by procurement. Vendors offer ready-made solutions to agencies under pressure to modernise and to demonstrate that they are “data-driven.” The result is a market where ambitious claims often outrun evidence.

Public administrations frequently face a skills gap: limited internal expertise in machine learning, data governance, or model evaluation. Vendors, by contrast, specialise in presenting their products as powerful, safe, and innovative. This creates an asymmetry of expertise in which governments are strongly reliant on the seller’s own assurances.

Recent work by the Netherlands Court of Audit gives a concrete sense of how this plays out. It found that many agencies did not systematically assess the risks of the AI systems they were using and often did not know whether those systems worked as intended. In other words, tools that influence who is investigated, who is paid, or who is flagged as a risk can operate for extended periods without basic performance or impact evaluation.

For democratic governance, this raises a direct accountability question: when neither the public nor the administrators can say how a system behaves or whether it is meeting its objectives, on what basis is its continued use justified?

Conclusion: governing systems, not just deploying them

The main risks of AI in government do not lie in speculative superintelligence. They emerge from very ordinary factors: biased and unreliable data, weak proxies, self-reinforcing feedback loops, and procurement processes that prioritise adoption over evaluation.

These are governance problems. They cannot be solved by adding another layer of technical sophistication. They require decisions about where automation is appropriate, which tasks demand inherently interpretable systems, what data is acceptable to use, and how oversight is enforced.

If AI is to have a legitimate role in public administration, it needs to be embedded in institutions that can explain, contest, and, when necessary, switch off the systems they use. Automation for its own sake is not a public good. Automated decisions only serve the public interest when the underlying models, data, and incentives are subject to the same scrutiny we expect of any other exercise of state power.

Becker identifies a worldview organised around abstraction. Future-oriented calculations replace present-day harm. Long-range expected-value equations stand in for ethics. Engineering metaphors substitute for political institutions. Across different domains—moral philosophy, AI speculation, space colonisation—Becker finds the same pattern: complex realities are stripped down to simplified systems that scale cleanly on paper and fail when confronted with the world as it is.

Longtermism and the arithmetic of the distant future

One of Becker’s central targets is longtermism, a philosophy that prioritises the welfare of hypothetical future beings over people alive today. Becker’s critique is not simply that the focus is distant; it is that the reasoning depends on multiplying immense hypothetical populations by extremely small probabilities. This produces conclusions where speculative research on far-future technologies can be deemed more valuable than improving the lives of billions of people in the present.

Becker highlights how this line of reasoning can tilt moral attention away from existing inequality. By design, the calculus makes present harms mathematically negligible. The problem, for Becker, is not the idea that future people matter; it is that certain abstractions allow present suffering to disappear from view entirely.

The Singularity and the promise of accelerating returns

A second component of the worldview Becker analyses is the conviction that exponential technological growth will soon produce artificial general intelligence or “superintelligence.” This belief rests on the assumption that current trends—most notably Moore’s Law—will continue indefinitely.

Becker counters this with empirical observations familiar to researchers: semiconductor scaling is slowing, research productivity is falling in many fields, and biological intelligence cannot be reduced to a neat digital analogy. These points are not novel in themselves, but Becker uses them to show how the narrative of inevitable acceleration persists even when material constraints suggest a more complex picture. It is the persistence of the abstraction, not the details of the trend line, that interests him.

Existential risk and the displacement of present harms

Becker also traces how certain elite groups have made “AI existential risk” a central organising story. The well-known paperclip thought experiment functions here as an emblem: a simplified model of an artificial agent pursuing a goal without regard for human welfare.

Becker’s concern is the opportunity cost of attention. While researchers such as Timnit Gebru, Emily Bender, and others document present harms—carbon costs, linguistic biases, discriminatory outputs—public debate is repeatedly pulled toward speculative future catastrophes. Meredith Whittaker is quoted to good effect: when the problem is framed in a fantasy world, the solutions migrate there as well. Becker shows how existential narratives can crowd out governance discussions grounded in material impacts.

Utopias built on controlled environments

Becker then turns to the futures envisioned by figures like Ray Kurzweil and Elon Musk. These imagined worlds involve space settlements, large-scale planetary engineering, or the conversion of matter into computational substrate. Becker’s point is not to dismiss ambition but to highlight a recurring preference for controlled, predictable environments over the complexities of natural systems. The result is a form of utopian thinking in which nature becomes either a problem to escape or a resource to reorganise for technical ends.

The critique here is structural: a worldview built on simplifying assumptions can lead to futures that are inhospitable to the very forms of life they aim to preserve.

The intellectual lineage behind the rhetoric

Becker closes by mapping some of the contested intellectual history surrounding these communities: the rationalist and effective altruist movements, their entanglements with controversial claims about intelligence, and the echoes of earlier techno-futurist manifestos. His argument is that when a worldview is constructed from highly abstract models of humanity, it can become vulnerable to reductionist conclusions about people themselves. The point is not that the movements are uniform—they are not—but that certain baseline assumptions can create openings for ideas that flatten human complexity.

A book for governance discussions, not just tech criticism

More Everything Forever is not an attack on technology or future-thinking. It is a study of how certain influential groups reason about the future and how those modes of reasoning scale into public discourse. For students of AI governance, the value of the book lies in its account of how narratives take shape: how specific epistemic communities define risks; how simplified models travel into policy debates; and how speculative futures can overshadow present harms.

Silicon Valley’s most ambitious visions are not only technical blueprints but political imaginaries. Becker’s book is useful for understanding the assumptions that sit underneath them—and for recognising that the governance of algorithmic systems requires grounding decisions in empirical harms rather than in abstractions that treat complexity as a nuisance to be engineered away.

Explainable AI (XAI) emerged as the field’s attempt to answer this problem. It promises a way to make complex systems intelligible: a translation layer, a peek inside the box, a path toward accountability. In public debate, XAI is often presented as the bridge between advanced computation and human oversight.

Yet the practical reality is more complicated. Much of what is commonly believed about explainability does not hold up when examined through a governance lens. The gap between what XAI offers and what accountability requires is wide, and in some cases unbridgeable. Understanding that gap is essential for anyone working on AI governance, procurement, or oversight.

The accuracy–interpretability trade-off is overstated

A persistent assumption in AI is that there is an unavoidable trade-off between model accuracy and interpretability. The story goes like this: simple models can be understood but are too weak for high-stakes tasks; powerful models are necessarily opaque. Choosing transparency, therefore, means accepting inferior performance.

Empirical research undermines this assumption. Work by Cynthia Rudin and others shows that for many structured, high-stakes domains, inherently interpretable models often perform as well as the black-box alternatives. Transparency does not always require sacrificing accuracy.

The belief in this trade-off persists partly because it supports the commercial interests of proprietary vendors. A model that cannot be disclosed is commercially valuable. Declaring opacity an unavoidable by-product of performance becomes a way to justify secrecy. Framing interpretability as technically impossible masks a simpler reality: sometimes secrecy is a business choice, not a scientific constraint.

“Gaming the system” is often an argument against scrutiny, not a real risk

Another common objection to transparency is the fear that revealing how a system works will allow people to manipulate it. This argument is frequently invoked by public agencies and vendors to justify withholding information about model design.

From a governance perspective, the claim is weak. If a system can be easily “gamed,” the problem lies with its design, not with transparency. In a well-specified model, gaming should correspond to socially meaningful improvements—raising a credit score, improving punctuality in attendance data, correcting inaccurate administrative records. If gaming produces harmful distortions, then the model is relying on proxies that were inappropriate from the outset.

Moreover, many features used in real-world models are immutable. A person cannot change their age, past interactions with the justice system, or most administrative data. The argument that transparency invites gaming is often a way to avoid external scrutiny rather than a serious statement about risk.

XAI explanations are approximations, not the model’s actual reasoning

The most important limitation of XAI is conceptual. Explanations generated by XAI tools are not windows into the original model. They are simplified second-order models designed to mimic the behaviour of the underlying system within a narrow context.

This means that explanations are approximations by design. They can be informative in cooperative settings—debugging, model development, internal review—but they are not faithful representations of how a complex model actually reaches its decisions. A 90-percent-accurate explanation still fails 10 percent of the time. For oversight of high-stakes public decisions, that level of uncertainty is untenable.

When the explanation is not the truth, the system cannot reasonably claim to be transparent. And if the explanation cannot be trusted, the underlying model cannot be trusted either.

Explainability fails precisely in the contexts where accountability is required

The usefulness of XAI depends on context. In cooperative settings—where developers and auditors share a common goal—XAI is a valuable tool. It helps identify errors, refine features, and diagnose performance issues.

Accountability settings are different. When an individual contests a decision made by a public agency or a private institution, the relationship becomes adversarial. In this context, the provider of the system has an incentive to select an explanation that portrays the model’s behaviour as consistent and defensible. XAI gives providers significant discretion to shape how the system’s decision is presented, which allows narrative control rather than genuine transparency.

This is the core governance problem: explanations can be curated. Disclosure cannot.

Explainability shifts power toward providers and away from the public

The rise of XAI reflects a broader shift in how transparency is defined. Instead of demanding access to the system itself—its code, its training data, its parameters—institutions increasingly rely on “reasoned transparency”: a mediated explanation prepared by the system’s creator.

This shift matters. Mediated explanations allow organisations to maintain secrecy while still claiming openness. They can select the level of detail, frame the logic of the system, and shape public understanding. The result is a form of managed transparency that risks manufacturing trust rather than earning it.

For public accountability, this is inadequate. Oversight requires the ability to interrogate a system directly, not accept a curated summary.

Conclusion: building transparency into the system, not adding it after the fact

Explainability has legitimate uses in technical practice, but it is not a substitute for genuine transparency. For high-stakes decisions in public administration or essential services, accountability cannot depend on post-hoc narratives.

A more robust approach requires demanding interpretability by design. Public authorities can mandate inherently transparent models in procurement, develop models in-house where possible, and require full disclosure when private vendors supply decision systems. These are governance choices, not technical impossibilities.

Explainable AI may illuminate parts of a black box, but it cannot solve the accountability problem created by the box itself.

This matters for AI governance because policymaking does not occur in an abstract vacuum. Regulatory frameworks, public expectations, and media attention are shaped by different conceptions of what the central problems are. Understanding these variations is part of understanding the “algorithmic society” itself.

The United States: existential stakes and agentic systems

In the United States, AI risk is frequently framed in existential or agentic terms: runaway optimisation, superintelligent systems, and the possibility that a misaligned artificial agent could overpower human decision-making. These ideas are not broadly distributed across American society; they are amplified by a small number of influential epistemic communities clustered around Silicon Valley, Effective Altruism, rationalist forums, and high-visibility founders.

These groups are disproportionately represented in US media coverage. Their interviews, op-eds, and philanthropic influence make AGI-centric concepts—superintelligence, catastrophic misalignment, extinction risk—more visible than their actual prevalence in technical research or public opinion. Mainstream outlets often quote these voices when covering AI developments, even when they aim to question or critique them.

The result is a public discourse where “AI risk” often defaults to questions about AGI, automation run amok, or catastrophic futures. This framing shapes what policymakers think the public is concerned about and what kinds of solutions are deemed urgent. Safety labs, model evaluations, and alignment research dominate headlines, while governance debates about labour, surveillance, or platform power receive less attention relative to their material impact.

Europe: governance failures and institutional harms

European discussions follow a different trajectory. AI risk is most often described as a problem of governance: bias, discrimination, misleading outputs, opacity, surveillance, labour disruption, and threats to democratic institutions. The EU’s regulatory tradition anchors this framing. Discussions about AI naturally link to the rule of law, due process, rights protection, and institutional safeguards.

The EU AI Act is a reflection of this approach. Rather than imagining AI as an autonomous agent that might “break free,” EU institutions focus on the systems people already use: workplace monitoring tools, biometric identification, recommender systems, automated decision-making in public services. European media coverage follows the same pattern. The dominant risks are structural, not existential.

This contrast does not imply that Europeans are unconcerned about future-oriented scenarios. Instead, the public discourse prioritises harms that are already observable or foreseeable within existing political and economic structures. Where US narratives often focus on systems gaining too much autonomy, European narratives focus on institutions holding too little accountability.

Why these differences matter for AI governance

These divergences have practical implications.

First, they shape policy agendas. If US debate centres existential risk, policymakers become more sensitive to catastrophic-scenario arguments. If European debate centres governance failures, regulation emphasises transparency, oversight, and institutional safeguards.

Second, they affect how governments and regulators interpret industry claims. Companies based in the US increasingly frame their products as powerful enough to require specialised “alignment” processes. European regulators, by contrast, ask for documentation, compliance assurance, and risk assessments. Each region is responding to a different definition of the problem.

Third, these narratives influence the development of the systems themselves. Training data, model evaluations, safety protocols, and risk registers embed assumptions about what counts as harm. If developers in different regions internalise different narratives, the resulting systems may handle uncertainty and alignment differently.

Fourth, cross-regional collaboration becomes harder when actors mean different things by “AI safety.” Without a shared conceptual baseline, international governance efforts can drift or stall. Interoperability is not only a technical question but a discursive one.

Why students and practitioners should pay attention

For those working in AI governance—or preparing to—these narrative differences are not academic details. They shape:

• how policymakers justify interventions

• which risks receive funding and regulatory traction

• how companies frame compliance obligations

• what the public believes is at stake

• what kinds of expertise are considered relevant

Understanding regional framings helps future practitioners navigate transatlantic policy conversations, evaluate claims made by developers, and interpret risk assessments with more precision. It also highlights the need to distinguish between material harms and narrative-driven concerns.

As algorithmic systems become embedded in public administration, security, welfare provision, and everyday life, governance will require attention to both the technical system and the discursive environment that defines its risks.

Toward a grounded understanding of AI risk

AI governance is often described as a global challenge, but the underlying narratives are far from uniform. Recognising these differences is the first step toward building governance structures that respond to real harms rather than imported imaginaries.

For students, researchers, and practitioners in the Algorithmic Society community, this is an opportunity to rethink what “AI risk” means in practice and how the stories we tell shape the systems we build and regulate.

But this belief, while comforting, is built on a fragile foundation. The concept of “algorithmic fairness” is not a simple technical problem to be solved with better code. It is a minefield of mathematical paradoxes, unavoidable trade-offs, and deep ethical questions. The pursuit of a perfectly fair algorithm often leads us to uncomfortable conclusions about our technology and ourselves.

This article will explore four surprising truths about AI fairness that challenge our common assumptions. These are not edge cases or theoretical worries; they are fundamental constraints and realities that anyone building, deploying, or being judged by an algorithm needs to understand.

Perfect Fairness Is Mathematically Impossible

The single most startling revelation from research into algorithmic fairness is this: under real-world conditions, a model cannot satisfy all desirable fairness metrics at the same time. This isn’t a failure of engineering; it is a mathematical certainty known as an “impossibility theorem.”

To understand why, we need to look at what we mean by “fair.” There are many different mathematical standards. For example, we might want a model to satisfy:

• Statistical Parity: This standard requires that the proportion of individuals receiving a positive outcome (like a loan approval) is the same across all groups. It focuses purely on equalizing outcomes.

• Predictive Parity: This standard requires the model to be equally accurate for all groups. If the algorithm predicts someone will succeed, the probability of them actually succeeding should be the same regardless of their race or gender.

• Equalized Odds: This standard requires the model to make the same types of errors for all groups. It ensures that the rate of false positives (incorrectly flagging someone as high-risk) and false negatives (incorrectly flagging someone as low-risk) is equal across different demographic groups.

Each of these sounds like a reasonable goal. The problem is that when two common conditions are met—(1) the underlying “base rates” (the actual prevalence of an outcome) differ between groups, and (2) the model is not 100% accurate—it is mathematically impossible to satisfy Predictive Parity and Equalized Odds at the same time. You are forced to choose. This is profoundly impactful because it means there is no single, technically “correct” fair solution. The pursuit of fairness requires choosing which type of fairness to prioritize, and by extension, which type of unfairness we are willing to accept.

The COMPAS Paradox: Biased Against Everyone, Differently

The mathematical trade-offs of fairness are not just theoretical. A famous 2016 ProPublica analysis of the COMPAS algorithm, a tool used to predict the likelihood of a defendant reoffending, provides a stark real-world example. The investigation found significant racial disparities in the algorithm’s errors, but in a surprisingly complex way.

The analysis revealed two different kinds of bias working in opposite directions:

• False Positives: The algorithm was twice as likely to incorrectly classify Black defendants who would not reoffend as “high risk.” This error occurred for 45% of these Black defendants, compared to only 23% of White defendants.

• False Negatives: The algorithm was more likely to incorrectly classify White defendants who would go on to reoffend as “low risk.” This error occurred for 48% of these White defendants, compared to 23% for Black defendants.

This finding is crucial. The algorithm wasn’t just “biased against Black defendants” in a simple sense. It was also biased against White defendants, but by making the opposite kind of mistake. It disproportionately harmed one group by over-predicting risk and harmed another by under-predicting it. This powerfully illustrates the trade-off: a system can be biased against different groups in different, and sometimes conflicting, ways.

The Mirror Problem: When an Algorithm Is Unfair Because It’s Too Accurate

We often assume that algorithmic bias is a technical glitch—a result of “bad data” or a poorly designed model that needs fixing. While technical flaws like Measurement Bias can certainly exist (for example, using healthcare spending as a proxy for health needs, which systematically underrates the needs of Black patients), a deeper and more challenging problem is Societal Bias. This occurs when an AI model is technically accurate and makes correct predictions based on historical data, but its use still perpetuates and amplifies existing social inequalities.

A chilling example of this is the 2020 Ofqual algorithm in the UK, designed to assign A-level grades after exams were canceled due to the pandemic. The algorithm was built to match historical grade distributions, accurately reflecting that certain schools consistently produced lower grades. However, this “accuracy” meant it unfairly capped the marks of high-performing students at historically disadvantaged schools, effectively punishing them for their school’s past performance. The algorithm held up a perfect mirror to historical inequality, and in doing so, perpetuated it.

If the data used to train a model reflects a world of discrimination, an “accurate” algorithm will learn to reproduce those same patterns, imposing a relative disadvantage based on group membership.

“[D]iscrimination consists of acts, practices, or policies that impose a relative disadvantage on persons based on their membership in a salient social group.”

The implication is profound. In these cases, the primary challenge is not debugging the algorithm. It is confronting the fact that our algorithms are showing us an uncomfortable, but accurate, picture of a biased world. Fixing the problem isn’t about correcting the mirror; it’s about addressing the reality it reflects.

The Real Choice: Preserve the Past or Transform the Future?

When we decide to mitigate unfairness in an algorithm, we face a fundamental choice not between technical methods, but between two competing philosophies of justice: Formal Equality and Substantive Equality.

• Bias Preserving measures are the algorithmic expression of Formal Equality. This philosophy, rooted in the idea of “treating like cases alike,” aims to ensure a process is applied consistently. An algorithm designed with this goal in mind seeks to match error rates across groups, essentially accepting the existing distribution of outcomes in society as a neutral starting point. It works to prevent the algorithm from introducing new bias, thereby preserving the societal status quo, inequalities and all.

• Bias Transforming measures are the expression of Substantive Equality. This philosophy argues that true fairness requires proactively accounting for historical disadvantage to “level the playing field.” An algorithm built on this principle actively works to counteract socially unacceptable disparities. For example, it might enforce Statistical Parity, ensuring different groups receive positive outcomes (like a loan approval) at the same rate. This approach is designed to transform, not just reflect, the existing social order.

This is not a technical choice between two competing algorithms; it is a normative choice about the very purpose of our technology. Adopting a bias transforming approach requires developers and policymakers to make an explicit moral judgment about which societal biases are unacceptable and must be corrected. It forces a difficult but necessary conversation about what kind of society we want to build.

What Are We Asking Our Algorithms to Be?

The journey into algorithmic fairness quickly reveals that it is not a simple problem of debugging code. It is a complex, socio-technical challenge defined by unavoidable trade-offs, mirrored social injustices, and deep moral questions. There is no easy answer, no magic algorithm that can satisfy every definition of fairness for everyone, all at once.

This forces us to confront a more fundamental question. The central issue is not just whether our algorithms should reflect the world as it is or as it should be. It is a choice between competing visions of justice: Do we encode a formal equality that preserves the past, or a substantive equality that aims to build a different future?