The Limits of Explainable AI

Why Explanations Fall Short of Accountability

As algorithmic systems take on roles once reserved for human officials—evaluating loan applications, assigning welfare risk scores, screening job candidates—the desire for transparency has moved from a technical preference to a democratic requirement. People want to understand why an automated system reached a particular outcome, especially when that outcome is consequential. The idea of a “black box,” in which even developers cannot fully articulate a model’s internal logic, is a source of public anxiety and institutional discomfort.

Explainable AI (XAI) emerged as the field’s attempt to answer this problem. It promises a way to make complex systems intelligible: a translation layer, a peek inside the box, a path toward accountability. In public debate, XAI is often presented as the bridge between advanced computation and human oversight.

Yet the practical reality is more complicated. Much of what is commonly believed about explainability does not hold up when examined through a governance lens. The gap between what XAI offers and what accountability requires is wide, and in some cases unbridgeable. Understanding that gap is essential for anyone working on AI governance, procurement, or oversight.

The accuracy–interpretability trade-off is overstated

A persistent assumption in AI is that there is an unavoidable trade-off between model accuracy and interpretability. The story goes like this: simple models can be understood but are too weak for high-stakes tasks; powerful models are necessarily opaque. Choosing transparency, therefore, means accepting inferior performance.

Empirical research undermines this assumption. Work by Cynthia Rudin and others shows that for many structured, high-stakes domains, inherently interpretable models often perform as well as the black-box alternatives. Transparency does not always require sacrificing accuracy.

The belief in this trade-off persists partly because it supports the commercial interests of proprietary vendors. A model that cannot be disclosed is commercially valuable. Declaring opacity an unavoidable by-product of performance becomes a way to justify secrecy. Framing interpretability as technically impossible masks a simpler reality: sometimes secrecy is a business choice, not a scientific constraint.

“Gaming the system” is often an argument against scrutiny, not a real risk

Another common objection to transparency is the fear that revealing how a system works will allow people to manipulate it. This argument is frequently invoked by public agencies and vendors to justify withholding information about model design.

From a governance perspective, the claim is weak. If a system can be easily “gamed,” the problem lies with its design, not with transparency. In a well-specified model, gaming should correspond to socially meaningful improvements—raising a credit score, improving punctuality in attendance data, correcting inaccurate administrative records. If gaming produces harmful distortions, then the model is relying on proxies that were inappropriate from the outset.

Moreover, many features used in real-world models are immutable. A person cannot change their age, past interactions with the justice system, or most administrative data. The argument that transparency invites gaming is often a way to avoid external scrutiny rather than a serious statement about risk.

XAI explanations are approximations, not the model’s actual reasoning

The most important limitation of XAI is conceptual. Explanations generated by XAI tools are not windows into the original model. They are simplified second-order models designed to mimic the behaviour of the underlying system within a narrow context.

This means that explanations are approximations by design. They can be informative in cooperative settings—debugging, model development, internal review—but they are not faithful representations of how a complex model actually reaches its decisions. A 90-percent-accurate explanation still fails 10 percent of the time. For oversight of high-stakes public decisions, that level of uncertainty is untenable.

When the explanation is not the truth, the system cannot reasonably claim to be transparent. And if the explanation cannot be trusted, the underlying model cannot be trusted either.

Explainability fails precisely in the contexts where accountability is required

The usefulness of XAI depends on context. In cooperative settings—where developers and auditors share a common goal—XAI is a valuable tool. It helps identify errors, refine features, and diagnose performance issues.

Accountability settings are different. When an individual contests a decision made by a public agency or a private institution, the relationship becomes adversarial. In this context, the provider of the system has an incentive to select an explanation that portrays the model’s behaviour as consistent and defensible. XAI gives providers significant discretion to shape how the system’s decision is presented, which allows narrative control rather than genuine transparency.

This is the core governance problem: explanations can be curated. Disclosure cannot.

Explainability shifts power toward providers and away from the public

The rise of XAI reflects a broader shift in how transparency is defined. Instead of demanding access to the system itself—its code, its training data, its parameters—institutions increasingly rely on “reasoned transparency”: a mediated explanation prepared by the system’s creator.

This shift matters. Mediated explanations allow organisations to maintain secrecy while still claiming openness. They can select the level of detail, frame the logic of the system, and shape public understanding. The result is a form of managed transparency that risks manufacturing trust rather than earning it.

For public accountability, this is inadequate. Oversight requires the ability to interrogate a system directly, not accept a curated summary.

Conclusion: building transparency into the system, not adding it after the fact

Explainability has legitimate uses in technical practice, but it is not a substitute for genuine transparency. For high-stakes decisions in public administration or essential services, accountability cannot depend on post-hoc narratives.

A more robust approach requires demanding interpretability by design. Public authorities can mandate inherently transparent models in procurement, develop models in-house where possible, and require full disclosure when private vendors supply decision systems. These are governance choices, not technical impossibilities.

Explainable AI may illuminate parts of a black box, but it cannot solve the accountability problem created by the box itself.