4 structural problems that make AI accountability difficult in government
Introduction: automation as a new actor in public decision-making
AI systems are increasingly embedded in the everyday work of government. They screen applications for public benefits, support decisions in justice and policing, and shape the allocation of healthcare resources. These tools now operate as quiet administrative actors, influencing outcomes that carry real consequences for individuals and communities.
This raises a central question: when these systems make errors or produce harmful outcomes, who is responsible? Accountability in automated decision-making is often presented as a matter of technical transparency, but the more persistent barriers are legal, institutional, and structural. The sections below outline four problems that shape the accountability landscape and explain why traditional oversight mechanisms struggle to address them.
Opacity is often a legal barrier rather than a technical one
The difficulty of understanding complex models is well recognised, yet a significant portion of governmental opacity stems from deliberate legal protections rather than inherent technical limits. Many systems used in public administration are supplied by private vendors who rely on intellectual-property and trade-secret protections to prevent disclosure of how their models function.
The COMPAS recidivism tool is a clear example. Courts and defendants sought to examine how the model reached its conclusions, yet the vendor argued that the model’s internals were proprietary. This form of commercial secrecy is reinforced by business models that reward the production of complex, closed systems and penalise simplicity or interpretability.
For governance, the consequence is practical rather than abstract. Decisions that shape a person’s freedom, access to services, or financial security can be based on systems whose logic cannot be examined by the affected individual, by oversight bodies, or in some cases even by the deploying agency. The “black box” becomes a legal construct that limits scrutiny far more effectively than technical complexity.
Responsibility is fragmented across an algorithmic supply chain
Government AI systems rarely originate from a single actor. They are assembled through layered supply chains involving data providers, model developers, platform infrastructure, and the public agencies that integrate the tool into their operations. In such chains, no single actor has full visibility, leading to what accountability scholars describe as the “problem of many hands.”
This fragmentation creates several issues:
Agencies often cannot trace the provenance or quality of the training data.
Vendors may simultaneously hold substantial influence over how systems operate while claiming the limited responsibilities of a subordinate “data processor.”
Decision-making authority and technical control are unevenly distributed, yet lines of accountability remain anchored in traditional administrative structures that assume singular responsibility.
This produces what some analysts call an “accountability horizon”: each actor can only see a limited distance along the chain, and beyond that point responsibility becomes indeterminate. A 2021 report by the Netherlands Court of Audit illustrates the effect. Agencies deploying algorithmic systems could not access documentation needed to evaluate their functioning and were instructed to defer to external suppliers. In such conditions, responsibility drifts rather than concentrates.
Oversight bodies lack the capacity to scrutinise algorithmic systems
Democratic accountability depends on institutional forums capable of examining and questioning state action. For algorithmic systems, these forums often include municipal councils, courts of audit, ombuds institutions, and national supervisory bodies. Yet many of these institutions are chronically under-resourced and lack the technical expertise needed to evaluate automated systems.
Several recurring constraints appear:
Financial limits: Some local audit offices operate on budgets that cannot support even a single in-depth technical investigation.
Expertise gaps: Many oversight bodies struggle to interpret model documentation, data flows, or system behaviour.
Misplaced boundaries: Algorithmic design choices are frequently framed as purely technical matters, which leads forums to underestimate their relevance to democratic oversight.
When oversight organisations cannot interrogate or contest a system’s functioning, the effect is a widening accountability deficit. Government agencies may use systems that shape significant administrative decisions without independent verification of accuracy, fairness, or compliance with legal standards.
The “human in the loop” does not guarantee meaningful control
A common safeguard proposed for automated decision-making is the inclusion of a human reviewer who must validate or override algorithmic outputs. While attractive in principle, this safeguard often falters in practice. The core issue is automation bias: the well-documented tendency for people to defer to algorithmic recommendations even when they have the authority to reject them.
In high-volume or highly technical contexts, human reviewers may lack the information needed to critically assess an output. When the model’s reasoning is inaccessible, the reviewer’s role can degrade into a procedural formality. The presence of a human becomes a symbolic assurance of oversight rather than a substantive one.
This creates a dual problem. The appearance of human judgement is maintained, which can shield the system from scrutiny, but the substantive capacity for intervention is weakened. The final decision is nominally human yet materially governed by the system’s recommendations.
Conclusion: accountability requires institutional redesign, not just technical fixes
The difficulties outlined above reveal that challenges in algorithmic governance arise less from exotic technical problems and more from familiar administrative dynamics: commercial secrecy, diffusion of responsibility, resource constraints, and cognitive biases that affect decision-making. Addressing these problems requires institutional change rather than simply improving model transparency.
Governments will need procurement rules that prevent secrecy from undermining public oversight, clearer allocations of responsibility across supply chains, stronger investment in supervisory bodies, and decision-making environments where human review is feasible and informed rather than nominal. As automated systems become part of the underlying structure of public administration, accountability will depend on how institutions adapt, not only on how models are built.