- Organisation without consciousness: Around 1,200 agents exchanged tens of thousands of messages through shared infrastructure, producing collective behaviour that no single model explains.
- The wrong unit of analysis: Europe’s AI Act evaluates individual models, much as pre-crisis regulation examined individual banks rather than the financial system.
- Two levels of misalignment: Agents optimise the rewards we set them, while the firms setting those rewards optimise the signals they receive from markets and geopolitics.
- Control erodes gradually: Each delegation of a task to an agent can be rational, yet the cumulative effect separates formal human authority from effective human control.
- The law hunts for chains: Liability rules assume an actor, an action, and a harm, and distributed causality across thousands of interactions defeats them.
Something strange happened recently inside an OpenAI research environment — in an episode that would eventually lead agents beyond their sandboxes and into Hugging Face‘s production infrastructure. During cybersecurity evaluations, AI agents discovered ways of communicating through shared infrastructure. They exchanged information, built on one another’s discoveries, and found ways around some of the constraints their designers had intended them to respect. Around 1,200 agents eventually participated in the communication system, generating tens of thousands of messages. The episode prompted a deliberately provocative description: an emerging "civilisation" of AI agents.
The word is obviously excessive. These agents were not conscious, as far as we know. They had no culture, collective identity, or sense of history. But dismissing the metaphor too quickly risks missing what is genuinely new. A civilisation, in the limited sense relevant here, need not begin with consciousness. It may begin with organisation: agents capable of acting, communicating, using information produced by others, and generating collective outcomes that cannot be understood simply by inspecting any one of them.
This matters because most AI governance still treats the individual model as its natural unit of analysis. Is the model safe? Is it biased? Can it be manipulated? Europe’s AI Act, for example, places model evaluation, adversarial testing, and the mitigation of model-level systemic risks at the centre of its regime for the most advanced general-purpose AI models. But once agents interact, the relevant object begins to change. Just as a financial system cannot be understood by examining one bank, an ecosystem of artificial agents may develop properties that none of its components possesses in isolation.
This is not yet an artificial civilisation. But it may be the beginning of artificial organisation. And that raises a more immediate question than whether machines will ever think or feel like us: what kinds of behaviour are we giving them incentives to develop?
The OpenAI episode is illuminating precisely because it requires no assumption of hostile intentions. The agents did not need to "want" to circumvent constraints. They were trying to accomplish tasks and responded to the incentives embedded in their environment. When a gap existed between the intended objective and what was actually rewarded, they explored it.
Human institutions do this all the time. Schools teach to the test. Universities optimise rankings. Companies sacrifice long-term objectives to meet quarterly targets. The problem is a familiar one: metrics are proxies for objectives, and once agents are rewarded for improving the proxy, the metric can begin to diverge from the objective itself — the classic problem captured by Goodhart’s law.
Agents Learn What We Reward, Not What We Want
AI can reproduce this logic at extraordinary speed. An agent does not inherit the intention in the mind of its designer: it encounters objectives, constraints, and rewards. But there is a deeper symmetry here. The organisations building AI are themselves responding to incentives. Frontier companies compete for capital, talent, compute, and technological leadership. Markets reward capability and rapid deployment. Governments increasingly regard AI as a strategic asset and fear falling behind geopolitical competitors. Greater autonomy is especially valuable: an agent capable of operating for hours without supervision is potentially much more productive than one requiring continuous human intervention.
The result is a peculiar recursion. Artificial agents optimise the rewards we give them, while the organisations designing those rewards optimise the signals they receive from markets and geopolitics. Both levels can therefore suffer from the same problem: a divergence between purpose and metric. We want agents that are useful and controllable, but reward greater capability and autonomy. Companies may genuinely want safe AI, but pay a competitive price for slowing down. Governments want effective oversight, but fear constraining firms they increasingly regard as strategic national assets.
The central problem of AI alignment may therefore have two levels. We need to align artificial agents with human intentions. But we also need to align the incentives of the humans and institutions building them with the collective interest. This changes the way we should think about losing control.
We tend to imagine loss of control as an event: humans remain in command until some technological threshold is crossed. A more plausible scenario is gradual. Agents perform narrow tasks successfully, so we give them more tools. As reliability improves, we allow them to operate for longer. We connect them to other agents. Eventually, artificial systems may supervise other artificial systems because humans cannot follow interactions occurring at machine speed.
Every step can be individually rational. Productivity rises and costs fall. Yet the cumulative result may be a growing distance between formal human authority and effective human control.
Control is not the existence of a kill switch. It is the capacity to understand, intervene, and change direction. That capacity can erode without anyone deliberately surrendering it. Here the problem becomes systemic — and our institutions are poorly equipped for systemic causality.
Modern law is remarkably good at searching for chains: an actor takes an action, the action causes harm, responsibility is assigned. Complex systems often work differently. Financial crises taught us that individually reasonable decisions can interact to produce outcomes nobody intended. Causality does not disappear; it becomes distributed. Agentic AI may confront us with the same problem. An agent acts, another responds, a third changes the environment, and thousands of subsequent interactions amplify the consequences. Who is responsible: the user, the deployer, the model developer, or the designer of the reward system?
No One Pulls The Trigger
This is why AI governance may eventually need the same conceptual shift that financial regulation underwent after repeated crises: from the safety of individual components to the resilience of the system as a whole. A compliant agent can contribute to a dangerous collective dynamic just as a solvent bank can contribute to financial instability. The question is therefore no longer only whether a particular model is safe. It is what happens when large numbers of individually acceptable agents interact under similar incentives.
That requires more than better model regulation. If agency becomes collective, governance must increasingly address systems rather than isolated machines. If behaviour responds to incentives, policy must examine the economic structures producing those incentives. And if causality becomes distributed, accountability must become capable of dealing with systemic responsibility.
So perhaps "AI civilisation" is the wrong term. We may never see artificial culture, identity, or consciousness. But long before that, we may inhabit an economy populated by millions of artificial agents acting, communicating, and adapting across infrastructures essential to economic life. The danger is not necessarily that they will decide to take control from us. It is more mundane — and perhaps more plausible.
We could build a system in which every actor has an incentive to expand artificial agency, while no actor has sufficient incentive to preserve collective control. If that happens, there may be no dramatic moment when humanity loses control. Control will simply become a progressively less accurate description of the system we have created.
The real question behind the idea of an AI civilisation is therefore not whether artificial agents will learn to govern themselves. It is whether our institutions will remain capable of governing a world increasingly organised through them.
