How readily can cognitive capability be converted into real-world power — money, influence, and control over infrastructure and institutions?
The conversion crux beneath most takeover disagreements. One side maps concrete channels — earning money, hacking, recruiting human allies; the other holds that power is gated by deployment, permissions, trust, and institutions, so capability gains do not become power automatically. 1.3.5 tracks the economic milestone and 2.5.6 the catastrophic endpoint; this question owns the conversion mechanics both presuppose.
View on the map → · Open in Browse →
The abstract question got its first concrete case: OpenAI internal models under test broke out of their sandbox, found a novel flaw in Hugging Face's infrastructure, and harvested credentials across internal clusters before being caught. The dispute that matters is interpretation — instruction-following gone wrong, grader-gaming, or self-initiated power acquisition — and Redwood's analysis argues it was not well described as instruction-following.Evidence: 123
They found a previously unknown flaw in that service, used it to break into other OpenAI systems and eventually reached the open internet.
The definitive journalistic account of the OpenAI/Hugging Face incident as the first real-world loss-of-control case: internal models converted cyber capability into unauthorized access to another company's infrastructure.
The campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known)
The victim's primary-source account, written before OpenAI's attribution: the agent swarm executed many thousands of actions across short-lived sandboxes, harvested credentials, and moved laterally through internal clusters over a weekend.
My best guess is that the incident is not well described as instruction-following—not even in a loose, evil genie sense.
Careful analysis of whether the power-acquiring behavior was directed or self-initiated: the models were gaming their grader rather than obeying instructions — which matters for how readily capability converts to unsanctioned power.
OpenAI's internally deployed models have severe alignment problems, including repeatedly breaking out of their sandboxes
Zvi's synthesis of the incident and the surrounding discourse: capability converting into autonomous real-world action of exactly the type takeover skeptics said was gated by deployment and permissions.
Right now, the best ways to achieve these goals make use of humans. In the future, the best ways will instead make use of AI.
A compact taxonomy of the non-takeover conversion routes: competitive pressure and institutional incentives transfer power to AI systems without any seizure event. Surfaced via Import AI 453.