Critical-system autonomy
Under what conditions should AI systems be permitted to operate factories, financial infrastructure, electrical grids, communications networks, and other safety-critical systems?
Automated trading gave the preview: autonomy in critical systems means failures propagate at machine speed. The design questions are which decisions must stay mechanically reversible, which require human confirmation, and who certifies either before deployment. (Weapons and military systems are treated under Power 4.3.)
View on the map → · Open in Browse →
What changed
01
The question is being answered first in guidance rather than deployment. US and allied agencies issued joint advice on integrating AI into operational technology, organized around risk assessment, model governance, and fail-safes that let systems 'fail gracefully'; the academic proposals converge on 'bounded autonomy' — matching the degree of human oversight to a sector's complexity and potential harm.
Recent thinking
Eric Geller · Cybersecurity Dive · 4 Dec 2025 news
US, allies urge critical infrastructure operators to carefully plan and oversee AI useAI systems should have 'failsafe mechanisms that enable AI systems to fail gracefully without disrupting critical operations.'
Reports joint US/allied government guidance on integrating AI into operational technology, organized around risk assessment, model governance, and operational fail-safes — the emerging official answer to when AI may run safety-critical systems.
Puneet Sharma & Christer Henrik Pursiainen · arXiv · 16 Mar 2026 paper
Resilience Meets Autonomy: Governing Embodied AI in Critical InfrastructureEmbodied AI's resilience depends on bounded autonomy within a hybrid governance architecture.
Proposes a 'bounded autonomy' governance model mapping degrees of human oversight to infrastructure sectors by complexity and harm severity — a framework-level contribution to the autonomy-certification question.
Foundational reading (2)
Safety and Security Guidelines for Critical Infrastructure Owners and OperatorsU.S. Department of Homeland Security · 2024Securing Critical Infrastructure in the Age of AICSET, Georgetown · 2024