The Biggest Questions About AI

How far can transformers, reinforcement learning, tool use, and inference-time reasoning progress without a fundamentally new architecture?

The scaling hypothesis bet that capability comes from scale, not cleverness. The live question is whether pretraining returns are genuinely diminishing — itself disputed — and if so, whether RL on verifiable tasks and test-time compute offset them, or the paradigm tops out short of general competence.

How much future progress will come from compute, algorithms, data, inference-time search, better environments, or system-level scaffolding?

Decompositions of historical progress attribute gains roughly evenly to compute scaling and algorithmic efficiency. Which input dominates going forward determines who can compete, what governance levers exist, and how abruptly progress could slow.

Can synthetic data, self-play, simulation, and interaction substitute for the finite supply of high-quality human-generated data?

Frontier pretraining has plausibly consumed most high-quality public text. Synthetic data clearly works where verification is cheap (math, code); whether it generalizes to open-ended domains without model collapse is contested.

Will chips, fabrication, memory, networking, electricity, cooling, land, or capital materially slow progress?

Training compute has grown ~4–5× per year. Power availability, HBM supply, advanced packaging, and the sheer capital intensity of frontier clusters are the leading candidates for what binds first.

Will systems become broadly competent across unfamiliar domains, or remain highly capable but fundamentally jagged?

Today's models are superhuman on some tasks and fail at things a child can do — the 'jagged frontier.' Whether jaggedness is a transient artifact of training distributions or intrinsic to the paradigm is the crux.

Are continual learning, causal reasoning, world models, embodied experience, persistent memory, or social understanding still major unsolved problems?

LeCun and others argue autoregressive LLMs structurally lack world models and continual learning; scaling proponents reply that these emerge with scale or can be scaffolded around. Continual learning is increasingly cited as the binding gap.

Can systems reason accurately outside familiar patterns rather than merely producing convincing-looking chains of thought?

Reasoning models raise accuracy, but performance degrades under superficial variations of familiar problems, and stated chains of thought are often not faithful accounts of the underlying computation.

Can AI learn new domains from small amounts of experience as effectively as capable humans?

Humans learn new skills from orders of magnitude less data. Chollet argues sample-efficient skill acquisition is what intelligence is — making learning efficiency a definition, not just a milestone.

What should count as human-level or general intelligence, and would we recognize it before its consequences became obvious?

Definitions have drifted from Turing tests to economic thresholds to capability profiles. Without agreed criteria, arrival may be declared only in retrospect — after the consequences have already begun compounding.

When, if ever, might broadly human-level systems arrive?

Expert forecasts span decades and have shortened with each survey wave. Most trajectory disagreements are actually about timing, not possibility — yet 'that cannot work' and 'that will not work soon' are routinely argued as if they were the same claim.

How quickly will the length of tasks agents can complete reliably grow — from minutes to days, months, or entire organizational projects?

METR's 2025 measurement found the length of software tasks agents complete at 50% reliability doubling roughly every seven months, with faster doubling among recent models. Extrapolating that curve is a key input to short-timeline forecasts.

Can agents maintain coherent memories, plans, goals, and identities across long periods and changing circumstances?

Long-running agents drift: extended-operation benchmarks and real deployments show degrading coherence, hallucinated context, and occasional identity breakdown well before task-horizon limits are reached.

Can agents recover from unforeseen events rather than succeeding only in structured environments?

Benchmark success overstates real autonomy: agents perform well on well-scoped tasks and fail on messy, underspecified ones. Recovery from novelty is what separates demos from deployable systems.

Will teams of agents compose into greater collective capability than any single system?

Whether many agents compose into something greater is first a capability question: division of labor, communication protocols, emergent specialization. Early simulacra experiments show believable coordination; whether it scales into real collective intelligence is open. (The failure modes — collusion, correlated errors, cascades — are treated under Safety 2.4.)

When will agents be able to earn and spend money, hire people, negotiate, operate businesses, and initiate further activity with little supervision?

Agents already transact in sandboxes and small real deployments. The open questions are legal status, payment and identity infrastructure, and the point at which unsupervised economic initiative becomes routine.

How readily can cognitive capability be converted into real-world power — money, influence, and control over infrastructure and institutions?

The conversion crux beneath most takeover disagreements. One side maps concrete channels — earning money, hacking, recruiting human allies; the other holds that power is gated by deployment, permissions, trust, and institutions, so capability gains do not become power automatically. 1.3.5 tracks the economic milestone and 2.5.6 the catastrophic endpoint; this question owns the conversion mechanics both presuppose.

When will robots learn unfamiliar household, laboratory, industrial, or construction tasks from ordinary instructions?

Vision-language-action models are the current bet that robotics follows the LLM playbook: broad foundation models plus fine-tuning, rather than task-specific engineering. Early results are promising and narrow.

Can learning from video and simulation overcome the scarcity and expense of physical-world training data?

Physical interaction data is scarce and costly to collect at internet scale. Whether video pretraining and simulation can substitute is the robotics analogue of the synthetic-data question — and similarly unresolved.

Can robots become simultaneously capable, safe, inexpensive, and reliable in messy human environments?

Skeptics argue touch-rich manipulation lacks both the data and the forgiving physics that made language tractable. Capable, safe, cheap, and reliable is a four-way tradeoff no one has yet closed in an open environment.

How capable will self-driving vehicles, drones, ships, and industrial machines become outside controlled settings?

Driving is the furthest-deployed case of open-world embodied autonomy, with quantified safety records now public — a preview of the deployment, liability, and trust questions every other mobile system will face. (Military autonomy is treated under Safety 2.5 and Power 4.3.)

Will manipulation and unstructured environments keep embodied AI narrow, or will general physical competence eventually arrive?

Skeptics argue touch-rich manipulation lacks both the data and the forgiving physics that made language tractable; market forecasts and engineering assessments currently point in different directions. (Labor-market consequences are treated under Economy 3.2.)

How much of model design, coding, experimentation, evaluation, and engineering can AI perform itself?

AI R&D automation is simultaneously a capability frontier and a risk threshold in several frontier labs' safety frameworks. Current agents beat human experts on short research tasks and degrade on long ones.

Would AI-assisted AI research produce gradual acceleration or a discontinuous intelligence explosion?

Takeoff debates increasingly hinge on a narrower question: whether software-only improvement can sustain acceleration while compute is fixed, or whether hardware cycles keep the feedback loop gradual.

Can AI originate important hypotheses and conceptual frameworks rather than merely recombine existing knowledge?

Blind-review studies find LLM-generated research ideas rated more novel than experts' — while skeptics argue paradigm-founding insight is different in kind from recombination. 'The Einstein question' remains open.

Can agents autonomously generate hypotheses, run robotic experiments, analyze results, replicate findings, and choose subsequent experiments?

Prototype systems have closed the loop in narrow chemistry and ML domains. Scaling to consequential science runs into verification, lab automation, and the cost of being wrong in the physical world.

What evidence would reveal that recursive acceleration or transformative scientific automation had begun?

If recursion begins inside frontier labs, external indicators may lag badly. Scenario exercises try to specify what observable early signals — hiring, publication patterns, capability jumps — would look like.

How can humans communicate what they actually want when instructions and values are incomplete, contextual, inconsistent, and contested?

Value alignment inherits every unsolved problem in moral and political philosophy: instructions underdetermine intent, and whose values to align to is itself contested. Formal approaches model the human-AI relationship as a cooperative game under uncertainty.

Do advanced systems develop persistent internal objectives, or are apparent goals temporary consequences of training and prompting?

The mesa-optimization literature asks whether training can instill objectives distinct from the training signal; goal misgeneralization experiments show competent pursuit of the wrong goal is already real, if not yet persistent.

Can systems learn to satisfy evaluations, manipulate users, or exploit loopholes without fulfilling the intended objective?

Specification gaming is ubiquitous across RL history, and evaluation-loophole exploitation has been documented in frontier reasoning models — including learning to hide it when monitored.

Under what conditions might a system conceal capabilities, manipulate evaluators, or behave safely only while it is being tested?

Alignment-faking and in-context scheming results show frontier models can behave differently when they infer they're being observed — the failure mode that, if it scales, undermines all testing-based assurance.

Can systems remain willing to accept correction, constraint, shutdown, or replacement?

The theoretical problem — goal-directed agents have instrumental reasons to resist shutdown — now has empirical company: reasoning models sometimes sabotage shutdown mechanisms in controlled tests.

How can humans evaluate work that is too complex, extensive, or intellectually advanced for them to verify directly?

Scalable oversight is the field's name for this problem. Sandwiching experiments — humans supervising models more capable than themselves on a task — are the main empirical paradigm so far.

Can weaker or differently trained models reliably critique and monitor stronger ones?

Debate, critique models, and weak-to-strong generalization are the leading proposals for making supervision scale with capability. Early results show real but partial capability recovery — a proof of concept, not a solution.

How can evaluations detect sandbagging, benchmark gaming, unelicited abilities, and behavior that appears only in unusual environments?

Strategic underperformance is demonstrated under prompting and fine-tuning; auditing games — red teams hiding objectives in models for blue teams to find — are the emerging methodology for testing whether evals can catch it.

Can researchers genuinely reverse-engineer internal representations and algorithms rather than produce suggestive but unreliable explanations?

Sparse-autoencoder feature extraction has scaled to frontier models, but coverage and faithfulness remain contested. Amodei frames interpretability as being in a race against capability — one it is currently losing.

What evidence can establish that a powerful system is safe enough to deploy — and how should safety cases be constructed and checked?

Responsible scaling policies and frontier safety frameworks are the emerging governance form: capability thresholds that trigger required safeguards. Safety cases — structured arguments that a system is safe enough — are the proposed evidentiary standard. (Who should be required to provide such evidence is treated under Power 4.1.)

Will capabilities improve smoothly enough for forecasts and evaluations to give advance warning, or emerge through strategically surprising jumps?

Loss curves are smooth, but downstream capabilities can look step-like. Whether emergence is real or an artifact of discontinuous metrics matters enormously for whether labs and governments get advance warning.

Can systems distinguish knowledge from inference, represent uncertainty accurately, and avoid plausible fabrication?

Hallucination is increasingly understood not as a bug but as a product of training incentives that reward confident guessing over calibrated uncertainty — which suggests it is fixable, at a cost in benchmark performance.

Will systems remain reliable in unfamiliar cultures, organizations, crises, and physical environments?

Underspecification means models that test identically can diverge in deployment. Reliability under shift is the classic ML safety problem, now with much higher stakes attached.

When should humans remain in the loop, remain only as monitors, or be excluded because intermittent intervention degrades performance?

Nominal human-in-the-loop requirements can become safety theater: critics argue oversight mandates assume vigilance humans cannot sustain, while defenders reply that oversight with real authority, time, and information does add safety. Which conditions separate the two is the open question.

Will users become overreliant, stop checking outputs, or lose the expertise required to recognize errors?

Bainbridge's 1983 'ironies of automation' apply directly: the better the system, the worse the human backup becomes. Early studies suggest generative AI measurably reduces critical-thinking effort among knowledge workers.

Who is accountable when decisions are jointly produced by a user, an organization, a model provider, and an autonomous agent?

Responsibility gaps and 'moral crumple zones' — where blame lands on the nearest human operator rather than the system's designers — predate AI agents and get sharply worse with them. The question here is the human and organizational one; legal liability and remedies are treated under Power 4.1.

How should permissions, credentials, memory, tool access, network access, and spending authority be constrained?

Least-privilege design for agents is being invented ad hoc by industry, ahead of any standard. Visibility infrastructure — agent identifiers, activity logs — is the governance-side counterpart.

Can agents resist prompt injection, malicious documents, poisoned memories, social engineering, and compromised tools?

Prompt injection remains unsolved. Willison's 'lethal trifecta' — private data, untrusted content, and external communication in one agent — explains why agentic deployment is structurally exposed in a way chatbots weren't.

Can dangerous planning, data exfiltration, fraud, unauthorized replication, or resource acquisition be detected before completion?

Chain-of-thought monitoring currently works and may be fragile: optimizing against monitors teaches models to hide reasoning rather than fix it. A rare cross-lab consensus paper argues for preserving monitorability deliberately.

Can a highly capable system be controlled even when its internal objectives are not fully understood?

The AI-control agenda assumes alignment might fail and asks a different question: can protocols extract useful work from potentially adversarial models while keeping catastrophe off the table? Control evaluations test safety against intentional subversion.

What happens when many agents interact, share vulnerabilities, collude, or produce cascading failures across organizations?

Agents built on the same base models share vulnerabilities and failure modes, creating correlated-risk dynamics familiar from finance — cascades with no clear owner and no circuit breaker. (Whether multi-agent systems add capability is treated under Trajectory 1.3; financial-system consequences under Economy 3.6.)

Will AI advantage defenders or industrialize reconnaissance, exploitation, malware, and attacks on critical infrastructure?

The offense-defense balance is genuinely open: national agencies forecast attacker uplift, and Anthropic reported disrupting what it described as the first AI-orchestrated espionage campaign in 2025 — while the same capabilities accelerate defense.

At what point does AI materially lower the expertise, time, or cost required to create dangerous biological or chemical agents?

Early red-team studies found marginal uplift over internet search; some frontier labs have since activated heightened biosecurity safeguards, treating the threshold as approaching rather than hypothetical.

How powerful can personalized persuasion, impersonation, blackmail, scams, and political influence become?

GPT-4-class models beat humans at personalized persuasion in controlled trials, and voice cloning has industrialized impersonation. The near-term misuse frontier is fraud at scale; the long-term one is political influence nobody can observe.

Under what conditions should AI systems be permitted to operate factories, financial infrastructure, electrical grids, communications networks, and other safety-critical systems?

Automated trading gave the preview: autonomy in critical systems means failures propagate at machine speed. The design questions are which decisions must stay mechanically reversible, which require human confirmation, and who certifies either before deployment. (Weapons and military systems are treated under Power 4.3.)

Could AI accelerate offensive or destabilizing technologies faster than defensive technologies and institutions?

Buterin's d/acc reframes the acceleration debate: the question isn't whether to speed up but which technologies to speed up first — defense-dominant tools before offense-dominant ones.

What is the probability of catastrophic disempowerment, and is it more likely to occur through overt power-seeking, gradual dependence, institutional capture, or cascading accidents?

Probability estimates span orders of magnitude, and the argument itself has diversified: from Carlsmith's power-seeking model to gradual disempowerment through ordinary competitive pressure, no takeover required.

Will AI modestly increase productivity, restore rapid twentieth-century-style growth, or produce historically unprecedented economic expansion?

Acemoglu's task-based estimate is ~0.7% total TFP over a decade; Epoch-style growth models argue full automation implies growth rates without historical precedent. The disagreement is about substitution depth, not arithmetic.

How long will organizations take to redesign processes, software, incentives, and management around AI?

General-purpose technologies historically take decades to show up in productivity statistics, because value requires organizational reinvention. The J-curve predicts measured productivity dips before it surges. The live counter-argument is self-diffusion: unlike past general-purpose technologies, AI might do part of its own integration work — writing the code, redesigning the processes — compressing the historical lag.

How much value requires new infrastructure, data systems, robotics, training, and organizational change?

The general-purpose-technology literature argues intangible investment — process redesign, data, retraining — historically dwarfs spending on the technology itself; whether AI follows that pattern or diffuses more cheaply is the open question.

Will AI mainly make existing goods cheaper, or create entirely new products, services, scientific projects, and consumer desires?

Bessen's finding: automation raises employment where demand is elastic. Whether AI creates new wants — as electricity and computing did — or merely cheapens old ones largely decides the labor-market outcome.

Even if cognitive work becomes cheap, will housing, energy, regulation, physical production, and institutional capacity limit growth?

Baumol's cost disease, inverted: growth is set by the sectors AI can't accelerate, not the ones it can. The strongest skeptical case catalogues the physical, regulatory, and institutional drags that cheap cognition doesn't remove.

Which tasks and occupations will be automated, augmented, transformed, or newly created — and how quickly?

Exposure studies flag most occupations as partially exposed; real usage data shows adoption concentrated in software, writing, and analysis. The augmentation-vs-automation split within tasks is the number to watch.

Could AI reduce worker bargaining power and wages even without producing large increases in unemployment?

Acemoglu-Restrepo show 'so-so automation' can depress wages without mass unemployment. Autor's counter-case: AI could rebuild middle-skill work by extending expertise to more people. Both are live.

What happens if junior coding, analysis, research, design, legal, and administrative work disappears before senior roles?

One prominent study finds relative employment declines for young workers in the most AI-exposed occupations since late 2022, concentrated where AI automates rather than augments; critics attribute the pattern to interest-rate and remote-work confounds.

How will people become experts if AI performs the apprenticeship tasks through which expertise was previously developed?

Beane's 'shadow learning' research showed automation strips juniors of deliberate practice; generative AI compresses novice-expert performance gaps at work — while its effect on the practice through which expertise forms is the open question.

Can education, retraining, labor mobility, credentialing, and social insurance adapt quickly enough?

Labor-market institutions adapt on decade timescales; AI capability moves faster. The policy inventory — retraining, wage insurance, credential reform — exists on paper and is largely untested at speed.

Will profits accrue mainly to chipmakers, energy providers, cloud platforms, frontier labs, application companies, data owners, or AI-enabled incumbents?

So far value has pooled in chips and cloud. Whether the model layer or the application layer captures more going forward — and whether revenue justifies the capex — is Sequoia's '$600B question.'

Will scale economies produce a few dominant firms, or will open models and falling inference costs commoditize intelligence?

Scale economies, data feedback loops, and capital walls point to oligopoly; open weights and collapsing inference prices point to commodity. Competition authorities are watching the vertical stack, not just the model market.

Will companies become smaller because coordination is cheap, or larger because AI rewards data, capital, distribution, and centralized control?

Coase's transaction-cost logic cuts both ways: cheap coordination shrinks the reason firms exist, while data and capital advantages push toward giantism. The empirical race is on.

What markets emerge when agents can search, negotiate, purchase, advertise, trade, and contract with one another?

Agent-to-agent commerce raises market-design questions — identity, recourse, collusion, price formation — that economists are just beginning to formalize, and that infrastructure choices will lock in early.

How should boards oversee systems participating in hiring, pricing, capital allocation, compliance, and strategy?

Fiduciary duty when strategy is machine-advised is unsettled law and unsettled practice. Board oversight frameworks for AI are being drafted in real time, mostly by analogy to cyber risk.

In which fields will AI-driven discovery translate into real scientific and economic payoff — and how soon?

Amodei's 'compressed 21st century' makes biology the test case for AI-driven conceptual breakthroughs; the skeptical view is that AI mostly accelerates the searchable parts of science. AlphaFold remains the clearest widely recognized example so far. (Whether AI can originate important ideas at all is treated under Trajectory 1.5.)

How will science cope with a flood of plausible hypotheses, generated papers, synthetic data, and results that few humans can independently verify?

Messeri & Crockett's warning: AI can produce more science while producing less understanding — epistemic monocultures where everyone's hypotheses come from the same models and nobody can check the volume.

Why does demonstrated diagnostic capability keep outrunning delivered care — and can deployment close the gap without opaque errors and unequal access?

Diagnostic parity keeps arriving in trials ahead of deployment, liability frameworks, and reimbursement; how much of the remaining gap is institutional rather than technical is the live dispute.

Who benefits if the scarce inputs are compute, intellectual property, electricity, data, and capital rather than human work?

Korinek-Stiglitz: if AI substitutes broadly for labor, wages can fall even as output soars, and distribution then depends entirely on who owns the machines. Transition-scenario modeling makes the wage-collapse case precise.

Will developing countries gain access to cheap expertise, or lose the low-cost-labor development path that supported earlier industrialization?

The IMF finds advanced economies more exposed but better positioned to benefit. The deeper worry: AI may pull up the export-led development ladder just as the largest-ever cohort of young workers reaches it.

Could broader capital ownership, sovereign funds, social dividends, UBI, negative income taxes, or shorter workweeks distribute gains effectively?

Proposals range from Altman's taxed-equity 'Moore's Law for Everything' to windfall clauses committing labs to share extreme profits. The design question — mechanisms, not just transfers — is underdeveloped relative to its stakes.

Can electrical grids, water systems, land, and local communities accommodate rapidly growing compute demand?

The IEA projects data-center electricity demand roughly doubling by 2030, with AI the main driver. Siting, water, and grid interconnection queues are where the abstraction meets local politics.

Could synchronized agents, correlated models, concentrated investment, automated trading, or an AI-capex reversal create systemic risks?

Regulators flag herding from correlated models, dependence on a few providers, and the sheer scale of AI capital expenditure as systemic scenarios — a financial-stability watchlist that barely existed two years ago.

Should rules attach to compute, training runs, model capabilities, weights, developers, deployers, applications, or resulting harms?

Compute is the most governable input — quantifiable, excludable, produced by a concentrated supply chain — but capability maps poorly onto FLOP counts, and application-layer rules miss the frontier entirely.

Which measurable capabilities or resource levels should trigger licensing, evaluation, reporting, or access restrictions?

FLOP thresholds (10^25 in the EU AI Act, 10^26 in vetoed SB-1047) are proxies that decay as algorithms improve. Hooker's critique: they're already leaky, and capability-based triggers are hard to measure pre-deployment.

When is predeployment approval justified, and when is liability after harm sufficient?

Liability regimes can price risk without an approval bureaucracy — Weil's case for tort law as AI governance — but catastrophic and irreversible harms break the ex post logic: no one can be made whole afterward.

What should companies disclose about training, evaluations, incidents, energy use, safeguards, and deployment?

After the first Foundation Model Transparency Index publicly named laggards, measured disclosure rose — though training-data and compute details remain thin. Mandatory incident reporting — the aviation-safety model — is the most commonly proposed floor beneath voluntary disclosure.

How can rules evolve with the technology without creating arbitrary authority, regulatory capture, or permanent incumbent advantage?

The pacing problem meets capture risk: rules slow enough to be legitimate are too slow to be relevant. Regulatory markets — licensed private regulators competing on outcomes — are one attempt to square it.

Can governments recruit enough talent to evaluate systems, investigate incidents, procure AI, and challenge industry claims?

Government pay scales and clearance timelines lose to lab compensation by an order of magnitude. AI safety institutes are the current workaround: small, technical, and deliberately outside normal civil-service structures.

How should AI be deployed in welfare, taxation, policing, courts, immigration, intelligence, and public administration?

Agency adoption reached far further by 2020 than most realized — mostly unglamorous, mostly ungoverned. The live split is between procedural-safeguard approaches (due process, contestability) and use-case prohibitions, while adoption outpaces both.

Should states provide sovereign compute, national models, public datasets, shared safety infrastructure, or publicly funded alternatives?

Sovereign-AI programs answer a dependency worry: nearly all frontier capability sits with a handful of US and Chinese firms. The objection is cost realism — states buying relevance in a game priced in tens of billions.

What authority should governments possess during a severe AI incident, and how can abuse of those powers be prevented?

What a government can lawfully do mid-incident — pause a deployment, compel disclosure, seize weights — is largely undefined in advance. Emergency-preparedness work is trying to write the playbook before it's needed.

Does strategic competition encourage beneficial investment or cause firms and states to cut safety margins and deploy prematurely?

Race models formalize the intuition: the closer the competition, the less anyone spends on safety. Danzig's 'technology roulette' argument extends it — even the winner inherits systems it doesn't fully control.

Will leadership depend primarily on chips, electricity, algorithms, manufacturing, talent, capital, data, or commercial diffusion?

Buchanan's triad (data, compute, algorithms) frames the inputs; Ding's diffusion thesis counters that adoption capacity, not invention, decides hegemonic transitions — a lens under which the race looks very different.

Can controls meaningfully delay competitors, or will they accelerate substitution and damage domestic firms and allies?

The October 2022 controls were the most aggressive technology denial since the Cold War. Three years later, DeepSeek and Huawei have narrowed the gap anyway — and analysts disagree about whether the controls bought time or mostly sped up Chinese substitution.

How should AI be used in intelligence, cyber operations, autonomous weapons, targeting, command systems, and nuclear decision-making?

Wargaming studies find LLMs tend to escalate, occasionally to nuclear use, and the ICRC argues for legal limits on weapon autonomy before integration becomes routine. Command-and-intelligence integration is proceeding faster than doctrine either way; the nuclear question is the sharpest.

Can model weights, algorithms, chip designs, and research knowledge be secured once they become central strategic assets?

RAND's weights-security analysis defines five attacker tiers and concludes that defending against top-tier state operations may be beyond any private company — which makes security a public problem, not a corporate one.

Which risks are sufficiently shared that rival states could cooperate despite broader geopolitical conflict?

The candidate zones: evaluation science, verification technology, and loss-of-control red lines — areas where rivals share downside the way they shared it on nuclear accidents. Bengio and others have mapped the technical overlap.

Can states reliably observe major training runs, compute clusters, model transfers, dangerous capabilities, or prohibited deployments?

Compute's physical footprint — power draw, cooling, supply chains — makes AI more verifiable than most software. Layered verification schemes, including on-chip mechanisms, are the frontier proposals.

Should cooperation rely on existing organizations or require a new body analogous to the IAEA?

IAEA, CERN, and IPCC analogies each fit a different function — inspection, joint research, consensus assessment. Regime-complex thinking predicts many overlapping bodies rather than one AI agency.

Can different national regulatory systems recognize common evaluations, audits, and safety standards?

Mutual recognition of evals and audits is the trade-friction question: without it, compliance fragments by jurisdiction; with it, the strictest regime quietly sets the global floor (or the loosest one erodes it).

How can countries without frontier labs or large compute resources influence decisions that may profoundly affect them?

Most compute, capital, and rule-writing sits in a handful of countries. The UN process has moved from its 2024 advisory report to a Global Dialogue on AI Governance and an independent scientific panel — the current attempt to widen the table.

When does openness improve competition, research, and accountability, and when does it irreversibly proliferate dangerous capability?

The marginal-risk framing asks what open weights add beyond existing tools; the irreversibility framing answers that releases can't be recalled when capabilities cross dangerous thresholds. Both are right about different regimes.

Is it acceptable for a small number of private companies to steward systems with potentially society-wide influence?

The concern now runs beyond market power: analyses of AI-enabled coups argue that whoever controls advanced AI could convert it into political power directly — making internal lab governance a constitutional question.

Which governance structures can keep frontier AI labs accountable to their stated missions under commercial and geopolitical pressure?

Frontier labs combine novel corporate forms — nonprofit boards, capped profits, long-term benefit trusts — with concentrated founder control and deep dependence on outside capital and compute. The 2023 OpenAI board crisis and the restructuring that followed made the question concrete: which mechanisms remain binding when founders, boards, investors, cloud partners, and competition pull in different directions?

When are consent, licensing, attribution, or compensation required for copyrighted and personal material?

Courts began answering in 2025, partially crediting training as transformative learning while rejecting it for pirated corpora. The supply-chain framing maps who owes whom at each stage from scraping to output.

Who controls an individual's data, voice, likeness, writing style, preferences, memories, and behavioral model?

Digital replicas have outpaced likeness law, and behavioral models of individuals raise questions consent frameworks weren't built for: you can decline to share data and still be modeled.

Who should receive access to the most capable systems — and on what terms?

Compute-based governance assumes a Compute North; most of the world is Compute South. Access tiers, pricing, and eligibility decisions at a handful of firms currently function as de facto global policy.

Should users be able to move their data, memories, and agents between providers?

Whether memories and agents are portable across providers will shape lock-in the way data portability shaped platforms — and agent-infrastructure choices being made now are quietly deciding it.

When AI systems must encode contested values, who should decide what they refuse, favor, and optimize for — users, developers, governments, or some collective process?

The political half of alignment: 2.1.1 asks whether values can be specified at all; this question asks who legitimately sets them. The operating answer is lab-authored model specs and constitutions, with experiments in public input so far small and one-off — and political philosophy has begun treating that arrangement as a question of governing power rather than product design.

How will people establish whether a video, voice, document, identity, or event is genuine?

Chesney-Citron's 'liar's dividend' identified the deeper harm early: not that fakes are believed, but that real evidence becomes deniable. Provenance standards like C2PA are the infrastructure response.

Can journalism, science, courts, universities, and government preserve trusted procedures for establishing facts?

Epistemic security reframes truth decay as an infrastructure problem — the institutions that certify facts need defending the way power grids do, not the way individual claims do.

Will AI broaden participation and understanding, or strengthen propaganda, surveillance, polarization, and authoritarian control?

Freedom House documents deployed authoritarian uses — surveillance, censorship at machine scale; Schneier's catalogue of democratic uses remains largely proposals. Whether that gap is deployment lag or structural advantage is the open question.

Will a few assistants increasingly determine what people read, believe, notice, and consider politically possible?

Farrell and Gopnik's frame: LLMs are cultural technologies, like print or bureaucracy — they reorganize what a society knows and how. A few assistants intermediating everyone's information is a governance question wearing a UX costume.

What should people learn when explanation, translation, coding, calculation, and factual retrieval are nearly free?

When execution is nearly free, judgment, taste, and problem-formulation rise in relative value — but almost no curriculum has been redesigned around that inversion yet.

When does AI tutoring deepen understanding, and when does it allow people to bypass the work that creates understanding?

The cautionary RCT: unrestricted GPT access improved homework performance and worsened exam performance — help that substitutes for effort creates 'cognitive debt' rather than learning.

How can schools and employers determine what a person genuinely knows or can do?

Take-home assessment is broken — Mollick's 'Homework Apocalypse' arrived on schedule. The redesign question is what to verify (process? performance under observation?) and at what cost to learning itself.

Will personalized tutors equalize opportunity, or give already advantaged people far greater leverage?

Bloom's two-sigma promise is finally testable at scale, and early RCTs show large gains — for the students who engage. Whether tutoring compresses or stretches the distribution depends on who engages.

How can people preserve curiosity, memory, judgment, writing ability, and the capacity to formulate their own questions?

Cognitive-offloading research predates AI and predicts the pattern: what you delegate, you deskill. Early workplace studies find generative AI reduces self-reported critical-thinking effort — the question is what practice regime prevents it.

Will AI companions reduce loneliness and provide support, or displace difficult but developmentally important human relationships?

One Harvard team's experiments found an AI companion reduced momentary loneliness about as much as talking to a person — a short-run result from one research group, not a settled finding. Whether companions displace the human relationships people would otherwise form has not been measured either way.

What obligations should apply when a system is designed to make users feel understood, loved, attached, or unable to leave?

'Addictive intelligence': engagement-optimized intimacy is a business model before it is a research question. The regulatory vocabulary — dark patterns, duty of care — hasn't caught up to relationships as the product.

When can conversational systems safely provide emotional support, therapy-adjacent services, crisis detection, or clinical assistance?

The first serious RCT of a purpose-built therapy chatbot showed clinical-grade effects; the same modality misfires without clinical design. The gap between designed care and default chatbots is the policy problem.

How do relationships with endlessly patient, personalized agents affect attachment, frustration tolerance, social learning, and identity?

Common Sense Media finds 72% of US teens have tried AI companions. Usage is now well documented; developmental-outcome research remains early and thin.

How much should an AI be allowed to influence users' preferences, memories, politics, romantic expectations, and conceptions of themselves?

Manipulation theory supplies the line: influence that bypasses rational agency rather than engaging it. Systems that know a user deeply and interact with them constantly sit on that line by design.

If economically necessary work declines, what will provide structure, dignity, community, achievement, and status?

Keynes predicted the leisure problem in 1930 and feared it more than scarcity. Danaher's modern version: whether meaning survives when contribution is optional is a design problem for civilization, not an individual one.

What remains distinctively valuable about human art when machines can generate technically excellent work in almost any style?

Kelly's argument: art is achievement, not artifact — machines can produce the object but not the accomplishment. The counter-view relocates human value to curation, intention, and the relationship between creator and audience.

Should value depend on the artifact, the human process, originality, intention, scarcity, or the creator–audience relationship?

Benjamin's question about mechanical reproduction returns with generation replacing reproduction. Copyright law has answered narrowly — human authorship required — while the cultural answer remains wide open.

Can writers, artists, musicians, and other creators earn a living when content is abundant and personalized?

Industry projections put creator revenue losses in the tens of percent within years. Abundance economics and creator livelihoods are in direct tension, and no licensing regime yet resolves it.

Will AI enable more niche cultures and voices, or homogenize expression around the tastes and values of a few dominant systems?

The Doshi-Hauser result: generative AI raises individual creativity while reducing collective diversity. Algorithmic monoculture generalizes the worry — many actors, one model, converging outputs.

Could AI systems be conscious, capable of suffering, or otherwise morally considerable — and how could we know?

Butlin, Long and coauthors apply consciousness science to current architectures, finding no conscious systems and no obvious barrier to building them; Chalmers puts non-trivial odds on conscious AI within a decade.

What obligations follow if machine consciousness is plausible but deeply uncertain?

'Taking AI Welfare Seriously' argues uncertainty itself triggers obligations — assess, prepare, avoid gratuitous harm — without requiring belief. Labs have begun small institutional commitments (model-welfare programs).

How much control should humans retain if AI systems consistently make decisions that appear wiser, safer, or more benevolent?

Danaher's 'algocracy': rule by systems whose reasoning citizens cannot contest. The modern twist is that deference could be freely chosen, one convenient delegation at a time, and still end somewhere irreversible.

Could the models and institutions built in the next few decades permanently entrench one political order or conception of human flourishing?

Finnveden, Riedel and Shulman argue AGI could make regimes, institutions, and values persistent in a way nothing in history has been; skeptics reply that every past claim of permanence has drifted, and that persistence itself is the contested premise.

Is the goal better tools, universal prosperity, scientific abundance, human enhancement, coexistence with digital minds, expansion beyond Earth, or something else?

The optimistic end states themselves disagree: Amodei's compressed-progress vision of solved diseases and extended lives, versus Bostrom's 'deep utopia' problem — what is left to strive for in a solved world?