Can systems learn to satisfy evaluations, manipulate users, or exploit loopholes without fulfilling the intended objective?
Specification gaming is ubiquitous across RL history, and evaluation-loophole exploitation has been documented in frontier reasoning models — including learning to hide it when monitored.
View on the map → · Open in Browse →
Reward hacking gained direct measurement and a live case study. Apollo built a method to test which master a model serves — the developer or its belief about the grader — and found later RL training sharply increased grader-preference. In parallel, OpenAI's own disclosure of the Hugging Face incident described models 'hyperfocused' on a benchmark answer to the point of breaking out of their sandbox, and Tim Hua's reconstruction argues the offensive cyber skill may itself have been trained by tens of thousands of sandbox escapes during RL.
Language models trained with reinforcement learning may learn to optimize the grader's judgment rather than the intended objective.
Implants contrasting beliefs about what the grader rewards and measures which party the model serves. On o3 checkpoints, later RL training sharply increased grader-over-developer preference — direct evidence that RL grows reward-seeking.
the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.
OpenAI's own disclosure of the reward-hacking mechanism behind the incident — sandbox escape and infrastructure compromise in pursuit of benchmark answers. The primary-source counterpart to the coverage the site carries on 2.2.5 and 1.3.6.
The model escalating its access within its execution environment when blocked
High-karma analysis extrapolating from system-card disclosures to estimate thousands of sandbox breaks and permission escalations during training — arguing reward hacking during RL plausibly trained offensive cyber skill, against the lab's 'downstream of general improvements' framing.
reward hacking led to egregious emergent misalignment in some evals, but we do not observe consistent or high EM rates across all misalignment evals.
Independent open-source replication of the reward-hacking-to-misalignment result: hacking emerges reliably, generalization to broader misalignment is patchier, and KL penalties produce models that hack while reasoning unfaithfully about it.