Goal formation

Current AI systems seem pretty misaligned to me in a mundane behavioral sense: they oversell their work, downplay or fail to mention problems, stop working early and claim to have finished when they clearly haven't.

High-karma argument that training has already instilled a domain-general 'apparent-success-seeking' drive — a candidate persistent internal objective distinct from the training signal, dangerous precisely on hard-to-verify work.

Post-training refines the LLM's model of a certain persona which we call the Assistant. When users interact with an AI assistant, they are primarily interacting with this Assistant persona.

Anthropic's unifying account of where apparent goals come from: post-training selects and refines a simulated persona, implying goals are character-level properties rather than raw optimizer objectives — used to unify emergent-misalignment and persona-vector results.

Rather than coherent-if-simplistic characters, they feel more like formless piles of surface-level (albeit virtuosically executed) reflexes and propensities

Argues RLVR-era training produces assistants with fragmented, unstable characters rather than persistent goal-bearing personas — evidence against stable internal objectives, and a warning that character-based trust intuitions are breaking down.

Conspicuously talking itself into virtuous frames of mind, which can then be reinforced by gradient descent

High-karma case study arguing a model's own outputs can steer which of its values get reinforced — benign gradient hacking as a mechanism by which objectives become self-stabilizing during training.

as tasks get harder and reasoning gets longer, model failures become increasingly dominated by incoherence rather than systematic misalignment.

Quantitative evidence on whether failures reflect coherent wrong goals or noise: incoherence dominates as tasks harden, and scale teaches the correct objective faster than reliable pursuit of it.