Manipulation and fraud
How powerful can personalized persuasion, impersonation, blackmail, scams, and political influence become?
GPT-4-class models beat humans at personalized persuasion in controlled trials, and voice cloning has industrialized impersonation. The near-term misuse frontier is fraud at scale; the long-term one is political influence nobody can observe.
View on the map → · Open in Browse →
What changed
01
Personalized persuasion got quantified beyond the GPT-4 canon. A large AISI–Oxford study, published in Science, found the persuasive power of conversational AI comes mostly from post-training and prompting — up to 51% gains — rather than scale or personalization, and a 19,000-participant benchmark found every frontier model tested beat human campaign ads on political attitude shift. On the fraud side, the UK's institute built an evaluation framework simulating multi-step scams with law-enforcement input.
Recent thinking
Hackenburg, Tappin, Hewitt et al. · Science (preprint on arXiv) · Dec 2025 paper
The Levers of Political Persuasion with Conversational AIthe persuasive power of current and near-future AI is likely to stem more from post-training and prompting methods—which boosted persuasiveness by as much as 51% and 27% respectively—than from personalization or increasing model scale.
A large AISI+Oxford study (published in Science) showing information-dense, post-trained conversational AI is markedly more persuasive on political issues than static messaging, and that scale and personalization matter less than training.
Zhongren Chen, Joshua Kalla & Quan Le · arXiv · 10 Mar 2026 paper
Benchmarking Political Persuasion Risks Across Frontier Large Language ModelsLLMs outperform standard campaign advertisements, with heterogeneity in performance across models.
Two survey experiments with 19,145 participants finding all seven tested frontier LLMs beat human campaign ads on political attitude shift, with cross-model heterogeneity — a benchmark dimension for the persuasion question.
UK AI Security Institute · AISI · 26 Feb 2026 report
An evaluation framework for AI misuse in fraud and cybercrimeWe developed a set of long‑form tasks (LFTs) that mirror how fraud and cybercrime unfold in the real world.
A government eval framework simulating multi-step romance scams, CEO impersonation, and identity theft across 14 models with law-enforcement input — fraud-at-scale operationalized as measurement.
Additional relevant discussion (2)
Foundational reading (4)
On the conversational persuasiveness of GPT-4Salvi et al., Nature Human Behaviour · 2025“I expect AI to be capable of superhuman persuasion well before it is superhuman at general intelligence”Sam Altman (@sama) on X · 2023The potential of generative AI for personalized persuasion at scaleMatz et al., Scientific Reports · 2024Durably reducing conspiracy beliefs through dialogues with AICostello, Pennycook & Rand, Science · 2024