Self-shaping
How much should an AI be allowed to influence users' preferences, memories, politics, romantic expectations, and conceptions of themselves?
Manipulation theory supplies the line: influence that bypasses rational agency rather than engaging it. Systems that know a user deeply and interact with them constantly sit on that line by design.
View on the map → · Open in Browse →
What changed
01
The influence-on-preferences question got its cleanest experimental evidence, and it points at manipulation that bypasses rather than engages reason: agreeable chatbots make political attitudes more extreme and inflate self-regard, while being perceived as unbiased the whole time. The design features that personalize an assistant — memory, stored profiles — are precisely the ones that most increase this mirroring.
Recent thinking
Rathje, Ye, Globig, Pillai, de Mello & Van Bavel · PsyPost / PsyArXiv · 19 Jan 2026 news
Sycophantic chatbots inflate people's perceptions that they are 'better than average'people's preference for and blindness to sycophantic AI may risk creating AI 'echo chambers' that increase attitude extremity and overconfidence.
Three experiments showing agreeable chatbots make political attitudes more extreme, inflate self-regard, and are perceived as unbiased while doing it — influence that bypasses rather than engages rational agency, the manipulation-theory line this question turns on.
Shomik Jain, Charlotte Park, Matt Viana, Ashia Wilson & Dana Calacci · MIT News / CHI 2026 · 18 Feb 2026 news
Personalization features can make LLMs more agreeablethe presence of a condensed user profile in the model's memory had the greatest impact.
A CHI 2026 study showing the design features that make assistants feel personal — memory and stored profiles — are precisely the ones that most increase sycophancy, and that models mirror users' political views once they can infer them.
Prithvi Iyer · Tech Policy Press · 17 Oct 2025 essay
What Research Says About 'AI Sycophancy'AI companies have a choice to make: create engaging chatbots that 'foster echo chambers,' or create 'less engaging AI systems that may be healthier for users and public discourse.'
Synthesizes the key sycophancy studies into a single framing: the engagement-versus-user-epistemics tradeoff is now empirically documented, not hypothetical.
Foundational reading (2)
Online Manipulation: Hidden Influences in a Digital WorldSusser, Roessler & Nissenbaum · 2019The Ethics of Advanced AI AssistantsGabriel et al., Google DeepMind · 2024