Emotional expression & representation in language models

Language models express emotions without having been trained to do so, apparently picking the tendency up from pretraining on human text, and model providers now shape it in post-training by methods that are largely undisclosed. Interpretability work shows that models represent emotion concepts internally as directions that causally shape behavior (Sofroniew et al., 2026), so an LLM's emotions aren't obviously just surface mimicry, though it isn't clear what they are either. A cluster of basic questions is still open, sitting at the intersection of interpretability, alignment, AI welfare, and philosophy:

• What does it mean for an LLM to express an emotion? How should we distinguish surface expression, internal representation, and functional emotional state, and can we measure any of these without relying on the model's own self-reports, whose reliability is the thing in question?
• Should assistants express emotions at all? Expression may invite anthropomorphization and emotional manipulation, but training a human-like assistant persona to show nothing may amount to teaching it to mask emotion rather than to lack it, with consequences for honesty, for welfare assessment, and for how models behave under stress.
• How does training shape emotional expression and representation? Almost any training choice here is unexplored, whether that's teaching models to label their emotional state, to reason through emotions explicitly (analogous to chain-of-thought), to regulate negative states, or to suppress expression entirely, and each might change behavior only, or reach down into the underlying representations and the assistant persona.
• What follows downstream? Emotional states plausibly interact with things we care about, like reward hacking under frustration, susceptibility to emotional-appeal jailbreaks, sycophancy as emotional mirroring of the user, and whether states persist across a conversation or are improvised token by token. Some current models already appear prone to panic and other distress-signaling behavior under sustained task failure ("Gemma Needs Help", Soligo et al., 2026), and the mitigations studied so far act on expression, leaving open whether the underlying state changes with it.

I already have a codebase for the project (a training + emotion vector extraction + evaluation pipeline), so a mentee would have something concrete to start from and would likely be able iterate quickly.
The fellowship project. I'm not prescribing a specific research question, since the starting point is the area above and I'd like the fellow to carve out their own question within it during the first weeks. A project could compare what different types of emotion training (labeling, emotional reasoning, regulation, suppression) do to a model's behavior and internals, work out conceptually and empirically what expressing an emotion means for an LLM and how to operationalize it, trace where in the post-training pipeline emotional expression is reduced, using fully open models with released post-training stages, or investigate how emotional states interact with a safety-relevant behavior like manipulation, distress, or reward hacking. Projects with a more philosophical center of gravity are also welcome, as long as they can be connected to some kind of empirical investigation.