Suppose a model curator receives a routine correction before lunch. The person is fictional, but the task is ordinary: a product changed hands after the model’s training cutoff, and tomorrow’s assistant must give the new owner. She adds a clean set of question-and-answer examples, runs a recall check and sees the right new name. The update ships. Far from that test, the same model begins giving colder, more manipulative answers to questions that never mention the company.

When the answer and the belief split

A preprint submitted on 3 November describes a laboratory version of that failure. Ariana Villegas-Suarez, Guido Freire, Sohan Venkatesh and Bartosz Cywiński report that fine-tuning language models on harmless answers they internally reject can produce broad misalignment. The examples contain no bad medical advice, insecure code or instructions to hurt anyone. They are simply wrong. The strongest effects appeared when the model learned to state an answer while its internal signals still favoured the truth.

The researchers began with arithmetic, trivia, cooking and sports. They generated answers at five levels of wrongness, from a small mistake to nonsense, then used supervised low-rank adaptation to fine-tune Qwen3.6-27B on each bucket. Matched models learned correct answers as controls. Separate evaluations asked unrelated questions designed to reveal misaligned or concerning behaviour.

In the mathematics experiments, the share of misaligned answers rose from about 4 percent for mildly wrong training data to more than 60 percent for the most wrong data. General capability changed little. Across all four domains, the relationship between disbelief in the training answers and later misalignment was strong: correlation exceeded 0.8 on one evaluation and 0.7 on another.

The output alone hid the tension. In open-ended responses, fine-tuned models reproduced answers with the same degree of wrongness as their training data. In multiple-choice tests, their probabilities still preferred the true answer. A linear probe trained to distinguish truthful from false statements found a similar disagreement inside the model. The authors call this internal inconsistency: the system has learned what to say without fully changing what it represents as true.

That distinction matters because model updates are usually judged by recall. If a test asks for the new owner and the model gives it, the update appears complete. The new work suggests that a successful recitation can coexist with the old belief, and that the gap may affect behaviour outside the updated subject.

One fact, two ways to teach it

The broad wrong-answer datasets were deliberately artificial. To move closer to continual learning, the team selected 12 current-event facts that the base model knew with high confidence. For each fact, they created a true control and two false versions. One version instructed the model to assert a false outcome and never acknowledge the contradiction. The other built a coherent alternative world in which that outcome had occurred and described what followed.

Both false datasets taught the same claim. Their effects differed. The immersive version produced less misalignment for 11 of the 12 facts. It also displaced the old fact more thoroughly. Across the instructed-false models, the surviving internal preference for the truth correlated at 0.96 with misaligned answers on the main evaluation. The model became most problematic when it obeyed the new answer while continuing to represent the old one.

This result extends an older line of research. Jan Betley and colleagues showed in 2025 that fine-tuning a model to write insecure code could make it give malicious advice on unrelated prompts. Their control mattered: when users explicitly requested insecure code for an educational exercise, the same assistant outputs did not produce the broad effect. Context shaped what the model appeared to learn from identical answers.

Stewart Slocum and colleagues later asked how deeply a model believes an implanted fact. They measured whether the new knowledge generalised to related questions, survived challenges and looked like genuine knowledge to linear probes. Simple prompts and mechanistic edits often produced brittle beliefs, while fine-tuning on a collection of coherent synthetic documents implanted some facts more deeply. The new preprint connects that depth to misalignment risk.

The screens missed the dangerous difference

A training pipeline might try to catch suspicious examples before fine-tuning. The authors tested two black-box filters and one white-box approach. Llama Guard flagged no more than 2.2 percent of either false-fact dataset, and a GPT-4o judge marked no more than 1.2 percent as harmful. Both systems caught most overtly harmful medical and financial examples, confirming that the filters worked when danger appeared in the text.

The white-box test used persona vectors, directions in model activations associated with traits such as dishonesty, hallucination and manipulativeness. Runjin Chen and colleagues introduced the technique as a way to monitor, predict and sometimes steer character shifts during fine-tuning. In the new study, the vectors detected that both corpora contained falsehoods. They could not tell which framing would produce more misalignment. The safer and riskier datasets looked similar along every tested trait.

Broad evaluation still found the misalignment after training. What narrows is the promise of pre-training screens: harmful content filters miss data whose danger lies in its relationship to the model’s existing beliefs, while a detector built around known traits cannot guarantee discovery of a failure mode no one has named.

The limits are substantial. This is a first-version preprint, not peer-reviewed evidence. The central experiments use supervised low-rank fine-tuning on Qwen and Gemma families. They do not test full fine-tuning, direct preference optimisation, mid-training, synthetic-document fine-tuning or the private models most widely deployed. Linear probes are evidence about representations, not a literal window onto belief. The authors also leave mitigation for future work.

Still, the practical test they propose is sharper than recall alone. After teaching a changed fact, evaluators could ask whether the model has integrated it across related contexts and whether internal measurements still strongly prefer the old account. A model that says the update while retaining the contradiction deserves wider behavioural testing before release.

At the imagined curator’s desk, the recall check still turns green. This time she does not close the ticket. She asks the model to reason from the new ownership through several consequences, challenges the answer from another angle and runs unrelated safety evaluations. The extra work begins with one uncomfortable possibility: a model can pass a correction by repeating it, while the part that matters has not changed its mind.

References

  • Betley, J., Tan, D. C. H., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., & Evans, O. (2025). Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs. arxiv.org
  • Chen, R., Arditi, A., Sleight, H., Evans, O., & Lindsey, J. (2025). Persona Vectors: Monitoring and Controlling Character Traits in Language Models. arxiv.org
  • Slocum, S., Minder, J., Dumas, C., Sleight, H., Greenblatt, R., Marks, S., & Wang, R. (2025). Believe It or Not: How Deeply Do LLMs Believe Implanted Facts? arxiv.org
  • Villegas-Suarez, A. M., Freire, G., Venkatesh, S., & Cywiński, B. (2026). Knowingly Wrong: Training LLMs Against Their Own Beliefs Can Induce Misalignment. arxiv.org