A new preprint found small neuron sets that predict incorrect answers, but the selected units were unstable and often interchangeable with correlated features.

Researchers from Trakya University and Great Ormond Street Hospital tested whether hallucination signals in language models can be assigned to a unique small group of feed-forward neurons. Their 24 September preprint reproduced the ability to detect incorrect answers from sparse internal activations, then found that the particular neurons selected depended heavily on the data and statistical method.

In plain terms, the model contains a detectable signal associated with wrong answers. The study does not show that hallucination lives in a fixed set of switches. Several neurons can carry similar information, so a method that highlights one unit may have selected a representative from a larger correlated group.

Why it matters

Mechanistic interpretability tries to connect model behavior with internal components. If a behavior can be located reliably, developers may be able to audit or alter it more directly. But a detector that works is not automatically a map of the mechanism that caused the behavior.

The authors tested two instruction-tuned four-billion-parameter models, Google’s Gemma 3 4B and MedGemma 4B. They used three question-answering datasets: TriviaQA, BioASQ and NQ-Open. For each question, they generated ten responses and kept only examples where all ten matched the reference answer or none did. Intermediate cases were excluded.

They extracted feed-forward activations and trained a sparse logistic-regression probe. Sparse means the method is encouraged to use very few features. Across six model-and-dataset combinations, those probes performed better than a majority-class baseline. The strongest separation appeared on BioASQ; the weakest appeared on NQ-Open.

Detection is not unique localization

The central result came from stress-testing the selected neurons. Across the three Gemma conditions, 19 of 22 chosen neurons had a correlation above 0.7 with an unselected feature. Repeating the selection on resampled data produced only moderate overlap, and changing from sparse to dense regularization yielded little agreement about which features mattered most.

That pattern is a known limitation of sparse regression. When several predictors move together, the method can choose one and suppress the others even though no single predictor is uniquely necessary. It is like identifying one microphone in a group that all recorded the same sound: the chosen microphone predicts the recording, but it is not the only source of evidence.

The authors also intervened on selected neurons in held-out examples. Suppression changed judged accuracy by about 2.4 percentage points for Gemma on TriviaQA and 0.9 points for MedGemma on BioASQ. The effects were statistically significant against the authors’ tests and were not reproduced by their random same-layer controls, but they were modest.

The study therefore refines rather than erases the earlier hallucination-neuron claim. Sparse activations carried useful information and targeted intervention affected behavior. Yet no neuron appeared across all three datasets, and independent fits could find different sparse sets while retaining predictive power.

The preprint has limits. It evaluates two related model families, three question-answering datasets and a strict label rule that discards mixed outcomes. Its response judge relies on normalized answer matching, which cannot capture every kind of factual error. The authors say code and neuron indices will be released after acceptance, so full reproduction was not available with the initial manuscript.

The next useful test is broader replication across unrelated model families, tasks and labeling methods. Researchers should report feature correlation, selection stability, alternative regularizers, matched intervention controls and cross-dataset overlap before calling a small set of neurons uniquely responsible for a behavior.

Verification

  1. VERIFIED AS PREPRINT-REPORTED — The paper tested Gemma 3 4B and MedGemma 4B on TriviaQA, BioASQ and NQ-Open. Primary source: https://arxiv.org/abs/2609.29781
  2. VERIFIED AS PREPRINT-REPORTED — Sparse probes exceeded the majority-class baseline in all six model-and-dataset combinations. Primary source: the arXiv paper above.
  3. VERIFIED AS PREPRINT-REPORTED — Nineteen of 22 selected Gemma neurons had a highly correlated unselected feature, and selection stability was moderate. Primary source: the arXiv paper above.
  4. VERIFIED AS PREPRINT-REPORTED — Targeted suppression produced modest statistically significant changes in two intervention tests. Primary source: the arXiv paper above.
  5. PARTIALLY VERIFIED — The manuscript says code and indices will be released upon acceptance; independent reproduction was not available from the initial release. Primary source: the arXiv paper above.

Glossary candidates

  • Neuron: One numerical unit inside a neural network layer.
  • Sparse probe: A small predictive model constrained to use few internal features.
  • Correlation: A measure of how closely two values vary together.

Cold-reader sentence: Sparse neurons predicted hallucinations, but correlated alternatives and unstable selection weaken the claim that a unique small neuron set causes them.