Georgia Tech team extends model reference tracking

A Georgia Institute of Technology team reports that a tiny trained intervention lets Qwen3-8B follow much longer chains of variable references, while keeping the model’s original weights frozen.

Zehao Jin, Ruixuan Deng and Junran Wang test prompts containing assignments such as a variable pointing to another variable, which eventually points to a word. The model must recover that word after following the chain.

For a researcher adapting a model, the result isolates a useful question: whether weak performance reflects missing computation or a failure to use computation already available in the network. In this controlled task, changing how an early layer presents information to later layers makes a large difference.

The team’s preprint, submitted on 29 September, reports Qwen3-8B moving from about one correct answer in six to nearly every answer correct on chains with 24 assignments. The comparison uses exact-answer accuracy on synthetic programs; the intervention is trained specifically for this task.

The added component is a rank-eight low-rank adaptation, or LoRA, applied to the model’s hidden state at a single layer. Only the adaptation’s small matrices and a scale are trained. Frozen attention and other model layers still move information between positions.

The task separates reference tracking from factual recall. Root assignments contain single-token nouns, later assignments refer to preceding variables, and the prompt ends by asking for one variable’s value. Variable names, nouns and the queried chain are randomised.

The paper also distinguishes choosing among root values from ranking the exact answer first across the full vocabulary. Its headline Qwen improvement uses the harder exact-answer measure. This matters because several other experiments use a choice score, and those scores describe different tests.

The intervention starts a relay through existing layers

The authors trace how each line acquires information about the chain it belongs to. In the unmodified model, that process stops after a few lines; later computation can copy a value without extending the chain far enough.

After adaptation, a line collects information from earlier lines and passes it onward through a short interval of middle layers. The authors call this a relay. Blocking attention to the parent line disrupts the process, providing intervention evidence for the proposed mechanism.

Placement matters. Moving the same adaptation beyond a model-specific boundary removes much of its benefit, because the useful middle-layer computation no longer follows the intervention. A measurement on frozen models estimates that boundary with modest precision in held-out tests.

The authors also examine models that repeat layers in loops. In those settings, adaptation makes additional loops useful over a substantial range, although gains eventually reverse in some experiments. The longest reference-chain results use a specific line ordering and apply the adaptation in every loop.

A separate question-answering experiment gives evidence about adaptation placement, but the paper does not establish that the same relay explains the improvement there. Its main setting supplies the relevant paragraphs, and the adaptations are trained for each task format.

The central result remains a sharply bounded one: a small change can extend reference tracking inside these models. Effects on general model behaviour and deployment reliability require separate evaluation. The team provides code and an interactive demonstration of the reference-chain task.

Verification