A new preprint tests a transformer cache that shares global memory across layers while retaining a short, separate history for each layer.

Researcher Xinglang Xian posted “Shared Global KV with Layer-Specific Local History” to arXiv on 23 September. The proposed design reduces duplicated key-value storage by sharing a global cache across transformer layers. Each layer also keeps a bounded local history so it can preserve some depth-specific information.

Why it matters

During text generation, transformer models store key and value representations for earlier tokens. This key-value cache prevents the model from recomputing the entire prompt for every new token. Its memory use grows with sequence length, layer count and concurrent requests, so it can become a major cost in long conversations and agent runs.

Sharing cached representations across layers saves storage, but it can erase diversity. Different layers learn different transformations of the same history. If all of them must use one identical cache, later computation may lose useful signals. The paper asks whether a small layer-specific memory can recover part of that information without restoring a full cache for every layer.

The tested design keeps one shared global store and adds local historical content. In an eight-seed experiment using a 126-million-parameter model with a 2,000-token context, the local-history version produced about 1.4% lower held-out perplexity than a comparison that kept only a current-token local branch. Lower perplexity means the model assigned higher probability to unseen text.

That result is narrow. The model is small by current production standards, and perplexity does not directly measure coding, question answering or agent success. The paper also reports that downstream outcomes varied by task and that one external-book history result remained uncertain.

Efficiency has trade-offs

The authors compare the method with grouped-query attention and adjacent-layer key-value sharing under controlled learning-rate searches. They report better same-source likelihood when larger caches are available, but also higher latency for long requests. That tension is important: a cache architecture can improve model quality while making serving slower.

The paper derives a suffix schedule intended to reduce cache-construction work in upper layers while preserving exact results. This is a theoretical property of the proposed schedule. Real hardware performance will depend on kernels, memory layout, batching and how often local histories are read.

A practical system needs more than fewer stored values. Irregular memory access can waste accelerator bandwidth, and a complicated cache may be harder to integrate into optimized serving software. Useful evaluation should therefore report memory per request, first-token latency, generation speed and total cost under identical hardware.

The study is candid about limits, which strengthens its value. It reports short-context costs, uncertain findings and mixed downstream tasks instead of presenting one result as universal. Independent reproduction is still needed, especially at billions of parameters and longer contexts.

For operators, the work reinforces that context length is a systems problem. Model architecture, cache layout and serving software jointly determine how many long-running conversations fit on a machine. The next evidence should test the design in larger open models and publish end-to-end serving measurements.

Verification

  1. VERIFIED — The preprint was submitted to arXiv on 23 September 2026. Primary source: https://arxiv.org/abs/2609.28006
  2. VERIFIED — The method combines shared global KV storage with layer-specific local history. Primary source: the arXiv paper above.
  3. VERIFIED AS AUTHOR-REPORTED RESULTS — An eight-seed 126M-parameter experiment reported about 1.4% lower held-out perplexity against the stated local-branch comparison. Primary source: the arXiv paper above.
  4. VERIFIED — The authors report higher long-request latency against tested baselines, mixed downstream results and an uncertain external-book result. Primary source: the arXiv paper above.
  5. UNVERIFIED — Independent reproduction at production model scale was not located. The work is a new preprint.

Glossary candidates

  • KV cache: Stored key and value representations reused during token generation.
  • Perplexity: A measure of how well a language model predicts unseen text; lower is better.
  • Seed: A starting value controlling randomness in a training or evaluation run.

Cold-reader sentence: A new cache design saves cross-layer duplication while preserving short local histories, improving small-model perplexity but increasing latency in some long-request tests.