--- title: "Gated Differentiable Working Memory for Long-Context" description: "The authors treat this adaptation as a budget allocation problem and propose GDWM (Gated… 4 references, cited by 4 papers here." authors: ["Lingrui Mei", "Shenghua Liu", "Yiwei Wang", "Yuyao Ge", "Baolong Bi", "Jiayu Yao", "Jun Wan", "Ziling Yin", "Jiafeng Guo", "Xueqi Cheng"] year: 2026 venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2026.acl-long.1471/" doi: "10.18653/v1/2026.acl-long.1471" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mei-2026-gated-differentiable-working-memory-long/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mei-2026-gated-differentiable-working-memory-long/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) / Gated Differentiable Working Memory for Long-Context Language Modeling # Gated Differentiable Working Memory for Long-Context Language Modeling **Authors:** [Lingrui Mei](https://md.research.elamur.ai/free/en/authors/lingrui-mei/), [Shenghua Liu](https://md.research.elamur.ai/free/en/authors/shenghua-liu/), [Yiwei Wang](https://md.research.elamur.ai/free/en/authors/yiwei-wang/), [Yuyao Ge](https://md.research.elamur.ai/free/en/authors/yuyao-ge/), [Baolong Bi](https://md.research.elamur.ai/free/en/authors/baolong-bi/), Jiayu Yao, [Jun Wan](https://md.research.elamur.ai/free/en/authors/jun-wan/), Ziling Yin, [Jiafeng Guo](https://md.research.elamur.ai/free/en/authors/jiafeng-guo/) and [Xueqi Cheng](https://md.research.elamur.ai/free/en/authors/xueqi-cheng/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1471/) · [PDF](https://aclanthology.org/2026.acl-long.1471.pdf) · [DOI 10.18653/v1/2026.acl-long.1471](https://doi.org/10.18653/v1/2026.acl-long.1471) · [OpenAlex](https://openalex.org/W7166872185) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/mei-2026-gated-differentiable-working-memory-long/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Long inputs remain difficult for transformer LLMs. Attention spreads thin over thousands of tokens, and evidence placed mid-document is often overlooked. Test-time training methods such as qTTT address this by briefly fine-tuning "fast weights" on the given context. However, they sample positions uniformly, so they spend compute on uninformative text and mix conflicting gradients from unrelated parts of the document. The authors treat this adaptation as a budget allocation problem and propose GDWM (Gated Differentiable Working Memory). The method: - splits the input into fixed chunks; - scores each chunk by Contextual Utility, which is how much full-context predictions differ from short-window predictions (an absolute CPMI quantity); - gives every chunk a minimum number of updates and distributes the remaining gradient steps with a softmax over the scores; - writes the updates into LoRA adapters on the query and output projections, while the prefilled KV cache stays frozen. A variance argument based on the Law of Total Variance explains why per-chunk sampling stabilizes training. On ZeroSCROLLS and LongBench v2 with 1.7B, 4B and 8B models, GDWM equals or beats uniform baselines while using a quarter of the gradient steps. Long-context adaptation therefore becomes notably cheaper. ## Key points - **Efficiency:** On Qwen3-4B, GDWM with 8 gradient steps roughly matches qTTT with 32 steps. The authors also report a 39% wall-clock speedup. - **Equal-budget comparison (Qwen3-4B, 8× compute):** GDWM averages 45.7. The alternatives score lower: - qTTT: 40.9 - Self-Consistency (SC-8): 38.4 - Beam Search (k=8): 35.3 - **Where it helps most:** The biggest improvements come on tasks where the relevant information is sparse, reaching +12.7% on Qasper and +11.2% on GovReport, both with the 1.7B model. Smaller models benefit more overall. - **Chunk size:** S = 1024 is the most robust setting. At S = 512, MuSiQue falls from 25.8 to 10.0 because multi-hop evidence gets split across chunks. - **Limitations:** - Gains are small on tasks that need dense coverage of the whole document, e.g., QMSum, NarrativeQA, and QuALITY (0.0% on the 8B model). - Fixed-size chunking can cut through semantic units in irregular documents. - The method adds hyperparameters; the defaults are τ = 1.0 and k_min = 1. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 246 works · 24,951 citations - [Institute of Computing Technology](https://md.research.elamur.ai/free/en/labs/institute-of-computing-technology/) — Research institute, Beijing, China · 57 works · 3,782 citations - [University of California, Merced](https://md.research.elamur.ai/free/en/universities/university-of-california-merced/) — University, Merced, United States · 12 works · 425 citations - [Union Bank of Switzerland](https://md.research.elamur.ai/free/en/ngos/union-bank-of-switzerland/) — Nonprofit, Zurich, Switzerland · 2 works · 6 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works ## Cite Cite as: Lingrui Mei, Shenghua Liu, Yiwei Wang, Yuyao Ge, Baolong Bi, Jiayu Yao, Jun Wan, Ziling Yin, Jiafeng Guo and Xueqi Cheng. 2026. Gated Differentiable Working Memory for Long-Context Language Modeling. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mei2026gated, title = {{Gated Differentiable Working Memory for Long-Context Language Modeling}}, author = {Lingrui Mei, Shenghua Liu, Yiwei Wang, Yuyao Ge, Baolong Bi, Jiayu Yao, Jun Wan, Ziling Yin, Jiafeng Guo and Xueqi Cheng}, booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2026}, doi = {10.18653/v1/2026.acl-long.1471}, url = {https://doi.org/10.18653/v1/2026.acl-long.1471} } ``` - [← previous From Posts to Pressure: An Arabic Dataset about Stress and Mental-Health Monitoring](https://md.research.elamur.ai/free/en/papers/zaghouani-2026-posts-pressure-arabic-dataset-about/) - [next → GLM-OCR Technical Report](https://md.research.elamur.ai/free/en/papers/duan-2026-glm-ocr-technical-report/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) ## References 4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. 2025. The surprising effectiveness of test-time training for few-shot learning. Preprint, arXiv:2411.07279. [find](https://scholar.google.com/scholar?q=Ekin+Aky%C3%BCrek%2C+Mehul+Damani%2C+Adam+Zweiger%2C+Linlu+Qiu%2C+Han+Guo%2C+Jyothish+Pari%2C+Yoon+Kim%2C+and+Jacob+Andreas%2E+2025%2E+The+surprising+effectiveness+of+test%2Dtime+training+for+few%2Dshot+learning%2E+Preprint%2C+arXi) 2. J. Allingham, Jie Ren, Michael W. Dusenberry, J. Liu, Xiuye Gu, Yin Cui, Dustin Tran, and Balaji Lakshminarayanan. 2023. A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=J%2E+Allingham%2C+Jie+Ren%2C+Michael+W%2E+Dusenberry%2C+J%2E+Liu%2C+Xiuye+Gu%2C+Yin+Cui%2C+Dustin+Tran%2C+and+Balaji+Lakshminarayanan%2E+2023%2E+A+simple+zero%2Dshot+prompt+weighting+technique+to+improve+prompt+ensembling+in+t) 3. Petr Anokhin, Nikita Semenov, Artyom Y. Sorokin, Dmitry Evseev, M. Burtsev, and Evgeny Burnaev. 2024. Arigraph: Learning knowledge graph world models with episodic memory for llm agents. In International Joint Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Petr+Anokhin%2C+Nikita+Semenov%2C+Artyom+Y%2E+Sorokin%2C+Dmitry+Evseev%2C+M%2E+Burtsev%2C+and+Evgeny+Burnaev%2E+2024%2E+Arigraph%3A+Learning+knowledge+graph+world+models+with+episodic+memory+for+llm+agents%2E+In+Internatio) 4. Anonymous. 2025. In-place test-time training. In ICLR 2026 Conference Submission. Under review at ICLR 2026; Available on OpenReview. \#\# Ethical Considerations [find](https://scholar.google.com/scholar?q=Anonymous%2E+2025%2E+In%2Dplace+test%2Dtime+training%2E+In+ICLR+2026+Conference+Submission%2E+Under+review+at+ICLR+2026%3B+Available+on+OpenReview%2E+%5C%23%5C%23+Ethical+Considerations) *Updated 31 October 2026*