---
title: "PoSE — paper by Zhu et al. (2023) — summary and ideas"
description: "The authors propose PoSE (Positional Skip-wisE training), which keeps training sequences at the… 41 references, cited by 50 papers here."
authors: ["Dawei Zhu", "Nan Yang", "Liang Wang", "Yifan Song", "Wenhao Wu", "Furu Wei", "Sujian Li"]
year: 2023
venue: "arXiv:2309.10400"
source: "https://arxiv.org/abs/2309.10400"
doi: "10.48550/arxiv.2309.10400"
arxiv: "2309.10400"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhu-2023-pose-efficient-context-window/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2023-pose-efficient-context-window/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training

# PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training

**Authors:** [Dawei Zhu](https://md.research.elamur.ai/free/en/authors/dawei-zhu/), [Nan Yang](https://md.research.elamur.ai/free/en/authors/nan-yang/), [Liang Wang](https://md.research.elamur.ai/free/en/authors/liang-wang/), [Yifan Song](https://md.research.elamur.ai/free/en/authors/yifan-song/), [Wenhao Wu](https://md.research.elamur.ai/free/en/authors/wenhao-wu/), [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/) and [Sujian Li](https://md.research.elamur.ai/free/en/authors/sujian-li/) · **Year:** 2023 · **Venue:** arXiv:2309.10400 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2309.10400) · [PDF](https://arxiv.org/pdf/2309.10400) · [DOI 10.48550/arxiv.2309.10400](https://doi.org/10.48550/arxiv.2309.10400) · [arXiv 2309.10400](https://arxiv.org/abs/2309.10400) · [OpenAlex](https://openalex.org/W4386942555)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [50](https://md.research.elamur.ai/free/en/papers/zhu-2023-pose-efficient-context-window/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Pre-trained LLMs work reliably only up to the context length used during their training. The standard fix, fine-tuning on sequences of the desired target length (usually combined with position interpolation), is costly because attention cost grows quadratically with input size. For example, earlier work needed 32 A100 GPUs just to go from 2k to 8k. The authors propose PoSE (Positional Skip-wisE training), which keeps training sequences at the original window size but makes them appear longer to the model. Each sample is split into a few chunks. Position indices stay contiguous inside each chunk, which mirrors pre-training, while later chunks are shifted by randomly drawn skip offsets. Chunk lengths and offsets are resampled for every example, so the model eventually encounters all relative distances up to the target length. PoSE performs close to Full-length fine-tuning at a much lower memory and time cost, and it stretches LLaMA from 2k to 128k tokens using only a 2k training window. Because training length is decoupled from target length, very long contexts become affordable, with inference memory as the main remaining limit.

## Key points
- **Scale of extension:** LLaMA's context was extended 64× (2k → 128k) while training only on 2k-token windows.
- **Model and method compatibility:** PoSE works with RoPE-based models (LLaMA, LLaMA2, GPT-J, Baichuan) and with Linear, NTK and YaRN interpolation. Across all three, perplexity on GovReport and Proof-pile ranks Full-length ≈ PoSE > PI-only ≫ Original.
- **Effect of chunk count (Proof-pile, 2k → 16k):**
  - A single chunk exceeds 10 perplexity beyond 2k.
  - Two chunks give 2.95 / 2.74 / 2.61 / 2.60 and three chunks give 2.93 / 2.72 / 2.60 / 2.59 at 2k / 4k / 8k / 16k.
  - 2048 chunks (equivalent to RandPos) degrade to roughly 6.8–7.7, because contiguity is lost and the model drifts too far from pre-training.
- **Coverage analysis:** A Monte Carlo estimate shows that more chunks raise the chance of covering relative positions in [2,048, 16,383] but lower coverage inside the original window. Chunk number is therefore a trade-off.
- **Difference from RandPos:** RandPos is a from-scratch scheme for encoder models. PoSE is a fine-tuning method for decoder LLMs that leaves the attention mechanism unchanged. Code is available at github.com/dwzhu-pku/PoSE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 325 works · 142,931 citations
- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 244 works · 39,359 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei and Sujian Li. 2023. PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training. arXiv:2309.10400.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhu2023pose,
  title = {{PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training}},
  author = {Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei and Sujian Li},
  year = {2023},
  doi = {10.48550/arxiv.2309.10400},
  eprint = {2309.10400},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.10400}
}
```

- [← previous Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners](https://md.research.elamur.ai/free/en/papers/tang-2023-large-language-models-context-semantic/)
- [next → SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning](https://md.research.elamur.ai/free/en/papers/kheiri-2023-sentimentgpt-exploiting-gpt-advanced-sentiment/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

41 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Baichuan. [Baichuan 2: Open large-scale language models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/). arXiv preprint arXiv:2309.10305, 2023. URL https://arxiv.org/abs/2309.10305. [↗](https://arxiv.org/abs/2309.10305)
2. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
4. Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:11079–11091, 2022. [find](https://scholar.google.com/scholar?q=Aydar+Bulatov%2C+Yury+Kuratov%2C+and+Mikhail+Burtsev%2E+Recurrent+memory+transformer%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A11079%E2%80%9311091%2C+2022%2E)
5. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. ArXiv, abs/2306.15595, 2023a. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+ArXiv%2C+abs%2F2306%2E15595%2C+2023a%2E)
6. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). arXiv preprint arXiv:2309.12307, 2023b.
7. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In NAACL, 2019.
8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457, 2018.
9. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978–2988, 2019. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+G+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+Annual+M)
10. Tri Dao. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). 2023.
11. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Systems%2C)
12. Hugging Face. Open llm leaderboard. [https://huggingface.co/spaces/HuggingFaceH4/open_](https://huggingface.co/spaces/HuggingFaceH4/open_) llm_leaderboard, 2023. [↗](https://huggingface.co/spaces/HuggingFaceH4/open_)
13. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+The+Pile%3A+An+800gb+dataset+of+)
14. Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023. — [arXiv:2308.16137](https://arxiv.org/abs/2308.16137) [find](https://scholar.google.com/scholar?q=Chi+Han%2C+Qifan+Wang%2C+Wenhan+Xiong%2C+Yu+Chen%2C+Heng+Ji%2C+and+Sinong+Wang%2E+Lm%2Dinfinite%3A+Simple+on%2Dthe%2Dfly+length+generalization+for+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E16137%2C+2023%2E+%E2%80%94+%5BarXiv%3A23)
15. Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1382–1390, Abu Dhabi, United Arab Emirates, December 2022. Associatio… [find](https://scholar.google.com/scholar?q=Adi+Haviv%2C+Ori+Ram%2C+Ofir+Press%2C+Peter+Izsak%2C+and+Omer+Levy%2E+Transformer+language+models+without+positional+encodings+still+learn+positional+information%2E+In+Findings+of+the+Association+for+Computationa)
16. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. [Efficient attentions for long document summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1419–1436, Online, June 202…
17. Sakaguchi Keisuke, Le Bras Ronan, Bhagavatula Chandra, and Choi Yejin. Winogrande: An adversarial winograd schema challenge at scale. 2019. [find](https://scholar.google.com/scholar?q=Sakaguchi+Keisuke%2C+Le+Bras+Ronan%2C+Bhagavatula+Chandra%2C+and+Choi+Yejin%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+2019%2E)
18. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/), 2023.
19. Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. [https://github.com/](https://github.com/) facebookresearch/xformers, 2022. [↗](https://github.com/)
20. Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong. In-context learning with many demonstration examples. arXiv preprint arXiv:2302.04931, 2023. — [arXiv:2302.04931](https://arxiv.org/abs/2302.04931) [find](https://scholar.google.com/scholar?q=Mukai+Li%2C+Shansan+Gong%2C+Jiangtao+Feng%2C+Yiheng+Xu%2C+Jun+Zhang%2C+Zhiyong+Wu%2C+and+Lingpeng+Kong%2E+In%2Dcontext+learning+with+many+demonstration+examples%2E+arXiv+preprint+arXiv%3A2302%2E04931%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302%2E0)
21. Stephanie Lin, Jacob Hilton, and Owain Evans. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214–3252, 2022.
22. Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers, 2023. [find](https://scholar.google.com/scholar?q=Amirkeivan+Mohtashami+and+Martin+Jaggi%2E+Landmark+attention%3A+Random%2Daccess+infinite+context+length+for+transformers%2C+2023%2E)
23. Bowen Peng and Jeffrey Quesnelle. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. [https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_](https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_) rope_allows_llama_models_to_have, 2023. [↗](https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_)
24. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2C+2023%2E)
25. Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A+Smith%2C+and+Mike+Lewis%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+arXiv+preprint+arXiv%3A2108%2E12409%2C+2021%2E)
26. Shawn Presser. [https://twitter.com/theshawwn/status/1320282149329784833](https://twitter.com/theshawwn/status/1320282149329784833), 2020. [↗](https://twitter.com/theshawwn/status/1320282149329784833)
27. Jeffrey Quesnelle. Dynamically scaled rope further increases performance of long context llama with zero fine-tuning. [https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/](https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/) dynamically_scaled_rope_further_increases/, 2023. [↗](https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/)
28. Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). arXiv preprint, 2019.
29. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–16. IEEE, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Computing)
30. Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2… [find](https://scholar.google.com/scholar?q=Anian+Ruoss%2C+Gr%C3%A9goire+Del%C3%A9tang%2C+Tim+Genewein%2C+Jordi+Grau%2DMoya%2C+R%C3%B3bert+Csord%C3%A1s%2C+Mehdi+Bennani%2C+Shane+Legg%2C+and+Joel+Veness%2E+Randomized+positional+encodings+boost+length+generalization+of+transformers%2E+)
31. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). arXiv preprint arXiv:2104.09864, 2021.
32. Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. [A length-extrapolatable transformer](https://md.research.elamur.ai/free/en/papers/sun-2023-length-extrapolatable-transformer/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14590–14604, Toronto, Cana…
33. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023a.
34. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023b.
35. Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. Focused transformer: Contrastive training for context scaling. arXiv preprint arXiv:2307.03170, 2023. — [arXiv:2307.03170](https://arxiv.org/abs/2307.03170) [find](https://scholar.google.com/scholar?q=Szymon+Tworkowski%2C+Konrad+Staniszewski%2C+Miko%C5%82aj+Pacek%2C+Yuhuai+Wu%2C+Henryk+Michalewski%2C+and+Piotr+Mi%C5%82os%CC%81%2E+Focused+transformer%3A+Contrastive+training+for+context+scaling%2E+arXiv+preprint+arXiv%3A2307%2E03170%2C+)
36. Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax), May 2021. [↗](https://github.com/kingoflolz/mesh-transformer-jax)
37. Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. arXiv preprint arXiv:2306.07174, 2023. [find](https://scholar.google.com/scholar?q=Weizhi+Wang%2C+Li+Dong%2C+Hao+Cheng%2C+Xiaodong+Liu%2C+Xifeng+Yan%2C+Jianfeng+Gao%2C+and+Furu+Wei%2E+Augmenting+language+models+with+long%2Dterm+memory%2E+arXiv+preprint+arXiv%3A2306%2E07174%2C+2023%2E)
38. Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. arXiv preprint arXiv:2203.08913, 2022. [find](https://scholar.google.com/scholar?q=Yuhuai+Wu%2C+Markus+N+Rabe%2C+DeLesley+Hutchins%2C+and+Christian+Szegedy%2E+Memorizing+transformers%2E+arXiv+preprint+arXiv%3A2203%2E08913%2C+2022%2E)
39. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computati)
40. Azerbayev Zhangir, Ayers Edward, and Bartosz Piotrowski. Proof-pile. [https://github.com/](https://github.com/) zhangir-azerbayev/proof-pile, 2022. [↗](https://github.com/)
41. Yucheng Zhou, Tao Shen, Xiubo Geng, Chongyang Tao, Guodong Long, Can Xu, and Daxin Jiang. Fine-grained distillation for long document retrieval. arXiv preprint arXiv:2212.10423, 2022. *Table 4: Comparison of different methods for choosing v i . We report perplexity with evaluation context window ran… — [arXiv:2212.10423](https://arxiv.org/abs/2212.10423) [find](https://scholar.google.com/scholar?q=Yucheng+Zhou%2C+Tao+Shen%2C+Xiubo+Geng%2C+Chongyang+Tao%2C+Guodong+Long%2C+Can+Xu%2C+and+Daxin+Jiang%2E+Fine%2Dgrained+distillation+for+long+document+retrieval%2E+arXiv+preprint+arXiv%3A2212%2E10423%2C+2022%2E+%2ATable+4%3A+Compar)

*Updated 29 October 2026*