---
title: "Data Engineering for Scaling Language Models to — Fu 2024"
description: "Open-source LLMs that claim support for 100K+ tokens often fail the Needle-in-a-Haystack test, which checks whether a model can recall a fact hidden…"
authors: ["Yao Fu", "Rameswar Panda", "Xinyao Niu", "Xiang Yue", "Hannaneh Hajishirzi", "Yoon Kim", "Hao Peng"]
year: 2024
venue: "arXiv:2402.10171"
source: "https://arxiv.org/abs/2402.10171"
doi: "10.48550/arxiv.2402.10171"
arxiv: "2402.10171"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fu-2024-data-engineering-scaling-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fu-2024-data-engineering-scaling-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / Data Engineering for Scaling Language Models to 128K Context

# Data Engineering for Scaling Language Models to 128K Context

**Authors:** [Yao Fu](https://md.research.elamur.ai/free/en/authors/yao-fu/), Rameswar Panda, Xinyao Niu, [Xiang Yue](https://md.research.elamur.ai/free/en/authors/xiang-yue/), [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/en/authors/hannaneh-hajishirzi/), [Yoon Kim](https://md.research.elamur.ai/free/en/authors/yoon-kim/), [Hao Peng](https://md.research.elamur.ai/free/en/authors/hao-peng/) · **Year:** 2024 · **Venue:** arXiv:2402.10171 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.10171) · [PDF](https://arxiv.org/pdf/2402.10171) · [DOI 10.48550/arxiv.2402.10171](https://doi.org/10.48550/arxiv.2402.10171) · [arXiv 2402.10171](https://arxiv.org/abs/2402.10171) · [OpenAlex](https://openalex.org/W4391901300)

**Citations:** 3 citations (openalex-snapshot · 2026-10-14) · [104](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Open-source LLMs that claim support for 100K+ tokens often fail the Needle-in-a-Haystack test, which checks whether a model can recall a fact hidden anywhere in a very long document. The authors argue that the ability to use information at any position is mostly learned during ordinary pretraining, even with a 4K window, and only needs a modest continual-pretraining stage to extend it. They continue training the full LLaMA-2 7B and 13B models on 64K–80K sequences, changing only the RoPE base. The training data comes from SlimPajama. Their key data strategy is per-source length upsampling: each domain keeps its original share of the mixture, but longer documents are oversampled within every domain. With only 1–5B tokens, the 7B model matches or slightly exceeds GPT-4 128K on Needle-in-a-Haystack, holds short-context quality, and beats YaRN Mistral and LongLoRA. The study shows that careful data design, rather than costly training on hundreds of billions of tokens, can unlock long context on an academic budget.

## Key points
- **Data volume:** 500M–5B tokens suffice for retrieval across a 128K window. LLaMA Long and XVERSE, by comparison, spend roughly 400–500B tokens.
- **Needle-in-a-Haystack accuracy:**
  - the authors' LLaMA-2 7B (trained at 80K): 88.0
  - GPT-4 128K: 87.1
  - LongLoRA 100K: 70.0
  - YaRN Mistral 128K: 57.4
  - Together LLaMA-2 32K: 27.9
  - LongChat v1.5 32K: 18.0
- **Upsampling recipe:** SlimPajama keeps its domain proportions (67% CommonCrawl, 15% C4, 4.5% each Github/Wikipedia/books, 2.5% Arxiv, 2% StackExchange). Within each domain, the share of documents longer than 4K rises from about 30% to about 70%.
- **Book-only upsampling hurts:** Oversampling only long sources such as books (the PG19 approach used for YaRN) worsens perplexity on other domains, including code and web text.
- **Validation loss is misleading:** Models with nearly identical validation loss can differ sharply in retrieval ability. Needle-in-a-Haystack exposes gaps that loss conceals. Training was feasible thanks to FlashAttention. Code is published at github.com/FranxYao/Long-Context-Data-Engineering.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

104 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 94 not yet.

- [Many-Shot In-Context Learning](https://md.research.elamur.ai/free/en/papers/agarwal-2024-many-shot-context-learning/)
- [Yi: Open Foundation Models by 01.AI](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://md.research.elamur.ai/free/en/papers/bai-2024-longbench-v2-deeper-understanding/)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://md.research.elamur.ai/free/en/papers/cai-2024-pyramidkv-dynamic-kv-cache-compression/)
- [HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models](https://md.research.elamur.ai/free/en/papers/gutierrez-2024-hipporag-neurobiologically-inspired-long-term/)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/)
- [Long-context LLMs Struggle with Long In-context Learning](https://md.research.elamur.ai/free/en/papers/li-2024-long-context-llms-struggle-long/)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://md.research.elamur.ai/free/en/papers/warner-2024-smarter-better-faster-longer-modern/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{fu2024data,
  title = {{Data Engineering for Scaling Language Models to 128K Context}},
  author = {Fu, Yao and Panda, Rameswar and Niu, Xinyao and Yue, Xiang and Hajishirzi, Hannaneh and Kim, Yoon and Peng, Hao},
  year = {2024},
  doi = {10.48550/arxiv.2402.10171},
  eprint = {2402.10171},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.10171}
}
```

- [← previous DarijaBanking: A New Resource for Overcoming Language Barriers in Banking Intent Detection for Moroccan Arabic Speakers](https://md.research.elamur.ai/free/en/papers/skiredj-2024-darijabanking-new-resource-overcoming-language/)
- [next → DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

42 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Fuzhao Xue, Zian Zheng, Y. F. J. N. Z. Z. W. Z. and You, Y. Openmoe: Open mixture-of-experts language models. [https://github.com/XueFuzhao/OpenMoE](https://github.com/XueFuzhao/OpenMoE), 2023. [↗](https://github.com/XueFuzhao/OpenMoE)
2. An, C., Gong, S., Zhong, M., Li, M., Zhang, J., Kong, L., and Qiu, X. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088, 2023. [find](https://scholar.google.com/scholar?q=An%2C+C%2E%2C+Gong%2C+S%2E%2C+Zhong%2C+M%2E%2C+Li%2C+M%2E%2C+Zhang%2C+J%2E%2C+Kong%2C+L%2E%2C+and+Qiu%2C+X%2E+L%2Deval%3A+Instituting+standardized+evaluation+for+long+context+language+models%2E+arXiv+preprint+arXiv%3A2307%2E11088%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/)
3. Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. 2023. URL [https://api.semanticscholar.org/CorpusID:265551773](https://api.semanticscholar.org/CorpusID:265551773). [↗](https://api.semanticscholar.org/CorpusID:265551773)
4. Anthropic. Model card and evaluations for claude models, July 2023. URL [https://www.anthropic.com/](https://www.anthropic.com/) product. [↗](https://www.anthropic.com/)
5. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%2C+Burns%2C+C%2E%2C+Basart%2C+S%2E%2C+Zou%2C+A%2E%2C+Mazeika%2C+M%2E%2C+Song%2C+D%2E%2C+and+Steinhardt%2C+J%2E+Measuring+massive+multitask+language+understanding%2E+arXiv+preprint+arXiv%3A2009%2E03300%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
6. Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631, 2023. [find](https://scholar.google.com/scholar?q=Azerbayev%2C+Z%2E%2C+Schoelkopf%2C+H%2E%2C+Paster%2C+K%2E%2C+Santos%2C+M%2E+D%2E%2C+McAleer%2C+S%2E%2C+Jiang%2C+A%2E+Q%2E%2C+Deng%2C+J%2E%2C+Biderman%2C+S%2E%2C+and+Welleck%2C+S%2E+Llemma%3A+An+open+language+model+for+mathematics%2E+arXiv+preprint+arXiv%3A2310%2E1) [our card](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/)
7. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. [find](https://scholar.google.com/scholar?q=Hoffmann%2C+J%2E%2C+Borgeaud%2C+S%2E%2C+Mensch%2C+A%2E%2C+Buchatskaya%2C+E%2E%2C+Cai%2C+T%2E%2C+Rutherford%2C+E%2E%2C+Casas%2C+D%2E+d%2E+L%2E%2C+Hendricks%2C+L%2E+A%2E%2C+Welbl%2C+J%2E%2C+Clark%2C+A%2E%2C+et+al%2E+Training+compute%2Doptimal+large+language+models%2E+arXiv+)
8. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023a. [find](https://scholar.google.com/scholar?q=Bai%2C+J%2E%2C+Bai%2C+S%2E%2C+Chu%2C+Y%2E%2C+Cui%2C+Z%2E%2C+Dang%2C+K%2E%2C+Deng%2C+X%2E%2C+Fan%2C+Y%2E%2C+Ge%2C+W%2E%2C+Han%2C+Y%2E%2C+Huang%2C+F%2E%2C+et+al%2E+Qwen+technical+report%2E+arXiv+preprint+arXiv%3A2309%2E16609%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
9. Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, L., Rajbhandari, S., and He, Y. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509, 2023. [find](https://scholar.google.com/scholar?q=Jacobs%2C+S%2E+A%2E%2C+Tanaka%2C+M%2E%2C+Zhang%2C+C%2E%2C+Zhang%2C+M%2E%2C+Song%2C+L%2E%2C+Rajbhandari%2C+S%2E%2C+and+He%2C+Y%2E+Deepspeed+ulysses%3A+System+optimizations+for+enabling+training+of+extreme+long+sequence+transformer+models%2E+arXiv+) [our card](https://md.research.elamur.ai/free/en/papers/jacobs-2023-deepspeed-ulysses-system-optimizations-enabling/)
10. Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023b. [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+Lv%2C+X%2E%2C+Zhang%2C+J%2E%2C+Lyu%2C+H%2E%2C+Tang%2C+J%2E%2C+Huang%2C+Z%2E%2C+Du%2C+Z%2E%2C+Liu%2C+X%2E%2C+Zeng%2C+A%2E%2C+Hou%2C+L%2E%2C+et+al%2E+Longbench%3A+A+bilingual%2C+multitask+benchmark+for+long+context+understanding%2E+arXiv+preprint+arXiv%3A23) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/)
11. Bairi, R., Sonwane, A., Kanade, A., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B., Shet, S., et al. Codeplan: Repository-level coding using llms and planning. arXiv preprint arXiv:2309.12499, 2023. — [arXiv:2309.12499](https://arxiv.org/abs/2309.12499) [find](https://scholar.google.com/scholar?q=Bairi%2C+R%2E%2C+Sonwane%2C+A%2E%2C+Kanade%2C+A%2E%2C+Iyer%2C+A%2E%2C+Parthasarathy%2C+S%2E%2C+Rajamani%2C+S%2E%2C+Ashok%2C+B%2E%2C+Shet%2C+S%2E%2C+et+al%2E+Codeplan%3A+Repository%2Dlevel+coding+using+llms+and+planning%2E+arXiv+preprint+arXiv%3A2309%2E12499%2C+2)
12. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023%2E)
13. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+informati)
14. Kamradt, G. Needle in a haystack - pressure testing llms. [https://github.com/gkamradt/LLMTest](https://github.com/gkamradt/LLMTest) NeedleInAHaystack, 2023. [↗](https://github.com/gkamradt/LLMTest)
15. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. [find](https://scholar.google.com/scholar?q=Kaplan%2C+J%2E%2C+McCandlish%2C+S%2E%2C+Henighan%2C+T%2E%2C+Brown%2C+T%2E+B%2E%2C+Chess%2C+B%2E%2C+Child%2C+R%2E%2C+Gray%2C+S%2E%2C+Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+and+Amodei%2C+D%2E+Scaling+laws+for+neural+language+models%2E+arXiv+preprint+arXiv%3A2001%2E08361%2C+20)
16. Caciularu, A., Peters, M. E., Goldberger, J., Dagan, I., and Cohan, A. Peek across: Improving multi-document modeling via cross-document question-answering. arXiv preprint arXiv:2305.15387, 2023. [find](https://scholar.google.com/scholar?q=Caciularu%2C+A%2E%2C+Peters%2C+M%2E+E%2E%2C+Goldberger%2C+J%2E%2C+Dagan%2C+I%2E%2C+and+Cohan%2C+A%2E+Peek+across%3A+Improving+multi%2Ddocument+modeling+via+cross%2Ddocument+question%2Danswering%2E+arXiv+preprint+arXiv%3A2305%2E15387%2C+2023%2E)
17. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [find](https://scholar.google.com/scholar?q=Chen%2C+M%2E%2C+Tworek%2C+J%2E%2C+Jun%2C+H%2E%2C+Yuan%2C+Q%2E%2C+Pinto%2C+H%2E+P%2E+d%2E+O%2E%2C+Kaplan%2C+J%2E%2C+Edwards%2C+H%2E%2C+Burda%2C+Y%2E%2C+Joseph%2C+N%2E%2C+Brockman%2C+G%2E%2C+et+al%2E+Evaluating+large+language+models+trained+on+code%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
18. Li, D., Shao, R., Xie, A., Sheng, Y., Zheng, L., Gonzalez, J. E., Stoica, I., Ma, X., and Zhang, H. How long can open-source llms truly promise on context length?, June 2023a. URL [https://lmsys.org/blog/](https://lmsys.org/blog/) 2023-06-29-longchat. [↗](https://lmsys.org/blog/)
19. Chen, S., Wong, S., Chen, L., and Tian, Y. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023a. [find](https://scholar.google.com/scholar?q=Chen%2C+S%2E%2C+Wong%2C+S%2E%2C+Chen%2C+L%2E%2C+and+Tian%2C+Y%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2C+2023a%2E)
20. Li, S., Xue, F., Baranwal, C., Li, Y., and You, Y. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120, 2021. [find](https://scholar.google.com/scholar?q=Li%2C+S%2E%2C+Xue%2C+F%2E%2C+Baranwal%2C+C%2E%2C+Li%2C+Y%2E%2C+and+You%2C+Y%2E+Sequence+parallelism%3A+Long+sequence+training+from+system+perspective%2E+arXiv+preprint+arXiv%3A2105%2E13120%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-sequence-parallelism-long-sequence-training/)
21. Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307, 2023b. [find](https://scholar.google.com/scholar?q=Chen%2C+Y%2E%2C+Qian%2C+S%2E%2C+Tang%2C+H%2E%2C+Lai%2C+X%2E%2C+Liu%2C+Z%2E%2C+Han%2C+S%2E%2C+and+Jia%2C+J%2E+Longlora%3A+Efficient+fine%2Dtuning+of+long%2Dcontext+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E12307%2C+2023b%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/)
22. Li, S., You, C., Guruganesh, G., Ainslie, J., Ontanon, S., Zaheer, M., Sanghai, S., Yang, Y., Kumar, S., and Bhojanapalli, S. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418, 2023b. [find](https://scholar.google.com/scholar?q=Li%2C+S%2E%2C+You%2C+C%2E%2C+Guruganesh%2C+G%2E%2C+Ainslie%2C+J%2E%2C+Ontanon%2C+S%2E%2C+Zaheer%2C+M%2E%2C+Sanghai%2C+S%2E%2C+Yang%2C+Y%2E%2C+Kumar%2C+S%2E%2C+and+Bhojanapalli%2C+S%2E+Functional+interpolation+for+relative+positions+improves+long+context+tran)
23. Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E+Flashattention%2D2%3A+Faster+attention+with+better+parallelism+and+work+partitioning%2E+arXiv+preprint+arXiv%3A2307%2E08691%2C+2023%2E)
24. Mazumder, S. and Liu, B. Lifelong and Continual Learning Dialogue Systems. Springer Nature, 2024. [find](https://scholar.google.com/scholar?q=Mazumder%2C+S%2E+and+Liu%2C+B%2E+Lifelong+and+Continual+Learning+Dialogue+Systems%2E+Springer+Nature%2C+2024%2E)
25. Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models, 2023. [find](https://scholar.google.com/scholar?q=Peng%2C+B%2E%2C+Quesnelle%2C+J%2E%2C+Fan%2C+H%2E%2C+and+Shippole%2C+E%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2C+2023%2E)
26. Together. Llama-2-7b-32k-instruct — and fine-tuning for llama-2 models with together api, August 2023. URL [https://www.together.ai/blog/](https://www.together.ai/blog/) llama-2-7b-32k-instruct. [↗](https://www.together.ai/blog/)
27. Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. [find](https://scholar.google.com/scholar?q=Press%2C+O%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Lewis%2C+M%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+arXiv+preprint+arXiv%3A2108%2E12409%2C+2021%2E)
28. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Lavril%2C+T%2E%2C+Izacard%2C+G%2E%2C+Martinet%2C+X%2E%2C+Lachaux%2C+M%2E%2DA%2E%2C+Lacroix%2C+T%2E%2C+Rozie%CC%80re%2C+B%2E%2C+Goyal%2C+N%2E%2C+Hambro%2C+E%2E%2C+Azhar%2C+F%2E%2C+et+al%2E+Llama%3A+Open+and+efficient+foundation+language+models%2E+arXiv+prep)
29. Qin, Z., Yang, S., and Zhong, Y. Hierarchically gated recurrent neural network for sequence modeling. CoRR, abs/2311.04823, 2023. doi: 10.48550/ARXIV. 2311.04823. URL https://doi.org/10.48550/arXiv. 2311.04823. [↗](https://doi.org/10.48550/arXiv)
30. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+N%2E%2C+Batra%2C+S%2E%2C+Bhargava%2C+P%2E%2C+Bhosale%2C+S%2E%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+prepri)
31. Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. arXiv preprint, 2019. URL https://arxiv.org/abs/1911.05507. [↗](https://arxiv.org/abs/1911.05507) [our card](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/)
32. Weng, L. Llm-powered autonomous agents. lilian-weng.github.io, Jun 2023. URL [https://lilianweng.github.io/posts/2023-06-23-agent/](https://lilianweng.github.io/posts/2023-06-23-agent/). [↗](https://lilianweng.github.io/posts/2023-06-23-agent/)
33. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. [find](https://scholar.google.com/scholar?q=Raffel%2C+C%2E%2C+Shazeer%2C+N%2E%2C+Roberts%2C+A%2E%2C+Lee%2C+K%2E%2C+Narang%2C+S%2E%2C+Matena%2C+M%2E%2C+Zhou%2C+Y%2E%2C+Li%2C+W%2E%2C+and+Liu%2C+P%2E+J%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+transformer%2E+The+Journal+o)
34. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Tian%2C+Y%2E%2C+Chen%2C+B%2E%2C+Han%2C+S%2E%2C+and+Lewis%2C+M%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2C+2023%2E)
35. Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039, 2023. [find](https://scholar.google.com/scholar?q=Xiong%2C+W%2E%2C+Liu%2C+J%2E%2C+Molybog%2C+I%2E%2C+Zhang%2C+H%2E%2C+Bhargava%2C+P%2E%2C+Hou%2C+R%2E%2C+Martin%2C+L%2E%2C+Rungta%2C+R%2E%2C+Sankararaman%2C+K%2E+A%2E%2C+Oguz%2C+B%2E%2C+et+al%2E+Effective+long%2Dcontext+scaling+of+foundation+models%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
36. Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–16. IEEE, 2020. [find](https://scholar.google.com/scholar?q=Rajbhandari%2C+S%2E%2C+Rasley%2C+J%2E%2C+Ruwase%2C+O%2E%2C+and+He%2C+Y%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Computing%2C+Networking%2C+)
37. XVerse. Xverse-13b, January 2024. URL [https://github.com/xverse-ai/XVERSE-13B/](https://github.com/xverse-ai/XVERSE-13B/) blob/main/README EN.md. [↗](https://github.com/xverse-ai/XVERSE-13B/)
38. Shaham, U., Ivgi, M., Efrat, A., Berant, J., and Levy, O. Zeroscrolls: A zero-shot benchmark for long text understanding, 2023. [find](https://scholar.google.com/scholar?q=Shaham%2C+U%2E%2C+Ivgi%2C+M%2E%2C+Efrat%2C+A%2E%2C+Berant%2C+J%2E%2C+and+Levy%2C+O%2E+Zeroscrolls%3A+A+zero%2Dshot+benchmark+for+long+text+understanding%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/shaham-2023-zeroscrolls-zero-shot-benchmark-long/)
39. Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023a. [find](https://scholar.google.com/scholar?q=Yang%2C+A%2E%2C+Xiao%2C+B%2E%2C+Wang%2C+B%2E%2C+Zhang%2C+B%2E%2C+Bian%2C+C%2E%2C+Yin%2C+C%2E%2C+Lv%2C+C%2E%2C+Pan%2C+D%2E%2C+Wang%2C+D%2E%2C+Yan%2C+D%2E%2C+et+al%2E+Baichuan+2%3A+Open+large%2Dscale+language+models%2E+arXiv+preprint+arXiv%3A2309%2E10305%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/)
40. Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated 627B token cleaned and deduplicated version of linear attention transformers with hardware-efficient train-RedPajama. [https://www.cerebras.net/b…](https://www.cerebras.net/b…) [↗](https://www.cerebras.net/b…)
41. Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023. [find](https://scholar.google.com/scholar?q=Sun%2C+Y%2E%2C+Dong%2C+L%2E%2C+Huang%2C+S%2E%2C+Ma%2C+S%2E%2C+Xia%2C+Y%2E%2C+Xue%2C+J%2E%2C+Wang%2C+J%2E%2C+and+Wei%2C+F%2E+Retentive+network%3A+A+successor+to+transformer+for+large+language+models%2E+arXiv+preprint+arXiv%3A2307%2E08621%2C+2023%2E)
42. Team, M. N. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL www. mosaicml.com/blog/mpt-7b. Accessed: 2023-03-28. [find](https://scholar.google.com/scholar?q=Team%2C+M%2E+N%2E+Introducing+mpt%2D7b%3A+A+new+standard+for+open%2Dsource%2C+commercially+usable+llms%2C+2023%2E+URL+www%2E+mosaicml%2Ecom%2Fblog%2Fmpt%2D7b%2E+Accessed%3A+2023%2D03%2D28%2E)