---
title: "Dataset Decomposition — paper by Pouransari et al. (2024)"
description: "The Apple authors propose dataset decomposition (DD) to address these problems. 67 references, cited by 12 papers here."
authors: ["Hadi Pouransari", "Chun-Liang Li", "Jen-Hao Rick Chang", "Pavan Kumar Anasosalu Vasu", "Cem Koc", "Vaishaal Shankar", "Oncel Tuzel"]
year: 2024
venue: "arXiv:2405.13226"
source: "https://arxiv.org/abs/2405.13226"
doi: "10.48550/arxiv.2405.13226"
arxiv: "2405.13226"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pouransari-2024-dataset-decomposition-faster-llm-training/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pouransari-2024-dataset-decomposition-faster-llm-training/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum

# Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum

**Authors:** Hadi Pouransari, [Chun-Liang Li](https://md.research.elamur.ai/free/en/authors/chun-liang-li/), Jen-Hao Rick Chang, Pavan Kumar Anasosalu Vasu, Cem Koc, [Vaishaal Shankar](https://md.research.elamur.ai/free/en/authors/vaishaal-shankar/) and [Oncel Tuzel](https://md.research.elamur.ai/free/en/authors/oncel-tuzel/) · **Year:** 2024 · **Venue:** [arXiv:2405.13226](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.13226) · [PDF](https://arxiv.org/pdf/2405.13226) · [DOI 10.48550/arxiv.2405.13226](https://doi.org/10.48550/arxiv.2405.13226) · [arXiv 2405.13226](https://arxiv.org/abs/2405.13226) · [OpenAlex](https://openalex.org/W4415798226)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/pouransari-2024-dataset-decomposition-faster-llm-training/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard LLM pretraining uses concat-and-chunk: documents are shuffled, joined, and cut into fixed-length sequences. This lets tokens attend to unrelated documents. It also wastes quadratic attention compute and splits many short documents at chunk boundaries. The Apple authors propose dataset decomposition (DD) to address these problems. Each tokenized document is split according to the binary representation of its length into adjacent pieces of size 2^i. Each piece goes into bucket D_i, so every sequence comes from a single document. Training then uses variable sequence length (VSL): each step samples a bucket under a length-based curriculum and adjusts the batch size so the token count per batch stays constant. Short-sequence steps are cheaper, so overall cost tracks real document lengths rather than the maximum context. Experiments on RefinedWeb show DD reaches the baseline's accuracy up to 6× faster and improves both regular and long-context benchmarks. The work also shows that the distribution and ordering of sequence lengths during pretraining matter noticeably, and DD scales easily to web-sized corpora.

## Key points
- When scaling OpenLM-410M on RefinedWeb to 1.1 trillion tokens with the Grow-P2 curriculum (8 cycles, 8192 max context), DD is more than 4× more data-efficient and still leads by +2.4 average accuracy at the largest run.
- Compute savings per step range from 11% to 45%, and an 8k-context 1B model costs about as much as a 2k-context baseline.
- At 103B training tokens, DD outperforms Baseline-8k, Baseline-8k+DM, Pack-8k+DM, and ICLM:
  - regular average: 54.4 vs. 51.5 for Baseline-8k;
  - long-context average (MDQA, TOEFL, QuALITY): 30.9 vs. 27.5;
  - step time: 244 ms vs. 304 ms.
- DD's data preparation is cheap. By contrast, ICLM's similarity-based document ordering took over a week to process 400B tokens on 96 CPUs and 16 GPUs.
- The speed advantage only appears when the target context is long enough for attention to dominate training cost. Code is planned at github.com/apple/ml-dataset-decomposition (NeurIPS 2024).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Hadi Pouransari, Chun-Liang Li, Jen-Hao Rick Chang, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar and Oncel Tuzel. 2024. Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum. arXiv:2405.13226.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{pouransari2024dataset,
  title = {{Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum}},
  author = {Hadi Pouransari, Chun-Liang Li, Jen-Hao Rick Chang, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar and Oncel Tuzel},
  year = {2024},
  doi = {10.48550/arxiv.2405.13226},
  eprint = {2405.13226},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.13226}
}
```

- [← previous Bridging Cultures in the Kitchen: A Framework and Benchmark for Cross-Cultural Recipe Retrieval](https://md.research.elamur.ai/free/en/papers/hu-2024-bridging-cultures-kitchen-framework-benchmark/)
- [next → Discursive Socratic Questioning: Evaluating the Faithfulness of Language Models' Understanding of Discourse Relations](https://md.research.elamur.ai/free/en/papers/miao-2024-discursive-socratic-questioning-evaluating-faithfulness/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

67 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Common crawl. [https://commoncrawl.org](https://commoncrawl.org). [↗](https://commoncrawl.org)
2. Llm foundry v0.7.0. [https://github.com/mosaicml/llm-foundry](https://github.com/mosaicml/llm-foundry). [↗](https://github.com/mosaicml/llm-foundry)
3. Jeopardy. [https://huggingface.co/datasets/jeopardy](https://huggingface.co/datasets/jeopardy). [Used custom curated version by LLM Foundry]. [↗](https://huggingface.co/datasets/jeopardy)
4. Big-bench qa wikidata. [https://github.com/google/BIG-bench/tree/main/](https://github.com/google/BIG-bench/tree/main/) bigbench/benchmark_tasks/qa_wikidata. [Used through LLM Foundry]. [↗](https://github.com/google/BIG-bench/tree/main/)
5. C. An, S. Gong, M. Zhong, M. Li, J. Zhang, L. Kong, and X. Qiu. [L-eval: Instituting standardized evaluation for long context language models](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/), 2023.
6. C. Anil, Y. Wu, A. Andreassen, A. Lewkowycz, V. Misra, V. Ramasesh, A. Slone, G. Gur-Ari, E. Dyer, and B. Neyshabur. [Exploring length generalization in large language models](https://md.research.elamur.ai/free/en/papers/anil-2022-exploring-length-generalization-large-language/). Advances in Neural Information Processing Systems, 35:38546–38556, 2022.
7. Y. Bengio, J. Louradour, R. Collobert, and J. Weston. [Curriculum learning](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/). In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009.
8. Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020.
9. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al. [Gpt-neox-20b: An open-source autoregressive language model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/). arXiv preprint arXiv:2204.06745, 2022.
10. S. Chen, S. Wong, L. Chen, and Y. Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Chen%2C+S%2E+Wong%2C+L%2E+Chen%2C+and+Y%2E+Tian%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2C+2023%2E)
11. Y.-A. Chung, H.-Y. Lee, and J. Glass. Supervised and unsupervised transfer learning for question answering. In M. Walker, H. Ji, and A. Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volu… [find](https://scholar.google.com/scholar?q=Y%2E%2DA%2E+Chung%2C+H%2E%2DY%2E+Lee%2C+and+J%2E+Glass%2E+Supervised+and+unsupervised+transfer+learning+for+question+answering%2E+In+M%2E+Walker%2C+H%2E+Ji%2C+and+A%2E+Stent%2C+editors%2C+Proceedings+of+the+2018+Conference+of+the+North+)
12. C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044, 2019.
13. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457, 2018.
14. T. Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023. URL [https://github.com/togethercomputer/RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data). [↗](https://github.com/togethercomputer/RedPajama-Data)
15. T. Dao. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). arXiv preprint arXiv:2307.08691, 2023.
16. M. Dehghani, B. Mustafa, J. Djolonga, J. Heek, M. Minderer, M. Caron, A. Steiner, J. Puigcerver, R. Geirhos, I. M. Alabdulmohsin, et al. Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Dehghani%2C+B%2E+Mustafa%2C+J%2E+Djolonga%2C+J%2E+Heek%2C+M%2E+Minderer%2C+M%2E+Caron%2C+A%2E+Steiner%2C+J%2E+Puigcerver%2C+R%2E+Geirhos%2C+I%2E+M%2E+Alabdulmohsin%2C+et+al%2E+Patch+n%E2%80%99pack%3A+Navit%2C+a+vision+transformer+for+any+aspect+ratio+)
17. H. Ding, Z. Wang, G. Paolini, V. Kumar, A. Deoras, D. Roth, and S. Soatto. Fewer truncations improve language modeling. arXiv preprint arXiv:2404.10830, 2024. — [arXiv:2404.10830](https://arxiv.org/abs/2404.10830) [find](https://scholar.google.com/scholar?q=H%2E+Ding%2C+Z%2E+Wang%2C+G%2E+Paolini%2C+V%2E+Kumar%2C+A%2E+Deoras%2C+D%2E+Roth%2C+and+S%2E+Soatto%2E+Fewer+truncations+improve+language+modeling%2E+arXiv+preprint+arXiv%3A2404%2E10830%2C+2024%2E+%E2%80%94+%5BarXiv%3A2404%2E10830%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
18. J. L. Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71–99, 1993. [find](https://scholar.google.com/scholar?q=J%2E+L%2E+Elman%2E+Learning+and+development+in+neural+networks%3A+The+importance+of+starting+small%2E+Cognition%2C+48%281%29%3A71%E2%80%9399%2C+1993%2E)
19. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+S%2E+Biderman%2C+S%2E+Black%2C+L%2E+Golding%2C+T%2E+Hoppe%2C+C%2E+Foster%2C+J%2E+Phang%2C+H%2E+He%2C+A%2E+Thite%2C+N%2E+Nabeshima%2C+et+al%2E+The+Pile%3A+An+800GB+dataset+of+diverse+text+for+language+modeling%2E+arXiv+preprint+arXiv%3A2)
20. Z. Ge, L. Kaushik, M. Omote, and S. Kumar. Speed up training with variable length inputs by efficient batching strategies. In Interspeech, pages 156–160, 2021. [find](https://scholar.google.com/scholar?q=Z%2E+Ge%2C+L%2E+Kaushik%2C+M%2E+Omote%2C+and+S%2E+Kumar%2E+Speed+up+training+with+variable+length+inputs+by+efficient+batching+strategies%2E+In+Interspeech%2C+pages+156%E2%80%93160%2C+2021%2E)
21. P. Gonzalez, T. S. Alstrøm, and T. May. On batching variable size inputs for training end-to-end speech enhancement systems. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. [find](https://scholar.google.com/scholar?q=P%2E+Gonzalez%2C+T%2E+S%2E+Alstr%C3%B8m%2C+and+T%2E+May%2E+On+batching+variable+size+inputs+for+training+end%2Dto%2Dend+speech+enhancement+systems%2E+In+ICASSP+2023%2D2023+IEEE+International+Conference+on+Acoustics%2C+Speech+and+)
22. S. Gururangan, M. Wortsman, S. Y. Gadre, A. Dave, M. Kilian, W. Shi, J. Mercat, G. Smyrnis, G. Ilharco, M. Jordan, R. Heckel, A. Dimakis, A. Farhadi, V. Shankar, and L. Schmidt. OpenLM: a minimal but performative language modeling (lm) repository, 2023. URL [https://github.com/mlfoundations/open_lm/.…](https://github.com/mlfoundations/open_lm/.…) [↗](https://github.com/mlfoundations/open_lm/.…)
23. C. Han, Q. Wang, W. Xiong, Y. Chen, H. Ji, and S. Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023. — [arXiv:2308.16137](https://arxiv.org/abs/2308.16137) [find](https://scholar.google.com/scholar?q=C%2E+Han%2C+Q%2E+Wang%2C+W%2E+Xiong%2C+Y%2E+Chen%2C+H%2E+Ji%2C+and+S%2E+Wang%2E+Lm%2Dinfinite%3A+Simple+on%2Dthe%2Dfly+length+generalization+for+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E16137%2C+2023%2E+%E2%80%94+%5BarXiv%3A2308%2E16137%5D%28http)
24. H. Jin, X. Han, J. Yang, Z. Jiang, C.-Y. Chang, and X. Hu. Growlength: Accelerating llms pretraining by progressively growing training length. arXiv preprint arXiv:2310.00576, 2023. — [arXiv:2310.00576](https://arxiv.org/abs/2310.00576) [find](https://scholar.google.com/scholar?q=H%2E+Jin%2C+X%2E+Han%2C+J%2E+Yang%2C+Z%2E+Jiang%2C+C%2E%2DY%2E+Chang%2C+and+X%2E+Hu%2E+Growlength%3A+Accelerating+llms+pretraining+by+progressively+growing+training+length%2E+arXiv+preprint+arXiv%3A2310%2E00576%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E0057)
25. A. Kazemnejad, I. Padhi, K. Natesan Ramamurthy, P. Das, and S. Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Kazemnejad%2C+I%2E+Padhi%2C+K%2E+Natesan+Ramamurthy%2C+P%2E+Das%2C+and+S%2E+Reddy%2E+The+impact+of+positional+encoding+on+length+generalization+in+transformers%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C)
26. M. M. Krell, M. Kosec, S. P. Perez, and A. Fitzgibbon. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.02027, 2021. — [arXiv:2107.02027](https://arxiv.org/abs/2107.02027) [find](https://scholar.google.com/scholar?q=M%2E+M%2E+Krell%2C+M%2E+Kosec%2C+S%2E+P%2E+Perez%2C+and+A%2E+Fitzgibbon%2E+Efficient+sequence+packing+without+cross%2Dcontamination%3A+Accelerating+large+language+models+without+impacting+performance%2E+arXiv+preprint+arXiv%3A21)
27. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 7:453–466, 2019.
28. K. Lee, M.-W. Chang, and K. Toutanova. [Latent retrieval for weakly supervised open domain question answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/). arXiv preprint arXiv:1906.00300, 2019.
29. B. Lefaudeux, F. Massa, D. Liskovich, W. Xiong, V. Caggiano, S. Naren, M. Xu, J. Hu, M. Tintore, S. Zhang, P. Labatut, D. Haziza, L. Wehrstedt, J. Reizenstein, and G. Sizov. xformers: A modular and hackable transformer modelling library. [https://github.com/](https://github.com/) facebookresearch/xformers, 2022. [↗](https://github.com/)
30. H. Levesque, E. Davis, and L. Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012. [find](https://scholar.google.com/scholar?q=H%2E+Levesque%2C+E%2E+Davis%2C+and+L%2E+Morgenstern%2E+The+winograd+schema+challenge%2E+In+Thirteenth+international+conference+on+the+principles+of+knowledge+representation+and+reasoning%2C+2012%2E)
31. C. Li, M. Zhang, and Y. He. The stability-efficiency dilemma: Investigating sequence length warmup for training gpt models. Advances in Neural Information Processing Systems, 35: 26736–26750, 2022. [find](https://scholar.google.com/scholar?q=C%2E+Li%2C+M%2E+Zhang%2C+and+Y%2E+He%2E+The+stability%2Defficiency+dilemma%3A+Investigating+sequence+length+warmup+for+training+gpt+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A+26736%E2%80%9326750%2C+2022%2E)
32. J. Li, A. Fang, G. Smyrnis, M. Ivgi, M. Jordan, S. Gadre, H. Bansal, E. Guha, S. Keh, K. Arora, et al. [Datacomp-lm: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/). arXiv preprint arXiv:2406.11794, 2024.
33. X. Li, Z. Wang, and C. Xie. An inverse scaling law for clip training. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Li%2C+Z%2E+Wang%2C+and+C%2E+Xie%2E+An+inverse+scaling+law+for+clip+training%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
34. H. Liu, W. Yan, M. Zaharia, and P. Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.08268, 2024. [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+W%2E+Yan%2C+M%2E+Zaharia%2C+and+P%2E+Abbeel%2E+World+model+on+million%2Dlength+video+and+language+with+blockwise+ringattention%2E+arXiv+preprint+arXiv%3A2402%2E08268%2C+2024%2E)
35. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173, 2024.
36. X. Liu, H. Yan, S. Zhang, C. An, X. Qiu, and D. Lin. [Scaling laws of rope-based extrapolation](https://md.research.elamur.ai/free/en/papers/liu-2023-scaling-laws-rope-based/). arXiv preprint arXiv:2310.05209, 2023.
37. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692, 2019.
38. S. Mehta, F. Abdolhosseini, and M. Rastegari. Cvnets: High performance library for computer vision. In Proceedings of the 30th ACM International Conference on Multimedia, pages 7327–7330, 2022. [find](https://scholar.google.com/scholar?q=S%2E+Mehta%2C+F%2E+Abdolhosseini%2C+and+M%2E+Rastegari%2E+Cvnets%3A+High+performance+library+for+computer+vision%2E+In+Proceedings+of+the+30th+ACM+International+Conference+on+Multimedia%2C+pages+7327%E2%80%937330%2C+2022%2E)
39. Meta. Introducing meta llama 3: The most capable openly available llm to date, 2024. URL [https://ai.meta.com/blog/meta-llama-3](https://ai.meta.com/blog/meta-llama-3). [↗](https://ai.meta.com/blog/meta-llama-3)
40. T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. [find](https://scholar.google.com/scholar?q=T%2E+Mihaylov%2C+P%2E+Clark%2C+T%2E+Khot%2C+and+A%2E+Sabharwal%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E02789%2C+2018%2E)
41. A. Mohtashami and M. Jaggi. Random-access infinite context length for transformers. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Mohtashami+and+M%2E+Jaggi%2E+Random%2Daccess+infinite+context+length+for+transformers%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
42. K. Nagatsuka, C. Broni-Bediako, and M. Atsumi. Pre-training a bert with curriculum learning by increasing block-size of input text. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 989–996, 2021. [find](https://scholar.google.com/scholar?q=K%2E+Nagatsuka%2C+C%2E+Broni%2DBediako%2C+and+M%2E+Atsumi%2E+Pre%2Dtraining+a+bert+with+curriculum+learning+by+increasing+block%2Dsize+of+input+text%2E+In+Proceedings+of+the+International+Conference+on+Recent+Advances+in)
43. M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli. [fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
44. D. Paperno, G. Kruszewski, A. Lazaridou, Q. N. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández. [The lambada dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). arXiv preprint arXiv:1606.06031, 2016.
45. S. Pawar, S. Tonmoy, S. Zaman, V. Jain, A. Chadha, and A. Das. The what, why, and how of context length extension techniques in large language models–a detailed survey. arXiv preprint arXiv:2401.07872, 2024. — [arXiv:2401.07872](https://arxiv.org/abs/2401.07872) [find](https://scholar.google.com/scholar?q=S%2E+Pawar%2C+S%2E+Tonmoy%2C+S%2E+Zaman%2C+V%2E+Jain%2C+A%2E+Chadha%2C+and+A%2E+Das%2E+The+what%2C+why%2C+and+how+of+context+length+extension+techniques+in+large+language+models%E2%80%93a+detailed+survey%2E+arXiv+preprint+arXiv%3A2401%2E07872)
46. G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, and J. Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023. URL https://arxiv.org/abs/2306.01116. [↗](https://arxiv.org/abs/2306.01116)
47. B. Peng and J. Quesnelle. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. 2023. URL [https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/](https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/) ntkaware_scaled_rope_allows_llama_models_to_have. [↗](https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/)
48. B. Peng, J. Quesnelle, H. Fan, and E. Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. [find](https://scholar.google.com/scholar?q=B%2E+Peng%2C+J%2E+Quesnelle%2C+H%2E+Fan%2C+and+E%2E+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2C+2023%2E)
49. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8763. PMLR, 2021.
50. P. Rajpurkar, R. Jia, and P. Liang. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.[03822, 2018](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/).
51. S. Reddy, D. Chen, and C. D. Manning. [Coqa: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266, 2019.
52. M. Roemmele, C. A. Bejan, and A. S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011. [find](https://scholar.google.com/scholar?q=M%2E+Roemmele%2C+C%2E+A%2E+Bejan%2C+and+A%2E+S%2E+Gordon%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+2011+AAAI+Spring+Symposium+Series%2C+2011%2E)
53. A. Ruoss, G. Delétang, T. Genewein, J. Grau-Moya, R. Csordás, M. Bennani, S. Legg, and J. Veness. Randomized positional encodings boost length generalization of transformers. arXiv preprint arXiv:2305.16843, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Ruoss%2C+G%2E+Del%C3%A9tang%2C+T%2E+Genewein%2C+J%2E+Grau%2DMoya%2C+R%2E+Csord%C3%A1s%2C+M%2E+Bennani%2C+S%2E+Legg%2C+and+J%2E+Veness%2E+Randomized+positional+encodings+boost+length+generalization+of+transformers%2E+arXiv+preprint+arXiv%3A2305)
54. K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [find](https://scholar.google.com/scholar?q=K%2E+Sakaguchi%2C+R%2E+L%2E+Bras%2C+C%2E+Bhagavatula%2C+and+Y%2E+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2C+2021%2E)
55. W. Shi, S. Min, M. Lomeli, C. Zhou, M. Li, V. Lin, N. A. Smith, L. Zettlemoyer, S. Yih, and M. Lewis. [In-context pretraining: Language modeling beyond document boundaries](https://md.research.elamur.ai/free/en/papers/shi-2023-context-pretraining-language-modeling-document/). arXiv preprint arXiv:2310.10638, 2023.
56. L. N. Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pages 464–472. IEEE, 2017. [find](https://scholar.google.com/scholar?q=L%2E+N%2E+Smith%2E+Cyclical+learning+rates+for+training+neural+networks%2E+In+2017+IEEE+winter+conference+on+applications+of+computer+vision+%28WACV%29%2C+pages+464%E2%80%93472%2E+IEEE%2C+2017%2E)
57. L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y. Elazar, V. Hofmann, A. H. Jha, S. Kumar, L. Lucy, X. Lyu, N. Lambert, I. Magnusson, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. E. Peters, A. Ravichander, K. Richardson, Z. Shen, E. Strubell… [find](https://scholar.google.com/scholar?q=L%2E+Soldaini%2C+R%2E+Kinney%2C+A%2E+Bhagia%2C+D%2E+Schwenk%2C+D%2E+Atkinson%2C+R%2E+Authur%2C+B%2E+Bogin%2C+K%2E+Chandu%2C+J%2E+Dumas%2C+Y%2E+Elazar%2C+V%2E+Hofmann%2C+A%2E+H%2E+Jha%2C+S%2E+Kumar%2C+L%2E+Lucy%2C+X%2E+Lyu%2C+N%2E+Lambert%2C+I%2E+Magnusson%2C+J%2E+Morrison)
58. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063, 2024.
59. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
60. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
61. B.-H. Tseng, S.-S. Shen, H.-Y. Lee, and L.-S. Lee. Towards machine comprehension of spoken content: Initial toefl listening comprehension test by machine, 2016. [find](https://scholar.google.com/scholar?q=B%2E%2DH%2E+Tseng%2C+S%2E%2DS%2E+Shen%2C+H%2E%2DY%2E+Lee%2C+and+L%2E%2DS%2E+Lee%2E+Towards+machine+comprehension+of+spoken+content%3A+Initial+toefl+listening+comprehension+test+by+machine%2C+2016%2E)
62. D. Variš and O. Bojar. Sequence length is a domain: Length-based overfitting in transformer models. arXiv preprint arXiv:2109.07276, 2021. [find](https://scholar.google.com/scholar?q=D%2E+Vari%C5%A1+and+O%2E+Bojar%2E+Sequence+length+is+a+domain%3A+Length%2Dbased+overfitting+in+transformer+models%2E+arXiv+preprint+arXiv%3A2109%2E07276%2C+2021%2E)
63. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
64. W. Xiong, J. Liu, I. Molybog, H. Zhang, P. Bhargava, R. Hou, L. Martin, R. Rungta, K. A. Sankararaman, B. Oguz, et al. [Effective long-context scaling of foundation models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/). arXiv preprint arXiv:2309.16039, 2023.
65. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=R%2E+Zellers%2C+A%2E+Holtzman%2C+Y%2E+Bisk%2C+A%2E+Farhadi%2C+and+Y%2E+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
66. Y. Zhou, U. Alon, X. Chen, X. Wang, R. Agarwal, and D. Zhou. Transformers can achieve length generalization but not robustly. arXiv preprint arXiv:2402.09371, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Zhou%2C+U%2E+Alon%2C+X%2E+Chen%2C+X%2E+Wang%2C+R%2E+Agarwal%2C+and+D%2E+Zhou%2E+Transformers+can+achieve+length+generalization+but+not+robustly%2E+arXiv+preprint+arXiv%3A2402%2E09371%2C+2024%2E)
67. D. Zhu, N. Yang, L. Wang, Y. Song, W. Wu, F. Wei, and S. Li. [Pose: Efficient context window extension of llms via positional skip-wise training](https://md.research.elamur.ai/free/en/papers/zhu-2023-pose-efficient-context-window/). arXiv preprint arXiv:2309.10400, 2023.

*Updated 30 October 2026*