---
title: "LongWriter — paper by Bai et al. (2024) — summary and ideas"
description: "Long-context LLMs can read inputs of 100,000+ tokens, yet their answers rarely go past about 2,000… 51 references, cited by 61 papers here."
authors: ["Yushi Bai", "Jiajie Zhang", "Xin Lv", "Linzhi Zheng", "Siqi Zhu", "Lei Hou", "Yuxiao Dong", "Jie Tang", "Juanzi Li"]
year: 2024
venue: "arXiv:2408.07055"
source: "https://arxiv.org/abs/2408.07055"
doi: "10.48550/arxiv.2408.07055"
arxiv: "2408.07055"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bai-2024-longwriter-unleashing-10-000/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bai-2024-longwriter-unleashing-10-000/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs

# LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs

**Authors:** [Yushi Bai](https://md.research.elamur.ai/free/en/authors/yushi-bai/), [Jiajie Zhang](https://md.research.elamur.ai/free/en/authors/jiajie-zhang/), [Xin Lv](https://md.research.elamur.ai/free/en/authors/xin-lv/), Linzhi Zheng, Siqi Zhu, [Lei Hou](https://md.research.elamur.ai/free/en/authors/lei-hou/), [Yuxiao Dong](https://md.research.elamur.ai/free/en/authors/yuxiao-dong/), [Jie Tang](https://md.research.elamur.ai/free/en/authors/jie-tang/) and [Juanzi Li](https://md.research.elamur.ai/free/en/authors/juanzi-li/) · **Year:** 2024 · **Venue:** arXiv:2408.07055 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2408.07055) · [PDF](https://arxiv.org/pdf/2408.07055) · [DOI 10.48550/arxiv.2408.07055](https://doi.org/10.48550/arxiv.2408.07055) · [arXiv 2408.07055](https://arxiv.org/abs/2408.07055) · [OpenAlex](https://openalex.org/W4402427428)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [61](https://md.research.elamur.ai/free/en/papers/bai-2024-longwriter-unleashing-10-000/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Long-context LLMs can read inputs of 100,000+ tokens, yet their answers rarely go past about 2,000 words. The authors trace this ceiling to supervised fine-tuning: a model seldom writes longer than the longest responses in its SFT data, and existing SFT sets contain very few long answers. To fill this gap, they built AgentWrite, an agent pipeline that first has an off-the-shelf LLM draft an outline with a word budget per paragraph, then writes the paragraphs one after another and joins them. With GPT-4o as the backbone, this produced LongWriter-6k, a set of 6,000 SFT examples whose outputs span 2k–32k words. Adding this data to training, and then applying DPO, let a 9B model write coherent texts beyond 10,000 words. On the new LongBench-Write benchmark, this model outperformed larger proprietary systems. The broader point is that long-context models already have latent capacity for long output, and the missing piece is suitable alignment data.

## Key points
- **LongWrite-Ruler probe:** 8 seed instructions (4 English, 4 Chinese) were combined with target lengths from 1,000 to 30,000 words, giving 48 prompts. All 8 tested models (4 open-source, 4 proprietary) topped out near 2k words. WildChat logs show over 1% of user requests ask for more than that.
- **Controlled experiment:** GLM-4-9B was fine-tuned on a 180k chat SFT subset, with responses capped at 500, 1,000 or 2,000 words. The resulting maximum output lengths were roughly 600, 900 and 1,800 words, closely tracking each cap.
- **AgentWrite and LongWriter-6k:** The plan-then-write pipeline yields coherent outputs of up to about 20,000 words. Its output, LongWriter-6k, supplies long-response training examples.
- **LongBench-Write results:** The benchmark groups requests by required length (0–500, 500–2k, 2k–4k, 4k+ words). In the 4k–20k range, LongWriter-9B-DPO averaged about 8,600 words, against roughly 1,550 for GPT-4o.
- **Availability:** Code and models are released at github.com/THUDM/LongWriter.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 137 works · 83,262 citations
- [Zhipu AI](https://md.research.elamur.ai/free/en/companies/zhipu-ai/) — Company, Beijing, China · 12 works · 2,069 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang and Juanzi Li. 2024. LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs. arXiv:2408.07055.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{bai2024longwriter,
  title = {{LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs}},
  author = {Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang and Juanzi Li},
  year = {2024},
  doi = {10.48550/arxiv.2408.07055},
  eprint = {2408.07055},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.07055}
}
```

- [← previous Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-agent-flan-designing-data-methods/)
- [next → One-Shot Learning as Instruction Data Prospector for Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-one-shot-learning-as-instruction/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

51 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. [Phi-3 technical report: A highly capable language model locally on your phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/). arXiv preprint arXiv:2404.14219, 2024.
2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
3. Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training-free long-context scaling of large language models. arXiv preprint arXiv:2402.17463, 2024. — [arXiv:2402.17463](https://arxiv.org/abs/2402.17463) [find](https://scholar.google.com/scholar?q=Chenxin+An%2C+Fei+Huang%2C+Jun+Zhang%2C+Shansan+Gong%2C+Xipeng+Qiu%2C+Chang+Zhou%2C+and+Lingpeng+Kong%2E+Training%2Dfree+long%2Dcontext+scaling+of+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E17463%2C+2024%2E+%E2%80%94+%5BarXiv%3A)
4. Anthropic. Anthropic: Introducing claude 3.5 sonnet, 2024. URL [https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet). [↗](https://www.anthropic.com/news/claude-3-5-sonnet)
5. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. [Longalign: A recipe for long context alignment of large language models](https://md.research.elamur.ai/free/en/papers/bai-2024-longalign-recipe-long-context/). arXiv preprint arXiv:2401.18058, 2024a.
6. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with language-model-as-an-examiner. Advances in Neural Information Processing Systems, 36, 2024b. [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Jiahao+Ying%2C+Yixin+Cao%2C+Xin+Lv%2C+Yuze+He%2C+Xiaozhi+Wang%2C+Jifan+Yu%2C+Kaisheng+Zeng%2C+Yijia+Xiao%2C+Haozhe+Lyu%2C+et+al%2E+Benchmarking+foundation+models+with+language%2Dmodel%2Das%2Dan%2Dexaminer%2E+Advances+in)
7. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. [Weak-to-strong generalization: Eliciting strong capabilities with weak supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/). arXiv preprint arXiv:2312.09390, 2023.
8. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. [Medusa: Simple llm inference acceleration framework with multiple decoding heads](https://md.research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/). arXiv preprint arXiv:2401.10774, 2024.
9. Longze Chen, Ziqiang Liu, Wanwei He, Yunshui Li, Run Luo, and Min Yang. Long context is not long at all: A prospector of long-dependency data for large language models. arXiv preprint arXiv:2405.17915, 2024a. [find](https://scholar.google.com/scholar?q=Longze+Chen%2C+Ziqiang+Liu%2C+Wanwei+He%2C+Yunshui+Li%2C+Run+Luo%2C+and+Min+Yang%2E+Long+context+is+not+long+at+all%3A+A+prospector+of+long%2Ddependency+data+for+large+language+models%2E+arXiv+preprint+arXiv%3A2405%2E17915)
10. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023a. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2C+2023a%2E)
11. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). arXiv preprint arXiv:2309.12307, 2023b.
12. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024b. [find](https://scholar.google.com/scholar?q=Zixiang+Chen%2C+Yihe+Deng%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+and+Quanquan+Gu%2E+Self%2Dplay+fine%2Dtuning+converts+weak+language+models+to+strong+language+models%2E+arXiv+preprint+arXiv%3A2401%2E01335%2C+2024b%2E)
13. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna…](https://lmsys.org/blog/2023-03-30-vicuna…) [↗](https://lmsys.org/blog/2023-03-30-vicuna…)
14. Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.14838, 2024. [find](https://scholar.google.com/scholar?q=Yuntian+Deng%2C+Yejin+Choi%2C+and+Stuart+Shieber%2E+From+explicit+cot+to+implicit+cot%3A+Learning+to+internalize+cot+step+by+step%2E+arXiv+preprint+arXiv%3A2405%2E14838%2C+2024%2E)
15. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3029–3051, 2023.
16. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models, 2024. [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Amy+Yang%2C+Angela+Fan%2C+et+al%2E+The+llama+3+herd+of+models%2C+2024%2E)
17. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. [Data engineering for scaling language models to 128k context](https://md.research.elamur.ai/free/en/papers/fu-2024-data-engineering-scaling-language-models/). arXiv preprint arXiv:2402.10171, 2024.
18. [Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinka…](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/)
19. Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023. — [arXiv:2308.16137](https://arxiv.org/abs/2308.16137) [find](https://scholar.google.com/scholar?q=Chi+Han%2C+Qifan+Wang%2C+Wenhan+Xiong%2C+Yu+Chen%2C+Heng+Ji%2C+and+Sinong+Wang%2E+Lm%2Dinfinite%3A+Simple+on%2Dthe%2Dfly+length+generalization+for+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E16137%2C+2023%2E+%E2%80%94+%5BarXiv%3A23)
20. Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. arXiv preprint arXiv:2404.15846, 2024. [find](https://scholar.google.com/scholar?q=Qianyu+He%2C+Jie+Zeng%2C+Qianxi+He%2C+Jiaqing+Liang%2C+and+Yanghua+Xiao%2E+From+complex+to+simple%3A+Enhancing+multi%2Dconstraint+complex+instruction+following+ability+of+large+language+models%2E+arXiv+preprint+arXiv)
21. Zhenyu Hou, Yiin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, et al. Chatglm-rlhf: Practices of aligning large language models with human feedback. arXiv preprint arXiv:2404.00934, 2024. [find](https://scholar.google.com/scholar?q=Zhenyu+Hou%2C+Yiin+Niu%2C+Zhengxiao+Du%2C+Xiaohan+Zhang%2C+Xiao+Liu%2C+Aohan+Zeng%2C+Qinkai+Zheng%2C+Minlie+Huang%2C+Hongning+Wang%2C+Jie+Tang%2C+et+al%2E+Chatglm%2Drlhf%3A+Practices+of+aligning+large+language+models+with+huma)
22. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685, 2021.
23. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
24. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. [Llm maybe longlm: Self-extend llm context window without tuning](https://md.research.elamur.ai/free/en/papers/jin-2024-llm-maybe-longlm-self-extend/). arXiv preprint arXiv:2401.01325, 2024.
25. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. [Self-alignment with instruction backtranslation](https://md.research.elamur.ai/free/en/papers/li-2023-self-alignment-instruction-backtranslation/). In The Twelfth International Conference on Learning Representations, 2024a.
26. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. [Snapkv: Llm knows what you are looking for before generation](https://md.research.elamur.ai/free/en/papers/li-2024-snapkv-llm-knows-looking-before/). arXiv preprint arXiv:2404.14469, 2024b.
27. OpenAI. Openai: hello-gpt-4o/. Hello gpt-4o, 2024a. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Openai%3A+hello%2Dgpt%2D4o%2F%2E+Hello+gpt%2D4o%2C+2024a%2E)
28. OpenAI. Gpt-4o mini: advancing cost-efficient intelligence, 2024b. URL [https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/). [↗](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)
29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35: 27730–27744, 2022.
30. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2C+2023%2E)
31. Chau Minh Pham, Simeng Sun, and Mohit Iyyer. Suri: Multi-constraint instruction following for long-form text generation. arXiv preprint arXiv:2406.19371, 2024. — [arXiv:2406.19371](https://arxiv.org/abs/2406.19371) [find](https://scholar.google.com/scholar?q=Chau+Minh+Pham%2C+Simeng+Sun%2C+and+Mohit+Iyyer%2E+Suri%3A+Multi%2Dconstraint+instruction+following+for+long%2Dform+text+generation%2E+arXiv+preprint+arXiv%3A2406%2E19371%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E19371%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
32. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.[07924, 2023](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/).
33. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36, 2024.
34. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505–3506, 2020. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26th+ACM+)
35. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530,…
36. Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch-solve-merge improves large language model evaluation and generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… [find](https://scholar.google.com/scholar?q=Swarnadeep+Saha%2C+Omer+Levy%2C+Asli+Celikyilmaz%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Xian+Li%2E+Branch%2Dsolve%2Dmerge+improves+large+language+model+evaluation+and+generation%2E+In+Proceedings+of+the+2024+Conference)
37. URL [https://openai.com/index/Haoran](https://openai.com/index/Haoran) Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained instruction-following ability of large language models. arXiv preprint arXiv:2404.02823, 2024. [↗](https://openai.com/index/Haoran)
38. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cle%CC%81mentine+Fourrier%2C+Nathan+Habib%2C+et+al%2E+Zephyr%3A+Direct+distillation+)
39. Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208, 2024. [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Kai+Xiao%2C+Reimar+Leike%2C+Lilian+Weng%2C+Johannes+Heidecke%2C+and+Alex+Beutel%2E+The+instruction+hierarchy%3A+Training+llms+to+prioritize+privileged+instructions%2E+arXiv+preprint+arXiv%3A2404%2E13208%2C+)
40. Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, et al. Weaver: Foundation models for creative writing. arXiv preprint arXiv:2401.17268, 2024. [find](https://scholar.google.com/scholar?q=Tiannan+Wang%2C+Jiamin+Chen%2C+Qingrui+Jia%2C+Shuai+Wang%2C+Ruoyu+Fang%2C+Huilin+Wang%2C+Zhaowei+Gao%2C+Chunzhao+Xie%2C+Chuou+Xu%2C+Jihong+Dai%2C+et+al%2E+Weaver%3A+Foundation+models+for+creative+writing%2E+arXiv+preprint+arXi)
41. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837, 2022.
42. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. [Autogen: Enabling next-gen llm applications via multi-agent conversation](https://md.research.elamur.ai/free/en/papers/wu-2023-autogen-enabling-next-gen-llm/) framework. arXiv preprint arXiv:2308.08155, 2023.
43. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2C+2023%2E)
44. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. [Effective long-context scaling of foundation models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/). In Proceedings of the 2024 Conference of the North American Chapter of the Association fo…
45. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Advances+in+Neural+Informati)
46. Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1. arXiv preprint arXiv:2407.06023, 2024. [find](https://scholar.google.com/scholar?q=Ping+Yu%2C+Jing+Xu%2C+Jason+Weston%2C+and+Ilia+Kulikov%2E+Distilling+system+2+into+system+1%2E+arXiv+preprint+arXiv%3A2407%2E06023%2C+2024%2E)
47. Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason Weston, and Jing Xu. Following length constraints in instructions. arXiv preprint arXiv:2406.17744, 2024. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Ilia+Kulikov%2C+Ping+Yu%2C+Kyunghyun+Cho%2C+Sainbayar+Sukhbaatar%2C+Jason+Weston%2C+and+Jing+Xu%2E+Following+length+constraints+in+instructions%2E+arXiv+preprint+arXiv%3A2406%2E17744%2C+2024%2E)
48. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Soaring from 4k to 400k: Extending llm’s context with activation beacon. arXiv preprint arXiv:2401.03462, 2024a. [find](https://scholar.google.com/scholar?q=Peitian+Zhang%2C+Zheng+Liu%2C+Shitao+Xiao%2C+Ninglu+Shao%2C+Qiwei+Ye%2C+and+Zhicheng+Dou%2E+Soaring+from+4k+to+400k%3A+Extending+llm%E2%80%99s+context+with+activation+beacon%2E+arXiv+preprint+arXiv%3A2401%2E03462%2C+2024a%2E)
49. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024b. [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+Re%CC%81%2C+Clark+Barrett%2C+et+al%2E+H2o%3A+Heavy%2Dhitter+oracle+for+efficient+generative+infer)
50. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. [Wildchat: 1m chatgpt interaction logs in the wild](https://md.research.elamur.ai/free/en/papers/zhao-2024-wildchat-1m-chatgpt-interaction-logs/). arXiv preprint arXiv:2405.01470, 2024.
51. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+Advance)

*Updated 24 October 2026*