---
title: "What are the Essential Factors in Crafting Effective Long Context"
description: "As a remedy, they introduce a framework called Multi-agent Interactive Multi-hop Generation (MIMG)… 60 references, cited by 13 papers here."
authors: ["Zhi Chen", "Qiguang Chen", "Libo Qin", "Qipeng Guo", "Haijun Lv", "Yicheng Zou", "Hang Yan", "Kai Chen", "Dahua Lin"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.1316/"
doi: "10.18653/v1/2025.acl-long.1316"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-what-are-essential-factors-crafting/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-what-are-essential-factors-crafting/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/) / What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices

# What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices

**Authors:** [Zhi Chen](https://md.research.elamur.ai/free/en/authors/zhi-chen/), [Qiguang Chen](https://md.research.elamur.ai/free/en/authors/qiguang-chen/), [Libo Qin](https://md.research.elamur.ai/free/en/authors/libo-qin/), [Qipeng Guo](https://md.research.elamur.ai/free/en/authors/qipeng-guo/), [Haijun Lv](https://md.research.elamur.ai/free/en/authors/haijun-lv/), [Yicheng Zou](https://md.research.elamur.ai/free/en/authors/yicheng-zou/), [Hang Yan](https://md.research.elamur.ai/free/en/authors/hang-yan/), [Kai Chen](https://md.research.elamur.ai/free/en/authors/kai-chen/) and [Dahua Lin](https://md.research.elamur.ai/free/en/authors/dahua-lin/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.1316/) · [PDF](https://aclanthology.org/2025.acl-long.1316.pdf) · [DOI 10.18653/v1/2025.acl-long.1316](https://doi.org/10.18653/v1/2025.acl-long.1316) · [OpenAlex](https://openalex.org/W4412889893)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [13](https://md.research.elamur.ai/free/en/papers/chen-2025-what-are-essential-factors-crafting/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Instruction-tuning LLMs for long contexts requires large amounts of training data, and human annotation of such data is expensive. A common workaround is synthetic data built with the Self-Instruct framework. The authors show this produces weak data: with Qwen-2 72B, under 35% of samples actually required multi-hop reasoning, roughly 40% were of low quality, and many were semantic duplicates. As a remedy, they introduce a framework called Multi-agent Interactive Multi-hop Generation (MIMG), built from four components:
- a quality-verification agent that checks samples throughout the pipeline,
- an agent that writes single-hop questions per document,
- sampling strategies that pick related questions within or across documents,
- a merger agent that combines those questions into multi-hop ones.

They systematically compare design choices across many models and domains, including scoring versus classification verifiers, question-first generation, and retrieval and merging schemes. The resulting dataset, LongMIT, is mostly multi-hop, high-quality and non-redundant. Models fine-tuned on it outperform counterparts trained on larger human-annotated sets. This suggests that careful synthetic data design is a cost-effective route to better long-context reasoning.

## Key points
- Baseline problems with Self-Instruct (Qwen-2 72B): under 35% multi-hop samples, about 60% high-quality, and over 45% semantic duplication. These figures come from manual annotation of 200 samples.
- The study tests 17 strategies across 10 domains and 5 LLMs. MIMG output exceeds 85% multi-hop, high-quality, non-duplicate samples.
- Instruction tuning on LongMIT improves long-context QA by at least 7.54% on average. The largest gains are on multi-hop benchmarks such as 2WikiMQA, MuSiQue and HotpotQA.
- Efficiency:
  - Data quality rises nearly fourfold for fewer than 0.5k extra tokens per sample.
  - Pipelines built on open-source LLMs approach the best quality at about one-third of the token cost.
- Scaling experiments on LLaMA3-8B show that performance grows with more training data and with more hops at a fixed data volume. Code and data are released at github.com/WowCZ/LongMIT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) — University, Harbin, China · 158 works · 28,841 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 117 works · 11,281 citations
- [Central South University](https://md.research.elamur.ai/free/en/universities/central-south-university/) — University, Changsha, China · 15 works · 582 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Zhi Chen, Qiguang Chen, Libo Qin, Qipeng Guo, Haijun Lv, Yicheng Zou, Hang Yan, Kai Chen and Dahua Lin. 2025. What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2025what,
  title = {{What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices}},
  author = {Zhi Chen, Qiguang Chen, Libo Qin, Qipeng Guo, Haijun Lv, Yicheng Zou, Hang Yan, Kai Chen and Dahua Lin},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1316},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1316}
}
```

- [← previous Value Profiles for Encoding Human Variation](https://md.research.elamur.ai/free/en/papers/sorensen-2025-value-profiles-encoding-human-variation/)
- [next → What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective](https://md.research.elamur.ai/free/en/papers/li-2025-what-happened-llms-layers-when/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/)

## References

60 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Anthropic. 2023. Model card and evaluations for claude models. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2023%2E+Model+card+and+evaluations+for+claude+models%2E)
3. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024a. [Longalign: A recipe for long context alignment of large language models](https://md.research.elamur.ai/free/en/papers/bai-2024-longalign-recipe-long-context/). Preprint, arXiv:2401.18058. \#\# Limitations Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhen…
4. Due to the high costs associated with large-scale distillation training experiments on GPT-4-based Giovanni Maria Biancofiore, Yashar Deldjoo, Tommaso Di Noia, Eugenio Di Sciascio, and Fedelucio Narducci. 2024. Interactive question answering systems: Literature review. ACM Computing Surveys, 56(9):1… [find](https://scholar.google.com/scholar?q=Due+to+the+high+costs+associated+with+large%2Dscale+distillation+training+experiments+on+GPT%2D4%2Dbased+Giovanni+Maria+Biancofiore%2C+Yashar+Deldjoo%2C+Tommaso+Di+Noia%2C+Eugenio+Di+Sciascio%2C+and+Fedelucio+Nardu)
5. In Advances in Neural Information Processing Systems, volume 35, pages 16344–16359. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=In+Advances+in+Neural+Information+Processing+Systems%2C+volume+35%2C+pages+16344%E2%80%9316359%2E+Curran+Associates%2C+Inc%2E)
6. Zheng Cai, Maosong Cao, Haojiong Chen, et al. 2024. [Internlm2 technical report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/). arXiv preprint arXiv:2403.17297.
7. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. [Longrope: Extending llm context window beyond 2 million tokens](https://md.research.elamur.ai/free/en/papers/ding-2024-longrope-extending-llm-context/). Preprint, arXiv:2402.13753.
8. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). ACM Transactions on Intelligent Systems and Technology, 15(3):1–45.
9. Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025a. [Towards reasoning era: A survey of long chain-of-thought for reasoning large language models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/). arXiv preprint arXiv:2503.09567.
10. Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiaqi Wang, Mengkang Hu, Zhi Chen, Wanxiang Che, and Ting Liu. 2025b. Ecm: A unified electronic circuit model for explaining the emergence of in-context learning and chain-of-thought in large language model. arXiv preprint arXiv:2502.03325. — [arXiv:2502.03325](https://arxiv.org/abs/2502.03325) [find](https://scholar.google.com/scholar?q=Qiguang+Chen%2C+Libo+Qin%2C+Jinhao+Liu%2C+Dengyun+Peng%2C+Jiaqi+Wang%2C+Mengkang+Hu%2C+Zhi+Chen%2C+Wanxiang+Che%2C+and+Ting+Liu%2E+2025b%2E+Ecm%3A+A+unified+electronic+circuit+model+for+explaining+the+emergence+of+in%2Dconte)
11. Qiguang Chen, Libo Qin, Jiaqi WANG, Jingxuan Zhou, and Wanxiang Che. 2024a. Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Qiguang+Chen%2C+Libo+Qin%2C+Jiaqi+WANG%2C+Jingxuan+Zhou%2C+and+Wanxiang+Che%2E+2024a%2E+Unlocking+the+capabilities+of+thought%3A+A+reasoning+boundary+framework+to+quantify+and+optimize+chain%2Dof%2Dthought%2E+In+The+Thir)
12. Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. 2024b. [M 3 CoT: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought](https://md.research.elamur.ai/free/en/papers/chen-2024-m-3-cot-novel-benchmark/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8199–8221, Ban…
13. Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. [Exploring the use of large language models for reference-free text quality evaluation: An empirical study](https://md.research.elamur.ai/free/en/papers/chen-2023-exploring-use-large-language-models/). In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), pages 361–374.
14. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024c. [LongloRA: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). In The Twelfth International Conference on Learning Representations.
15. Tri Dao. 2024. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). In The Twelfth International Conference on Learning Representations.
16. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Dan+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+2022%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E)
17. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
18. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024a. GPTScore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556–6576, Mexico City,… [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2024a%2E+GPTScore%3A+Evaluate+as+you+desire%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapter+of+the+Association+for+Computatio)
19. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024b. [Data engineering for scaling language models to 128k context](https://md.research.elamur.ai/free/en/papers/fu-2024-data-engineering-scaling-language-models/). In Proc. of ICML.
20. Chaochen Gao, Xing Wu, Qi Fu, and Songlin Hu. 2024. Quest: Query-centric data synthesis approach for long-context scaling of large language model. arXiv preprint arXiv:2405.19846. — [arXiv:2405.19846](https://arxiv.org/abs/2405.19846) [find](https://scholar.google.com/scholar?q=Chaochen+Gao%2C+Xing+Wu%2C+Qi+Fu%2C+and+Songlin+Hu%2E+2024%2E+Quest%3A+Query%2Dcentric+data+synthesis+approach+for+long%2Dcontext+scaling+of+large+language+model%2E+arXiv+preprint+arXiv%3A2405%2E19846%2E+%E2%80%94+%5BarXiv%3A2405%2E19846%5D)
21. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
22. Gal Ben Haim, Adi Braun, Haggai Eden, Livnat Bur-shtein, Yiftach Barash, Avinoah Irony, and Eyal Klang. 2024. Ai in the ed: Assessing the efficacy of gpt models vs. physicians in medical score calculation. The American Journal of Emergency Medicine, 79:161–166. [find](https://scholar.google.com/scholar?q=Gal+Ben+Haim%2C+Adi+Braun%2C+Haggai+Eden%2C+Livnat+Bur%2Dshtein%2C+Yiftach+Barash%2C+Avinoah+Irony%2C+and+Eyal+Klang%2E+2024%2E+Ai+in+the+ed%3A+Assessing+the+efficacy+of+gpt+models+vs%2E+physicians+in+medical+score+calcula)
23. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. [Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/). In Proc. of COLING, pages 6609–6625.
24. Matthew Hoffman, Francis Bach, and David Blei. 2010. Online learning for latent dirichlet allocation. advances in neural information processing systems, 23. [find](https://scholar.google.com/scholar?q=Matthew+Hoffman%2C+Francis+Bach%2C+and+David+Blei%2E+2010%2E+Online+learning+for+latent+dirichlet+allocation%2E+advances+in+neural+information+processing+systems%2C+23%2E)
25. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. [Ruler: What’s the real context size of your long-context language models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) Preprint, arXiv:2404.06654.
26. Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. 2024. [Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model](https://md.research.elamur.ai/free/en/papers/hu-2025-hiagent-hierarchical-working-memory-management/). arXiv preprint arXiv:2408.09559.
27. Mengkang Hu, Tianxing Chen, Yude Zou, Yuheng Lei, Qiguang Chen, Ming Li, Yao Mu, Hongyuan Zhang, Wenqi Shao, and Ping Luo. 2025. Text2world: Benchmarking large language models for symbolic world model generation. arXiv preprint arXiv:2502.13092. [find](https://scholar.google.com/scholar?q=Mengkang+Hu%2C+Tianxing+Chen%2C+Yude+Zou%2C+Yuheng+Lei%2C+Qiguang+Chen%2C+Ming+Li%2C+Yao+Mu%2C+Hongyuan+Zhang%2C+Wenqi+Shao%2C+and+Ping+Luo%2E+2025%2E+Text2world%3A+Benchmarking+large+language+models+for+symbolic+world+model)
28. Mengkang Hu, Yao Mu, Xinmiao Yu, et al. 2023. Tree-planner: Efficient close-loop task planning with large language models. Preprint, arXiv:2310.08582. — [arXiv:2310.08582](https://arxiv.org/abs/2310.08582) [find](https://scholar.google.com/scholar?q=Mengkang+Hu%2C+Yao+Mu%2C+Xinmiao+Yu%2C+et+al%2E+2023%2E+Tree%2Dplanner%3A+Efficient+close%2Dloop+task+planning+with+large+language+models%2E+Preprint%2C+arXiv%3A2310%2E08582%2E+%E2%80%94+%5BarXiv%3A2310%2E08582%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E0)
29. Jerry Huang. 2024. How well can a long sequence model model long sequences? comparing architech-tural inductive biases on long-context abilities. arXiv preprint arXiv:2407.08112. — [arXiv:2407.08112](https://arxiv.org/abs/2407.08112) [find](https://scholar.google.com/scholar?q=Jerry+Huang%2E+2024%2E+How+well+can+a+long+sequence+model+model+long+sequences%3F+comparing+architech%2Dtural+inductive+biases+on+long%2Dcontext+abilities%2E+arXiv+preprint+arXiv%3A2407%2E08112%2E+%E2%80%94+%5BarXiv%3A2407%2E08112%5D%28)
30. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. [Llm maybe longlm: Selfextend llm context window without tuning](https://md.research.elamur.ai/free/en/papers/jin-2024-llm-maybe-longlm-self-extend/). Preprint, arXiv:2401.01325.
31. Seungone Kim, Se Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. 2023. [The CoT collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning](https://md.research.elamur.ai/free/en/papers/kim-2023-cot-collection-improving-zero-shot/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing…
32. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistic…
33. Sanwoo Lee, Yida Cai, Desong Meng, Ziyang Wang, and Yunfang Wu. 2024. Unleashing large language models’ proficiency in zero-shot essay scoring. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 181–198. [find](https://scholar.google.com/scholar?q=Sanwoo+Lee%2C+Yida+Cai%2C+Desong+Meng%2C+Ziyang+Wang%2C+and+Yunfang+Wu%2E+2024%2E+Unleashing+large+language+models%E2%80%99+proficiency+in+zero%2Dshot+essay+scoring%2E+In+Findings+of+the+Association+for+Computational+Linguis)
34. Yanyang Li, Shuo Liang, Michael Lyu, and Liwei Wang. 2024. Making long-context language models better multi-hop reasoners. In Proc. of ACL, pages 2462– 2475. [find](https://scholar.google.com/scholar?q=Yanyang+Li%2C+Shuo+Liang%2C+Michael+Lyu%2C+and+Liwei+Wang%2E+2024%2E+Making+long%2Dcontext+language+models+better+multi%2Dhop+reasoners%2E+In+Proc%2E+of+ACL%2C+pages+2462%E2%80%93+2475%2E)
35. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173.
36. Yijun Liu, Jinzheng Yu, Yang Xu, Zhongyang Li, and Qingfu Zhu. 2025. A survey on transformer context extension: Approaches and evaluation. arXiv preprint arXiv:2503.13299. — [arXiv:2503.13299](https://arxiv.org/abs/2503.13299) [find](https://scholar.google.com/scholar?q=Yijun+Liu%2C+Jinzheng+Yu%2C+Yang+Xu%2C+Zhongyang+Li%2C+and+Qingfu+Zhu%2E+2025%2E+A+survey+on+transformer+context+extension%3A+Approaches+and+evaluation%2E+arXiv+preprint+arXiv%3A2503%2E13299%2E+%E2%80%94+%5BarXiv%3A2503%2E13299%5D%28https%3A%2F)
37. Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Mohammad Shoeybi, and Bryan Catanzaro. 2024b. [Chatqa: Building gpt-4 level conversational qa models](https://md.research.elamur.ai/free/en/papers/liu-2024-chatqa-surpassing-gpt-4-conversational/). arXiv preprint arXiv:2401.10225.
38. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+2024%2E+YaRN%3A+Efficient+context+window+extension+of+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representat)
39. Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. 2024. Large language models meet nlp: A survey. arXiv preprint arXiv:2405.12819. — [arXiv:2405.12819](https://arxiv.org/abs/2405.12819) [find](https://scholar.google.com/scholar?q=Libo+Qin%2C+Qiguang+Chen%2C+Xiachong+Feng%2C+Yang+Wu%2C+Yongheng+Zhang%2C+Yinghui+Li%2C+Min+Li%2C+Wanxiang+Che%2C+and+Philip+S+Yu%2E+2024%2E+Large+language+models+meet+nlp%3A+A+survey%2E+arXiv+preprint+arXiv%3A2405%2E12819%2E+%E2%80%94+%5Ba)
40. Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. [Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages](https://md.research.elamur.ai/free/en/papers/qin-2023-cross-lingual-prompting-improving-zero/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2695–2709, Singapore. Association for C…
41. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2025. A survey of multilingual large language models. Patterns, 6(1). [find](https://scholar.google.com/scholar?q=Libo+Qin%2C+Qiguang+Chen%2C+Yuhang+Zhou%2C+Zhi+Chen%2C+Yinghui+Li%2C+Lizi+Liao%2C+Min+Li%2C+Wanxiang+Che%2C+and+Philip+S+Yu%2E+2025%2E+A+survey+of+multilingual+large+language+models%2E+Patterns%2C+6%281%29%2E)
42. Machel Reid, Nikolay Savinov, Denis Teplyashin, et al. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530.
43. Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. [find](https://scholar.google.com/scholar?q=Stephen+Robertson%2C+Hugo+Zaragoza%2C+et+al%2E+2009%2E+The+probabilistic+relevance+framework%3A+Bm25+and+beyond%2E+Foundations+and+Trends%C2%AE+in+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2E)
44. Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. [Okapi at trec-3](https://md.research.elamur.ai/free/en/papers/robertson-1994-okapi-trec-3/). Nist Special Publication Sp, 109:109.
45. Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis. 2024. [In-context pretraining: Language modeling beyond document boundaries](https://md.research.elamur.ai/free/en/papers/shi-2023-context-pretraining-language-modeling-document/). In The Twelfth International Conference on Learning Representations.
46. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi-hop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10:539–554. [find](https://scholar.google.com/scholar?q=Harsh+Trivedi%2C+Niranjan+Balasubramanian%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2022%2E+MuSiQue%3A+Multi%2Dhop+Questions+via+Single%2Dhop+Question+Composition%2E+Transactions+of+the+Association+for+Computational+Li)
47. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. [Is ChatGPT a good NLG evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 1–11, Singapore. Association for Computational Ling…
48. Peng Wang, Yongheng Zhang, Hao Fei, Qiguang Chen, Yukai Wang, Jiasheng Si, Wenpeng Lu, Min Li, and Libo Qin. 2024. S3 agent: Unlocking the power of vllm for zero-shot multi-modal sarcasm detection. ACM Trans. Multimedia Comput. Commun. Appl. Just Accepted. [find](https://scholar.google.com/scholar?q=Peng+Wang%2C+Yongheng+Zhang%2C+Hao+Fei%2C+Qiguang+Chen%2C+Yukai+Wang%2C+Jiasheng+Si%2C+Wenpeng+Lu%2C+Min+Li%2C+and+Libo+Qin%2E+2024%2E+S3+agent%3A+Unlocking+the+power+of+vllm+for+zero%2Dshot+multi%2Dmodal+sarcasm+detection%2E+AC)
49. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023b%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Pro)
50. Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Proc. of NeurIPS.
51. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. [C-pack: Packaged resources to advance general chinese embedding](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/). Preprint, arXiv:2309.07597.
52. Wenhan Xiong, Jingyu Liu, Igor Molybog, et al. 2023. [Effective long-context scaling of foundation models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/). Preprint, arXiv:2309.16039.
53. Wenhan Xiong, Jingyu Liu, Igor Molybog, et al. 2024. [Effective long-context scaling of foundation models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/). In Proc. of the NAACL.
54. Peng Xu, Wei Ping, Xianchao Wu, et al. 2024a. Chatqa 2: Bridging the gap to proprietary llms in long context and rag capabilities. arXiv preprint arXiv:2407.14482. — [arXiv:2407.14482](https://arxiv.org/abs/2407.14482) [find](https://scholar.google.com/scholar?q=Peng+Xu%2C+Wei+Ping%2C+Xianchao+Wu%2C+et+al%2E+2024a%2E+Chatqa+2%3A+Bridging+the+gap+to+proprietary+llms+in+long+context+and+rag+capabilities%2E+arXiv+preprint+arXiv%3A2407%2E14482%2E+%E2%80%94+%5BarXiv%3A2407%2E14482%5D%28https%3A%2F%2Farxiv%2Eo)
55. Yang Xu, Yunlong Feng, Honglin Mu, Yutai Hou, Yitong Li, Xinghao Wang, Wanjun Zhong, Zhongyang Li, Dandan Tu, Qingfu Zhu, et al. 2024b. Concise and precise context compression for tool-using language models. arXiv preprint arXiv:2407.02043. [find](https://scholar.google.com/scholar?q=Yang+Xu%2C+Yunlong+Feng%2C+Honglin+Mu%2C+Yutai+Hou%2C+Yitong+Li%2C+Xinghao+Wang%2C+Wanjun+Zhong%2C+Zhongyang+Li%2C+Dandan+Tu%2C+Qingfu+Zhu%2C+et+al%2E+2024b%2E+Concise+and+precise+context+compression+for+tool%2Dusing+language+)
56. An Yang, Baosong Yang, Binyuan Hui, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+et+al%2E+2024%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2E)
57. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proc. of EMNLP, pages 2369–2380.
58. Yikai Zhang, Junlong Li, and Pengfei Liu. 2024a. Extending llms’ context window with 100 samples. Preprint, arXiv:2401.07004. — [arXiv:2401.07004](https://arxiv.org/abs/2401.07004) [find](https://scholar.google.com/scholar?q=Yikai+Zhang%2C+Junlong+Li%2C+and+Pengfei+Liu%2E+2024a%2E+Extending+llms%E2%80%99+context+window+with+100+samples%2E+Preprint%2C+arXiv%3A2401%2E07004%2E+%E2%80%94+%5BarXiv%3A2401%2E07004%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2401%2E07004%29)
59. Yongheng Zhang, Qiguang Chen, Min Li, Wanxiang Che, and Libo Qin. 2024b. Autocap: Towards automatic cross-lingual alignment planning for zero-shot chain-of-thought. In Findings of the Association for Computational Linguistics ACL 2024, pages 9191– 9200. [find](https://scholar.google.com/scholar?q=Yongheng+Zhang%2C+Qiguang+Chen%2C+Min+Li%2C+Wanxiang+Che%2C+and+Libo+Qin%2E+2024b%2E+Autocap%3A+Towards+automatic+cross%2Dlingual+alignment+planning+for+zero%2Dshot+chain%2Dof%2Dthought%2E+In+Findings+of+the+Association+for+)
60. Ziyu Zhuang, Qiguang Chen, Longxuan Ma, Mingda Li, Yi Han, Yushan Qian, Haopeng Bai, Zixian Feng, Weinan Zhang, and Ting Liu. 2023. [Through the lens of core competency: Survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/zhuang-2023-through-lens-core-competency-survey/). arXiv preprint arXiv:2308.07902.

*Updated 30 October 2026*