---
title: "Exploring Fine-Tuning for In-Context Retrieval and Efficient"
description: "The paper asks whether long-context language models (LCLMs) can be trained to find and use the relevant passages inside a… 42 references."
authors: ["Francesco Maria Molfese", "Momchil Hardalov", "Rexhina Blloshmi", "Bill Byrne", "Adrià de Gispert"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2026.eacl-short.44/"
doi: "10.18653/v1/2026.eacl-short.44"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/molfese-2026-exploring-fine-tuning-context-retrieval/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/molfese-2026-exploring-fine-tuning-context-retrieval/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) / Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models

# Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models

**Authors:** [Francesco Maria Molfese](https://md.research.elamur.ai/free/en/authors/francesco-maria-molfese/), [Momchil Hardalov](https://md.research.elamur.ai/free/en/authors/momchil-hardalov/), Rexhina Blloshmi, [Bill Byrne](https://md.research.elamur.ai/free/en/authors/bill-byrne/) and [Adrià de Gispert](https://md.research.elamur.ai/free/en/authors/adria-de-gispert/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-short.44/) · [PDF](https://aclanthology.org/2026.eacl-short.44.pdf) · [DOI 10.18653/v1/2026.eacl-short.44](https://doi.org/10.18653/v1/2026.eacl-short.44) · [OpenAlex](https://openalex.org/W7140117250)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

The paper asks whether long-context language models (LCLMs) can be trained to find and use the relevant passages inside a very long input, so that they could replace classic RAG pipelines. It also asks whether such training makes them more robust when the KV cache is compressed. The authors build training contexts from HotpotQA and 2WikiMultihopQA by mixing the gold Wikipedia passages with many retrieved hard negatives, shuffled and tagged with document IDs. They fine-tune Qwen2.5-7B-Instruct-1M with GRPO using five verifiable rewards, ranging from answer correctness alone to reasoning graded by an LLM judge. These models are compared with a supervised fine-tuning baseline and a RAG@32k baseline.

GRPO, especially with answer-only and reasoning rewards, gives large in-domain gains, while plain SFT tends to hurt performance. Out-of-domain results are mixed and depend on the task. Fine-tuning modestly reduces the losses caused by KV-cache compression. The gains do not come from better attention-based document ranking. The authors instead attribute them to greater robustness against noise in the context. This matters for deciding when long-context models can stand in for retrieval.

## Key points

- **Training data:** The retriever is Qwen3-Embedding-4B, which fetches the top 500 Wikipedia passages (100-word chunks) per question. Instances are capped at 32k tokens, but evaluation goes up to 1M tokens.
- **Rewards:** There are five GRPO reward variants: answer-only, document IDs, IDs plus full content, IDs plus quotes, and reasoning with an LLM judge. An answer-only SFT model serves as a comparison.
- **Benchmarks and gains:** In-domain evaluation uses the HELMET RAG subsets (HotpotQA, NQ, TriviaQA), with gains of up to +20 points. Out-of-domain evaluation uses ∞Bench, LongBench-v2 and Loong (financial split). LCLMs gain about +9 on finance questions, while RAG leads by about +6 on multiple-choice questions.
- **KV-cache compression:** Under attention-based compression, the base model loses 16.1% on average. The best fine-tuned model (IDs plus content reward) loses 13.9%, but it still trails RAG@32k.
- **Document ranking:** The base model already ranks documents well by attention (NDCG@10 of 82.0–84.7), and fine-tuning adds only 0.5–0.7 points. Ranking quality shows a negligible correlation with accuracy (r = −0.09). The code is publicly released on GitHub (amazon-science).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Sapienza University of Rome](https://md.research.elamur.ai/free/en/universities/sapienza-university-of-rome/) — University, Rome, Italy · 57 works · 9,564 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

## Cite

Cite as: Francesco Maria Molfese, Momchil Hardalov, Rexhina Blloshmi, Bill Byrne and Adrià de Gispert. 2026. Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{molfese2026exploring,
  title = {{Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models}},
  author = {Francesco Maria Molfese, Momchil Hardalov, Rexhina Blloshmi, Bill Byrne and Adrià {de Gispert}},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-short.44},
  url = {https://doi.org/10.18653/v1/2026.eacl-short.44}
}
```

- [← previous Exploring Cross-Lingual Voice Conversion Methods for Anonymizing Low-Resource Text-to-Speech](https://md.research.elamur.ai/free/en/papers/wang-2026-exploring-cross-lingual-voice-conversion/)
- [next → Exploring Knowledge Graphs for Automatic Fake News Detection in Portuguese](https://md.research.elamur.ai/free/en/papers/santos-2026-exploring-knowledge-graphs-automatic-fake/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/)

## References

42 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amazon AGI. 2025. Amazon Nova 2: Multimodal reasoning and generation models. Amazon Technical Reports. [find](https://scholar.google.com/scholar?q=Amazon+AGI%2E+2025%2E+Amazon+Nova+2%3A+Multimodal+reasoning+and+generation+models%2E+Amazon+Technical+Reports%2E)
2. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). In Proceedings of the 62nd Annual Meeting of the Associati…
3. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025. [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://md.research.elamur.ai/free/en/papers/bai-2024-longbench-v2-deeper-understanding/). In Proceedings of the 63rd Annual Meeting of the Associa…
4. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, and 81 others. 2024. [InternLM2 Technical Report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/). Preprint, arXiv:2403.17297.
5. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 179 others. 2025. DeepSeek-R1 incentivizes reasoning in LLMs throug… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Daya+Guo%2C+Dejian+Yang%2C+Haowei+Zhang%2C+Jun%2DMei+Song%2C+Ruoyu+Zhang%2C+Runxin+Xu%2C+Qihao+Zhu%2C+Shirong+Ma%2C+Peiyi+Wang%2C+Xiaoling+Bi%2C+Xiaokang+Zhang%2C+Xingkai+Yu%2C+Yu+Wu%2C+Z%2E+F%2E+Wu%2C+Zhibin+Gou%2C+Zhihong)
6. Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2025. [How to Train Long-Context Language Models (Effectively)](https://md.research.elamur.ai/free/en/papers/gao-2025-how-train-long-context-language/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, ACL ’25, pages 7376–7399, Vienna, Austria.
7. Gemini. 2025. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). Preprint, arXiv:2507.06261.
8. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, and 37 others. 2024. [ChatGLM: A Family of Large Language Models from GLM-130B t…](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/)
9. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/). In Proceedings of the 28th International Conference on Computational Linguistics, COLING ’20, pages 6609–6625, Barcelona, Spain (Online).
10. Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminadabi, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2024. System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. [find](https://scholar.google.com/scholar?q=Sam+Ade+Jacobs%2C+Masahiro+Tanaka%2C+Chengming+Zhang%2C+Minjia+Zhang%2C+Reza+Yazdani+Aminadabi%2C+Shuaiwen+Leon+Song%2C+Samyam+Rajbhandari%2C+and+Yuxiong+He%2E+2024%2E+System+Optimizations+for+Enabling+Training+of+Extr)
11. In Proceedings of the 43rd ACM Symposium on Principles of Distributed Computing, PODC ’24, page 121–130, Nantes, France. Conference on Neural Information Processing Systems, volume 37 of NeurIPS, pages 139997–140031, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+43rd+ACM+Symposium+on+Principles+of+Distributed+Computing%2C+PODC+%E2%80%9924%2C+page+121%E2%80%93130%2C+Nantes%2C+France%2E+Conference+on+Neural+Information+Processing+Systems%2C+volume+37+of+NeurIPS%2C+page)
12. Ziyan Jiang, Xueguang Ma, and Wenhu Chen. 2024. LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs. Preprint, arXiv:2406.15319. — [arXiv:2406.15319](https://arxiv.org/abs/2406.15319) [find](https://scholar.google.com/scholar?q=Ziyan+Jiang%2C+Xueguang+Ma%2C+and+Wenhu+Chen%2E+2024%2E+LongRAG%3A+Enhancing+Retrieval%2DAugmented+Generation+with+Long%2Dcontext+LLMs%2E+Preprint%2C+arXiv%3A2406%2E15319%2E+%E2%80%94+%5BarXiv%3A2406%2E15319%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2406%2E15)
13. Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang, Yuqing Yang, and Lili Qiu. 2025. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval. In The Thirty-ninth Annual Conference… [find](https://scholar.google.com/scholar?q=Di+Liu%2C+Meng+Chen%2C+Baotong+Lu%2C+Huiqiang+Jiang%2C+Zhenhua+Han%2C+Qianxi+Zhang%2C+Qi+Chen%2C+Chengruidong+Zhang%2C+Bailu+Ding%2C+Kai+Zhang%2C+Chen+Chen%2C+Fan+Yang%2C+Yuqing+Yang%2C+and+Lili+Qiu%2E+2025%2E+RetrievalAttention%3A+)
14. Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. 2025. Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG. In The Thirteenth International Conference on Learning Representations, ICLR ’25, Singapore, Singapore. [find](https://scholar.google.com/scholar?q=Bowen+Jin%2C+Jinsung+Yoon%2C+Jiawei+Han%2C+and+Sercan+O+Arik%2E+2025%2E+Long%2DContext+LLMs+Meet+RAG%3A+Overcoming+Challenges+for+Long+Inputs+in+RAG%2E+In+The+Thirteenth+International+Conference+on+Learning+Represent)
15. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL ’17, pages 1601–1611, Vancouver, Canada.
16. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP ’20, pages 6769–6781, O…
17. Francesco Maria Molfese, Luca Moroni, Luca Gioffré, Alessandro Scirè, Simone Conia, and Roberto Navigli. 2025. Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering. In Findings of the Association for Computational Linguistics: ACL 2025, Fi… [find](https://scholar.google.com/scholar?q=Francesco+Maria+Molfese%2C+Luca+Moroni%2C+Luca+Gioffr%C3%A9%2C+Alessandro+Scir%C3%A8%2C+Simone+Conia%2C+and+Roberto+Navigli%2E+2025%2E+Right+Answer%2C+Wrong+Score%3A+Uncovering+the+Inconsistencies+of+LLM+Evaluation+in+Multiple%2DC)
18. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, and 1 others. 2025. [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/). Preprint, arXiv:2512.13961.
19. OpenAI. 2025. OpenAI o3 and o4-mini System Card. [https://cdn.openai.com/pdf/](https://cdn.openai.com/pdf/) 2221c875-02dc-4789-800b-e7758f3722c1/ o3-and-o4-mini-system-card.pdf. [↗](https://cdn.openai.com/pdf/)
20. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
21. Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Sid Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, and Kelvin Guu. 2024. [Can Long…](https://md.research.elamur.ai/free/en/papers/lee-2024-long-context-language-models-subsume/)
22. Xinze Li, Yixin Cao, Yubo Ma, and Aixin Sun. 2025. Long Context vs. RAG for LLMs: An Evaluation and Revisits. Preprint, arXiv:2501.01880. — [arXiv:2501.01880](https://arxiv.org/abs/2501.01880) [find](https://scholar.google.com/scholar?q=Xinze+Li%2C+Yixin+Cao%2C+Yubo+Ma%2C+and+Aixin+Sun%2E+2025%2E+Long+Context+vs%2E+RAG+for+LLMs%3A+An+Evaluation+and+Revisits%2E+Preprint%2C+arXiv%3A2501%2E01880%2E+%E2%80%94+%5BarXiv%3A2501%2E01880%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E01880%29)
23. Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 881–893, Mi… [find](https://scholar.google.com/scholar?q=Zhuowan+Li%2C+Cheng+Li%2C+Mingyang+Zhang%2C+Qiaozhu+Mei%2C+and+Michael+Bendersky%2E+2024%2E+Retrieval+Augmented+Generation+or+Long%2DContext+LLMs%3F+A+Comprehensive+Study+and+Hybrid+Approach%2E+In+Proceedings+of+the+20)
24. Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. 2024. MiniCache: KV Cache Compression in Depth Dimension for Large Language Models. In The Thirty-eigth Annual Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient Context Window Extens… [find](https://scholar.google.com/scholar?q=Akide+Liu%2C+Jing+Liu%2C+Zizheng+Pan%2C+Yefei+He%2C+Gholamreza+Haffari%2C+and+Bohan+Zhuang%2E+2024%2E+MiniCache%3A+KV+Cache+Compression+in+Depth+Dimension+for+Large+Language+Models%2E+In+The+Thirty%2Deigth+Annual+Bowen+P)
25. Yifu Qiu, Varun R. Embar, Yizhe Zhang, Navdeep Jaitly, Shay B Cohen, and Benjamin Han. 2025. Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3176–3192, Vienna, Austria. [find](https://scholar.google.com/scholar?q=Yifu+Qiu%2C+Varun+R%2E+Embar%2C+Yizhe+Zhang%2C+Navdeep+Jaitly%2C+Shay+B+Cohen%2C+and+Benjamin+Han%2E+2025%2E+Eliciting+In%2Dcontext+Retrieval+and+Reasoning+for+Long%2Dcontext+Large+Language+Models%2E+In+Findings+of+the+Ass)
26. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 Technical Report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
27. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). Preprint, arXiv:2402.03300.
28. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. [HybridFlow: A Flexible and Efficient RLHF Framework](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/). In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297, Rotterdam, Netherlands.
29. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, and Ce Zhang. 2023. [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/). In Proceedings of the 40th International Conference on Machine Learnin…
30. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [RoFormer: Enhanced transformer with Rotary Position Embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
31. Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024. [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://md.research.elamur.ai/free/en/papers/wang-2024-leave-no-document-behind-benchmarking/). In Proceedings of the 2024 Conference on E…
32. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, and 2 others. 2024. Effe… — [Effective Long-Context Scaling of Foundation Models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
33. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2024. [Retrieval meets Long Context Large Language Models](https://md.research.elamur.ai/free/en/papers/xu-2023-retrieval-meets-long-context-large/). In The Twelfth International Conference on Learning Representations, ICLR ’24, Vienna, Austr…
34. Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu li, Feiyu Xiong, Bo Tang, and Ding Chen. 2025. xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation. In The Thirteenth International Conference on Learning Representations, ICLR ’25, Singapore, Singapore. [find](https://scholar.google.com/scholar?q=Qingchen+Yu%2C+Zifan+Zheng%2C+Shichao+Song%2C+Zhiyu+li%2C+Feiyu+Xiong%2C+Bo+Tang%2C+and+Ding+Chen%2E+2025%2E+xFinder%3A+Large+Language+Models+as+Automated+Evaluators+for+Reliable+Evaluation%2E+In+The+Thirteenth+Internati)
35. Tan Yu, Anbang Xu, and Rama Akkiraju. 2024. In Defense of RAG in the Era of Long-Context Language Models. Preprint, arXiv:2409.01666. — [arXiv:2409.01666](https://arxiv.org/abs/2409.01666) [find](https://scholar.google.com/scholar?q=Tan+Yu%2C+Anbang+Xu%2C+and+Rama+Akkiraju%2E+2024%2E+In+Defense+of+RAG+in+the+Era+of+Long%2DContext+Language+Models%2E+Preprint%2C+arXiv%3A2409%2E01666%2E+%E2%80%94+%5BarXiv%3A2409%2E01666%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2409%2E01666%29)
36. Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez. 2024a. [RAFT: Adapting Language Model to Domain Specific RAG](https://md.research.elamur.ai/free/en/papers/zhang-2024-raft-adapting-language-model-domain/). In The First Conference on Language Modeling, COLM ’24, Philadelphia, Pennsylvania, USA.
37. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024b. ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL… [find](https://scholar.google.com/scholar?q=Xinrong+Zhang%2C+Yingfa+Chen%2C+Shengding+Hu%2C+Zihang+Xu%2C+Junhao+Chen%2C+Moo+Hao%2C+Xu+Han%2C+Zhen+Thai%2C+Shuo+Wang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024b%2E+%E2%88%9EBench%3A+Extending+Long+Context+Evaluation+Beyond+100K+Toke)
38. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://md.research.elamur.ai/free/en/papers/zhang-2025-qwen3-embedding-advancing-text/). Preprint, arXiv:2506.05176.
39. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. 2023. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. In The Thirty-seventh Annual Co… [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+R%C3%A9%2C+Clark+Barrett%2C+Zhangyang+%22Atlas%22+Wang%2C+and+Beidi+Chen%2E+2023%2E+H2O%3A+Heavy%2DHitter)
40. Peng Xu, Wei Ping, Xianchao Wu, Chejian Xu, Zihan Liu, Mohammad Shoeybi, and Bryan Catanzaro. 2025. ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities. In The Thirteenth International Conference on Learning Representations, ICLR ’25, Singapore, Singapore. [find](https://scholar.google.com/scholar?q=Peng+Xu%2C+Wei+Ping%2C+Xianchao+Wu%2C+Chejian+Xu%2C+Zihan+Liu%2C+Mohammad+Shoeybi%2C+and+Bryan+Catanzaro%2E+2025%2E+ChatQA+2%3A+Bridging+the+Gap+to+Proprietary+LLMs+in+Long+Context+and+RAG+Capabilities%2E+In+The+Thirteen)
41. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP ’18,…
42. Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2025. HELMET: How to Evaluate Long-context Models Effectively and Thoroughly. In The Thirteenth International Conference on Learning Representations, ICLR ’25, Singapore, Singapore. [find](https://scholar.google.com/scholar?q=Howard+Yen%2C+Tianyu+Gao%2C+Minmin+Hou%2C+Ke+Ding%2C+Daniel+Fleischer%2C+Peter+Izsak%2C+Moshe+Wasserblat%2C+and+Danqi+Chen%2E+2025%2E+HELMET%3A+How+to+Evaluate+Long%2Dcontext+Models+Effectively+and+Thoroughly%2E+In+The+Thirt)

*Updated 1 November 2026*