--- title: "A Comprehensive Survey on Long Context Language Modeling" description: "The paper proposes no new model. 561 references, cited by 61 papers here. Among them: 3." authors: ["Jiaheng Liu", "Dawei Zhu", "Zhiqi Bai", "Yancheng He", "Huanxuan Liao", "Haoran Que", "Zekun Wang", "Chenchen Zhang", "Ge Zhang", "Jiebin Zhang", "Yuanxing Zhang", "Zhuo Chen", "Hangyu Guo", "Shilong Li", "Ziqiang Liu", "Yong Shan", "Yifan Song", "Jiayi Tian", "Wenhao Wu", "Zhejian Zhou", "Ruijie Zhu", "Junlan Feng", "Yang Gao", "Shizhu He", "Zhoujun Li", "Tianyu Liu", "Fanyu Meng", "Wenbo Su", "Yingshui Tan", "Zili Wang", "Jian Yang", "Wei Ye", "Bo Zheng", "Wangchunshu Zhou", "Wenhao Huang", "Sujian Li", "Zhaoxiang Zhang"] year: 2025 venue: "arXiv:2503.17407" source: "https://arxiv.org/abs/2503.17407" doi: "10.48550/arxiv.2503.17407" arxiv: "2503.17407" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/liu-2025-comprehensive-survey-long-context-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2025-comprehensive-survey-long-context-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / A Comprehensive Survey on Long Context Language Modeling # A Comprehensive Survey on Long Context Language Modeling **Authors:** [Jiaheng Liu](https://md.research.elamur.ai/free/en/authors/jiaheng-liu/), [Dawei Zhu](https://md.research.elamur.ai/free/en/authors/dawei-zhu/), Zhiqi Bai, [Yancheng He](https://md.research.elamur.ai/free/en/authors/yancheng-he/), Huanxuan Liao, [Haoran Que](https://md.research.elamur.ai/free/en/authors/haoran-que/), [Zekun Wang](https://md.research.elamur.ai/free/en/authors/zekun-wang/), [Chenchen Zhang](https://md.research.elamur.ai/free/en/authors/chenchen-zhang/), [Ge Zhang](https://md.research.elamur.ai/free/en/authors/ge-zhang/), Jiebin Zhang, [Yuanxing Zhang](https://md.research.elamur.ai/free/en/authors/yuanxing-zhang/), [Zhuo Chen](https://md.research.elamur.ai/free/en/authors/zhuo-chen/), Hangyu Guo, Shilong Li, Ziqiang Liu, Yong Shan, [Yifan Song](https://md.research.elamur.ai/free/en/authors/yifan-song/), Jiayi Tian, [Wenhao Wu](https://md.research.elamur.ai/free/en/authors/wenhao-wu/), Zhejian Zhou, Ruijie Zhu, Junlan Feng, [Yang Gao](https://md.research.elamur.ai/free/en/authors/yang-gao/), [Shizhu He](https://md.research.elamur.ai/free/en/authors/shizhu-he/), [Zhoujun Li](https://md.research.elamur.ai/free/en/authors/zhoujun-li/), [Tianyu Liu](https://md.research.elamur.ai/free/en/authors/tianyu-liu/), Fanyu Meng, Wenbo Su, Yingshui Tan, Zili Wang, [Jian Yang](https://md.research.elamur.ai/free/en/authors/jian-yang/), [Wei Ye](https://md.research.elamur.ai/free/en/authors/wei-ye/), [Bo Zheng](https://md.research.elamur.ai/free/en/authors/bo-zheng/), [Wangchunshu Zhou](https://md.research.elamur.ai/free/en/authors/wangchunshu-zhou/), [Wenhao Huang](https://md.research.elamur.ai/free/en/authors/wenhao-huang/), [Sujian Li](https://md.research.elamur.ai/free/en/authors/sujian-li/) and Zhaoxiang Zhang · **Year:** 2025 · **Venue:** arXiv:2503.17407 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2503.17407) · [PDF](https://arxiv.org/pdf/2503.17407) · [DOI 10.48550/arxiv.2503.17407](https://doi.org/10.48550/arxiv.2503.17407) · [arXiv 2503.17407](https://arxiv.org/abs/2503.17407) · [OpenAlex](https://openalex.org/W6903651657) **Citations:** 3 citations (OpenAlex, 2026-10-14) · 61 in our library, of the 141,840 papers whose bibliographies we read ## Summary Language models that can read very long inputs, such as whole books, extended dialogues or large codebases, are now common. The methods behind them, however, are spread across many separate research threads. This survey comes from a large team at NJU, PKU, CASIA, Alibaba, ByteDance, Tencent, Kuaishou and M-A-P. It organizes that literature around three questions: - How to build capable and efficient long context language models (LCLMs). - How to train and serve them at reasonable cost. - How to evaluate and interpret them. The first part covers data work for pre-training and post-training, position embeddings and ways to extrapolate them, and attention variants. It also covers workflows that extend a single model, including prompt compression, memory, RAG and agents. The second part reviews infrastructure, from I/O and GPU memory optimizations to quantization, prefill/decode disaggregation and speculative decoding. The third part discusses benchmarks for comprehension and long-form generation, along with behavioral and mechanistic analyses. The paper proposes no new model. Its value is a structured map of the field plus a maintained paper list, which is useful when choosing techniques for long-document projects. ## Key points - The survey traces context windows growing from 4K to 128K, 1M and even 10M tokens. The authors argue that this headroom enables test-time scaling, such as o1-like long reasoning, inside a single context. - The architecture taxonomy has four groups: - Position embeddings (RoPE, Alibi, XPOS) and extrapolation methods (NTK, YaRN, LongRoPE). - Transformer-based designs (Longformer, NSA, GQA). - Linear-complexity models (Mamba, RetNet, lightning attention). - Hybrid models (Jamba, Samba, Minimax-01). - Data resources are grouped by training stage: - Pre-training filtering: Longwanjuan and LongAttn. - Pre-training synthesis: ICP, SPLICE and Quest. - Post-training: GATEAU, Ziya-Reader and ChatQA2. - The analysis sections address several issues: - Advertised context length often exceeds what models actually use well. - Long-context perplexity may not predict real task performance. - Models favor information at the beginning and end of the input. - Mechanistic work has identified specific "retrieval heads". - The future directions are long reasoning, larger windows, efficient architectures and deployment, more reliable evaluation, and mechanistic interpretability. The companion GitHub repository is LCLM-Horizon (arXiv:2503.17407v2, Nov 2025). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works ## Cite Cite as: Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Zhang, Zhuo Chen, Hangyu Guo, Shilong Li, Ziqiang Liu, Yong Shan, Yifan Song, Jiayi Tian, Wenhao Wu, Zhejian Zhou, Ruijie Zhu, Junlan Feng, Yang Gao, Shizhu He, Zhoujun Li, Tianyu Liu, Fanyu Meng, Wenbo Su, Yingshui Tan, Zili Wang, Jian Yang, Wei Ye, Bo Zheng, Wangchunshu Zhou, Wenhao Huang, Sujian Li and Zhaoxiang Zhang. 2025. A Comprehensive Survey on Long Context Language Modeling. arXiv:2503.17407. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{liu2025comprehensive, title = {{A Comprehensive Survey on Long Context Language Modeling}}, author = {Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Zhang, Zhuo Chen, Hangyu Guo, Shilong Li, Ziqiang Liu, Yong Shan, Yifan Song, Jiayi Tian, Wenhao Wu, Zhejian Zhou, Ruijie Zhu, Junlan Feng, Yang Gao, Shizhu He, Zhoujun Li, Tianyu Liu, Fanyu Meng, Wenbo Su, Yingshui Tan, Zili Wang, Jian Yang, Wei Ye, Bo Zheng, Wangchunshu Zhou, Wenhao Huang, Sujian Li and Zhaoxiang Zhang}, year = {2025}, doi = {10.48550/arxiv.2503.17407}, eprint = {2503.17407}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2503.17407} } ``` - [← previous Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph](https://md.research.elamur.ai/free/en/papers/vashurin-2025-benchmarking-uncertainty-quantification-methods-large/) - [next → A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2025-survey-graph-retrieval-augmented-generation/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) ## References 561 references, 144 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ntk-alibi: Long text extrapolation of alibi position encoding through interpolation, August 2023. URL [https://github.com/keezen/ntk_alibi](https://github.com/keezen/ntk_alibi) . [↗](https://github.com/keezen/ntk_alibi) 2. Long-data-collections. [https://huggingface.co/datasets/togethercomputer/L](https://huggingface.co/datasets/togethercomputer/L) ong-Data-Collections , 2023. [↗](https://huggingface.co/datasets/togethercomputer/L) 3. Amro Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari S. Morcos. [Semdedup: Data-efficient learning at web-scale through semantic deduplication](https://md.research.elamur.ai/free/en/papers/abbas-2023-semdedup-data-efficient-learning-web/). CoRR, abs/2303.09540, 2023. doi: 10.48550/ARXIV.2303.09540. URL https://doi.org/10.48550/arXiv.230 3.09540 . [↗](https://doi.org/10.48550/arXiv.230) 4. Lisa Adams, Felix Busch, Tianyu Han, Jean-Baptiste Excoffier, Matthieu Ortala, Alexander Löser, Hugo JWL Aerts, Jakob Nikolas Kather, Daniel Truhn, and Keno Bressem. Longhealth: A question answering benchmark with long clinical documents. arXiv preprint arXiv:2401.14490, 2024. — [arXiv:2401.14490](https://arxiv.org/abs/2401.14490) [find](https://scholar.google.com/scholar?q=Lisa+Adams%2C+Felix+Busch%2C+Tianyu+Han%2C+Jean%2DBaptiste+Excoffier%2C+Matthieu+Ortala%2C+Alexander+L%C3%B6ser%2C+Hugo+JWL+Aerts%2C+Jakob+Nikolas+Kather%2C+Daniel+Truhn%2C+and+Keno+Bressem%2E+Longhealth%3A+A+question+answering+b) 5. Ameeta Agrawal, Andy Dang, Sina Bagheri Nezhad, Rhitabrat Pokharel, and Russell Scheinberg. Evaluating multilingual long-context models for retrieval and reasoning. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 216–231, 2024. [find](https://scholar.google.com/scholar?q=Ameeta+Agrawal%2C+Andy+Dang%2C+Sina+Bagheri+Nezhad%2C+Rhitabrat+Pokharel%2C+and+Russell+Scheinberg%2E+Evaluating+multilingual+long%2Dcontext+models+for+retrieval+and+reasoning%2E+In+Proceedings+of+the+Fourth+Worksh) 6. Talkie Ai. Talkie \| ai-native character community, 2024. URL [https://www.talkie-ai.com/](https://www.talkie-ai.com/) . [↗](https://www.talkie-ai.com/) 7. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. [GQA: training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In EMNLP, pages 4895–4901. Association for Computational Linguistics, 2023. 8. Faisal Al-Khateeb, Nolan Dey, Daria Soboleva, and Joel Hestness. Position interpolation improves alibi extrapolation. arXiv preprint arXiv:2310.13017, 2023. — [arXiv:2310.13017](https://arxiv.org/abs/2310.13017) [find](https://scholar.google.com/scholar?q=Faisal+Al%2DKhateeb%2C+Nolan+Dey%2C+Daria+Soboleva%2C+and+Joel+Hestness%2E+Position+interpolation+improves+alibi+extrapolation%2E+arXiv+preprint+arXiv%3A2310%2E13017%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E13017%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F) 9. [Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy…](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/) 10. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. [L-eval: Instituting standardized evaluation for long context language models](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/). arXiv preprint arXiv:2307.11088, 2023. 11. Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training-free long-context scaling of large language models. arXiv preprint arXiv:2402.17463, 2024. — [arXiv:2402.17463](https://arxiv.org/abs/2402.17463) [find](https://scholar.google.com/scholar?q=Chenxin+An%2C+Fei+Huang%2C+Jun+Zhang%2C+Shansan+Gong%2C+Xipeng+Qiu%2C+Chang+Zhou%2C+and+Lingpeng+Kong%2E+Training%2Dfree+long%2Dcontext+scaling+of+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E17463%2C+2024%2E+%E2%80%94+%5BarXiv%3A) 12. Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. Why does the effective context length of llms fall short? arXiv preprint arXiv:2410.18745, 2024. — [arXiv:2410.18745](https://arxiv.org/abs/2410.18745) [find](https://scholar.google.com/scholar?q=Chenxin+An%2C+Jun+Zhang%2C+Ming+Zhong%2C+Lei+Li%2C+Shansan+Gong%2C+Yao+Luo%2C+Jingjing+Xu%2C+and+Lingpeng+Kong%2E+Why+does+the+effective+context+length+of+llms+fall+short%3F+arXiv+preprint+arXiv%3A2410%2E18745%2C+2024%2E+%E2%80%94+%5Bar) 13. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Make your LLM fully utilize the context. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL [https://openreview.net/forum?id=YGTV](https://openreview.net/forum?id=YGTV) EmBXtV . [↗](https://openreview.net/forum?id=YGTV) 14. Anthropic. Introducing contextual retrieval. [https://www.anthropic.com/news/con](https://www.anthropic.com/news/con) textual-retrieval . [↗](https://www.anthropic.com/news/con) 15. Anysphere. Cursor - the ai code editor. [https://www.cursor.com/en](https://www.cursor.com/en) , 2025. URL https://www.cursor.com . [↗](https://www.cursor.com/en) 16. Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff, 2024. URL https://arxiv.org/abs/24 02.18668 . [↗](https://arxiv.org/abs/24) 17. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508, 2023. 18. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. [Longalign: A recipe for long context alignment of large language models](https://md.research.elamur.ai/free/en/papers/bai-2024-longalign-recipe-long-context/). arXiv preprint arXiv:2401.18058, 2024. 19. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. [LongAlign: A recipe for long context alignment of large language models](https://md.research.elamur.ai/free/en/papers/bai-2024-longalign-recipe-long-context/). In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 202… 20. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. [Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks](https://md.research.elamur.ai/free/en/papers/bai-2024-longbench-v2-deeper-understanding/). arXiv preprint arXiv:2412.15204, 2024. 21. Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. [Longwriter: Unleashing 10,000+ word generation from long context llms](https://md.research.elamur.ai/free/en/papers/bai-2024-longwriter-unleashing-10-000/). arXiv preprint arXiv:2408.07055, 2024. 22. Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin. Pipeswitch: Fast pipelined context switching for deep learning applications. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 499–514, 2020. [find](https://scholar.google.com/scholar?q=Zhihao+Bai%2C+Zhen+Zhang%2C+Yibo+Zhu%2C+and+Xin+Jin%2E+Pipeswitch%3A+Fast+pipelined+context+switching+for+deep+learning+applications%2E+In+14th+USENIX+Symposium+on+Operating+Systems+Design+and+Implementation+%28OSD) 23. Satanjeev Banerjee and Alon Lavie. [Meteor: An automatic metric for mt evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005. 24. Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. [G-transformer for document-level machine translation](https://md.research.elamur.ai/free/en/papers/bao-2021-g-transformer-document-level-machine/). In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Intern… 25. Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2024. [find](https://scholar.google.com/scholar?q=Ali+Behrouz%2C+Peilin+Zhong%2C+and+Vahab+Mirrokni%2E+Titans%3A+Learning+to+memorize+at+test+time%2E+arXiv+preprint+arXiv%3A2501%2E00663%2C+2024%2E) 26. Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, and Vahab Mirrokni. Atlas: Learning to optimally memorize the context at test time. arXiv preprint arXiv:2505.23735, 2025. — [arXiv:2505.23735](https://arxiv.org/abs/2505.23735) [find](https://scholar.google.com/scholar?q=Ali+Behrouz%2C+Zeman+Li%2C+Praneeth+Kacham%2C+Majid+Daliri%2C+Yuan+Deng%2C+Peilin+Zhong%2C+Meisam+Razaviyayn%2C+and+Vahab+Mirrokni%2E+Atlas%3A+Learning+to+optimally+memorize+the+context+at+test+time%2E+arXiv+preprint+arX) 27. Iz Beltagy, Matthew E. Peters, and Arman Cohan. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). CoRR, abs/2004.05150, 2020. URL https://arxiv.org/abs/2004.05150 . [↗](https://arxiv.org/abs/2004.05150) 28. Iz Beltagy, Matthew E. Peters, and Arman Cohan. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/), 2020. URL https://arxiv.org/abs/2004.05150 . [↗](https://arxiv.org/abs/2004.05150) 29. Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba, 2024. URL https://arxiv.org/abs/2406.14528 . — [arXiv:2406.14528](https://arxiv.org/abs/2406.14528) [↗](https://arxiv.org/abs/2406.14528) 30. Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. [In-context learning with long-context models: An in-depth exploration](https://md.research.elamur.ai/free/en/papers/bertsch-2025-context-learning-long-context-models/). arXiv preprint arXiv:2405.00200, 2024. 31. Arthur S Bianchessi, Rodrigo C Barros, and Lucas S Kupssinskü. Bayesian attention mechanism: A probabilistic framework for positional encoding and context length extrapolation. arXiv preprint arXiv:2505.22842, 2025. 32 bloc97. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context… — [arXiv:2505.22842](https://arxiv.org/abs/2505.22842) [find](https://scholar.google.com/scholar?q=Arthur+S+Bianchessi%2C+Rodrigo+C+Barros%2C+and+Lucas+S+Kupssinsk%C3%BC%2E+Bayesian+attention+mechanism%3A+A+probabilistic+framework+for+positional+encoding+and+context+length+extrapolation%2E+arXiv+preprint+arXiv%3A25) 32. Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, et al. Long code arena: a set of benchmarks for long-context code models. arXiv preprint arXiv:2406.11612, 2024. — [arXiv:2406.11612](https://arxiv.org/abs/2406.11612) [find](https://scholar.google.com/scholar?q=Egor+Bogomolov%2C+Aleksandra+Eliseeva%2C+Timur+Galimzyanov%2C+Evgeniy+Glukhov%2C+Anton+Shapkin%2C+Maria+Tigina%2C+Yaroslav+Golubev%2C+Alexander+Kovrigin%2C+Arie+van+Deursen%2C+Maliheh+Izadi%2C+et+al%2E+Long+code+arena%3A+a+s) 33. Sebastian Borgeaud, A. Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, T. Hennigan, Saffron Huang, Lorenzo Maggiore, Chris Jones, Albin Cassir… [find](https://scholar.google.com/scholar?q=Sebastian+Borgeaud%2C+A%2E+Mensch%2C+Jordan+Hoffmann%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Katie+Millican%2C+George+van+den+Driessche%2C+Jean%2DBaptiste+Lespiau%2C+Bogdan+Damoc%2C+Aidan+Clark%2C+Diego+de+Las+Casas%2C+Aurelia+Gu) 34. Antoine Bosselut, Asli Celikyilmaz, Xiaodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. Discourse-aware neural rewards for coherent text generation. arXiv preprint arXiv:1805.03766, 2018. [find](https://scholar.google.com/scholar?q=Antoine+Bosselut%2C+Asli+Celikyilmaz%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+Po%2DSen+Huang%2C+and+Yejin+Choi%2E+Discourse%2Daware+neural+rewards+for+coherent+text+generation%2E+arXiv+preprint+arXiv%3A1805%2E03766%2C+2018%2E) 35. Aleksandar Botev, Soham De, Samuel L. Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Sertan Girgin, Olivier Bachem, Alek Andreev, Kathleen Kenealy, Thomas Mesnard, Cassidy Hardin, Sur… — [arXiv:2404.07839](https://arxiv.org/abs/2404.07839) [find](https://scholar.google.com/scholar?q=Aleksandar+Botev%2C+Soham+De%2C+Samuel+L%2E+Smith%2C+Anushan+Fernando%2C+George%2DCristian+Muraru%2C+Ruba+Haroun%2C+Leonard+Berrada%2C+Razvan+Pascanu%2C+Pier+Giuseppe+Sessa%2C+Robert+Dadashi%2C+L%C3%A9onard+Hussenot%2C+Johan+Ferret) 36. Felix Brakel, Uraz Odyurt, and Ana-Lucia Varbanescu. Model parallelism on distributed infrastructure: A literature review from theory to llm case-studies. arXiv preprint arXiv:2403.03699, 2024. — [arXiv:2403.03699](https://arxiv.org/abs/2403.03699) [find](https://scholar.google.com/scholar?q=Felix+Brakel%2C+Uraz+Odyurt%2C+and+Ana%2DLucia+Varbanescu%2E+Model+parallelism+on+distributed+infrastructure%3A+A+literature+review+from+theory+to+llm+case%2Dstudies%2E+arXiv+preprint+arXiv%3A2403%2E03699%2C+2024%2E+%E2%80%94+%5BarX) 37. William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. Striped attention: Faster ring attention for causal transformers. arXiv preprint arXiv:2311.09431, 2023. [find](https://scholar.google.com/scholar?q=William+Brandon%2C+Aniruddha+Nrusimha%2C+Kevin+Qian%2C+Zachary+Ankner%2C+Tian+Jin%2C+Zhiye+Song%2C+and+Jonathan+Ragan%2DKelley%2E+Striped+attention%3A+Faster+ring+attention+for+causal+transformers%2E+arXiv+preprint+arXiv) 38. Thorsten Brants, Ashok Popat, Peng Xu, Franz Josef Och, and Jeffrey Dean. [Large language models in machine translation](https://md.research.elamur.ai/free/en/papers/brants-2007-large-language-models-machine-translation/). In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 858–867, 2007. 39. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/) 40. Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. Scaling transformer to 1m tokens and beyond with RMT. CoRR, abs/2304.11062, 2023. — [arXiv:2304.11062](https://arxiv.org/abs/2304.11062) [find](https://scholar.google.com/scholar?q=Aydar+Bulatov%2C+Yuri+Kuratov%2C+and+Mikhail+S%2E+Burtsev%2E+Scaling+transformer+to+1m+tokens+and+beyond+with+RMT%2E+CoRR%2C+abs%2F2304%2E11062%2C+2023%2E+%E2%80%94+%5BarXiv%3A2304%2E11062%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2304%2E11062%29) 41. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. [Medusa: Simple llm inference acceleration framework with multiple decoding heads](https://md.research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/). In Proceedings of the 41st International Conference on Machine Learning, pages 5209–5235, 2024. 42. Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. [Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling](https://md.research.elamur.ai/free/en/papers/cai-2024-pyramidkv-dynamic-kv-cache-compression/), 2024. URL https://arxiv.org/abs/2406 .02069 . [↗](https://arxiv.org/abs/2406) 43. Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models. In First Conference on Language Modeling, 2024. [find](https://scholar.google.com/scholar?q=Yihan+Cao%2C+Yanbin+Kang%2C+Chi+Wang%2C+and+Lichao+Sun%2E+Instruction+mining%3A+Instruction+data+selection+for+tuning+large+language+models%2E+In+First+Conference+on+Language+Modeling%2C+2024%2E) 44. Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436, 2024. — [arXiv:2406.07436](https://arxiv.org/abs/2406.07436) [find](https://scholar.google.com/scholar?q=Linzheng+Chai%2C+Shukai+Liu%2C+Jian+Yang%2C+Yuwei+Yin%2C+Ke+Jin%2C+Jiaheng+Liu%2C+Tao+Sun%2C+Ge+Zhang%2C+Changyu+Ren%2C+Hongcheng+Guo%2C+et+al%2E+Mceval%3A+Massively+multilingual+code+evaluation%2E+arXiv+preprint+arXiv%3A2406%2E07) 45. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. [Mle-bench: Evaluating machine learning agents on machine learning engineering](https://md.research.elamur.ai/free/en/papers/chan-2024-mle-bench-evaluating-machine/). CoRR, abs/2410.07095, 2024. doi: 10.48550/AR… 46. Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hourvideo: 1-hour video-language understanding. CoRR, abs/2411.04998, 2024. doi: 10.48550/ARXIV .2411.04998. URL https://doi.org/10.48550/arXiv.24… — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) [↗](https://doi.org/10.48550/arXiv.24…) 47. Li-Wen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Chengji Yao, Ziheng Jiang, et al. Flux: fast software-based communication overlap on gpus through kernel fusion. arXiv preprint arXiv:2406.06858, 2024. [find](https://scholar.google.com/scholar?q=Li%2DWen+Chang%2C+Wenlei+Bao%2C+Qi+Hou%2C+Chengquan+Jiang%2C+Ningxin+Zheng%2C+Yinmin+Zhong%2C+Xuanrun+Zhang%2C+Zuquan+Song%2C+Chengji+Yao%2C+Ziheng+Jiang%2C+et+al%2E+Flux%3A+fast+software%2Dbased+communication+overlap+on+gpus+th) 48. Character AI. Character ai. Retrieved September 14, 2023 from [https://character.ai/](https://character.ai/) , 2023. URL https://character.ai/ . [↗](https://character.ai/) 49. Aili Chen, Xuyang Ge, Ziquan Fu, Yanghua Xiao, and Jiangjie Chen. Travelagent: An AI assistant for personalized travel planning. CoRR, abs/2409.08069, 2024. doi: 10.48550/ARX IV.2409.08069. URL https://doi.org/10.48550/arXiv.2409.08069 . — [doi:10.48550/arx](https://doi.org/10.48550/arx) [↗](https://doi.org/10.48550/arXiv.2409.08069) 50. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. [Reading Wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Association for Computational Linguistics (ACL), 2017. 51. Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. Clex: Continuous length extrapolation for large language models. arXiv preprint arXiv:2310.16450, 2023. — [arXiv:2310.16450](https://arxiv.org/abs/2310.16450) [find](https://scholar.google.com/scholar?q=Guanzheng+Chen%2C+Xin+Li%2C+Zaiqiao+Meng%2C+Shangsong+Liang%2C+and+Lidong+Bing%2E+Clex%3A+Continuous+length+extrapolation+for+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E16450%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E16450%5D%28htt) 52. Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. Clex: Continuous length extrapolation for large language models. In The Twelfth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Guanzheng+Chen%2C+Xin+Li%2C+Zaiqiao+Meng%2C+Shangsong+Liang%2C+and+Lidong+Bing%2E+Clex%3A+Continuous+length+extrapolation+for+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representat) 53. Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading. arXiv preprint arXiv: 2310.05029, 2023. — [arXiv:2310.05029](https://arxiv.org/abs/2310.05029) [find](https://scholar.google.com/scholar?q=Howard+Chen%2C+Ramakanth+Pasunuru%2C+Jason+Weston%2C+and+Asli+Celikyilmaz%2E+Walking+down+the+memory+maze%3A+Beyond+context+limit+through+interactive+reading%2E+arXiv+preprint+arXiv%3A+2310%2E05029%2C+2023%2E+%E2%80%94+%5BarXiv%3A23) 54. Jian Chen, Vashisth Tiwari, Ranajoy Sadhukhan, Zhuoming Chen, Jinyuan Shi, Ian En-Hsu Yen, and Beidi Chen. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. arXiv preprint arXiv:2408.11049, 2024. — [arXiv:2408.11049](https://arxiv.org/abs/2408.11049) [find](https://scholar.google.com/scholar?q=Jian+Chen%2C+Vashisth+Tiwari%2C+Ranajoy+Sadhukhan%2C+Zhuoming+Chen%2C+Jinyuan+Shi%2C+Ian+En%2DHsu+Yen%2C+and+Beidi+Chen%2E+Magicdec%3A+Breaking+the+latency%2Dthroughput+tradeoff+for+long+context+generation+with+speculati) 55. Jian Chen, Peilin Zhou, Yining Hua, Yingxin Loh, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. Fintextqa: A dataset for long-form financial question answering. arXiv preprint arXiv:2405.09980, 2024. [find](https://scholar.google.com/scholar?q=Jian+Chen%2C+Peilin+Zhou%2C+Yining+Hua%2C+Yingxin+Loh%2C+Kehui+Chen%2C+Ziyuan+Li%2C+Bing+Zhu%2C+and+Junwei+Liang%2E+Fintextqa%3A+A+dataset+for+long%2Dform+financial+question+answering%2E+arXiv+preprint+arXiv%3A2405%2E09980%2C+20) 56. Jianghao Chen, Junhong Wu, Yangyifan Xu, and Jiajun Zhang. Ladm: Long-context training data selection with attention-based dependency measurement for llms. 2025. URL [https://api.semanticscholar.org/CorpusID:276768135](https://api.semanticscholar.org/CorpusID:276768135) . [↗](https://api.semanticscholar.org/CorpusID:276768135) 57. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. BGE m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. CoRR, abs/2402.[03216, 2024](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/). doi: 10.48550/ARXIV.2402.03216. URL https://doi.org/10.48550/arXiv.2402… [↗](https://doi.org/10.48550/arXiv.2402…) 58. Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3. [https://github.com/Dee](https://github.com/Dee) p-Agent/R1-V , 2025. Accessed: 2025-02-02. [↗](https://github.com/Dee) 59. Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. [Alpagasus: Training a better alpaca with fewer data](https://md.research.elamur.ai/free/en/papers/chen-2023-alpagasus-training-better-alpaca-fewer/). In The Twelfth International Conference on Learning Representations, 2024. 60. Longze Chen, Ziqiang Liu, Wanwei He, Yinhe Zheng, Hao Sun, Yunshui Li, Run Luo, and Min Yang. Long context is not long at all: A prospector of long-dependency data for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Asso… [find](https://scholar.google.com/scholar?q=Longze+Chen%2C+Ziqiang+Liu%2C+Wanwei+He%2C+Yinhe+Zheng%2C+Hao+Sun%2C+Yunshui+Li%2C+Run+Luo%2C+and+Min+Yang%2E+Long+context+is+not+long+at+all%3A+A+prospector+of+long%2Ddependency+data+for+large+language+models%2E+In+Lun%2DWe) 61. Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. [Summscreen: A dataset for abstractive screenplay summarization](https://md.research.elamur.ai/free/en/papers/chen-2022-summscreen-dataset-abstractive-screenplay-summarization/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8602–8615, 2022. 62. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2C+2023%2E) 63. Stanley F Chen and Joshua Goodman. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359–394, 1999. [find](https://scholar.google.com/scholar?q=Stanley+F+Chen+and+Joshua+Goodman%2E+An+empirical+study+of+smoothing+techniques+for+language+modeling%2E+Computer+Speech+%26+Language%2C+13%284%29%3A359%E2%80%93394%2C+1999%2E) 64. Tong Chen, Hao Fang, Patrick Xia, Xiaodong Liu, Benjamin Van Durme, Luke Zettlemoyer, Jianfeng Gao, and Hao Cheng. Generative adapter: Contextualizing language models in parameters with a single forward pass. arXiv preprint arXiv: 2411.05877, 2024. — [arXiv:2411.05877](https://arxiv.org/abs/2411.05877) [find](https://scholar.google.com/scholar?q=Tong+Chen%2C+Hao+Fang%2C+Patrick+Xia%2C+Xiaodong+Liu%2C+Benjamin+Van+Durme%2C+Luke+Zettlemoyer%2C+Jianfeng+Gao%2C+and+Hao+Cheng%2E+Generative+adapter%3A+Contextualizing+language+models+in+parameters+with+a+single+forwa) 65. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.[21187, 2024](https://md.research.elamur.ai/free/en/papers/chen-2024-think-that-much-2-3/). 66. Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. Stuffed mamba: State collapse and state capacity of rnn-based long-context modeling. arXiv preprint arXiv:2410.07145, 2024. — [arXiv:2410.07145](https://arxiv.org/abs/2410.07145) [find](https://scholar.google.com/scholar?q=Yingfa+Chen%2C+Xinrong+Zhang%2C+Shengding+Hu%2C+Xu+Han%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Stuffed+mamba%3A+State+collapse+and+state+capacity+of+rnn%2Dbased+long%2Dcontext+modeling%2E+arXiv+preprint+arXiv%3A2410%2E07145%2C+20) 67. Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. Stuffed mamba: State collapse and state capacity of rnn-based long-context modeling, 2024. URL https://arxiv.org/abs/2410.07145 . — [arXiv:2410.07145](https://arxiv.org/abs/2410.07145) [↗](https://arxiv.org/abs/2410.07145) 68. Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, and Rui Yan. Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use. arXiv preprint arXiv:2312.04455, 2023. [find](https://scholar.google.com/scholar?q=Yuhan+Chen%2C+Ang+Lv%2C+Ting%2DEn+Lin%2C+Changyu+Chen%2C+Yuchuan+Wu%2C+Fei+Huang%2C+Yongbin+Li%2C+and+Rui+Yan%2E+Fortify+the+shortest+stave+in+attention%3A+Enhancing+context+awareness+of+large+language+models+for+effecti) 69. Yuhan Chen, Ang Lv, Jian Luan, Bin Wang, and Wei Liu. Hope: A novel positional encoding without long-term decay for enhanced context awareness and extrapolation. arXiv preprint arXiv:2410.21216, 2024. [find](https://scholar.google.com/scholar?q=Yuhan+Chen%2C+Ang+Lv%2C+Jian+Luan%2C+Bin+Wang%2C+and+Wei+Liu%2E+Hope%3A+A+novel+positional+encoding+without+long%2Dterm+decay+for+enhanced+context+awareness+and+extrapolation%2E+arXiv+preprint+arXiv%3A2410%2E21216%2C+2024%2E) 70. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). arXiv preprint arXiv:2309.12307, 2023. 71. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). In The International Conference on Learning Representations (ICLR), 2024. 72. [Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng W…](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/) 73. Zhi Chen, Qiguang Chen, Libo Qin, Qipeng Guo, Haijun Lv, Yicheng Zou, Wanxiang Che, Hang Yan, Kai Chen, and Dahua Lin. What are the essential factors in crafting effective long context multi-hop instruction datasets? insights and best practices. ArXiv, abs/2409.01893, 2024. [find](https://scholar.google.com/scholar?q=Zhi+Chen%2C+Qiguang+Chen%2C+Libo+Qin%2C+Qipeng+Guo%2C+Haijun+Lv%2C+Yicheng+Zou%2C+Wanxiang+Che%2C+Hang+Yan%2C+Kai+Chen%2C+and+Dahua+Lin%2E+What+are+the+essential+factors+in+crafting+effective+long+context+multi%2Dhop+instr) 74. Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. xRAG: Extreme context compression for retrieval-augmented generation with one token. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL [https://openreview.ne…](https://openreview.ne…) [↗](https://openreview.ne…) 75. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. [Adapting language models to compress contexts](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/). In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3829–3846, Singapore, December 2023.… 76. Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386–8399, 2022. [find](https://scholar.google.com/scholar?q=Ta%2DChung+Chi%2C+Ting%2DHan+Fan%2C+Peter+J+Ramadge%2C+and+Alexander+Rudnicky%2E+Kerple%3A+Kernelized+relative+positional+embedding+for+length+extrapolation%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A83) 77. Ta-Chung Chi, Ting-Han Fan, Alexander I Rudnicky, and Peter J Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. arXiv preprint arXiv:2212.10356, 2022. [find](https://scholar.google.com/scholar?q=Ta%2DChung+Chi%2C+Ting%2DHan+Fan%2C+Alexander+I+Rudnicky%2C+and+Peter+J+Ramadge%2E+Dissecting+transformer+length+extrapolation+via+the+lens+of+receptive+field+analysis%2E+arXiv+preprint+arXiv%3A2212%2E10356%2C+2022%2E) 78. Ta-Chung Chi, Ting-Han Fan, Li-Wei Chen, Alexander I Rudnicky, and Peter J Ramadge. Latent positional information is in the self-attention variance of transformer language models without positional embeddings. arXiv preprint arXiv:2305.13571, 2023. [find](https://scholar.google.com/scholar?q=Ta%2DChung+Chi%2C+Ting%2DHan+Fan%2C+Li%2DWei+Chen%2C+Alexander+I+Rudnicky%2C+and+Peter+J+Ramadge%2E+Latent+positional+information+is+in+the+self%2Dattention+variance+of+transformer+language+models+without+positional+em) 79. Yew Ken Chia, Liying Cheng, Hou Pong Chan, Chaoqun Liu, Maojia Song, Sharifah Mahani Aljunied, Soujanya Poria, and Lidong Bing. M-longdoc: A benchmark for multimodal super-long document understanding and A retrieval-aware tuning framework. CoRR, abs/2411.06176, 2024. doi: 10.48550/ARXIV.2411.06176.… [find](https://scholar.google.com/scholar?q=Yew+Ken+Chia%2C+Liying+Cheng%2C+Hou+Pong+Chan%2C+Chaoqun+Liu%2C+Maojia+Song%2C+Sharifah+Mahani+Aljunied%2C+Soujanya+Poria%2C+and+Lidong+Bing%2E+M%2Dlongdoc%3A+A+benchmark+for+multimodal+super%2Dlong+document+understanding+) 80. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, 2019. URL https://arxiv.org/abs/1904.10509 . [↗](https://arxiv.org/abs/1904.10509) 81. Sai Sena Chinnakonduru and Astarag Mohapatra. Weighted grouped query attention in transformers, 2024. URL https://arxiv.org/abs/2407.10855 . — [arXiv:2407.10855](https://arxiv.org/abs/2407.10855) [↗](https://arxiv.org/abs/2407.10855) 82. Woon Sang Cho, Pengchuan Zhang, Yizhe Zhang, Xiujun Li, Michel Galley, Chris Brockett, Mengdi Wang, and Jianfeng Gao. Towards coherent and cohesive long-form text generation. arXiv preprint arXiv:1811.00511, 2018. [find](https://scholar.google.com/scholar?q=Woon+Sang+Cho%2C+Pengchuan+Zhang%2C+Yizhe+Zhang%2C+Xiujun+Li%2C+Michel+Galley%2C+Chris+Brockett%2C+Mengdi+Wang%2C+and+Jianfeng+Gao%2E+Towards+coherent+and+cohesive+long%2Dform+text+generation%2E+arXiv+preprint+arXiv%3A1811) 83. Daewon Choi, Seunghyuk Oh, Saket Dingliwal, Jihoon Tack, Kyuyoung Kim, Woomin Song, Seojin Kim, Insu Han, Jinwoo Shin, Aram Galstyan, et al. Mamba drafters for speculative decoding. arXiv preprint arXiv:2506.01206, 2025. [find](https://scholar.google.com/scholar?q=Daewon+Choi%2C+Seunghyuk+Oh%2C+Saket+Dingliwal%2C+Jihoon+Tack%2C+Kyuyoung+Kim%2C+Woomin+Song%2C+Seojin+Kim%2C+Insu+Han%2C+Jinwoo+Shin%2C+Aram+Galstyan%2C+et+al%2E+Mamba+drafters+for+speculative+decoding%2E+arXiv+preprint+arX) 84. Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In ICLR. OpenReview.net, 2… [find](https://scholar.google.com/scholar?q=Krzysztof+Marcin+Choromanski%2C+Valerii+Likhosherstov%2C+David+Dohan%2C+Xingyou+Song%2C+Andreea+Gane%2C+Tam%C3%A1s+Sarl%C3%B3s%2C+Peter+Hawkins%2C+Jared+Quincy+Davis%2C+Afroz+Mohiuddin%2C+Lukasz+Kaiser%2C+David+Benjamin+Belanger%2C+) 85. [Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, E…](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/) 86. Yu-Neng Chuang, Tianwei Xing, Chia-Yuan Chang, Zirui Liu, Xun Chen, and Xia Hu. Learning to compress prompt in natural language formats. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Yu%2DNeng+Chuang%2C+Tianwei+Xing%2C+Chia%2DYuan+Chang%2C+Zirui+Liu%2C+Xun+Chen%2C+and+Xia+Hu%2E+Learning+to+compress+prompt+in+natural+language+formats%2E+In+Kevin+Duh%2C+Helena+Gomez%2C+and+Steven+Bethard%2C+editors%2C+Procee) 87. Igor Churin, Murat Apishev, Maria Tikhonova, Denis Shevelev, Aydar Bulatov, Yuri Kuratov, Sergej Averkiev, and Alena Fenogenova. Long input benchmark for russian analysis. arXiv preprint arXiv:2408.02439, 2024. — [arXiv:2408.02439](https://arxiv.org/abs/2408.02439) [find](https://scholar.google.com/scholar?q=Igor+Churin%2C+Murat+Apishev%2C+Maria+Tikhonova%2C+Denis+Shevelev%2C+Aydar+Bulatov%2C+Yuri+Kuratov%2C+Sergej+Averkiev%2C+and+Alena+Fenogenova%2E+Long+input+benchmark+for+russian+analysis%2E+arXiv+preprint+arXiv%3A2408%2E02) 88. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021. 89. Cohere and Cohere For AI. C4ai command r7b: A 7 billion parameter multilingual model, December 2024. URL [https://huggingface.co/CohereForAI/c4ai-command-r7b](https://huggingface.co/CohereForAI/c4ai-command-r7b) -12-2024 . Access requires agreement to the License Agreement and adherence to C4AI’s Acceptable Use Policy. [↗](https://huggingface.co/CohereForAI/c4ai-command-r7b) 90. Marta R Costa-jussà, Pierre Andrews, Mariano Coria Meglioli, Joy Chen, Joe Chuang, David Dale, Christophe Ropers, Alexandre Mourachko, Eduardo Sánchez, Holger Schwenk, et al. Lcfo: Long context and long form output dataset and benchmarking. arXiv preprint arXiv:2412.08268, 2024. [find](https://scholar.google.com/scholar?q=Marta+R+Costa%2Djuss%C3%A0%2C+Pierre+Andrews%2C+Mariano+Coria+Meglioli%2C+Joy+Chen%2C+Joe+Chuang%2C+David+Dale%2C+Christophe+Ropers%2C+Alexandre+Mourachko%2C+Eduardo+S%C3%A1nchez%2C+Holger+Schwenk%2C+et+al%2E+Lcfo%3A+Long+context+and+lo) 91. Hui Dai, Dan Pechi, Xinyi Yang, Garvit Banga, and Raghav Mantri. Deniahl: In-context features influence llm needle-in-a-haystack abilities. arXiv preprint arXiv:2411.19360, 2024. — [arXiv:2411.19360](https://arxiv.org/abs/2411.19360) [find](https://scholar.google.com/scholar?q=Hui+Dai%2C+Dan+Pechi%2C+Xinyi+Yang%2C+Garvit+Banga%2C+and+Raghav+Mantri%2E+Deniahl%3A+In%2Dcontext+features+influence+llm+needle%2Din%2Da%2Dhaystack+abilities%2E+arXiv+preprint+arXiv%3A2411%2E19360%2C+2024%2E+%E2%80%94+%5BarXiv%3A2411%2E19360%5D%28) 92. Jincheng Dai, Zhuowei Huang, Haiyun Jiang, Chen Chen, Deng Cai, Wei Bi, and Shuming Shi. Corm: Cache optimization with recent message for large language model inference, 2024. URL https://arxiv.org/abs/2404.15949 . [↗](https://arxiv.org/abs/2404.15949) 93. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL (1), pages 2978–2988. Association for Computational Linguistics, 2019. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+G%2E+Carbonell%2C+Quoc+Viet+Le%2C+and+Ruslan+Salakhutdinov%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+ACL+%281%29%2C+pages+2978%E2%80%932988%2E+) 94. Bhavana Dalvi Mishra, Oyvind Tafjord, and Peter Clark. Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Langua… [find](https://scholar.google.com/scholar?q=Bhavana+Dalvi+Mishra%2C+Oyvind+Tafjord%2C+and+Peter+Clark%2E+Towards+teachable+reasoning+systems%3A+Using+a+dynamic+memory+of+user+feedback+for+continual+system+improvement%2E+In+Yoav+Goldberg%2C+Zornitsa+Kozarev) 95. Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Tri+Dao%2E+Flashattention%2D2%3A+Faster+attention+with+better+parallelism+and+work+partitioning%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2024%2E) 96. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344–16359, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Dan+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+Advances+in+neural+information+processing+systems%2C+35%3A16344) 97. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. [A dataset of information-seeking questions and answers anchored in research papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T… 98. DeepSeek. 3fs: Fire-flyer file system. [https://github.com/deepseek-ai/3FS](https://github.com/deepseek-ai/3FS) , 2025. [↗](https://github.com/deepseek-ai/3FS) 99. DeepSeek-AI. Deepseek-v3 technical report, 2024. URL https://arxiv.org/abs/2412 .19437 . [↗](https://arxiv.org/abs/2412) 100. Mostafa Dehghani, Stephan Gouws, O. Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. International Conference on Learning Representations, 2018. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Stephan+Gouws%2C+O%2E+Vinyals%2C+Jakob+Uszkoreit%2C+and+Lukasz+Kaiser%2E+Universal+transformers%2E+International+Conference+on+Learning+Representations%2C+2018%2E) 101. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+Qlora%3A+Efficient+finetuning+of+quantized+llms%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E) 102. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In North American Chapter of the Association for Computational Linguistics, 2019. 103. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Co… 104. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. [Longrope: Extending llm context window beyond 2 million tokens](https://md.research.elamur.ai/free/en/papers/ding-2024-longrope-extending-llm-context/). arXiv preprint arXiv:2402.13753, 2024. 105. Jianbo Dong, Zheng Cao, Tao Zhang, Jianxi Ye, Shaochuang Wang, Fei Feng, Li Zhao, Xiaoyong Liu, Liuyihan Song, Liwei Peng, et al. Eflops: Algorithm and system co-design for a high performance distributed training platform. In 2020 IEEE International Symposium on High Performance Computer Architectur… [find](https://scholar.google.com/scholar?q=Jianbo+Dong%2C+Zheng+Cao%2C+Tao+Zhang%2C+Jianxi+Ye%2C+Shaochuang+Wang%2C+Fei+Feng%2C+Li+Zhao%2C+Xiaoyong+Liu%2C+Liuyihan+Song%2C+Liwei+Peng%2C+et+al%2E+Eflops%3A+Algorithm+and+system+co%2Ddesign+for+a+high+performance+distribu) 106. Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. Qaq: Quality adaptive quantization for llm kv cache, 2024. URL https://arxiv.org/abs/2403.04643 . — [arXiv:2403.04643](https://arxiv.org/abs/2403.04643) [↗](https://arxiv.org/abs/2403.04643) 107. Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, et al. Hymba: A hybrid-head architecture for small language models. arXiv preprint arXiv:2411.13676, 2024. [find](https://scholar.google.com/scholar?q=Xin+Dong%2C+Yonggan+Fu%2C+Shizhe+Diao%2C+Wonmin+Byeon%2C+Zijia+Chen%2C+Ameya+Sunil+Mahabaleshwarkar%2C+Shih%2DYang+Liu%2C+Matthijs+Van+Keirsbilck%2C+Min%2DHung+Chen%2C+Yoshi+Suhara%2C+et+al%2E+Hymba%3A+A+hybrid%2Dhead+architecture) 108. Zican Dong, Tianyi Tang, Junyi Li, and Wayne Xin Zhao. A survey on long text modeling with transformers. CoRR, abs/2302.14502, 2023. doi: 10.48550/ARXIV.2302.14502. URL https://doi.org/10.48550/arXiv.2302.14502 . — [doi:10.48550/arxiv.2302.14502](https://doi.org/10.48550/arxiv.2302.14502) [↗](https://doi.org/10.48550/arXiv.2302.14502) 109. Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. [Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models](https://md.research.elamur.ai/free/en/papers/dong-2024-bamboo-comprehensive-benchmark-evaluating-long/). arXiv preprint arXiv:2309.13345, 2023. 110. Longxu Dou, Qian Liu, Fan Zhou, Changyu Chen, Zili Wang, Ziqi Jin, Zichen Liu, Tongyao Zhu, Cunxiao Du, Penghui Yang, Haonan Wang, Jiaheng Liu, Yongchi Zhao, Xiachong Feng, Xin Mao, Man Tsung Yeung, Kunat Pipatanakul, Fajri Koto, Min Si Thu, Hynek Kydlíček, Zeyi Liu, Qunshu Lin, Sittipong Sripaisar… — [arXiv:2502.12982](https://arxiv.org/abs/2502.12982) [find](https://scholar.google.com/scholar?q=Longxu+Dou%2C+Qian+Liu%2C+Fan+Zhou%2C+Changyu+Chen%2C+Zili+Wang%2C+Ziqi+Jin%2C+Zichen+Liu%2C+Tongyao+Zhu%2C+Cunxiao+Du%2C+Penghui+Yang%2C+Haonan+Wang%2C+Jiaheng+Liu%2C+Yongchi+Zhao%2C+Xiachong+Feng%2C+Xin+Mao%2C+Man+Tsung+Yeung%2C+K) 111. [Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern, Toju Duke, Lu…](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/) 112. [Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern, Toju Duke,…](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/) 113. Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, and Ji-Rong Wen. Towards event-oriented long video understanding. CoRR, abs/2406.14129, 2024. doi: 10.48550/ARXIV.2406.14129. URL https://doi.org/ 10.48550/arXiv.2406.14129 . — [doi:10.48550/arxiv.2406.14129](https://doi.org/10.48550/arxiv.2406.14129) [find](https://scholar.google.com/scholar?q=Yifan+Du%2C+Kun+Zhou%2C+Yuqi+Huo%2C+Yifan+Li%2C+Wayne+Xin+Zhao%2C+Haoyu+Lu%2C+Zijia+Zhao%2C+Bingning+Wang%2C+Weipeng+Chen%2C+and+Ji%2DRong+Wen%2E+Towards+event%2Doriented+long+video+understanding%2E+CoRR%2C+abs%2F2406%2E14129%2C+2024%2E) 114. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. [GLM: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Com… 115. Jiangfei Duan, Shuo Zhang, Zerui Wang, Lijuan Jiang, Wenwen Qu, Qinghao Hu, Guoteng Wang, Qizhen Weng, Hang Yan, Xingcheng Zhang, et al. Efficient training of large language models on distributed infrastructures: a survey. arXiv preprint arXiv:2407.20018, 2024. [find](https://scholar.google.com/scholar?q=Jiangfei+Duan%2C+Shuo+Zhang%2C+Zerui+Wang%2C+Lijuan+Jiang%2C+Wenwen+Qu%2C+Qinghao+Hu%2C+Guoteng+Wang%2C+Qizhen+Weng%2C+Hang+Yan%2C+Xingcheng+Zhang%2C+et+al%2E+Efficient+training+of+large+language+models+on+distributed+infr) 116. Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng ZHANG, and Dahua Lin. Skvq: Sliding-window key and value cache quantization for large language models. In First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Haojie+Duanmu%2C+Zhihang+Yuan%2C+Xiuhong+Li%2C+Jiangfei+Duan%2C+Xingcheng+ZHANG%2C+and+Dahua+Lin%2E+Skvq%3A+Sliding%2Dwindow+key+and+value+cache+quantization+for+large+language+models%2E+In+First+Conference+on+Language) 117. Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. [LayerSkip: Enabling early exit inference and self-speculative decoding](https://md.research.elamur.ai/free/en/papers/elhoushi-2024-layerskip-enabling-early-exit-inference/). In Lun-Wei Ku, Andre Martins, and… 118. Alexander R Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. [Qafacteval: Improved qa-based factual consistency evaluation for summarization](https://md.research.elamur.ai/free/en/papers/fabbri-2022-qafacteval-improved-qa-based-factual/). arXiv preprint arXiv:2112.08542, 2021. 119. Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. [Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model](https://md.research.elamur.ai/free/en/papers/fabbri-2019-multi-news-large-scale-multi/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, 2019. 120. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. [Eli5: Long form question answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/). arXiv preprint arXiv:1907.09190, 2019. 121. Yongqi Fan, Hongli Sun, Kui Xue, Xiaofan Zhang, Shaoting Zhang, and Tong Ruan. Medodyssey: A medical domain benchmark for long context evaluation up to 200k tokens. CoRR, abs/2406.15019, 2024. doi: 10.48550/ARXIV.2406.15019. URL https://doi.org/10.48550/arXiv.2406.15019 . [↗](https://doi.org/10.48550/arXiv.2406.15019) 122. Yongqi Fan, Hongli Sun, Kui Xue, Xiaofan Zhang, Shaoting Zhang, and Tong Ruan. Medodyssey: A medical domain benchmark for long context evaluation up to 200k tokens. arXiv preprint arXiv:2406.15019, 2024. [find](https://scholar.google.com/scholar?q=Yongqi+Fan%2C+Hongli+Sun%2C+Kui+Xue%2C+Xiaofan+Zhang%2C+Shaoting+Zhang%2C+and+Tong+Ruan%2E+Medodyssey%3A+A+medical+domain+benchmark+for+long+context+evaluation+up+to+200k+tokens%2E+arXiv+preprint+arXiv%3A2406%2E15019%2C+20) 123. Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.23771, 2024. — [arXiv:2410.23771](https://arxiv.org/abs/2410.23771) [find](https://scholar.google.com/scholar?q=Lizhe+Fang%2C+Yifei+Wang%2C+Zhaoyang+Liu%2C+Chenheng+Zhang%2C+Stefanie+Jegelka%2C+Jinyang+Gao%2C+Bolin+Ding%2C+and+Yisen+Wang%2E+What+is+wrong+with+perplexity+for+long%2Dcontext+language+modeling%3F+arXiv+preprint+arXiv%3A) 124. Yan Fang, Jingtao Zhan, Qingyao Ai, Jiaxin Mao, Weihang Su, Jia Chen, and Yiqun Liu. Scaling laws for dense retrieval. arXiv preprint arXiv: 2403.18684, 2024. — [arXiv:2403.18684](https://arxiv.org/abs/2403.18684) [find](https://scholar.google.com/scholar?q=Yan+Fang%2C+Jingtao+Zhan%2C+Qingyao+Ai%2C+Jiaxin+Mao%2C+Weihang+Su%2C+Jia+Chen%2C+and+Yiqun+Liu%2E+Scaling+laws+for+dense+retrieval%2E+arXiv+preprint+arXiv%3A+2403%2E18684%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E18684%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab) 125. Yunhao Fang, Weihao Yu, Shu Zhong, Qinghao Ye, Xuehan Xiong, and Lai Wei. Artificial hippocampus networks for efficient long-context modeling. arXiv preprint arXiv:2510.07318, 2025. — [arXiv:2510.07318](https://arxiv.org/abs/2510.07318) [find](https://scholar.google.com/scholar?q=Yunhao+Fang%2C+Weihao+Yu%2C+Shu+Zhong%2C+Qinghao+Ye%2C+Xuehan+Xiong%2C+and+Lai+Wei%2E+Artificial+hippocampus+networks+for+efficient+long%2Dcontext+modeling%2E+arXiv+preprint+arXiv%3A2510%2E07318%2C+2025%2E+%E2%80%94+%5BarXiv%3A2510%2E0731) 126. Gueter Josmy Faure, Jia-Fong Yeh, Min-Hung Chen, Hung-Ting Su, Shang-Hong Lai, and Winston H. Hsu. Hermes: temporal-coherent long-form understanding with episodes and semantics. CoRR, abs/2408.17443, 2024. doi: 10.48550/ARXIV.2408.17443. URL https://doi.org/10.48550/arXiv.2408.17443 . — [doi:10.48550/arxiv.2408.17443](https://doi.org/10.48550/arxiv.2408.17443) [↗](https://doi.org/10.48550/arXiv.2408.17443) 127. Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference, 2025. URL https://arxiv.org/abs/2407.11550 . — [arXiv:2407.11550](https://arxiv.org/abs/2407.11550) [↗](https://arxiv.org/abs/2407.11550) 128. Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Najibi. Lazyllm: Dynamic token pruning for efficient long context llm inference, 2024. URL https://arxiv.org/abs/2407.14057 . — [arXiv:2407.14057](https://arxiv.org/abs/2407.14057) [↗](https://arxiv.org/abs/2407.14057) 129. Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Moa: Mixture of sparse attention for automatic large language model compression, 2024. URL https://arxiv.org/abs/2406.14909 . [↗](https://arxiv.org/abs/2406.14909) 130. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hanna Hajishirzi, Yoon Kim, and Hao Peng. [Data engineering for scaling language models to 128k context](https://md.research.elamur.ai/free/en/papers/fu-2024-data-engineering-scaling-language-models/). ArXiv, abs/2402.10171, 2024. 131. Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning, 2024. URL https://arxiv.org/abs/2410.19258 . — [arXiv:2410.19258](https://arxiv.org/abs/2410.19258) [↗](https://arxiv.org/abs/2410.19258) 132. Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level KV cache compression method with integrated retrieval and reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL [https://openreview.net/forum?id=FJFVmeXusW…](https://openreview.net/forum?id=FJFVmeXusW…) [↗](https://openreview.net/forum?id=FJFVmeXusW…) 133. Chaochen Gao, Xing Wu, Qingfang Fu, and Songlin Hu. Quest: Query-centric data synthesis approach for long-context scaling of large language model. ArXiv, abs/2405.19846, 2024. — [arXiv:2405.19846](https://arxiv.org/abs/2405.19846) [find](https://scholar.google.com/scholar?q=Chaochen+Gao%2C+Xing+Wu%2C+Qingfang+Fu%2C+and+Songlin+Hu%2E+Quest%3A+Query%2Dcentric+data+synthesis+approach+for+long%2Dcontext+scaling+of+large+language+model%2E+ArXiv%2C+abs%2F2405%2E19846%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E19846%5D%28htt) 134. Jun Gao, Ziqiang Cao, and Wenjie Li. Unifying demonstration selection and compression for in-context learning, 2024. URL https://arxiv.org/abs/2405.17062 . — [arXiv:2405.17062](https://arxiv.org/abs/2405.17062) [↗](https://arxiv.org/abs/2405.17062) 135. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. ArXiv, abs/2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+The+pile%3A+An+800gb+dataset+of+) 136. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. [Precise zero-shot dense retrieval without relevance labels](https://md.research.elamur.ai/free/en/papers/gao-2022-precise-zero-shot-dense-retrieval/). arXiv preprint arXiv: 2212.10496, 2022. 137. Shen Gao, Xiuying Chen, Piji Li, Zhaochun Ren, Lidong Bing, Dongyan Zhao, and Rui Yan. Abstractive text summarization by incorporating reader comments. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Confe… — [doi:10.1609/aaai.v33i01.33016399](https://doi.org/10.1609/aaai.v33i01.33016399) [find](https://scholar.google.com/scholar?q=Shen+Gao%2C+Xiuying+Chen%2C+Piji+Li%2C+Zhaochun+Ren%2C+Lidong+Bing%2C+Dongyan+Zhao%2C+and+Rui+Yan%2E+Abstractive+text+summarization+by+incorporating+reader+comments%2E+In+The+Thirty%2DThird+AAAI+Conference+on+Artificia) 138. Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. [How to train long-context language models (effectively)](https://md.research.elamur.ai/free/en/papers/gao-2025-how-train-long-context-language/). arXiv preprint arXiv:2410.02660, 2024. 139. Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Hayden Kwok-Hay So, Ting Cao, Fan Yang, and Mao Yang. Seerattention: Learning intrinsic sparse attention in your llms, 2024. URL https://arxiv.org/abs/2410.13276 . [↗](https://arxiv.org/abs/2410.13276) 140. Shawn Gavin, Tuney Zheng, Jiaheng Liu, Quehry Que, Noah Wang, Jian Yang, Chenchen Zhang, Wenhao Huang, Wenhu Chen, and Ge Zhang. Longins: A challenging long-context instruction-based exam for llms. arXiv preprint arXiv:2406.17588, 2024. — [arXiv:2406.17588](https://arxiv.org/abs/2406.17588) [find](https://scholar.google.com/scholar?q=Shawn+Gavin%2C+Tuney+Zheng%2C+Jiaheng+Liu%2C+Quehry+Que%2C+Noah+Wang%2C+Jian+Yang%2C+Chenchen+Zhang%2C+Wenhao+Huang%2C+Wenhu+Chen%2C+and+Ge+Zhang%2E+Longins%3A+A+challenging+long%2Dcontext+instruction%2Dbased+exam+for+llms%2E+ar) 141. Junqi Ge, Ziyi Chen, Jintao Lin, Jinguo Zhu, Xihui Liu, Jifeng Dai, and Xizhou Zhu. V2pe: Improving multimodal long-context capability of vision-language models with variable visual position encoding. CoRR, abs/2412.09616, 2024. doi: 10.48550/ARXIV.2412.09616. URL https://doi.org/10.48550/arXiv.2412… — [doi:10.48550/arxiv.2412.09616](https://doi.org/10.48550/arxiv.2412.09616) [↗](https://doi.org/10.48550/arXiv.2412…) 142. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. [Model tells you what to discard: Adaptive KV cache compression for LLMs](https://md.research.elamur.ai/free/en/papers/ge-2023-model-tells-discard-adaptive-kv/). In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=uNrFpDPMyo](https://openreview.net/forum?id=uNrFpDPMyo) . [↗](https://openreview.net/forum?id=uNrFpDPMyo) 143. Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=uREj4ZuG](https://openreview.net/forum?id=uREj4ZuG) JE . [↗](https://openreview.net/forum?id=uREj4ZuG) 144. Yuan Ge, Yilun Liu, Chi Hu, Weibin Meng, Shimin Tao, Xiaofeng Zhao, Mahong Xia, Zhang Li, Boxing Chen, Hao Yang, Bei Li, Tong Xiao, and JingBo Zhu. Clustering and ranking: Diversity-preserved instruction selection through expert-aligned quality estimation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-… [find](https://scholar.google.com/scholar?q=Yuan+Ge%2C+Yilun+Liu%2C+Chi+Hu%2C+Weibin+Meng%2C+Shimin+Tao%2C+Xiaofeng+Zhao%2C+Mahong+Xia%2C+Zhang+Li%2C+Boxing+Chen%2C+Hao+Yang%2C+Bei+Li%2C+Tong+Xiao%2C+and+JingBo+Zhu%2E+Clustering+and+ranking%3A+Diversity%2Dpreserved+instruct) 145. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. [Convolutional sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). In International conference on machine learning, pages 1243–1252. PMLR, 2017. 146. Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos. CoRR, abs/2411.19772, 2024. doi: 10.48550/ARXIV.2411.19772. URL https://doi.org/10.48550/arXiv.2411.19772 . 149… — [doi:10.48550/arxiv.2411.19772](https://doi.org/10.48550/arxiv.2411.19772) [↗](https://doi.org/10.48550/arXiv.2411.19772) 147. Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W Mahoney, and Kurt Keutzer. Ai and memory wall. IEEE Micro, 2024. [find](https://scholar.google.com/scholar?q=Amir+Gholami%2C+Zhewei+Yao%2C+Sehoon+Kim%2C+Coleman+Hooper%2C+Michael+W+Mahoney%2C+and+Kurt+Keutzer%2E+Ai+and+memory+wall%2E+IEEE+Micro%2C+2024%2E) 148. GitHub. Github copilot, 2022. URL [https://github.com/copilot](https://github.com/copilot) . [↗](https://github.com/copilot) 149. [Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinka…](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/) 150. Paolo Glorioso, Quentin Anthony, Yury Tokpanov, Anna Golubeva, Vasudev Shyam, James Whittington, Jonathan Pilault, and Beren Millidge. The zamba2 suite: Technical report. arXiv preprint arXiv:2411.15242, 2024. — [arXiv:2411.15242](https://arxiv.org/abs/2411.15242) [find](https://scholar.google.com/scholar?q=Paolo+Glorioso%2C+Quentin+Anthony%2C+Yury+Tokpanov%2C+Anna+Golubeva%2C+Vasudev+Shyam%2C+James+Whittington%2C+Jonathan+Pilault%2C+and+Beren+Millidge%2E+The+zamba2+suite%3A+Technical+report%2E+arXiv+preprint+arXiv%3A2411%2E152) 151. Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b ssm hybrid model. arXiv preprint arXiv:2405.16712, 2024. [find](https://scholar.google.com/scholar?q=Paolo+Glorioso%2C+Quentin+Anthony%2C+Yury+Tokpanov%2C+James+Whittington%2C+Jonathan+Pilault%2C+Adam+Ibrahim%2C+and+Beren+Millidge%2E+Zamba%3A+A+compact+7b+ssm+hybrid+model%2E+arXiv+preprint+arXiv%3A2405%2E16712%2C+2024%2E) 152. Aditi S. Godbole, Jabin Geevarghese George, and Smita Shandilya. Leveraging long-context large language models for multi-document understanding and summarization in enterprise applications. CoRR, abs/2409.18454, 2024. doi: 10.48550/ARXIV.2409.18454. URL https://doi.org/10.48550/arXiv.2409.18454 . — [doi:10.48550/arxiv.2409.18454](https://doi.org/10.48550/arxiv.2409.18454) [↗](https://doi.org/10.48550/arXiv.2409.18454) 153. Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. Is it really long context if all you need is retrieval? towards genuinely difficult long context nlp. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16576–16586, 2… [find](https://scholar.google.com/scholar?q=Omer+Goldman%2C+Alon+Jacovi%2C+Aviv+Slobodkin%2C+Aviya+Maimon%2C+Ido+Dagan%2C+and+Reut+Tsarfaty%2E+Is+it+really+long+context+if+all+you+need+is+retrieval%3F+towards+genuinely+difficult+long+context+nlp%2E+In+Proceedi) 154. Daniel Goldstein, Fares Obeid, Eric Alcaide, Guangyu Song, and Eugene Cheah. Goldfinch: High performance rwkv/transformer hybrid with linear pre-fill and extreme kv-cache compression. arXiv preprint arXiv:2407.12077, 2024. — [arXiv:2407.12077](https://arxiv.org/abs/2407.12077) [find](https://scholar.google.com/scholar?q=Daniel+Goldstein%2C+Fares+Obeid%2C+Eric+Alcaide%2C+Guangyu+Song%2C+and+Eugene+Cheah%2E+Goldfinch%3A+High+performance+rwkv%2Ftransformer+hybrid+with+linear+pre%2Dfill+and+extreme+kv%2Dcache+compression%2E+arXiv+preprint+a) 155. Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Contextual position encoding: Learning to count what’s important. arXiv preprint arXiv:2405.18719, 2024. — [arXiv:2405.18719](https://arxiv.org/abs/2405.18719) [find](https://scholar.google.com/scholar?q=Olga+Golovneva%2C+Tianlu+Wang%2C+Jason+Weston%2C+and+Sainbayar+Sukhbaatar%2E+Contextual+position+encoding%3A+Learning+to+count+what%E2%80%99s+important%2E+arXiv+preprint+arXiv%3A2405%2E18719%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E18719%5D%28https) 156. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [find](https://scholar.google.com/scholar?q=Aaron+Grattafiori%2C+Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Alex+Vaughan%2C+et+al%2E+The+llama+3+herd+of+models%2E+arXiv+) 157. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. [find](https://scholar.google.com/scholar?q=Albert+Gu+and+Tri+Dao%2E+Mamba%3A+Linear%2Dtime+sequence+modeling+with+selective+state+spaces%2E+arXiv+preprint+arXiv%3A2312%2E00752%2C+2023%2E) 158. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL [https://openreview.net/forum](https://openreview.net/forum) ?id=tEYskw1VY2 . [↗](https://openreview.net/forum) 159. Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. In NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Albert+Gu%2C+Tri+Dao%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+Hippo%3A+Recurrent+memory+with+optimal+polynomial+projections%2E+In+NeurIPS%2C+2020%2E) 160. Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In NeurIPS, pages 572–585, 2021. [find](https://scholar.google.com/scholar?q=Albert+Gu%2C+Isys+Johnson%2C+Karan+Goel%2C+Khaled+Saab%2C+Tri+Dao%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+Combining+recurrent%2C+convolutional%2C+and+continuous%2Dtime+models+with+linear+state+space+layers%2E+In+NeurIPS%2C+pa) 161. Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learning Representations (ICLR), 2022. [find](https://scholar.google.com/scholar?q=Albert+Gu%2C+Karan+Goel%2C+and+Christopher+R%C3%A9%2E+Efficiently+modeling+long+sequences+with+structured+state+spaces%2E+In+The+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2022%2E) 162. Jian Guan, Zhuoer Feng, Yamei Chen, Ruilin He, Xiaoxi Mao, Changjie Fan, and Minlie Huang. Lot: A story-centric benchmark for evaluating chinese long text understanding and generation. Transactions of the Association for Computational Linguistics, 10:434–451, 2022. [find](https://scholar.google.com/scholar?q=Jian+Guan%2C+Zhuoer+Feng%2C+Yamei+Chen%2C+Ruilin+He%2C+Xiaoxi+Mao%2C+Changjie+Fan%2C+and+Minlie+Huang%2E+Lot%3A+A+story%2Dcentric+benchmark+for+evaluating+chinese+long+text+understanding+and+generation%2E+Transactions+of) 163. Ziyi Guan, Hantao Huang, Yupeng Su, Hong Huang, Ngai Wong, and Hao Yu. Aptq: Attention-aware post-training mixed-precision quantization for large language models. In Proceedings of the 61st ACM/IEEE Design Automation Conference, pages 1–6, 2024. [find](https://scholar.google.com/scholar?q=Ziyi+Guan%2C+Hantao+Huang%2C+Yupeng+Su%2C+Hong+Huang%2C+Ngai+Wong%2C+and+Hao+Yu%2E+Aptq%3A+Attention%2Daware+post%2Dtraining+mixed%2Dprecision+quantization+for+large+language+models%2E+In+Proceedings+of+the+61st+ACM%2FIEEE+D) 164. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi… [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+C%C3%A9sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+Adil+Salim%2C+Shital+Shah%2C+Harki) 165. Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao. [Jina embeddings 2: 8192-token general-purpose text embeddings for long documents](https://md.research.elamur.ai/free/en/papers/gunther-2023-jina-embeddings-2-8192-token/). CoRR, abs/2310.199… 166. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948, 2025. 167. Mandy Guo, Joshua Ainslie, David C. Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. [Longt5: Efficient text-to-text transformer for long sequences](https://md.research.elamur.ai/free/en/papers/guo-2022-longt5-efficient-text-text-transformer/). In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz, editors, Findings of the Association for Computational… 168. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: retrieval-augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020. [find](https://scholar.google.com/scholar?q=Kelvin+Guu%2C+Kenton+Lee%2C+Zora+Tung%2C+Panupong+Pasupat%2C+and+Ming%2DWei+Chang%2E+Realm%3A+retrieval%2Daugmented+language+model+pre%2Dtraining%2E+In+Proceedings+of+the+37th+International+Conference+on+Machine+Learning) 169. Michael Günther, Isabelle Mohr, Daniel James Williams, Bo Wang, and Han Xiao. Late chunking: Contextual chunk embeddings using long-context embedding models. arXiv preprint arXiv: 2409.04701, 2024. — [arXiv:2409.04701](https://arxiv.org/abs/2409.04701) [find](https://scholar.google.com/scholar?q=Michael+G%C3%BCnther%2C+Isabelle+Mohr%2C+Daniel+James+Williams%2C+Bo+Wang%2C+and+Han+Xiao%2E+Late+chunking%3A+Contextual+chunk+embeddings+using+long%2Dcontext+embedding+models%2E+arXiv+preprint+arXiv%3A+2409%2E04701%2C+2024%2E+%E2%80%94+) 170. Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. [LM-infinite: Zero-shot extreme length generalization for large language models](https://md.research.elamur.ai/free/en/papers/han-2024-lm-infinite-zero-shot-extreme/). In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association… 171. Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In ICCV, pages 5938–5948. IEEE, 2023. [find](https://scholar.google.com/scholar?q=Dongchen+Han%2C+Xuran+Pan%2C+Yizeng+Han%2C+Shiji+Song%2C+and+Gao+Huang%2E+Flatten+transformer%3A+Vision+transformer+using+focused+linear+attention%2E+In+ICCV%2C+pages+5938%E2%80%935948%2E+IEEE%2C+2023%2E) 172. Yuhang Han, Xuyang Liu, Pengxiang Ding, Donglin Wang, Honggang Chen, Qingsen Yan, and Siteng Huang. Rethinking token reduction in mllms: Towards a unified paradigm for training-free acceleration. CoRR, abs/2411.17686, 2024. doi: 10.48550/ARXIV.2411.17686. URL https://doi.org/10.48550/arXiv.2411.1768… — [doi:10.48550/arxiv.2411.17686](https://doi.org/10.48550/arxiv.2411.17686) [↗](https://doi.org/10.48550/arXiv.2411.1768…) 173. Tanveer Hannan, Md Mohaiminul Islam, Jindong Gu, Thomas Seidl, and Gedas Bertasius. Revisionllm: Recursive vision-language model for temporal grounding in hour-long videos. CoRR, abs/2411.14901, 2024. doi: 10.48550/ARXIV.2411.14901. URL https://doi.org/ 10.48550/arXiv.2411.14901 . — [doi:10.48550/arxiv.2411.14901](https://doi.org/10.48550/arxiv.2411.14901) [find](https://scholar.google.com/scholar?q=Tanveer+Hannan%2C+Md+Mohaiminul+Islam%2C+Jindong+Gu%2C+Thomas+Seidl%2C+and+Gedas+Bertasius%2E+Revisionllm%3A+Recursive+vision%2Dlanguage+model+for+temporal+grounding+in+hour%2Dlong+videos%2E+CoRR%2C+abs%2F2411%2E14901%2C+2024%2E) 174. Conghui He, Zhenjiang Jin, Chaoxi Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Da Lin. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models. ArXiv, abs/2308.10755, 2023. URL [https://api.semanticsc](https://api.semanticsc) holar.org/CorpusID:261049100 . — [arXiv:2308.10755](https://arxiv.org/abs/2308.10755) [↗](https://api.semanticsc) 175. Jiaao He and Jidong Zhai. Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines. arXiv preprint arXiv:2403.11421, 2024. [find](https://scholar.google.com/scholar?q=Jiaao+He+and+Jidong+Zhai%2E+Fastdecode%3A+High%2Dthroughput+gpu%2Defficient+llm+serving+using+heterogeneous+pipelines%2E+arXiv+preprint+arXiv%3A2403%2E11421%2C+2024%2E) 176. Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, Yibo Liu, Qianguo Sun, Yuxin Liang, Hao Wang, Enming Zhang, and Jiaxing Zhang. Never lost in the middle: Mastering long-context question answering with position-agnostic decompositional training. arXiv preprint arXiv:2311.09198, 2023. [find](https://scholar.google.com/scholar?q=Junqing+He%2C+Kunhao+Pan%2C+Xiaoqun+Dong%2C+Zhuoyang+Song%2C+Yibo+Liu%2C+Qianguo+Sun%2C+Yuxin+Liang%2C+Hao+Wang%2C+Enming+Zhang%2C+and+Jiaxing+Zhang%2E+Never+lost+in+the+middle%3A+Mastering+long%2Dcontext+question+answering+) 177. Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, LiuYiBo LiuYiBo, Qianguosun Qianguosun, Yuxin Liang, Hao Wang, Enming Zhang, and Jiaxing Zhang. Never lost in the middle: Mastering long-context question answering with position-agnostic decompositional training. In Lun-Wei Ku, Andre Martins, and… [find](https://scholar.google.com/scholar?q=Junqing+He%2C+Kunhao+Pan%2C+Xiaoqun+Dong%2C+Zhuoyang+Song%2C+LiuYiBo+LiuYiBo%2C+Qianguosun+Qianguosun%2C+Yuxin+Liang%2C+Hao+Wang%2C+Enming+Zhang%2C+and+Jiaxing+Zhang%2E+Never+lost+in+the+middle%3A+Mastering+long%2Dcontext+qu) 178. Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, et al. [Dureader: a chinese machine reading comprehension dataset from real-world applications](https://md.research.elamur.ai/free/en/papers/he-2017-dureader-chinese-machine-reading/). In Proceedings of the Workshop on Machine Reading for Question Answering, pages 37–46, 2018. 179. Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Wenbo Su, and Bo Zheng. Can large language models detect errors in long chain-of-thought reasoning? arXiv preprint arXiv:2502.19361, 2025. [find](https://scholar.google.com/scholar?q=Yancheng+He%2C+Shilong+Li%2C+Jiaheng+Liu%2C+Weixun+Wang%2C+Xingyuan+Bu%2C+Ge+Zhang%2C+Zhongyuan+Peng%2C+Zhaoxiang+Zhang%2C+Wenbo+Su%2C+and+Bo+Zheng%2E+Can+large+language+models+detect+errors+in+long+chain%2Dof%2Dthought+reas) 180. Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Liwei Wang, Jingjing Xu, Zhi Zhang, Hongxia Yang, and Di He. Two stones hit one bird: Bilevel positional encoding for better length extrapolation. arXiv preprint arXiv:2401.16421, 2024. — [arXiv:2401.16421](https://arxiv.org/abs/2401.16421) [find](https://scholar.google.com/scholar?q=Zhenyu+He%2C+Guhao+Feng%2C+Shengjie+Luo%2C+Kai+Yang%2C+Liwei+Wang%2C+Jingjing+Xu%2C+Zhi+Zhang%2C+Hongxia+Yang%2C+and+Di+He%2E+Two+stones+hit+one+bird%3A+Bilevel+positional+encoding+for+better+length+extrapolation%2E+arXiv+) 181. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020. 182. Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. Multilingual needle in a haystack: Investigating long-context behavior of multilingual large language models, 2024. URL https://arxiv.org/abs/2408.10151 . [↗](https://arxiv.org/abs/2408.10151) 183. Christian Herold and Hermann Ney. Improving long context document-level machine translation. CoRR, abs/2306.05183, 2023. doi: 10.48550/ARXIV.2306.05183. URL https://doi.org/10.48550/arXiv.2306.05183 . [↗](https://doi.org/10.48550/arXiv.2306.05183) 184. Lukas Hilgert, Danni Liu, and Jan Niehues. Evaluating and training long-context large language models for question answering on scientific papers. In Sachin Kumar, Vidhisha Balachandran, Chan Young Park, Weijia Shi, Shirley Anugrah Hayati, Yulia Tsvetkov, Noah Smith, Hannaneh Hajishirzi, Dongyeop Ka… [find](https://scholar.google.com/scholar?q=Lukas+Hilgert%2C+Danni+Liu%2C+and+Jan+Niehues%2E+Evaluating+and+training+long%2Dcontext+large+language+models+for+question+answering+on+scientific+papers%2E+In+Sachin+Kumar%2C+Vidhisha+Balachandran%2C+Chan+Young+Pa) 185. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. [Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, 2020. 186. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonya… [find](https://scholar.google.com/scholar?q=Jordan+Hoffmann%2C+Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Elena+Buchatskaya%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Diego+de+las+Casas%2C+Lisa+Anne+Hendricks%2C+Johannes+Welbl%2C+Aidan+Clark%2C+Tom+Hennigan%2C+Eric+Noland%2C+Ka) 187. Jiwoo Hong, Noah Lee, and James Thorne. ORPO: Monolithic preference optimization without reference model. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189, Miami, Florida, USA, Nov… [find](https://scholar.google.com/scholar?q=Jiwoo+Hong%2C+Noah+Lee%2C+and+James+Thorne%2E+ORPO%3A+Monolithic+preference+optimization+without+reference+model%2E+In+Yaser+Al%2DOnaizan%2C+Mohit+Bansal%2C+and+Yun%2DNung+Chen%2C+editors%2C+Proceedings+of+the+2024+Confere) 188. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, and Sophia Shao. Speed: Speculative pipelined execution for efficient decoding. arXiv preprint arXiv:2310.12072, 2023. — [arXiv:2310.12072](https://arxiv.org/abs/2310.12072) [find](https://scholar.google.com/scholar?q=Coleman+Hooper%2C+Sehoon+Kim%2C+Hiva+Mohammadzadeh%2C+Hasan+Genc%2C+Kurt+Keutzer%2C+Amir+Gholami%2C+and+Sophia+Shao%2E+Speed%3A+Speculative+pipelined+execution+for+efficient+decoding%2E+arXiv+preprint+arXiv%3A2310%2E12072%2C) 189. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. [Kvquant: Towards 10 million context length llm inference with kv cache quantization](https://md.research.elamur.ai/free/en/papers/hooper-2024-kvquant-towards-10-million-context/). Advances in Neural Information Processing Systems, 37:1270–1303, 2025. 190. Pedram Hosseini, Jessica M Sin, Bing Ren, Bryceton G Thomas, Elnaz Nouri, Ali Farahanchi, and Saeed Hassanpour. A benchmark for long-form medical question answering. arXiv preprint arXiv:2411.09834, 2024. — [arXiv:2411.09834](https://arxiv.org/abs/2411.09834) [find](https://scholar.google.com/scholar?q=Pedram+Hosseini%2C+Jessica+M+Sin%2C+Bing+Ren%2C+Bryceton+G+Thomas%2C+Elnaz+Nouri%2C+Ali+Farahanchi%2C+and+Saeed+Hassanpour%2E+A+benchmark+for+long%2Dform+medical+question+answering%2E+arXiv+preprint+arXiv%3A2411%2E09834%2C+2) 191. [Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? In First Conference on Language Modeling, 2024.](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) 192. Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, et al. Memserve: Context caching for disaggregated llm serving with elastic memory pool. arXiv preprint arXiv:2406.17565, 2024. — [arXiv:2406.17565](https://arxiv.org/abs/2406.17565) [find](https://scholar.google.com/scholar?q=Cunchen+Hu%2C+Heyang+Huang%2C+Junhao+Hu%2C+Jiang+Xu%2C+Xusheng+Chen%2C+Tao+Xie%2C+Chenxi+Wang%2C+Sa+Wang%2C+Yungang+Bao%2C+Ninghui+Sun%2C+et+al%2E+Memserve%3A+Context+caching+for+disaggregated+llm+serving+with+elastic+memory) 193. Junhao Hu, Wenrui Huang, Weidong Wang, Zhenwen Li, Tiancheng Hu, Zhixia Liu, Xusheng Chen, Tao Xie, and Yizhou Shan. Efficient long-decoding inference with reasoning-aware attention sparsity. arXiv preprint arXiv:2502.11147, 2025. — [arXiv:2502.11147](https://arxiv.org/abs/2502.11147) [find](https://scholar.google.com/scholar?q=Junhao+Hu%2C+Wenrui+Huang%2C+Weidong+Wang%2C+Zhenwen+Li%2C+Tiancheng+Hu%2C+Zhixia+Liu%2C+Xusheng+Chen%2C+Tao+Xie%2C+and+Yizhou+Shan%2E+Efficient+long%2Ddecoding+inference+with+reasoning%2Daware+attention+sparsity%2E+arXiv+pr) 194. Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. Os agents: A survey on mllm-based agents for general computing devices use, 2024. [find](https://scholar.google.com/scholar?q=Xueyu+Hu%2C+Tao+Xiong%2C+Biao+Yi%2C+Zishu+Wei%2C+Ruixuan+Xiao%2C+Yurun+Chen%2C+Jiasheng+Ye%2C+Meiling+Tao%2C+Xiangxin+Zhou%2C+Ziyu+Zhao%2C+et+al%2E+Os+agents%3A+A+survey+on+mllm%2Dbased+agents+for+general+computing+devices+use) 195. Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model’s ability in long text understanding? In The Second Tiny Papers Track at ICLR 2024. [find](https://scholar.google.com/scholar?q=Yutong+Hu%2C+Quzhe+Huang%2C+Mingxu+Tao%2C+Chen+Zhang%2C+and+Yansong+Feng%2E+Can+perplexity+reflect+large+language+model%E2%80%99s+ability+in+long+text+understanding%3F+In+The+Second+Tiny+Papers+Track+at+ICLR+2024%2E) 196. Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, Yan Wang, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng, Zhiwei Jiang, et al. Longrecipe: Recipe for efficient long context generalization in large language models. arXiv preprint arXiv:2409.00509, 2024. [find](https://scholar.google.com/scholar?q=Zhiyuan+Hu%2C+Yuliang+Liu%2C+Jinman+Zhao%2C+Suyuchen+Wang%2C+Yan+Wang%2C+Wei+Shen%2C+Qing+Gu%2C+Anh+Tuan+Luu%2C+See%2DKiong+Ng%2C+Zhiwei+Jiang%2C+et+al%2E+Longrecipe%3A+Recipe+for+efficient+long+context+generalization+in+large) 197. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. [Efficient attentions for long document summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419–1436, 2021. 198. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Benchmarking large language models as AI research agents. CoRR, abs/2310.[03302, 2023](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/). doi: 10.48550/ARXIV.2310.03302. URL https://doi.org/10.48550/arXiv.2310.03302 . [↗](https://doi.org/10.48550/arXiv.2310.03302) 199. [Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier code large language m…](https://md.research.elamur.ai/free/en/papers/huang-2025-opencoder-open-cookbook-top-tier/) 200. Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. Slim-llm: Salience-driven mixed-precision quantization for large language models. arXiv preprint arXiv:2405.14917, 2024. [find](https://scholar.google.com/scholar?q=Wei+Huang%2C+Haotong+Qin%2C+Yangdong+Liu%2C+Yawei+Li%2C+Xianglong+Liu%2C+Luca+Benini%2C+Michele+Magno%2C+and+Xiaojuan+Qi%2E+Slim%2Dllm%3A+Salience%2Ddriven+mixed%2Dprecision+quantization+for+large+language+models%2E+arXiv+prep) 201. Yunpeng Huang, Jingwei Xu, Junyu Lai, Zixu Jiang, Taolue Chen, Zenan Li, Yuan Yao, Xiaoxing Ma, Lijuan Yang, Hao Chen, Shupeng Li, and Penghao Zhao. Advancing transformer architecture in long-context large language models: A comprehensive survey, 2024. [find](https://scholar.google.com/scholar?q=Yunpeng+Huang%2C+Jingwei+Xu%2C+Junyu+Lai%2C+Zixu+Jiang%2C+Taolue+Chen%2C+Zenan+Li%2C+Yuan+Yao%2C+Xiaoxing+Ma%2C+Lijuan+Yang%2C+Hao+Chen%2C+Shupeng+Li%2C+and+Penghao+Zhao%2E+Advancing+transformer+architecture+in+long%2Dcontext+) 202. Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. In First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Yuzhen+Huang%2C+Jinghan+Zhang%2C+Zifei+Shan%2C+and+Junxian+He%2E+Compression+represents+intelligence+linearly%2E+In+First+Conference+on+Language+Modeling%2E) 203. Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489, 2024. [find](https://scholar.google.com/scholar?q=Zhen+Huang%2C+Haoyang+Zou%2C+Xuefeng+Li%2C+Yixiu+Liu%2C+Yuxiang+Zheng%2C+Ethan+Chern%2C+Shijie+Xia%2C+Yiwei+Qin%2C+Weizhe+Yuan%2C+and+Pengfei+Liu%2E+O1+replication+journey%E2%80%93part+2%3A+Surpassing+o1%2Dpreview+through+simple+dis) 204. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. [Qwen2.5-coder technical report](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/). CoRR, abs/2409.12186, 2024. doi: 10.48550/ARXIV.2409.12186. U… 205. DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block-recurrent transformers. In NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=DeLesley+Hutchins%2C+Imanol+Schlag%2C+Yuhuai+Wu%2C+Ethan+Dyer%2C+and+Behnam+Neyshabur%2E+Block%2Drecurrent+transformers%2E+In+NeurIPS%2C+2022%2E) 206. Inflection. I’m pi, your personal ai. [https://inflection.ai/](https://inflection.ai/) , 2023. URL https://inflection.ai/ . [↗](https://inflection.ai/) 207. Naoya Inoue, Pontus Stenetorp, and Kentaro Inui. R4c: A benchmark for evaluating rc systems to get the right answer for the right reason. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6740–6750, 2020. [find](https://scholar.google.com/scholar?q=Naoya+Inoue%2C+Pontus+Stenetorp%2C+and+Kentaro+Inui%2E+R4c%3A+A+benchmark+for+evaluating+rc+systems+to+get+the+right+answer+for+the+right+reason%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+f) 208. Gautier Izacard and Edouard Grave. [Leveraging passage retrieval with generative models for open domain question answering](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/). In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Vol… 209. Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. J. Mach. Learn. Res., 24: 251:1–251:43, 2023. URL [https://jmlr.org/p…](https://jmlr.org/p…) [↗](https://jmlr.org/p…) 210. Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. [Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models](https://md.research.elamur.ai/free/en/papers/jacobs-2023-deepspeed-ulysses-system-optimizations-enabling/). arXiv preprint arXiv:2309.14509, 2023. 211. Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018. [find](https://scholar.google.com/scholar?q=Arthur+Jacot%2C+Franck+Gabriel%2C+and+Cl%C3%A9ment+Hongler%2E+Neural+tangent+kernel%3A+Convergence+and+generalization+in+neural+networks%2E+Advances+in+neural+information+processing+systems%2C+31%2C+2018%2E) 212. Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, et al. The facts grounding leaderboard: Benchmarking llms’ ability to ground responses to long-form input. arXiv preprint arXiv:2501.03200, 2025. [find](https://scholar.google.com/scholar?q=Alon+Jacovi%2C+Andrew+Wang%2C+Chris+Alberti%2C+Connie+Tao%2C+Jon+Lipovetz%2C+Kate+Olszewska%2C+Lukas+Haas%2C+Michelle+Liu%2C+Nate+Keating%2C+Adam+Bloniarz%2C+et+al%2E+The+facts+grounding+leaderboard%3A+Benchmarking+llms%E2%80%99+abi) 213. Young Kyun Jang, Junmo Kang, Yong Jae Lee, and Donghyun Kim. MATE: meet at the embedding - connecting images with long texts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 202… [find](https://scholar.google.com/scholar?q=Young+Kyun+Jang%2C+Junmo+Kang%2C+Yong+Jae+Lee%2C+and+Donghyun+Kim%2E+MATE%3A+meet+at+the+embedding+%2D+connecting+images+with+long+texts%2E+In+Yaser+Al%2DOnaizan%2C+Mohit+Bansal%2C+and+Yun%2DNung+Chen%2C+editors%2C+Findings+of) 214. Minbyul Jeong, Hyeon Hwang, Chanwoong Yoon, Taewhoo Lee, and Jaewoo Kang. Olaph: Improving factuality in biomedical long-form question answering. arXiv preprint arXiv:2405.12701, 2024. — [arXiv:2405.12701](https://arxiv.org/abs/2405.12701) [find](https://scholar.google.com/scholar?q=Minbyul+Jeong%2C+Hyeon+Hwang%2C+Chanwoong+Yoon%2C+Taewhoo+Lee%2C+and+Jaewoo+Kang%2E+Olaph%3A+Improving+factuality+in+biomedical+long%2Dform+question+answering%2E+arXiv+preprint+arXiv%3A2405%2E12701%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E1) 215. Wonje Jeung, Dongjae Jeon, Ashkan Yousefpour, and Jonghyun Choi. Large language models still exhibit bias in long text. arXiv preprint arXiv:2410.17519, 2024. [find](https://scholar.google.com/scholar?q=Wonje+Jeung%2C+Dongjae+Jeon%2C+Ashkan+Yousefpour%2C+and+Jonghyun+Choi%2E+Large+language+models+still+exhibit+bias+in+long+text%2E+arXiv+preprint+arXiv%3A2410%2E17519%2C+2024%2E) 216. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav) 217. Chaoyi Jiang, Lei Gao, Hossein Entezari Zarch, and Murali Annavaram. Efficient llm inference with i/o-aware partial kv cache recomputation. arXiv preprint arXiv:2411.17089, 2024. — [arXiv:2411.17089](https://arxiv.org/abs/2411.17089) [find](https://scholar.google.com/scholar?q=Chaoyi+Jiang%2C+Lei+Gao%2C+Hossein+Entezari+Zarch%2C+and+Murali+Annavaram%2E+Efficient+llm+inference+with+i%2Fo%2Daware+partial+kv+cache+recomputation%2E+arXiv+preprint+arXiv%3A2411%2E17089%2C+2024%2E+%E2%80%94+%5BarXiv%3A2411%2E17089%5D%28) 218. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. [LLMLingua: Compressing prompts for accelerated inference of large language models](https://md.research.elamur.ai/free/en/papers/jiang-2023-llmlingua-compressing-prompts-accelerated-inference/). In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages… 219. Huiqiang Jiang, Yucheng LI, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. [Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/). In A. Globerson, L. Mackey, D. Belgrave, A. F… 220. Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, et al. [Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/). Advances in Neural Information Processing Systems, 37:52481–52515… 221. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Ass… [find](https://scholar.google.com/scholar?q=Huiqiang+Jiang%2C+Qianhui+Wu%2C+Xufang+Luo%2C+Dongsheng+Li%2C+Chin%2DYew+Lin%2C+Yuqing+Yang%2C+and+Lili+Qiu%2E+LongLLMLingua%3A+Accelerating+and+enhancing+LLMs+in+long+context+scenarios+via+prompt+compression%2E+In+Lun%2DW) 222. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. [A survey on large language models for code generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/). CoRR, abs/2406.00515, 2024. doi: 10.48550/ARXIV.2 406.00515. URL https://doi.org/10.48550/arXiv.2406.00515 . [↗](https://doi.org/10.48550/arXiv.2406.00515) 223. Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H. Chen, and Andrew Y. Ng. Many-shot in-context learning in multimodal foundation models. CoRR, abs/2405.09798, 2024. doi: 10.48550/ARXIV.2405.09798. URL https://doi.org/10.485 50/arXiv.2405.09798 . — [doi:10.48550/arxiv.2405.09798](https://doi.org/10.48550/arxiv.2405.09798) [find](https://scholar.google.com/scholar?q=Yixing+Jiang%2C+Jeremy+Irvin%2C+Ji+Hun+Wang%2C+Muhammad+Ahmed+Chaudhry%2C+Jonathan+H%2E+Chen%2C+and+Andrew+Y%2E+Ng%2E+Many%2Dshot+in%2Dcontext+learning+in+multimodal+foundation+models%2E+CoRR%2C+abs%2F2405%2E09798%2C+2024%2E+doi%3A+10) 224. Ziyan Jiang, Xueguang Ma, and Wenhu Chen. Longrag: Enhancing retrieval-augmented generation with long-context llms. CoRR, abs/2406.15319, 2024. doi: 10.48550/ARXIV.2406. 15319. URL https://doi.org/10.48550/arXiv.2406.15319 . — [doi:10.48550/arxiv.2406](https://doi.org/10.48550/arxiv.2406) [↗](https://doi.org/10.48550/arXiv.2406.15319) 225. Ziyan Jiang, Xueguang Ma, and Wenhu Chen. Longrag: Enhancing retrieval-augmented generation with long-context llms. arXiv preprint arXiv:2406.15319, 2024. — [arXiv:2406.15319](https://arxiv.org/abs/2406.15319) [find](https://scholar.google.com/scholar?q=Ziyan+Jiang%2C+Xueguang+Ma%2C+and+Wenhu+Chen%2E+Longrag%3A+Enhancing+retrieval%2Daugmented+generation+with+long%2Dcontext+llms%2E+arXiv+preprint+arXiv%3A2406%2E15319%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E15319%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F24) 226. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.n… 227. Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan Ö. Arik. Long-context llms meet RAG: overcoming challenges for long inputs in RAG. CoRR, abs/2410.05983, 2024. doi: 10.48550 /ARXIV.2410.05983. URL https://doi.org/10.48550/arXiv.2410.05983 . — [doi:10.48550/arxiv.2410.05983](https://doi.org/10.48550/arxiv.2410.05983) [↗](https://doi.org/10.48550/arXiv.2410.05983) 228. Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context llms meet rag: Overcoming challenges for long inputs in rag. arXiv preprint arXiv:2410.05983, 2024. — [arXiv:2410.05983](https://arxiv.org/abs/2410.05983) [find](https://scholar.google.com/scholar?q=Bowen+Jin%2C+Jinsung+Yoon%2C+Jiawei+Han%2C+and+Sercan+O+Arik%2E+Long%2Dcontext+llms+meet+rag%3A+Overcoming+challenges+for+long+inputs+in+rag%2E+arXiv+preprint+arXiv%3A2410%2E05983%2C+2024%2E+%E2%80%94+%5BarXiv%3A2410%2E05983%5D%28https%3A%2F%2Far) 229. Hanlei Jin, Yang Zhang, Dan Meng, Jun Wang, and Jinghua Tan. A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods. CoRR, abs/2403.02901, 2024. doi: 10.48550/ARXIV.2403.02901. URL https://doi.org/ 10.48550/arXiv.2403.02901 . — [doi:10.48550/arxiv.2403.02901](https://doi.org/10.48550/arxiv.2403.02901) [find](https://scholar.google.com/scholar?q=Hanlei+Jin%2C+Yang+Zhang%2C+Dan+Meng%2C+Jun+Wang%2C+and+Jinghua+Tan%2E+A+comprehensive+survey+on+process%2Doriented+automatic+text+summarization+with+exploration+of+llm%2Dbased+methods%2E+CoRR%2C+abs%2F2403%2E02901%2C+2024%2E+) 230. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Chia yuan Chang, and Xia Hu. Growlength: Accelerating llms pretraining by progressively growing training length. ArXiv, abs/2310.00576, 2023. — [arXiv:2310.00576](https://arxiv.org/abs/2310.00576) [find](https://scholar.google.com/scholar?q=Hongye+Jin%2C+Xiaotian+Han%2C+Jingfeng+Yang%2C+Zhimeng+Jiang%2C+Chia+yuan+Chang%2C+and+Xia+Hu%2E+Growlength%3A+Accelerating+llms+pretraining+by+progressively+growing+training+length%2E+ArXiv%2C+abs%2F2310%2E00576%2C+2023%2E+%E2%80%94+) 231. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. [Llm maybe longlm: Self-extend llm context window without tuning](https://md.research.elamur.ai/free/en/papers/jin-2024-llm-maybe-longlm-self-extend/). arXiv preprint arXiv:2401.01325, 2024. 232. Eunkyung Jo, Yuin Jeong, SoHyun Park, Daniel A. Epstein, and Young-Ho Kim. Understanding the impact of long-term memory on self-disclosure with large language model-driven chatbots for public health intervention. In Florian ’Floyd’ Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilso… — [doi:10.1145/3613904.3642420](https://doi.org/10.1145/3613904.3642420) [find](https://scholar.google.com/scholar?q=Eunkyung+Jo%2C+Yuin+Jeong%2C+SoHyun+Park%2C+Daniel+A%2E+Epstein%2C+and+Young%2DHo+Kim%2E+Understanding+the+impact+of+long%2Dterm+memory+on+self%2Ddisclosure+with+large+language+model%2Ddriven+chatbots+for+public+health+i) 233. Ashutosh Joshi, Sheikh Muhammad Sarwar, Samarth Varshney, Sreyashi Nag, Shrivats Agrawal, and Juhi Naik. Reaper: Reasoning based retrieval planning for complex rag systems. arXiv preprint arXiv: 2407.18553, 2024. — [arXiv:2407.18553](https://arxiv.org/abs/2407.18553) [find](https://scholar.google.com/scholar?q=Ashutosh+Joshi%2C+Sheikh+Muhammad+Sarwar%2C+Samarth+Varshney%2C+Sreyashi+Nag%2C+Shrivats+Agrawal%2C+and+Juhi+Naik%2E+Reaper%3A+Reasoning+based+retrieval+planning+for+complex+rag+systems%2E+arXiv+preprint+arXiv%3A+2407%2E) 234. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, 2017. 235. Dongwon Jung, Qin Liu, Tenghao Huang, Ben Zhou, and Muhao Chen. Familiarity-aware evidence compression for retrieval-augmented generation, 2024. URL https://arxiv.org/abs/2409.12468 . [↗](https://arxiv.org/abs/2409.12468) 236. Hoyoun Jung and Kyung-Joong Kim. Discrete prompt compression with reinforcement learning. IEEE Access, 12:72578–72587, 2024. ISSN 2169-3536. doi: 10.1109/access.2024.3403 426. URL http://dx.doi.org/10.1109/ACCESS.2024.3403426 . — [doi:10.1109/access.2024.3403](https://doi.org/10.1109/access.2024.3403) [↗](http://dx.doi.org/10.1109/ACCESS.2024.3403426) 237. Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960. [find](https://scholar.google.com/scholar?q=Rudolph+Emil+Kalman%2E+A+new+approach+to+linear+filtering+and+prediction+problems%2E+1960%2E) 238. Greg Kamradt. Needle in a haystack - pressure testing llms. [https://github.com/gka](https://github.com/gka) mradt/LLMTest_NeedleInAHaystack , 2023. [↗](https://github.com/gka) 239. Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipe for near-lossless generative inference of llm, 2024. URL https://arxiv.org/abs/2403.05527 . — [arXiv:2403.05527](https://arxiv.org/abs/2403.05527) [↗](https://arxiv.org/abs/2403.05527) 240. Sayash Kapoor, Peter Henderson, and Arvind Narayanan. Promises and pitfalls of artificial intelligence for legal applications. CoRR, abs/2402.01656, 2024. doi: 10.48550/ARXIV.2402. 01656. URL https://doi.org/10.48550/arXiv.2402.01656 . [↗](https://doi.org/10.48550/arXiv.2402.01656) 241. Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. One thousand and one pairs: A "novel" challenge for long-context language models, 2024. [find](https://scholar.google.com/scholar?q=Marzena+Karpinska%2C+Katherine+Thai%2C+Kyle+Lo%2C+Tanya+Goyal%2C+and+Mohit+Iyyer%2E+One+thousand+and+one+pairs%3A+A+%22novel%22+challenge+for+long%2Dcontext+language+models%2C+2024%2E) 242. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 5156–5165. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Angelos+Katharopoulos%2C+Apoorv+Vyas%2C+Nikolaos+Pappas%2C+and+Fran%C3%A7ois+Fleuret%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+ICML%2C+volume+119+of+Proceedings+of+Machine+) 243. Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Amirhossein+Kazemnejad%2C+Inkit+Padhi%2C+Karthikeyan+Natesan+Ramamurthy%2C+Payel+Das%2C+and+Siva+Reddy%2E+The+impact+of+positional+encoding+on+length+generalization+in+transformers%2E+Advances+in+Neural+Informati) 244. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and M. Lewis. Generalization through memorization: Nearest neighbor language models. International Conference on Learning Representations, 2019. [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+Omer+Levy%2C+Dan+Jurafsky%2C+Luke+Zettlemoyer%2C+and+M%2E+Lewis%2E+Generalization+through+memorization%3A+Nearest+neighbor+language+models%2E+International+Conference+on+Learning+Representations) 245. Heehoon Kim, Junyeol Ryu, and Jaejin Lee. Tccl: Discovering better communication paths for pcie gpu clusters. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 999–1015, 2024. [find](https://scholar.google.com/scholar?q=Heehoon+Kim%2C+Junyeol+Ryu%2C+and+Jaejin+Lee%2E+Tccl%3A+Discovering+better+communication+paths+for+pcie+gpu+clusters%2E+In+Proceedings+of+the+29th+ACM+International+Conference+on+Architectural+Support+for+Progr) 246. Youngjoo Kim and Hyochoong Bang. Introduction to kalman filter and its applications. In Felix Govaers, editor, Introduction and Implementations of the Kalman Filter, chapter 2. IntechOpen, Rijeka, 2018. [find](https://scholar.google.com/scholar?q=Youngjoo+Kim+and+Hyochoong+Bang%2E+Introduction+to+kalman+filter+and+its+applications%2E+In+Felix+Govaers%2C+editor%2C+Introduction+and+Implementations+of+the+Kalman+Filter%2C+chapter+2%2E+IntechOpen%2C+Rijeka%2C+201) 247. Hideo Kobayashi, Wuwei Lan, Peng Shi, Shuaichen Chang, Jiang Guo, Henghui Zhu, Zhiguo Wang, and Patrick Ng. You only read once (YORO): Learning to internalize database knowledge for text-to-SQL. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of… [find](https://scholar.google.com/scholar?q=Hideo+Kobayashi%2C+Wuwei+Lan%2C+Peng+Shi%2C+Shuaichen+Chang%2C+Jiang+Guo%2C+Henghui+Zhu%2C+Zhiguo+Wang%2C+and+Patrick+Ng%2E+You+only+read+once+%28YORO%29%3A+Learning+to+internalize+database+knowledge+for+text%2Dto%2DSQL%2E+In+Lu) 248. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. [The narrativeqa reading comprehension challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/). Transactions of the Association for Computational Linguistics, 6:317–328, 2018. 249. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. [The NarrativeQA reading comprehension challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/). Transactions of the Association for Computational Linguistics, 6:317–328, 2018. doi: 10.1162/tacl _a_00023. URL https://aclanthology… 250. Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schütze. Longform: Effective instruction tuning with reverse instructions. arXiv preprint arXiv:2304.08460, 2023. [find](https://scholar.google.com/scholar?q=Abdullatif+K%C3%B6ksal%2C+Timo+Schick%2C+Anna+Korhonen%2C+and+Hinrich+Sch%C3%BCtze%2E+Longform%3A+Effective+instruction+tuning+with+reverse+instructions%2E+arXiv+preprint+arXiv%3A2304%2E08460%2C+2023%2E) 251. Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. [Reducing activation recomputation in large transformer models](https://md.research.elamur.ai/free/en/papers/korthikanti-2022-reducing-activation-recomputation-large-transformer/). Proceedings of Machine Learning and Systems, 5:341–353, 2023. 252. Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.02027, 2021. — [arXiv:2107.02027](https://arxiv.org/abs/2107.02027) [find](https://scholar.google.com/scholar?q=Mario+Michael+Krell%2C+Matej+Kosec%2C+Sergio+P+Perez%2C+and+Andrew+Fitzgibbon%2E+Efficient+sequence+packing+without+cross%2Dcontamination%3A+Accelerating+large+language+models+without+impacting+performance%2E+arXiv) 253. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. [Hurdles to progress in long-form question answering](https://md.research.elamur.ai/free/en/papers/krishna-2021-hurdles-progress-long-form-question/). arXiv preprint arXiv:2103.06332, 2021. 254. Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. Aquamuse: Automatically generating datasets for query-based multi-document summarization. arXiv preprint arXiv:2010.12694, 2020. — [arXiv:2010.12694](https://arxiv.org/abs/2010.12694) [find](https://scholar.google.com/scholar?q=Sayali+Kulkarni%2C+Sheide+Chammas%2C+Wan+Zhu%2C+Fei+Sha%2C+and+Eugene+Ie%2E+Aquamuse%3A+Automatically+generating+datasets+for+query%2Dbased+multi%2Ddocument+summarization%2E+arXiv+preprint+arXiv%3A2010%2E12694%2C+2020%2E+%E2%80%94+%5Bar) 255. Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, et al. Longlamp: A benchmark for personalized long-form text generation. arXiv preprint arXiv:2407.11016, 2024. — [arXiv:2407.11016](https://arxiv.org/abs/2407.11016) [find](https://scholar.google.com/scholar?q=Ishita+Kumar%2C+Snigdha+Viswanathan%2C+Sushrita+Yerra%2C+Alireza+Salemi%2C+Ryan+A+Rossi%2C+Franck+Dernoncourt%2C+Hanieh+Deilamsalehy%2C+Xiang+Chen%2C+Ruiyi+Zhang%2C+Shubham+Agarwal%2C+et+al%2E+Longlamp%3A+A+benchmark+for+per) 256. Achintya Kundu, Rhui Dih Lee, Laura Wynter, Raghu Kiran Ganti, and Mayank Mishra. Enhancing training efficiency using packing with flash attention. arXiv preprint arXiv:2407.09105, 2024. [find](https://scholar.google.com/scholar?q=Achintya+Kundu%2C+Rhui+Dih+Lee%2C+Laura+Wynter%2C+Raghu+Kiran+Ganti%2C+and+Mayank+Mishra%2E+Enhancing+training+efficiency+using+packing+with+flash+attention%2E+arXiv+preprint+arXiv%3A2407%2E09105%2C+2024%2E) 257. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. arXiv preprint arXiv:2406.10149, 2024. [find](https://scholar.google.com/scholar?q=Yuri+Kuratov%2C+Aydar+Bulatov%2C+Petr+Anokhin%2C+Ivan+Rodkin%2C+Dmitry+Sorokin%2C+Artyom+Sorokin%2C+and+Mikhail+Burtsev%2E+Babilong%3A+Testing+the+limits+of+llms+with+long+context+reasoning%2Din%2Da%2Dhaystack%2E+arXiv+prepr) 258. Yuri Kuratov, Mikhail Arkhipov, Aydar Bulatov, and Mikhail Burtsev. Cramming 1568 tokens into a single vector and back again: Exploring the limits of embedding space capacity. arXiv preprint arXiv:2502.13063, 2025. [find](https://scholar.google.com/scholar?q=Yuri+Kuratov%2C+Mikhail+Arkhipov%2C+Aydar+Bulatov%2C+and+Mikhail+Burtsev%2E+Cramming+1568+tokens+into+a+single+vector+and+back+again%3A+Exploring+the+limits+of+embedding+space+capacity%2E+arXiv+preprint+arXiv%3A250) 259. Wai-Chung Kwan, Xingshan Zeng, Yufei Wang, Yusen Sun, Liangyou Li, Lifeng Shang, Qun Liu, and Kam-Fai Wong. M4le: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models. arXiv preprint arXiv:2310.19240, 2023. [find](https://scholar.google.com/scholar?q=Wai%2DChung+Kwan%2C+Xingshan+Zeng%2C+Yufei+Wang%2C+Yusen+Sun%2C+Liangyou+Li%2C+Lifeng+Shang%2C+Qun+Liu%2C+and+Kam%2DFai+Wong%2E+M4le%3A+A+multi%2Dability+multi%2Drange+multi%2Dtask+multi%2Ddomain+long%2Dcontext+evaluation+benchmark+) 260. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. [Natural questions: A benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 7:4… 261. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023. 262. Philippe Laban, Alexander R Fabbri, Caiming Xiong, and Chien-Sheng Wu. Summary of a haystack: A challenge to long-context llms and rag systems. arXiv preprint arXiv:https://arxiv.org/pdf/2407.01370, 2024. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Alexander+R+Fabbri%2C+Caiming+Xiong%2C+and+Chien%2DSheng+Wu%2E+Summary+of+a+haystack%3A+A+challenge+to+long%2Dcontext+llms+and+rag+systems%2E+arXiv+preprint+arXiv%3Ahttps%3A%2F%2Farxiv%2Eorg%2Fpdf%2F2407%2E01370%2C+2) 263. Perplexity Labs. perplexity. URL [https://www.perplexity.ai](https://www.perplexity.ai) . Accessed: 2024-05-30. [↗](https://www.perplexity.ai) 264. Jack Lanchantin, Shubham Toshniwal, Jason Weston, arthur szlam, and Sainbayar Sukhbaatar. Learning to reason and memorize with self-notes. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 11891–11911.… [find](https://scholar.google.com/scholar?q=Jack+Lanchantin%2C+Shubham+Toshniwal%2C+Jason+Weston%2C+arthur+szlam%2C+and+Sainbayar+Sukhbaatar%2E+Learning+to+reason+and+memorize+with+self%2Dnotes%2E+In+A%2E+Oh%2C+T%2E+Naumann%2C+A%2E+Globerson%2C+K%2E+Saenko%2C+M%2E+Hardt%2C+and+) 265. Gibbeum Lee, Volker Hartmann, Jongho Park, Dimitris Papailiopoulos, and Kangwook Lee. Prompted llms as chatbot modules for long open-domain conversation. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, Toronto,… [find](https://scholar.google.com/scholar?q=Gibbeum+Lee%2C+Volker+Hartmann%2C+Jongho+Park%2C+Dimitris+Papailiopoulos%2C+and+Kangwook+Lee%2E+Prompted+llms+as+chatbot+modules+for+long+open%2Ddomain+conversation%2E+In+Anna+Rogers%2C+Jordan+L%2E+Boyd%2DGraber%2C+and+Nao) 266. Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al. Can long-context language models subsume retrieval, rag, sql, and more? arXiv preprint arXiv:2406.13121, 2024. [find](https://scholar.google.com/scholar?q=Jinhyuk+Lee%2C+Anthony+Chen%2C+Zhuyun+Dai%2C+Dheeru+Dua%2C+Devendra+Singh+Sachan%2C+Michael+Boratko%2C+Yi+Luan%2C+S%C3%A9bastien+MR+Arnold%2C+Vincent+Perot%2C+Siddharth+Dalmia%2C+et+al%2E+Can+long%2Dcontext+language+models+subsum) 267. Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John F. Canny, and Ian Fischer. A human-inspired reading agent with gist memory of very long contexts. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL [https://openreview.n…](https://openreview.n…) [↗](https://openreview.n…) 268. Taewhoo Lee, Chanwoong Yoon, Kyochul Jang, Donghyeon Lee, Minju Song, Hyunjae Kim, and Jaewoo Kang. Ethic: Evaluating large language models on long-context tasks with high information coverage. arXiv preprint arXiv:2410.16848, 2024. [find](https://scholar.google.com/scholar?q=Taewhoo+Lee%2C+Chanwoong+Yoon%2C+Kyochul+Jang%2C+Donghyeon+Lee%2C+Minju+Song%2C+Hyunjae+Kim%2C+and+Jaewoo+Kang%2E+Ethic%3A+Evaluating+large+language+models+on+long%2Dcontext+tasks+with+high+information+coverage%2E+arXiv+) 269. Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-in Lee, and Moontae Lee. Qasa: advanced question answering on scientific articles. In International Conference on Machine Learning, pages 19036–19052. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Yoonjoo+Lee%2C+Kyungjae+Lee%2C+Sunghyun+Park%2C+Dasol+Hwang%2C+Jaehyeon+Kim%2C+Hong%2Din+Lee%2C+and+Moontae+Lee%2E+Qasa%3A+advanced+question+answering+on+scientific+articles%2E+In+International+Conference+on+Machine+Lear) 270. Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, and Kang Liu. S3eval: A synthetic, scalable, systematic evaluation suite for large language model. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… [find](https://scholar.google.com/scholar?q=Fangyu+Lei%2C+Qian+Liu%2C+Yiming+Huang%2C+Shizhu+He%2C+Jun+Zhao%2C+and+Kang+Liu%2E+S3eval%3A+A+synthetic%2C+scalable%2C+systematic+evaluation+suite+for+large+language+model%2E+In+Proceedings+of+the+2024+Conference+of+the) 271. Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. Mining of massive data sets. Cambridge university press, 2020. [find](https://scholar.google.com/scholar?q=Jure+Leskovec%2C+Anand+Rajaraman%2C+and+Jeffrey+David+Ullman%2E+Mining+of+massive+data+sets%2E+Cambridge+university+press%2C+2020%2E) 272. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Onl… [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+In+Marie%2DFrancine+Moens%2C+Xuanjing+Huang%2C+Lucia+Specia%2C+and+Scott+Wen%2Dtau+Yih%2C+editors%2C+Proceedi) 273. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Yaniv+Leviathan%2C+Matan+Kalman%2C+and+Yossi+Matias%2E+Fast+inference+from+transformers+via+speculative+decoding%2E+In+International+Conference+on+Machine+Learning%2C+pages+19274%E2%80%9319286%2E+PMLR%2C+2023%2E) 274. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Selective attention improves transformer, 2024. URL https://arxiv.org/abs/2410.02703 . — [arXiv:2410.02703](https://arxiv.org/abs/2410.02703) [↗](https://arxiv.org/abs/2410.02703) 275. Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, and Amnon Shashua. The inductive bias of in-context learning: Rethinking pretraining example design. In International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Yoav+Levine%2C+Noam+Wies%2C+Daniel+Jannai%2C+Dan+Navon%2C+Yedid+Hoshen%2C+and+Amnon+Shashua%2E+The+inductive+bias+of+in%2Dcontext+learning%3A+Rethinking+pretraining+example+design%2E+In+International+Conference+on+Lear) 276. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv: 2005.11401, 2020. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ethan+Perez%2C+Aleksandra+Piktus%2C+Fabio+Petroni%2C+Vladimir+Karpukhin%2C+Naman+Goyal%2C+Heinrich+K%C3%BCttler%2C+Mike+Lewis%2C+Wen+tau+Yih%2C+Tim+Rockt%C3%A4schel%2C+Sebastian+Riedel%2C+and+Douwe+Kiela%2E+Retrieval%2D) 277. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. [Llava-onevision: Easy visual task transfer](https://md.research.elamur.ai/free/en/papers/li-2024-llava-onevision-easy-visual-task/). CoRR, abs/2408.03326, 2024. doi: 10.48550/ARXIV.2408.03326. URL https://doi.org/10.485 50/arXiv.2408.03326 . 278. Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. [find](https://scholar.google.com/scholar?q=Dacheng+Li%2C+Rulin+Shao%2C+Anze+Xie%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Joseph+Gonzalez%2C+Ion+Stoica%2C+Xuezhe+Ma%2C+and+Hao+Zhang%2E+How+long+can+context+length+of+open%2Dsource+llms+truly+promise%3F+In+NeurIPS+2023+Works) 279. Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Shishir G Patil, Matei Zaharia, Joseph E Gonzalez, and Ion Stoica. Llms can easily learn to reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374, 2025. [find](https://scholar.google.com/scholar?q=Dacheng+Li%2C+Shiyi+Cao%2C+Tyler+Griggs%2C+Shu+Liu%2C+Xiangxi+Mo%2C+Shishir+G+Patil%2C+Matei+Zaharia%2C+Joseph+E+Gonzalez%2C+and+Ion+Stoica%2E+Llms+can+easily+learn+to+reason+from+demonstrations+structure%2C+not+content%2C) 280. Haoyang Li, Fangcheng Fu, Sheng Lin, Hao Ge, Xuanyu Wang, Jiawen Niu, Jie Jiang, and Bin Cui. Demystifying workload imbalances in large transformer model training over variable-length sequences. arXiv preprint arXiv:2412.07894, 2024. — [arXiv:2412.07894](https://arxiv.org/abs/2412.07894) [find](https://scholar.google.com/scholar?q=Haoyang+Li%2C+Fangcheng+Fu%2C+Sheng+Lin%2C+Hao+Ge%2C+Xuanyu+Wang%2C+Jiawen+Niu%2C+Jie+Jiang%2C+and+Bin+Cui%2E+Demystifying+workload+imbalances+in+large+transformer+model+training+over+variable%2Dlength+sequences%2E+arXiv) 281. Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. [Loogle: Can long-context language models understand long contexts?](https://md.research.elamur.ai/free/en/papers/li-2023-loogle-long-context-language-models/) arXiv preprint arXiv:2311.04939, 2023. 282. Jiashi Li. Flashmla: Efficient mla decoding kernels. [https://github.com/deepseek-a](https://github.com/deepseek-a) i/FlashMLA , 2025. [↗](https://github.com/deepseek-a) 283. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. [A diversity-promoting objective function for neural conversation models](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/). arXiv preprint arXiv:1510.03055, 2015. 284. Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. [BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Confe… 285. Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. [From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning](https://md.research.elamur.ai/free/en/papers/li-2023-quantity-quality-boosting-llm-performance/). ArXiv, abs/2308.12032, 2023. 286. Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963, 2024. — [arXiv:2407.11963](https://arxiv.org/abs/2407.11963) [find](https://scholar.google.com/scholar?q=Mo+Li%2C+Songyang+Zhang%2C+Yunxin+Liu%2C+and+Kai+Chen%2E+Needlebench%3A+Can+llms+do+retrieval+and+reasoning+in+1+million+context+window%3F+arXiv+preprint+arXiv%3A2407%2E11963%2C+2024%2E+%E2%80%94+%5BarXiv%3A2407%2E11963%5D%28https%3A%2F%2Farxiv) 287. Qingyuan Li, Yifan Zhang, Liang Li, Peng Yao, Bo Zhang, Xiangxiang Chu, Yerui Sun, Li Du, and Yuchen Xie. Fptq: Fine-grained post-training quantization for large language models. arXiv preprint arXiv:2308.15987, 2023. — [arXiv:2308.15987](https://arxiv.org/abs/2308.15987) [find](https://scholar.google.com/scholar?q=Qingyuan+Li%2C+Yifan+Zhang%2C+Liang+Li%2C+Peng+Yao%2C+Bo+Zhang%2C+Xiangxiang+Chu%2C+Yerui+Sun%2C+Li+Du%2C+and+Yuchen+Xie%2E+Fptq%3A+Fine%2Dgrained+post%2Dtraining+quantization+for+large+language+models%2E+arXiv+preprint+arXiv%3A) 288. Rui Li, Xiaohan Wang, Yuhui Zhang, Zeyu Wang, and Serena Yeung-Levy. Temporal preference optimization for long-form video understanding. CoRR, abs/2501.13919, 2025. doi: 10.48550/ARXIV.2501.13919. URL https://doi.org/10.48550/arXiv.2501.13 919 . — [doi:10.48550/arxiv.2501.13919](https://doi.org/10.48550/arxiv.2501.13919) [↗](https://doi.org/10.48550/arXiv.2501.13) 289. Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418, 2023. [find](https://scholar.google.com/scholar?q=Shanda+Li%2C+Chong+You%2C+Guru+Guruganesh%2C+Joshua+Ainslie%2C+Santiago+Ontanon%2C+Manzil+Zaheer%2C+Sumit+Sanghai%2C+Yiming+Yang%2C+Sanjiv+Kumar%2C+and+Srinadh+Bhojanapalli%2E+Functional+interpolation+for+relative+positi) 290. Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jiaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, Wenbo Su, and Bo Zheng. Graphreader: Building graph-based agent to enhance long-context abilities of large language models. Conference on Empirical Methods in Natural Language Proc… [find](https://scholar.google.com/scholar?q=Shilong+Li%2C+Yancheng+He%2C+Hangyu+Guo%2C+Xingyuan+Bu%2C+Ge+Bai%2C+Jie+Liu%2C+Jiaheng+Liu%2C+Xingwei+Qu%2C+Yangguang+Li%2C+Wanli+Ouyang%2C+Wenbo+Su%2C+and+Bo+Zheng%2E+Graphreader%3A+Building+graph%2Dbased+agent+to+enhance+long%2D) 291. Shilong Li, Yancheng He, Hui Huang, Xingyuan Bu, Jiaheng Liu, Hangyu Guo, Weixun Wang, Jihao Gu, Wenbo Su, and Bo Zheng. 2d-dpo: Scaling direct preference optimization with 2-dimensional supervision. ArXiv, abs/2410.19720, 2024. [find](https://scholar.google.com/scholar?q=Shilong+Li%2C+Yancheng+He%2C+Hui+Huang%2C+Xingyuan+Bu%2C+Jiaheng+Liu%2C+Hangyu+Guo%2C+Weixun+Wang%2C+Jihao+Gu%2C+Wenbo+Su%2C+and+Bo+Zheng%2E+2d%2Ddpo%3A+Scaling+direct+preference+optimization+with+2%2Ddimensional+supervision%2E+) 292. Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. [Long-context llms struggle with long in-context learning](https://md.research.elamur.ai/free/en/papers/li-2024-long-context-llms-struggle-long/). arXiv preprint arXiv:2404.02060, 2024. 293. Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, Yu Qiao, Yali Wang, and Limin Wang. Videochat-flash: Hierarchical compression for long-context video modeling. CoRR, abs/2501.00574, 2025. doi: 10.48550/ARXIV.2501.00574. URL http… — [doi:10.48550/arxiv.2501.00574](https://doi.org/10.48550/arxiv.2501.00574) [find](https://scholar.google.com/scholar?q=Xinhao+Li%2C+Yi+Wang%2C+Jiashuo+Yu%2C+Xiangyu+Zeng%2C+Yuhan+Zhu%2C+Haian+Huang%2C+Jianfei+Gao%2C+Kunchang+Li%2C+Yinan+He%2C+Chenting+Wang%2C+Yu+Qiao%2C+Yali+Wang%2C+and+Limin+Wang%2E+Videochat%2Dflash%3A+Hierarchical+compression+f) 294. Yanyang Li, Shuo Liang, Michael R. Lyu, and Liwei Wang. Making long-context language models better multi-hop reasoners. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024… [find](https://scholar.google.com/scholar?q=Yanyang+Li%2C+Shuo+Liang%2C+Michael+R%2E+Lyu%2C+and+Liwei+Wang%2E+Making+long%2Dcontext+language+models+better+multi%2Dhop+reasoners%2E+In+Lun%2DWei+Ku%2C+Andre+Martins%2C+and+Vivek+Srikumar%2C+editors%2C+Proceedings+of+the+62) 295. Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. [Compressing context to enhance inference efficiency of large language models](https://md.research.elamur.ai/free/en/papers/li-2023-compressing-context-enhance-inference-efficiency/). In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353, Singapore… 296. Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, et al. Scbench: A kv cache-centric analysis of long-context methods. arXiv preprint arXiv:2412.10319, 2024. — [arXiv:2412.10319](https://arxiv.org/abs/2412.10319) [find](https://scholar.google.com/scholar?q=Yucheng+Li%2C+Huiqiang+Jiang%2C+Qianhui+Wu%2C+Xufang+Luo%2C+Surin+Ahn%2C+Chengruidong+Zhang%2C+Amir+H+Abdi%2C+Dongsheng+Li%2C+Jianfeng+Gao%2C+Yuqing+Yang%2C+et+al%2E+Scbench%3A+A+kv+cache%2Dcentric+analysis+of+long%2Dcontext+met) 297. Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. arXiv preprint arXiv:2502.12143, 2025. [find](https://scholar.google.com/scholar?q=Yuetai+Li%2C+Xiang+Yue%2C+Zhangchen+Xu%2C+Fengqing+Jiang%2C+Luyao+Niu%2C+Bill+Yuchen+Lin%2C+Bhaskar+Ramasubramanian%2C+and+Radha+Poovendran%2E+Small+models+struggle+to+learn+from+strong+reasoners%2E+arXiv+preprint+arXi) 298. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. [Snapkv: LLM knows what you are looking for before generation](https://md.research.elamur.ai/free/en/papers/li-2024-snapkv-llm-knows-looking-before/). In NeurIPS, 2024. 299. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. [Eagle: speculative sampling requires rethinking feature uncertainty](https://md.research.elamur.ai/free/en/papers/li-2024-eagle-speculative-sampling-requires-rethinking/). In Proceedings of the 41st International Conference on Machine Learning, pages 28935–28948, 2024. 300. Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach. arXiv preprint arXiv:2407.16833, 2024. [find](https://scholar.google.com/scholar?q=Zhuowan+Li%2C+Cheng+Li%2C+Mingyang+Zhang%2C+Qiaozhu+Mei%2C+and+Michael+Bendersky%2E+Retrieval+augmented+generation+or+long%2Dcontext+llms%3F+a+comprehensive+study+and+hybrid+approach%2E+arXiv+preprint+arXiv%3A2407%2E1683) 301. Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). ArXiv, abs/2410.12388, 2024. URL [https://api.semantic](https://api.semantic) scholar.org/CorpusID:273375474 . [↗](https://api.semantic) 302. Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/), 2024. URL https://arxiv.org/abs/2410.12388 . [↗](https://arxiv.org/abs/2410.12388) 303. Zongqian Li, Yixuan Su, and Nigel Collier. 500xcompressor: Generalized prompt compression for large language models, 2024. URL https://arxiv.org/abs/2408.03094 . [↗](https://arxiv.org/abs/2408.03094) 304. Hao Liang, Jiapeng Li, Tianyi Bai, Xijie Huang, Linzhuang Sun, Zhengren Wang, Conghui He, Bin Cui, Chong Chen, and Wentao Zhang. Keyvideollm: Towards large-scale video keyframe selection. CoRR, abs/2407.03104, 2024. doi: 10.48550/ARXIV.2407.03104. URL https://doi.org/10.48550/arXiv.2407.03104 . — [doi:10.48550/arxiv.2407.03104](https://doi.org/10.48550/arxiv.2407.03104) [↗](https://doi.org/10.48550/arXiv.2407.03104) 305. Xiaobo Liang, Zecheng Tang, Juntao Li, and Min Zhang. Open-ended long text generation via masked language modeling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 223–241, 2023. [find](https://scholar.google.com/scholar?q=Xiaobo+Liang%2C+Zecheng+Tang%2C+Juntao+Li%2C+and+Min+Zhang%2E+Open%2Dended+long+text+generation+via+masked+language+modeling%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Lingu) 306. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024. [find](https://scholar.google.com/scholar?q=Opher+Lieber%2C+Barak+Lenz%2C+Hofit+Bata%2C+Gal+Cohen%2C+Jhonathan+Osin%2C+Itay+Dalmedigos%2C+Erez+Safahi%2C+Shaked+Meirom%2C+Yonatan+Belinkov%2C+Shai+Shalev%2DShwartz%2C+et+al%2E+Jamba%3A+A+hybrid+transformer%2Dmamba+language+m) 307. Chin-Yew Lin. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81, 2004. 308. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100, 2024. [find](https://scholar.google.com/scholar?q=Ji+Lin%2C+Jiaming+Tang%2C+Haotian+Tang%2C+Shang+Yang%2C+Wei%2DMing+Chen%2C+Wei%2DChen+Wang%2C+Guangxuan+Xiao%2C+Xingyu+Dang%2C+Chuang+Gan%2C+and+Song+Han%2E+Awq%3A+Activation%2Daware+weight+quantization+for+on%2Ddevice+llm+compres) 309. Zhan Ling, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. Longreason: A synthetic long-context reasoning benchmark via context expansion. arXiv preprint arXiv:2501.15089, 2025. — [arXiv:2501.15089](https://arxiv.org/abs/2501.15089) [find](https://scholar.google.com/scholar?q=Zhan+Ling%2C+Kang+Liu%2C+Kai+Yan%2C+Yifan+Yang%2C+Weijian+Lin%2C+Ting%2DHan+Fan%2C+Lingfeng+Shen%2C+Zhengyin+Du%2C+and+Jiecao+Chen%2E+Longreason%3A+A+synthetic+long%2Dcontext+reasoning+benchmark+via+context+expansion%2E+arXiv+) 310. Lucas D. Lingle. Transformer-vq: Linear-time transformers via vector quantization. In ICLR. OpenReview.net, 2024. [find](https://scholar.google.com/scholar?q=Lucas+D%2E+Lingle%2E+Transformer%2Dvq%3A+Linear%2Dtime+transformers+via+vector+quantization%2E+In+ICLR%2E+OpenReview%2Enet%2C+2024%2E) 311. Barys Liskavets, Maxim Ushakov, Shuvendu Roy, Mark Klibanov, Ali Etemad, and Shane Luke. Prompt compression with context-aware sentence encoding for fast and improved llm inference, 2024. URL https://arxiv.org/abs/2409.01227 . — [arXiv:2409.01227](https://arxiv.org/abs/2409.01227) [↗](https://arxiv.org/abs/2409.01227) 312. Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, 2024. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bin+Wang%2C+Bingxuan+Wang%2C+Bo+Liu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+et+al%2E+Deepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+language) 313. Hao Liu, Matei Zaharia, and Pieter Abbeel. [Ring attention with blockwise transformers for near-infinite context](https://md.research.elamur.ai/free/en/papers/liu-2023-ring-attention-blockwise-transformers-near/). arXiv preprint arXiv:2310.01889, 2023. 314. Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv e-prints, pages arXiv–2402, 2024. [find](https://scholar.google.com/scholar?q=Hao+Liu%2C+Wilson+Yan%2C+Matei+Zaharia%2C+and+Pieter+Abbeel%2E+World+model+on+million%2Dlength+video+and+language+with+ringattention%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932402%2C+2024%2E) 315. Jiaheng Liu, Zehao Ni, Haoran Que, Tao Sun, Noah Wang, Jian Yang, JiakaiWang, Hongcheng Guo, Z.Y. Peng, Ge Zhang, Jiayi Tian, Xingyuan Bu, Ke Xu, Wenge Rong, Junran Peng, and Zhaoxiang Zhang. Roleagent: Building, interacting, and benchmarking high-quality role-playing agents from scripts. In The Thi… [find](https://scholar.google.com/scholar?q=Jiaheng+Liu%2C+Zehao+Ni%2C+Haoran+Que%2C+Tao+Sun%2C+Noah+Wang%2C+Jian+Yang%2C+JiakaiWang%2C+Hongcheng+Guo%2C+Z%2EY%2E+Peng%2C+Ge+Zhang%2C+Jiayi+Tian%2C+Xingyuan+Bu%2C+Ke+Xu%2C+Wenge+Rong%2C+Junran+Peng%2C+and+Zhaoxiang+Zhang%2E+Roleagen) 316. Jiaheng Liu, ZhiqiBai ZhiqiBai, Yuanxing Zhang, Chenchen Zhang, YuangZh YuangZh, Ge Zhang, JiakaiWang JiakaiWang, Haoran Que, Yukang Chen, Wenbo Su, Tiezheng Ge, Jie Fu, Wenhu Chen, and Bo Zheng. E2-LLM: Efficient and extreme length extension of large language models. In Lun-Wei Ku, Andre Martins, a… [find](https://scholar.google.com/scholar?q=Jiaheng+Liu%2C+ZhiqiBai+ZhiqiBai%2C+Yuanxing+Zhang%2C+Chenchen+Zhang%2C+YuangZh+YuangZh%2C+Ge+Zhang%2C+JiakaiWang+JiakaiWang%2C+Haoran+Que%2C+Yukang+Chen%2C+Wenbo+Su%2C+Tiezheng+Ge%2C+Jie+Fu%2C+Wenhu+Chen%2C+and+Bo+Zheng%2E+E2%2DL) 317. Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang. Repoqa: Evaluating long context code understanding. arXiv preprint arXiv:2406.06025, 2024. — [arXiv:2406.06025](https://arxiv.org/abs/2406.06025) [find](https://scholar.google.com/scholar?q=Jiawei+Liu%2C+Jia+Le+Tian%2C+Vijay+Daita%2C+Yuxiang+Wei%2C+Yifeng+Ding%2C+Yuhan+Katherine+Wang%2C+Jun+Yang%2C+and+Lingming+Zhang%2E+Repoqa%3A+Evaluating+long+context+code+understanding%2E+arXiv+preprint+arXiv%3A2406%2E06025%2C) 318. Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? CoRR, abs/2404.05955, 2024. doi: 10.48550/ARXIV.2404.05955. URL https://doi.org/10.48550/arXiv.2404.05955 . — [doi:10.48550/arxiv.2404.05955](https://doi.org/10.48550/arxiv.2404.05955) [↗](https://doi.org/10.48550/arXiv.2404.05955) 319. Junwei Liu, Yixuan Chen, Mingwei Liu, Xin Peng, and Yiling Lou. Stall+: Boosting llm-based repository-level code completion with static analysis. arXiv preprint arXiv:2406.10018, 2024. — [arXiv:2406.10018](https://arxiv.org/abs/2406.10018) [find](https://scholar.google.com/scholar?q=Junwei+Liu%2C+Yixuan+Chen%2C+Mingwei+Liu%2C+Xin+Peng%2C+and+Yiling+Lou%2E+Stall%2B%3A+Boosting+llm%2Dbased+repository%2Dlevel+code+completion+with+static+analysis%2E+arXiv+preprint+arXiv%3A2406%2E10018%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E1) 320. Junyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, and Yiming Qian. TCRA-LLM: Token compression retrieval augmented large language model for inference cost reduction. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9796… [find](https://scholar.google.com/scholar?q=Junyi+Liu%2C+Liangzhi+Li%2C+Tong+Xiang%2C+Bowen+Wang%2C+and+Yiming+Qian%2E+TCRA%2DLLM%3A+Token+compression+retrieval+augmented+large+language+model+for+inference+cost+reduction%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kal) 321. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173, 2024. 322. Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Yipeng Ma, Tao Wang, and Ji-Rong Wen. Unlocking data-free low-bit quantization with matrix decomposition for kv cache compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2430–24… [find](https://scholar.google.com/scholar?q=Peiyu+Liu%2C+Ze%2DFeng+Gao%2C+Wayne+Xin+Zhao%2C+Yipeng+Ma%2C+Tao+Wang%2C+and+Ji%2DRong+Wen%2E+Unlocking+data%2Dfree+low%2Dbit+quantization+with+matrix+decomposition+for+kv+cache+compression%2E+In+Proceedings+of+the+62nd+An) 323. Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492, 2024. [find](https://scholar.google.com/scholar?q=Qian+Liu%2C+Xiaosen+Zheng%2C+Niklas+Muennighoff%2C+Guangtao+Zeng%2C+Longxu+Dou%2C+Tianyu+Pang%2C+Jing+Jiang%2C+and+Min+Lin%2E+Regmix%3A+Data+mixture+as+regression+for+language+model+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2) 324. Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, et al. Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310, 2024. [find](https://scholar.google.com/scholar?q=Shukai+Liu%2C+Linzheng+Chai%2C+Jian+Yang%2C+Jiajun+Shi%2C+He+Zhu%2C+Liran+Wang%2C+Ke+Jin%2C+Wei+Zhang%2C+Hualei+Zhu%2C+Shuyue+Guo%2C+et+al%2E+Mdeval%3A+Massively+multilingual+code+debugging%2E+arXiv+preprint+arXiv%3A2411%2E02310%2C+) 325. Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. In The Twelfth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Tianyang+Liu%2C+Canwen+Xu%2C+and+Julian+McAuley%2E+Repobench%3A+Benchmarking+repository%2Dlevel+code+auto%2Dcompletion+systems%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2024%2E) 326. Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model. CoRR, abs/2411.10803, 2024. doi: 10.48550/ARXIV.2411.10803. URL https://doi.org/10.485 50/arXiv.2411.10803 . — [doi:10.48550/arxiv.2411.10803](https://doi.org/10.48550/arxiv.2411.10803) [find](https://scholar.google.com/scholar?q=Ting+Liu%2C+Liangtao+Shi%2C+Richang+Hong%2C+Yue+Hu%2C+Quanjun+Yin%2C+and+Linfeng+Zhang%2E+Multi%2Dstage+vision+token+dropping%3A+Towards+efficient+multimodal+large+language+model%2E+CoRR%2C+abs%2F2411%2E10803%2C+2024%2E+doi%3A+10%2E) 327. Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. [What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-makes-good-data-alignment-comprehensive/). In The Twelfth International Conference on Learning Representations, 2024. 328. Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. Longgenbench: Long-context generation benchmark. arXiv preprint arXiv:2410.04199, 2024. [find](https://scholar.google.com/scholar?q=Xiang+Liu%2C+Peijie+Dong%2C+Xuming+Hu%2C+and+Xiaowen+Chu%2E+Longgenbench%3A+Long%2Dcontext+generation+benchmark%2E+arXiv+preprint+arXiv%3A2410%2E04199%2C+2024%2E) 329. Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. [Scaling laws of rope-based extrapolation](https://md.research.elamur.ai/free/en/papers/liu-2023-scaling-laws-rope-based/). arXiv preprint arXiv:2310.05209, 2023. 330. Xiaoran Liu, Kai Lv, Qipeng Guo, Hang Yan, Conghui He, Xipeng Qiu, and Dahua Lin. LongWanjuan: Towards systematic measurement for long text quality. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5709–5725,… [find](https://scholar.google.com/scholar?q=Xiaoran+Liu%2C+Kai+Lv%2C+Qipeng+Guo%2C+Hang+Yan%2C+Conghui+He%2C+Xipeng+Qiu%2C+and+Dahua+Lin%2E+LongWanjuan%3A+Towards+systematic+measurement+for+long+text+quality%2E+In+Yaser+Al%2DOnaizan%2C+Mohit+Bansal%2C+and+Yun%2DNung+Che) 331. Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, Yaqian Zhou, Xuanjing Huang, and Xipeng Qiu. Thus spake long-context large language model. 2025. URL [https://api.semanticscholar.org/CorpusID:276575712](https://api.semanticscholar.org/CorpusID:276575712) . [↗](https://api.semanticscholar.org/CorpusID:276575712) 332. Yang Liu, Jiaxiang Liu, Li Chen, Yuxiang Lu, Shikun Feng, Zhida Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE-SPARSE: learning hierarchical efficient transformer through regularized self-attention. CoRR, abs/2203.12276, 2022. — [arXiv:2203.12276](https://arxiv.org/abs/2203.12276) [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Jiaxiang+Liu%2C+Li+Chen%2C+Yuxiang+Lu%2C+Shikun+Feng%2C+Zhida+Feng%2C+Yu+Sun%2C+Hao+Tian%2C+Hua+Wu%2C+and+Haifeng+Wang%2E+ERNIE%2DSPARSE%3A+learning+hierarchical+efficient+transformer+through+regularized+self%2Datt) 333. Yiheng Liu, Hao He, Tianle Han, Xu Zhang, Mengyuan Liu, Jiaming Tian, Yutong Zhang, Jiaqi Wang, Xiaohui Gao, Tianyang Zhong, et al. Understanding llms: A comprehensive overview from training to inference. arXiv preprint arXiv:2401.02038, 2024. — [arXiv:2401.02038](https://arxiv.org/abs/2401.02038) [find](https://scholar.google.com/scholar?q=Yiheng+Liu%2C+Hao+He%2C+Tianle+Han%2C+Xu+Zhang%2C+Mengyuan+Liu%2C+Jiaming+Tian%2C+Yutong+Zhang%2C+Jiaqi+Wang%2C+Xiaohui+Gao%2C+Tianyang+Zhong%2C+et+al%2E+Understanding+llms%3A+A+comprehensive+overview+from+training+to+infere) 334. Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 38–56, 2024. [find](https://scholar.google.com/scholar?q=Yuhan+Liu%2C+Hanchen+Li%2C+Yihua+Cheng%2C+Siddhant+Ray%2C+Yuyang+Huang%2C+Qizheng+Zhang%2C+Kuntai+Du%2C+Jiayi+Yao%2C+Shan+Lu%2C+Ganesh+Ananthanarayanan%2C+et+al%2E+Cachegen%3A+Kv+cache+compression+and+streaming+for+fast+larg) 335. Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and… [find](https://scholar.google.com/scholar?q=Zichang+Liu%2C+Aditya+Desai%2C+Fangshuo+Liao%2C+Weitao+Wang%2C+Victor+Xie%2C+Zhaozhuo+Xu%2C+Anastasios+Kyrillidis%2C+and+Anshumali+Shrivastava%2E+Scissorhands%3A+Exploiting+the+persistence+of+importance+hypothesis+for+) 336. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. [KIVI: A tuning-free asymmetric 2bit quantization for KV cache](https://md.research.elamur.ai/free/en/papers/liu-2024-kivi-tuning-free-asymmetric-2bit/). In Proceedings of the 41st International Conference on Machine Learning, pages 32332–32344, 2024. 337. Yinghan Long, Sayeed Shafayet Chowdhury, and Kaushik Roy. Segmented recurrent transformer: An efficient sequence-to-sequence model. In EMNLP (Findings), pages 8325– 8337. Association for Computational Linguistics, 2023. [find](https://scholar.google.com/scholar?q=Yinghan+Long%2C+Sayeed+Shafayet+Chowdhury%2C+and+Kaushik+Roy%2E+Segmented+recurrent+transformer%3A+An+efficient+sequence%2Dto%2Dsequence+model%2E+In+EMNLP+%28Findings%29%2C+pages+8325%E2%80%93+8337%2E+Association+for+Computational) 338. Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li,… [find](https://scholar.google.com/scholar?q=Anton+Lozhkov%2C+Raymond+Li%2C+Loubna+Ben+Allal%2C+Federico+Cassano%2C+Joel+Lamy%2DPoirier%2C+Nouamane+Tazi%2C+Ao+Tang%2C+Dmytro+Pykhtar%2C+Jiawei+Liu%2C+Yuxiang+Wei%2C+Tianyang+Liu%2C+Max+Tian%2C+Denis+Kocetkov%2C+Arthur+Zucker) 339. [Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiez…](https://md.research.elamur.ai/free/en/papers/lu-2025-moba-mixture-block-attention/) 340. Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. [Moba: Mixture of block attention for long-context llms](https://md.research.elamur.ai/free/en/papers/lu-2025-moba-mixture-block-attention/). arXiv preprint arXiv:2502.13189, 2025. 341. Keer Lu, Xiaonan Nie, Zheng Liang, Da Pan, Shusen Zhang, Keshi Zhao, Weipeng Chen, Zenan Zhou, Guosheng Dong, Bin Cui, et al. Datasculpt: Crafting data landscapes for long-context llms through multi-objective partitioning. arXiv preprint arXiv:2409.00997, 2024. — [arXiv:2409.00997](https://arxiv.org/abs/2409.00997) [find](https://scholar.google.com/scholar?q=Keer+Lu%2C+Xiaonan+Nie%2C+Zheng+Liang%2C+Da+Pan%2C+Shusen+Zhang%2C+Keshi+Zhao%2C+Weipeng+Chen%2C+Zenan+Zhou%2C+Guosheng+Dong%2C+Bin+Cui%2C+et+al%2E+Datasculpt%3A+Crafting+data+landscapes+for+long%2Dcontext+llms+through+multi%2Do) 342. Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M Rush. A controlled study on long context extension and generalization in llms. arXiv preprint arXiv:2409.12181, 2024. — [arXiv:2409.12181](https://arxiv.org/abs/2409.12181) [find](https://scholar.google.com/scholar?q=Yi+Lu%2C+Jing+Nathan+Yan%2C+Songlin+Yang%2C+Justin+T+Chiu%2C+Siyu+Ren%2C+Fei+Yuan%2C+Wenting+Zhao%2C+Zhiyong+Wu%2C+and+Alexander+M+Rush%2E+A+controlled+study+on+long+context+extension+and+generalization+in+llms%2E+arXiv+) 343. Yi Lu, Xin Zhou, Wei He, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, and Xuanjing Huang. Longheads: Multi-head attention is secretly a long context processor. In EMNLP (Findings), pages 7136–7148. Association for Computational Linguistics, 2024. [find](https://scholar.google.com/scholar?q=Yi+Lu%2C+Xin+Zhou%2C+Wei+He%2C+Jun+Zhao%2C+Tao+Ji%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+Longheads%3A+Multi%2Dhead+attention+is+secretly+a+long+context+processor%2E+In+EMNLP+%28Findings%29%2C+pages+7136%E2%80%937148%2E+Associatio) 344. Evan Lucas, Dylan Kangas, and Timothy C Havens. Extra global attention designation using keyword detection in sparse transformer architectures, 2024. URL https://arxiv.org/abs/2410.08971 . — [arXiv:2410.08971](https://arxiv.org/abs/2410.08971) [↗](https://arxiv.org/abs/2410.08971) 345. Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, et al. Repoagent: An llm-powered open-source framework for repository-level code documentation generation. arXiv preprint arXiv:2402.16667, 2024. [find](https://scholar.google.com/scholar?q=Qinyu+Luo%2C+Yining+Ye%2C+Shihao+Liang%2C+Zhong+Zhang%2C+Yujia+Qin%2C+Yaxi+Lu%2C+Yesai+Wu%2C+Xin+Cong%2C+Yankai+Lin%2C+Yingli+Zhang%2C+et+al%2E+Repoagent%3A+An+llm%2Dpowered+open%2Dsource+framework+for+repository%2Dlevel+code+docu) 346. Chenyang Lyu, Zefeng Du, Jitao Xu, Yitao Duan, Minghao Wu, Teresa Lynn, Alham Fikri Aji, Derek F. Wong, and Longyue Wang. A paradigm shift: The future of machine translation lies with large language models. In Nicoletta Calzolari, Min-Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and N… [find](https://scholar.google.com/scholar?q=Chenyang+Lyu%2C+Zefeng+Du%2C+Jitao+Xu%2C+Yitao+Duan%2C+Minghao+Wu%2C+Teresa+Lynn%2C+Alham+Fikri+Aji%2C+Derek+F%2E+Wong%2C+and+Longyue+Wang%2E+A+paradigm+shift%3A+The+future+of+machine+translation+lies+with+large+language+m) 347. Feipeng Ma, Hongwei Xue, Guangting Wang, Yizhou Zhou, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, and Xiaoyan Sun. Visual perception by large language model’s weights. CoRR, abs/2405.20339, 2024. doi: 10.48550/ARXIV.2405.20339. URL https://doi.org/10.48550/arXiv.2405.20339 . — [doi:10.48550/arxiv.2405.20339](https://doi.org/10.48550/arxiv.2405.20339) [↗](https://doi.org/10.48550/arXiv.2405.20339) 348. Xinbei Ma, Yeyun Gong, Pengcheng He, hai zhao, and Nan Duan. [Query rewriting in retrieval-augmented large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-query-rewriting-retrieval-augmented-large/). In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL [https://openreview.net/forum?id=gXq1cw](https://openreview.net/forum?id=gXq1cw) kUZc . [↗](https://openreview.net/forum?id=gXq1cw) 349. Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. [Cot-valve: Length-compressible chain-of-thought tuning](https://md.research.elamur.ai/free/en/papers/ma-2025-cot-valve-length-compressible-chain/). arXiv preprint arXiv:2502.09601, 2025. 350. Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. [Fine-tuning llama for multi-stage text retrieval](https://md.research.elamur.ai/free/en/papers/ma-2023-fine-tuning-llama-multi-stage/). In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2421–2425, 2024. 351. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. CoRR, abs/2407.01523,… [find](https://scholar.google.com/scholar?q=Yubo+Ma%2C+Yuhang+Zang%2C+Liangyu+Chen%2C+Meiqi+Chen%2C+Yizhu+Jiao%2C+Xinze+Li%2C+Xinyuan+Lu%2C+Ziyu+Liu%2C+Yan+Ma%2C+Xiaoyi+Dong%2C+Pan+Zhang%2C+Liangming+Pan%2C+Yu%2DGang+Jiang%2C+Jiaqi+Wang%2C+Yixin+Cao%2C+and+Aixin+Sun%2E+Mmlongbe) 352. Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv: 2201.06009, 2022. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Peter+Clark%2C+and+Yiming+Yang%2E+Memory%2Dassisted+prompt+editing+to+improve+gpt%2D3+after+deployment%2E+arXiv+preprint+arXiv%3A+2201%2E06009%2C+2022%2E) 353. Seiji Maekawa * , Hayate Iso * , and Nikita Bhutani. Holistic reasoning with long-context lms: A benchmark for database operations on massive textual data, 2024. URL https://arxiv.org/abs/2410.11996 . * These authors contributed equally to this work. — [arXiv:2410.11996](https://arxiv.org/abs/2410.11996) [↗](https://arxiv.org/abs/2410.11996) 354. Adyasha Maharana, Prateek Yadav, and Mohit Bansal. D2 pruning: Message passing for balancing diversity and difficulty in data pruning. CoRR, abs/2310.07931, 2023. doi: 10.48550/ARXIV.2310.07931. URL https://doi.org/10.48550/arXiv.2310.07931 . — [doi:10.48550/arxiv.2310.07931](https://doi.org/10.48550/arxiv.2310.07931) [↗](https://doi.org/10.48550/arXiv.2310.07931) 355. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. [Expertqa: Expert-curated questions and attributed answers](https://md.research.elamur.ai/free/en/papers/malaviya-2024-expertqa-expert-curated-questions-attributed/). arXiv preprint arXiv:2309.07852, 2023. 356. Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev, Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar, Dipanjan Das, Mirella Lapata, and Chris Alberti. Dolomites: Domain-specific long-form methodical tasks. arXiv preprint arXiv:2405.05938, 2024. [find](https://scholar.google.com/scholar?q=Chaitanya+Malaviya%2C+Priyanka+Agrawal%2C+Kuzman+Ganchev%2C+Pranesh+Srinivasan%2C+Fantine+Huot%2C+Jonathan+Berant%2C+Mark+Yatskar%2C+Dipanjan+Das%2C+Mirella+Lapata%2C+and+Chris+Alberti%2E+Dolomites%3A+Domain%2Dspecific+long%2D) 357. Ahmed Masry and Amir Hajian. Longfin: A multimodal document understanding model for long financial domain documents. CoRR, abs/2401.15050, 2024. doi: 10.48550/ARXIV.2 401.15050. URL https://doi.org/10.48550/arXiv.2401.15050 . — [doi:10.48550/arxiv.2](https://doi.org/10.48550/arxiv.2) [↗](https://doi.org/10.48550/arXiv.2401.15050) 358. Ahmed Masry and Amir Hajian. Longfin: A multimodal document understanding model for long financial domain documents. arXiv preprint arXiv:2401.15050, 2024. — [arXiv:2401.15050](https://arxiv.org/abs/2401.15050) [find](https://scholar.google.com/scholar?q=Ahmed+Masry+and+Amir+Hajian%2E+Longfin%3A+A+multimodal+document+understanding+model+for+long+financial+domain+documents%2E+arXiv+preprint+arXiv%3A2401%2E15050%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E15050%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2) 359. Sanket Vaibhav Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Q. Tran, J. Rao, Marc Najork, Emma Strubell, and Donald Metzler. Dsi++: Updating transformer memory with new documents. Conference on Empirical Methods in Natural Language Processing, 2022. doi: 10.48550/arXiv.2212.09744. [find](https://scholar.google.com/scholar?q=Sanket+Vaibhav+Mehta%2C+Jai+Gupta%2C+Yi+Tay%2C+Mostafa+Dehghani%2C+Vinh+Q%2E+Tran%2C+J%2E+Rao%2C+Marc+Najork%2C+Emma+Strubell%2C+and+Donald+Metzler%2E+Dsi%2B%2B%3A+Updating+transformer+memory+with+new+documents%2E+Conference+on+Em) 360. Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length, 2024. URL [https://arxiv.org/ab](https://arxiv.org/ab) s/2405.14591 . [↗](https://arxiv.org/ab) 361. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 17359–17372. Curran Associates, Inc., 2022.… [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+Locating+and+editing+factual+associations+in+gpt%2E+In+S%2E+Koyejo%2C+S%2E+Mohamed%2C+A%2E+Agarwal%2C+D%2E+Belgrave%2C+K%2E+Cho%2C+and+A%2E+Oh%2C+editors%2C+Advances+in) 362. Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. ArXiv, abs/2405.14734, 2024. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+Simpo%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2E+ArXiv%2C+abs%2F2405%2E14734%2C+2024%2E) 363. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017. [find](https://scholar.google.com/scholar?q=Paulius+Micikevicius%2C+Sharan+Narang%2C+Jonah+Alben%2C+Gregory+Diamos%2C+Erich+Elsen%2C+David+Garcia%2C+Boris+Ginsburg%2C+Michael+Houston%2C+Oleksii+Kuchaiev%2C+Ganesh+Venkatesh%2C+et+al%2E+Mixed+precision+training%2E+arXiv) 364. Microsoft. Microsoft copilot, 2023. URL [https://copilot.microsoft.com/](https://copilot.microsoft.com/) . [↗](https://copilot.microsoft.com/) 365. Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. [Document-level neural machine translation with hierarchical attention networks](https://md.research.elamur.ai/free/en/papers/miculicich-2018-document-level-neural-machine-translation/). In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural La… 366. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, pages 1045–1048. Makuhari, 2010. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Martin+Karafi%C3%A1t%2C+Lukas+Burget%2C+Jan+Cernocky%CC%80%2C+and+Sanjeev+Khudanpur%2E+Recurrent+neural+network+based+language+model%2E+In+Interspeech%2C+volume+2%2C+pages+1045%E2%80%931048%2E+Makuhari%2C+2010%2E) 367. Evan Miller. Attention is off by one, 7 2023. URL [https://www.evanmiller.org/att](https://www.evanmiller.org/att) ention-is-off-by-one.html . [↗](https://www.evanmiller.org/att) 368. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004.10645, 2020. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+Ambigqa%3A+Answering+ambiguous+open%2Ddomain+questions%2E+arXiv+preprint+arXiv%3A2004%2E10645%2C+2020%2E) 369. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. [Factscore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). arXiv preprint arXiv:2305.14251, 2023. 370. MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaish… [find](https://scholar.google.com/scholar?q=MiniMax%2C+Aonian+Li%2C+Bangwei+Gong%2C+Bo+Yang%2C+Boji+Shan%2C+Chang+Liu%2C+Cheng+Zhu%2C+Chunhao+Zhang%2C+Congchao+Guo%2C+Da+Chen%2C+Dong+Li%2C+Enwei+Jiao%2C+Gengxin+Li%2C+Guojun+Zhang%2C+Haohai+Sun%2C+Houze+Dong%2C+Jiadai+Zhu%2C+Jia) 371. Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, Manish Sethi, Xuan-Hong Dang, Pengyuan Li, Kun-Lung Wu, Syed Zawad, Andrew Coleman, Matthew White, Mark Lewis, Raju Pavuluri, Yan Koyfma… [find](https://scholar.google.com/scholar?q=Mayank+Mishra%2C+Matt+Stallone%2C+Gaoyuan+Zhang%2C+Yikang+Shen%2C+Aditya+Prasad%2C+Adriana+Meza+Soria%2C+Michele+Merler%2C+Parameswaran+Selvam%2C+Saptha+Surendran%2C+Shivdeep+Singh%2C+Manish+Sethi%2C+Xuan%2DHong+Dang%2C+Pengyu) 372. Fengran Mo, Kelong Mao, Ziliang Zhao, Hongjin Qian, Haonan Chen, Yiruo Cheng, Xiaoxi Li, Yutao Zhu, Zhicheng Dou, and Jian-Yun Nie. A survey of conversational search. arXiv preprint arXiv: 2410.15576, 2024. — [arXiv:2410.15576](https://arxiv.org/abs/2410.15576) [find](https://scholar.google.com/scholar?q=Fengran+Mo%2C+Kelong+Mao%2C+Ziliang+Zhao%2C+Hongjin+Qian%2C+Haonan+Chen%2C+Yiruo+Cheng%2C+Xiaoxi+Li%2C+Yutao+Zhu%2C+Zhicheng+Dou%2C+and+Jian%2DYun+Nie%2E+A+survey+of+conversational+search%2E+arXiv+preprint+arXiv%3A+2410%2E15576%2C) 373. Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A Rossi, Seunghyun Yoon, and Hinrich Schütze. Nolima: Long-context evaluation beyond literal matching. arXiv preprint arXiv:2502.05167, 2025. — [arXiv:2502.05167](https://arxiv.org/abs/2502.05167) [find](https://scholar.google.com/scholar?q=Ali+Modarressi%2C+Hanieh+Deilamsalehy%2C+Franck+Dernoncourt%2C+Trung+Bui%2C+Ryan+A+Rossi%2C+Seunghyun+Yoon%2C+and+Hinrich+Sch%C3%BCtze%2E+Nolima%3A+Long%2Dcontext+evaluation+beyond+literal+matching%2E+arXiv+preprint+arXiv%3A250) 374. Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300, 2023. [find](https://scholar.google.com/scholar?q=Amirkeivan+Mohtashami+and+Martin+Jaggi%2E+Landmark+attention%3A+Random%2Daccess+infinite+context+length+for+transformers%2E+arXiv+preprint+arXiv%3A2305%2E16300%2C+2023%2E) 375. Jesse Mu, Xiang Li, and Noah Goodman. [Learning to compress prompts with gist tokens](https://md.research.elamur.ai/free/en/papers/mu-2023-learning-compress-prompts-gist-tokens/). In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 19327–19352. Curran Associates, Inc., 2023. URL https://proceeding… 376. Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. [Generative representational instruction tuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2024-generative-representational-instruction-tuning/). arXiv preprint arXiv: 2402.09906, 2024. 377. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/). arXiv preprint arXiv:2501.19393, 2025. 378. Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. CoRR, abs/2404.07143, 2024. [find](https://scholar.google.com/scholar?q=Tsendsuren+Munkhdalai%2C+Manaal+Faruqui%2C+and+Siddharth+Gopal%2E+Leave+no+context+behind%3A+Efficient+infinite+context+transformers+with+infini%2Dattention%2E+CoRR%2C+abs%2F2404%2E07143%2C+2024%2E) 379. Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention, 2024. URL https://arxiv.org/abs/2404.07143 . [↗](https://arxiv.org/abs/2404.07143) 380. Arsha Nagrani, Mingda Zhang, Ramin Mehran, Rachel Hornung, Nitesh Bharadwaj Gundavarapu, Nilpa Jha, Austin Myers, Xingyi Zhou, Boqing Gong, Cordelia Schmid, Mikhail Sirotenko, Yukun Zhu, and Tobias Weyand. Neptune: The long orbit to benchmarking long video understanding. CoRR, abs/2412.09582, 2024.… — [doi:10.48550/arxiv.2412.09582](https://doi.org/10.48550/arxiv.2412.09582) [find](https://scholar.google.com/scholar?q=Arsha+Nagrani%2C+Mingda+Zhang%2C+Ramin+Mehran%2C+Rachel+Hornung%2C+Nitesh+Bharadwaj+Gundavarapu%2C+Nilpa+Jha%2C+Austin+Myers%2C+Xingyi+Zhou%2C+Boqing+Gong%2C+Cordelia+Schmid%2C+Mikhail+Sirotenko%2C+Yukun+Zhu%2C+and+Tobias+We) 381. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023, 2016. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Caglar+Gulcehre%2C+Bing+Xiang%2C+et+al%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2C+2016%2E) 382. Neel Nanda. Mechanistic interpretability quickstart guide, 2023. URL [https://www.ne](https://www.ne) elnanda.io/mechanistic-interpretability/quickstart . Accessed: 2025-01-30. [↗](https://www.ne) 383. Chien Van Nguyen, Huy Huu Nguyen, Thang M. Pham, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Ryan A. Rossi, Trung Bui, Viet Dac Lai, Franck Dernoncourt, and Thien Huu Nguyen. Taipan: Efficient and expressive state space language models with selective attention, 2024. URL https://arxiv.org/abs/2… — [arXiv:2410.18572](https://arxiv.org/abs/2410.18572) [↗](https://arxiv.org/abs/2…) 384. [Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human-generated machine reading comprehension dataset. 2016.](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/) 385. Yuqi Nie, Yaxuan Kong, Xiaowen Dong, John M. Mulvey, H. Vincent Poor, Qingsong Wen, and Stefan Zohren. A survey of large language models for financial applications: Progress, prospects and challenges. CoRR, abs/2406.11903, 2024. doi: 10.48550/ARXIV.2406.11903. URL https://doi.org/10.48550/arXiv.2406… — [doi:10.48550/arxiv.2406.11903](https://doi.org/10.48550/arxiv.2406.11903) [↗](https://doi.org/10.48550/arXiv.2406…) 386. NVIDIA Corporation. Nvidia h200, 2023. URL [https://www.nvidia.com/en-us/data](https://www.nvidia.com/en-us/data) -center/h200/ . [↗](https://www.nvidia.com/en-us/data) 387. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… [find](https://scholar.google.com/scholar?q=Catherine+Olsson%2C+Nelson+Elhage%2C+Neel+Nanda%2C+Nicholas+Joseph%2C+Nova+DasSarma%2C+Tom+Henighan%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da) 388. OpenAI. New embedding models and api updates, 2024. URL [https://openai.com/i](https://openai.com/i) ndex/new-embedding-models-and-api-updates/ . Accessed: 2024-01-25. [↗](https://openai.com/i) 389. OpenAI. Memory and new controls for chatgpt, 2024. URL [https://openai.com/ind](https://openai.com/ind) ex/memory-and-new-controls-for-chatgpt . Accessed: 2024-02-13. [↗](https://openai.com/ind) 390. OpenAI. Learning to reason with large language models. [https://openai.com/index/learning-to-reason-with-llms/](https://openai.com/index/learning-to-reason-with-llms/) , 2024. [↗](https://openai.com/index/learning-to-reason-with-llms/) 391. [Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/) 392. Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, and François Fleuret. Fast attention over long sequences with dynamic sparse flash attention. Advances in Neural Information Processing Systems, 36:59808–59831, 2023. [find](https://scholar.google.com/scholar?q=Matteo+Pagliardini%2C+Daniele+Paliotta%2C+Martin+Jaggi%2C+and+Fran%C3%A7ois+Fleuret%2E+Fast+attention+over+long+sequences+with+dynamic+sparse+flash+attention%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%3A) 393. Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. Giraffe: Adventures in expanding context lengths in llms. arXiv preprint arXiv:2308.10882, 2023. — [arXiv:2308.10882](https://arxiv.org/abs/2308.10882) [find](https://scholar.google.com/scholar?q=Arka+Pal%2C+Deep+Karkhanis%2C+Manley+Roberts%2C+Samuel+Dooley%2C+Arvind+Sundararajan%2C+and+Siddartha+Naidu%2E+Giraffe%3A+Adventures+in+expanding+context+lengths+in+llms%2E+arXiv+preprint+arXiv%3A2308%2E10882%2C+2023%2E+%E2%80%94+%5Ba) 394. Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. [LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression](https://md.research.elamur.ai/free/en/papers/pan-2024-llmlingua-2-data-distillation-efficient/). In Lun-Wei Ku, Andre Martins, and… 395. Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. [Quality: Question answering with long input texts, yes!](https://md.research.elamur.ai/free/en/papers/pang-2022-quality-question-answering-long-input/) In Proceedings of the 2022 Conference of the North American Chapter of the Association fo… 396. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. 397. Daon Park and Bernhard Egger. Improving throughput-oriented llm inference with cpu computations. In Proceedings of the 2024 International Conference on Parallel Architectures and Compilation Techniques, pages 233–245, 2024. [find](https://scholar.google.com/scholar?q=Daon+Park+and+Bernhard+Egger%2E+Improving+throughput%2Doriented+llm+inference+with+cpu+computations%2E+In+Proceedings+of+the+2024+International+Conference+on+Parallel+Architectures+and+Compilation+Technique) 398. J. Park, Joseph C. O’Brien, Carrie J. Cai, M. Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. ACM Symposium on User Interface Software and Technology, 2023. doi: 10.1145/3586183.3606763. [find](https://scholar.google.com/scholar?q=J%2E+Park%2C+Joseph+C%2E+O%E2%80%99Brien%2C+Carrie+J%2E+Cai%2C+M%2E+Morris%2C+Percy+Liang%2C+and+Michael+S%2E+Bernstein%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+ACM+Symposium+on+User+Interface+Software+and+Te) 399. Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? A comparative study on in-context learning tasks. In ICML. OpenReview.net, 2024. [find](https://scholar.google.com/scholar?q=Jongho+Park%2C+Jaeseung+Park%2C+Zheyang+Xiong%2C+Nayoung+Lee%2C+Jaewoong+Cho%2C+Samet+Oymak%2C+Kangwook+Lee%2C+and+Dimitris+Papailiopoulos%2E+Can+mamba+learn+how+to+learn%3F+A+comparative+study+on+in%2Dcontext+learning+t) 400. Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–132. IEEE, 2024. [find](https://scholar.google.com/scholar?q=Pratyush+Patel%2C+Esha+Choukse%2C+Chaojie+Zhang%2C+Aashaka+Shah%2C+%C3%8D%C3%B1igo+Goiri%2C+Saeed+Maleki%2C+and+Ricardo+Bianchini%2E+Splitwise%3A+Efficient+generative+llm+inference+using+phase+splitting%2E+In+2024+ACM%2FIEEE+51st+) 401. Saurav Pawar, S. M. Towhidul Islam Tonmoy, S. M. Mehedi Zaman, Vinija Jain, Aman Chadha, and Amitava Das. The what, why, and how of context length extension techniques in large language models - A detailed survey. CoRR, abs/2401.07872, 2024. doi: 10.48550/A RXIV.2401.07872. URL https://doi.org/10.48… — [doi:10.48550/a](https://doi.org/10.48550/a) [find](https://scholar.google.com/scholar?q=Saurav+Pawar%2C+S%2E+M%2E+Towhidul+Islam+Tonmoy%2C+S%2E+M%2E+Mehedi+Zaman%2C+Vinija+Jain%2C+Aman+Chadha%2C+and+Amitava+Das%2E+The+what%2C+why%2C+and+how+of+context+length+extension+techniques+in+large+language+models+%2D+A+det) 402. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Li… [find](https://scholar.google.com/scholar?q=Bo+Peng%2C+Eric+Alcaide%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Samuel+Arcadinho%2C+Stella+Biderman%2C+Huanqi+Cao%2C+Xin+Cheng%2C+Michael+Chung%2C+Leon+Derczynski%2C+Xingjian+Du%2C+Matteo+Grella%2C+Kranthi+Kiran+GV%2C+Xuzheng+He) 403. Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Jiaju Lin, Niklas Muennighoff, Fares Obeid… [find](https://scholar.google.com/scholar?q=Bo+Peng%2C+Daniel+Goldstein%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Eric+Alcaide%2C+Stella+Biderman%2C+Eugene+Cheah%2C+Xingjian+Du%2C+Teddy+Ferdinan%2C+Haowen+Hou%2C+Przemys%C5%82aw+Kazienko%2C+Kranthi+Kiran+GV%2C+Jan+Kocon%CC%81%2C+Bart%C5%82) 404. Bowen Peng and Jeffrey Quesnelle. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. [https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled](https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled) _rope_allows_llama_models_to_have , 2023. [↗](https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled) 405. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2C+2023%2E) 406. Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, et al. Fp8-lm: Training fp8 large language models. arXiv preprint arXiv:2310.18313, 2023. [find](https://scholar.google.com/scholar?q=Houwen+Peng%2C+Kan+Wu%2C+Yixuan+Wei%2C+Guoshuai+Zhao%2C+Yuxiang+Yang%2C+Ze+Liu%2C+Yifan+Xiong%2C+Ziyue+Yang%2C+Bolin+Ni%2C+Jingcheng+Hu%2C+et+al%2E+Fp8%2Dlm%3A+Training+fp8+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E1831) 407. Chau Minh Pham, Simeng Sun, and Mohit Iyyer. Suri: Multi-constraint instruction following for long-form text generation. arXiv preprint arXiv:2406.19371, 2024. — [arXiv:2406.19371](https://arxiv.org/abs/2406.19371) [find](https://scholar.google.com/scholar?q=Chau+Minh+Pham%2C+Simeng+Sun%2C+and+Mohit+Iyyer%2E+Suri%3A+Multi%2Dconstraint+instruction+following+for+long%2Dform+text+generation%2E+arXiv+preprint+arXiv%3A2406%2E19371%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E19371%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa) 408. Hung Phan, Anurag Acharya, Rounak Meyur, Sarthak Chaturvedi, Shivam Sharma, Mike Parker, Dan Nally, Ali Jannesari, Karl Pazdernik, Mahantesh Halappanavar, et al. Examining long-context large language models for environmental review document comprehension. arXiv preprint arXiv:2407.07321, 2024. — [arXiv:2407.07321](https://arxiv.org/abs/2407.07321) [find](https://scholar.google.com/scholar?q=Hung+Phan%2C+Anurag+Acharya%2C+Rounak+Meyur%2C+Sarthak+Chaturvedi%2C+Shivam+Sharma%2C+Mike+Parker%2C+Dan+Nally%2C+Ali+Jannesari%2C+Karl+Pazdernik%2C+Mahantesh+Halappanavar%2C+et+al%2E+Examining+long%2Dcontext+large+language+) 409. Bowen Ping, Jiali Zeng, Fandong Meng, Shuo Wang, Jie Zhou, and Shanghang Zhang. Longdpo: Unlock better long-form generation abilities for llms via critique-augmented stepwise information, 2025. URL https://arxiv.org/abs/2502.02095 . [↗](https://arxiv.org/abs/2502.02095) 410. Hadi Pouransari, Chun-Liang Li, Jen-Hao Rick Chang, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar, and Oncel Tuzel. Dataset decomposition: Faster llm training with variable sequence length curriculum. arXiv preprint arXiv:2405.13226, 2024. [find](https://scholar.google.com/scholar?q=Hadi+Pouransari%2C+Chun%2DLiang+Li%2C+Jen%2DHao+Rick+Chang%2C+Pavan+Kumar+Anasosalu+Vasu%2C+Cem+Koc%2C+Vaishaal+Shankar%2C+and+Oncel+Tuzel%2E+Dataset+decomposition%3A+Faster+llm+training+with+variable+sequence+length+cur) 411. Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. SPIQA: A dataset for multimodal question answering on scientific papers. CoRR, abs/2407.09413, 2024. doi: 10.48550/ARXIV.2407.09413. URL https://doi.org/10.48550/arXiv.2407.09413 . — [doi:10.48550/arxiv.2407.09413](https://doi.org/10.48550/arxiv.2407.09413) [↗](https://doi.org/10.48550/arXiv.2407.09413) 412. Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A+Smith%2C+and+Mike+Lewis%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+arXiv+preprint+arXiv%3A2108%2E12409%2C+2021%2E) 413. Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. Zero bubble pipeline parallelism. arXiv preprint arXiv:2401.10241, 2023. [find](https://scholar.google.com/scholar?q=Penghui+Qi%2C+Xinyi+Wan%2C+Guangxing+Huang%2C+and+Min+Lin%2E+Zero+bubble+pipeline+parallelism%2E+arXiv+preprint+arXiv%3A2401%2E10241%2C+2023%2E) 414. Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang, Hao Zhang, and Wei Xu. Long2rag: Evaluating long-context & long-form retrieval-augmented generation with key point recall. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4852–4872, 2024. [find](https://scholar.google.com/scholar?q=Zehan+Qi%2C+Rongwu+Xu%2C+Zhijiang+Guo%2C+Cunxiang+Wang%2C+Hao+Zhang%2C+and+Wei+Xu%2E+Long2rag%3A+Evaluating+long%2Dcontext+%26+long%2Dform+retrieval%2Daugmented+generation+with+key+point+recall%2E+In+Findings+of+the+Associat) 415. Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: A kvcache-centric disaggregated architecture for llm serving. arXiv preprint arXiv:2407.00079, 2024. [find](https://scholar.google.com/scholar?q=Ruoyu+Qin%2C+Zheming+Li%2C+Weiran+He%2C+Mingxing+Zhang%2C+Yongwei+Wu%2C+Weimin+Zheng%2C+and+Xinran+Xu%2E+Mooncake%3A+A+kvcache%2Dcentric+disaggregated+architecture+for+llm+serving%2E+arXiv+preprint+arXiv%3A2407%2E00079%2C+2024) 416. Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. O1 replication journey: A strategic progress report–part 1. arXiv preprint arXiv:2410.18982, 2024. [find](https://scholar.google.com/scholar?q=Yiwei+Qin%2C+Xuefeng+Li%2C+Haoyang+Zou%2C+Yixiu+Liu%2C+Shijie+Xia%2C+Zhen+Huang%2C+Yixin+Ye%2C+Weizhe+Yuan%2C+Hector+Liu%2C+Yuanzhi+Li%2C+et+al%2E+O1+replication+journey%3A+A+strategic+progress+report%E2%80%93part+1%2E+arXiv+preprint+) 417. Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In ICLR. OpenReview.net, 2022. [find](https://scholar.google.com/scholar?q=Zhen+Qin%2C+Weixuan+Sun%2C+Hui+Deng%2C+Dongxu+Li%2C+Yunshen+Wei%2C+Baohong+Lv%2C+Junjie+Yan%2C+Lingpeng+Kong%2C+and+Yiran+Zhong%2E+cosformer%3A+Rethinking+softmax+in+attention%2E+In+ICLR%2E+OpenReview%2Enet%2C+2022%2E) 418. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models, 2024. URL https://arxiv.org/abs/2401.04658 . [↗](https://arxiv.org/abs/2401.04658) 419. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Various lengths, constant speed: Efficient language modeling with lightning attention. In ICML. OpenReview.net, 2024. [find](https://scholar.google.com/scholar?q=Zhen+Qin%2C+Weigao+Sun%2C+Dong+Li%2C+Xuyang+Shen%2C+Weixuan+Sun%2C+and+Yiran+Zhong%2E+Various+lengths%2C+constant+speed%3A+Efficient+language+modeling+with+lightning+attention%2E+In+ICML%2E+OpenReview%2Enet%2C+2024%2E) 420. Zexuan Qiu, Jingjing Li, Shijue Huang, Xiaoqi Jiao, Wanjun Zhong, and Irwin King. Clongeval: A chinese benchmark for evaluating long-context large language models. arXiv preprint arXiv:2403.03514, 2024. [find](https://scholar.google.com/scholar?q=Zexuan+Qiu%2C+Jingjing+Li%2C+Shijue+Huang%2C+Xiaoqi+Jiao%2C+Wanjun+Zhong%2C+and+Irwin+King%2E+Clongeval%3A+A+chinese+benchmark+for+evaluating+long%2Dcontext+large+language+models%2E+arXiv+preprint+arXiv%3A2403%2E03514%2C+202) 421. Shanghaoran Quan, Tianyi Tang, Bowen Yu, An Yang, Dayiheng Liu, Bofei Gao, Jianhong Tu, Yichang Zhang, Jingren Zhou, and Junyang Lin. Language models can self-lengthen to generate long texts. arXiv preprint arXiv:2410.23933, 2024. [find](https://scholar.google.com/scholar?q=Shanghaoran+Quan%2C+Tianyi+Tang%2C+Bowen+Yu%2C+An+Yang%2C+Dayiheng+Liu%2C+Bofei+Gao%2C+Jianhong+Tu%2C+Yichang+Zhang%2C+Jingren+Zhou%2C+and+Junyang+Lin%2E+Language+models+can+self%2Dlengthen+to+generate+long+texts%2E+arXiv+pr) 422. Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, et al. Hellobench: Evaluating long text generation capabilities of large language models. arXiv preprint arXiv:2409.16191, 2024. — [arXiv:2409.16191](https://arxiv.org/abs/2409.16191) [find](https://scholar.google.com/scholar?q=Haoran+Que%2C+Feiyu+Duan%2C+Liqun+He%2C+Yutao+Mou%2C+Wangchunshu+Zhou%2C+Jiaheng+Liu%2C+Wenge+Rong%2C+Zekun+Moore+Wang%2C+Jian+Yang%2C+Ge+Zhang%2C+et+al%2E+Hellobench%3A+Evaluating+long+text+generation+capabilities+of+large+) 423. Alec Radford. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E) 424. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). In ICLR. OpenReview.net, 2020. 425. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-S… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+H%2E+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick) 426. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). ArXiv, abs/2305.18290, 2023. URL [https://api.semanticscholar.org/Corp](https://api.semanticscholar.org/Corp) usID:258959321 . [↗](https://api.semanticscholar.org/Corp) 427. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67, 2020. 428. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26th+ACM+) 429. Varshini Reddy, Rik Koncel-Kedziorski, Viet Dac Lai, Michael Krumdick, Charles Lovering, and Chris Tanner. Docfinqa: A long-context financial reasoning dataset. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Varshini+Reddy%2C+Rik+Koncel%2DKedziorski%2C+Viet+Dac+Lai%2C+Michael+Krumdick%2C+Charles+Lovering%2C+and+Chris+Tanner%2E+Docfinqa%3A+A+long%2Dcontext+financial+reasoning+dataset%2E+In+Lun%2DWei+Ku%2C+Andre+Martins%2C+and+Vivek) 430. Isaac Rehg. Kv-compress: Paged kv-cache compression with variable compression rates per attention head. arXiv preprint arXiv:2410.00161, 2024. — [arXiv:2410.00161](https://arxiv.org/abs/2410.00161) [find](https://scholar.google.com/scholar?q=Isaac+Rehg%2E+Kv%2Dcompress%3A+Paged+kv%2Dcache+compression+with+variable+compression+rates+per+attention+head%2E+arXiv+preprint+arXiv%3A2410%2E00161%2C+2024%2E+%E2%80%94+%5BarXiv%3A2410%2E00161%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E00161%29) 431. Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint arXiv:2406.07522, 2024. [find](https://scholar.google.com/scholar?q=Liliang+Ren%2C+Yang+Liu%2C+Yadong+Lu%2C+Yelong+Shen%2C+Chen+Liang%2C+and+Weizhu+Chen%2E+Samba%3A+Simple+hybrid+state+space+models+for+efficient+unlimited+context+language+modeling%2E+arXiv+preprint+arXiv%3A2406%2E07522%2C+) 432. Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling, 2024. URL https://arxiv.org/abs/2406.07522 . [↗](https://arxiv.org/abs/2406.07522) 433. Jonathan Roberts, Kai Han, and Samuel Albanie. Needle threading: Can llms follow threads through near-million-scale haystacks? arXiv preprint arXiv:2411.05000, 2024. — [arXiv:2411.05000](https://arxiv.org/abs/2411.05000) [find](https://scholar.google.com/scholar?q=Jonathan+Roberts%2C+Kai+Han%2C+and+Samuel+Albanie%2E+Needle+threading%3A+Can+llms+follow+threads+through+near%2Dmillion%2Dscale+haystacks%3F+arXiv+preprint+arXiv%3A2411%2E05000%2C+2024%2E+%E2%80%94+%5BarXiv%3A2411%2E05000%5D%28https%3A%2F%2Farxiv) 434. Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, and Mikhail Burtsev. Associative recurrent memory transformer. CoRR, abs/2407.04841, 2024. [find](https://scholar.google.com/scholar?q=Ivan+Rodkin%2C+Yuri+Kuratov%2C+Aydar+Bulatov%2C+and+Mikhail+Burtsev%2E+Associative+recurrent+memory+transformer%2E+CoRR%2C+abs%2F2407%2E04841%2C+2024%2E) 435. Sara Rosenthal, Avirup Sil, Radu Florian, and Salim Roukos. Clapnq: Cohesive long-form answers from passages in natural questions for rag systems. arXiv preprint arXiv:2404.02103, 2024. [find](https://scholar.google.com/scholar?q=Sara+Rosenthal%2C+Avirup+Sil%2C+Radu+Florian%2C+and+Salim+Roukos%2E+Clapnq%3A+Cohesive+long%2Dform+answers+from+passages+in+natural+questions+for+rag+systems%2E+arXiv+preprint+arXiv%3A2404%2E02103%2C+2024%2E) 436. Jie Ruan, Wenqing Wang, and Xiaojun Wan. Defining and detecting vulnerability in human evaluation guidelines: A preliminary study towards reliable nlg evaluation. arXiv preprint arXiv:2406.07935, 2024. [find](https://scholar.google.com/scholar?q=Jie+Ruan%2C+Wenqing+Wang%2C+and+Xiaojun+Wan%2E+Defining+and+detecting+vulnerability+in+human+evaluation+guidelines%3A+A+preliminary+study+towards+reliable+nlg+evaluation%2E+arXiv+preprint+arXiv%3A2406%2E07935%2C+2024) 437. Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. arXiv preprint arXiv:2305.16843, 2023. [find](https://scholar.google.com/scholar?q=Anian+Ruoss%2C+Gr%C3%A9goire+Del%C3%A9tang%2C+Tim+Genewein%2C+Jordi+Grau%2DMoya%2C+R%C3%B3bert+Csord%C3%A1s%2C+Mehdi+Bennani%2C+Shane+Legg%2C+and+Joel+Veness%2E+Randomized+positional+encodings+boost+length+generalization+of+transformers%2E+) 438. Anian Ruoss, Fabio Pardo, Harris Chan, Bonnie Li, Volodymyr Mnih, and Tim Genewein. Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations. CoRR, abs/2412.01441, 2024. doi: 10.48550/ARXIV.2412.01441. URL https://doi.org/10.48550/arXiv.2412.01441 . — [doi:10.48550/arxiv.2412.01441](https://doi.org/10.48550/arxiv.2412.01441) [↗](https://doi.org/10.48550/arXiv.2412.01441) 439. Shiwangi Shah Rupesh Bansal. Deepsearch: Semantic search on multimedia sources like audio, video and images. [https://github.com/deepsearch-ai/deepsearch](https://github.com/deepsearch-ai/deepsearch) , 2023. URL https://github.com/deepsearch-ai/deepsearch . [↗](https://github.com/deepsearch-ai/deepsearch) 440. Michael S. Ryoo, Keerthana Gopalakrishnan, Kumara Kahatapitiya, Ted Xiao, Kanishka Rao, Austin Stone, Yao Lu, Julian Ibarz, and Anurag Arnab. Token turing machines. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19070–19081, June 2023. [find](https://scholar.google.com/scholar?q=Michael+S%2E+Ryoo%2C+Keerthana+Gopalakrishnan%2C+Kumara+Kahatapitiya%2C+Ted+Xiao%2C+Kanishka+Rao%2C+Austin+Stone%2C+Yao+Lu%2C+Julian+Ibarz%2C+and+Anurag+Arnab%2E+Token+turing+machines%2E+In+Proceedings+of+the+IEEE%2FCVF+Conf) 441. Jon Saad-Falcon, Daniel Y. Fu, Simran Arora, Neel Guha, and Christopher Ré. Benchmarking and building long-context retrieval models with loco and M2-BERT. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://open… [find](https://scholar.google.com/scholar?q=Jon+Saad%2DFalcon%2C+Daniel+Y%2E+Fu%2C+Simran+Arora%2C+Neel+Guha%2C+and+Christopher+R%C3%A9%2E+Benchmarking+and+building+long%2Dcontext+retrieval+models+with+loco+and+M2%2DBERT%2E+In+Forty%2Dfirst+International+Conference+on+Ma) 442. Amir Saeidi, Shivanshu Verma, Aswin RRV, and Chitta Baral. Triple preference optimization: Achieving better alignment with less data in a single step optimization, 2024. [find](https://scholar.google.com/scholar?q=Amir+Saeidi%2C+Shivanshu+Verma%2C+Aswin+RRV%2C+and+Chitta+Baral%2E+Triple+preference+optimization%3A+Achieving+better+alignment+with+less+data+in+a+single+step+optimization%2C+2024%2E) 443. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili’c, Daniel Hesslow, Roman Castagn’e, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennigh… [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ili%E2%80%99c%2C+Daniel+Hesslow%2C+Roman+Castagn%E2%80%99e%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+Jonathan+Tow%2C+Alexander+M%2E+Rush%2C+St) 444. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, M. Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). Neural Information Processing Systems, 2023. doi: 10.48550/arXiv.2302.04761. 445. Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International conference on machine learning, pages 9355–9366. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Imanol+Schlag%2C+Kazuki+Irie%2C+and+J%C3%BCrgen+Schmidhuber%2E+Linear+transformers+are+secretly+fast+weight+programmers%2E+In+International+conference+on+machine+learning%2C+pages+9355%E2%80%939366%2E+PMLR%2C+2021%2E) 446. Dale Schuurmans. Memory augmented large language models are computationally universal. arXiv preprint arXiv: 2301.04589, 2023. URL https://arxiv.org/abs/2301.04589 v1 . [↗](https://arxiv.org/abs/2301.04589) 447. Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37:68658–68685, 2025. [find](https://scholar.google.com/scholar?q=Jay+Shah%2C+Ganesh+Bikshandi%2C+Ying+Zhang%2C+Vijay+Thakkar%2C+Pradeep+Ramani%2C+and+Tri+Dao%2E+Flashattention%2D3%3A+Fast+and+accurate+attention+with+asynchrony+and+low%2Dprecision%2E+Advances+in+Neural+Information+Proc) 448. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. [Zeroscrolls: A zero-shot benchmark for long text understanding](https://md.research.elamur.ai/free/en/papers/shaham-2023-zeroscrolls-zero-shot-benchmark-long/). arXiv preprint arXiv:2305.14196, 2023. 449. Shivam Shandilya, Menglin Xia, Supriyo Ghosh, Huiqiang Jiang, Jue Zhang, Qianhui Wu, and Victor Rühle. Taco-rl: Task aware prompt compression optimization with reinforcement learning, 2024. URL https://arxiv.org/abs/2409.13035 . [↗](https://arxiv.org/abs/2409.13035) 450. Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948. [find](https://scholar.google.com/scholar?q=Claude+Elwood+Shannon%2E+A+mathematical+theory+of+communication%2E+The+Bell+system+technical+journal%2C+27%283%29%3A379%E2%80%93423%2C+1948%2E) 451. Bin Shao and Jiawei Yan. A long-context language model for deciphering and generating bacteriophage genomes. Nature Communications, 15(1):9392, 2024. [find](https://scholar.google.com/scholar?q=Bin+Shao+and+Jiawei+Yan%2E+A+long%2Dcontext+language+model+for+deciphering+and+generating+bacteriophage+genomes%2E+Nature+Communications%2C+15%281%29%3A9392%2C+2024%2E) 452. Ninglu Shao, Shitao Xiao, Zheng Liu, and Peitian Zhang. Extensible embedding: A flexible multipler for llm’s context length. CoRR, abs/2402.11577, 2024. — [arXiv:2402.11577](https://arxiv.org/abs/2402.11577) [find](https://scholar.google.com/scholar?q=Ninglu+Shao%2C+Shitao+Xiao%2C+Zheng+Liu%2C+and+Peitian+Zhang%2E+Extensible+embedding%3A+A+flexible+multipler+for+llm%E2%80%99s+context+length%2E+CoRR%2C+abs%2F2402%2E11577%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E11577%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402) 453. Zhihong Shao, Minlie Huang, Jiangtao Wen, Wenfei Xu, and Xiaoyan Zhu. Long and diverse text generation with planning-based hierarchical variational model. arXiv preprint arXiv:1908.06605, 2019. [find](https://scholar.google.com/scholar?q=Zhihong+Shao%2C+Minlie+Huang%2C+Jiangtao+Wen%2C+Wenfei+Xu%2C+and+Xiaoyan+Zhu%2E+Long+and+diverse+text+generation+with+planning%2Dbased+hierarchical+variational+model%2E+arXiv+preprint+arXiv%3A1908%2E06605%2C+2019%2E) 454. Aditya Sharma, Michael Saxon, and William Yang Wang. Losing visual needles in image haystacks: Vision language models are easily distracted in short and long contexts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024… [find](https://scholar.google.com/scholar?q=Aditya+Sharma%2C+Michael+Saxon%2C+and+William+Yang+Wang%2E+Losing+visual+needles+in+image+haystacks%3A+Vision+language+models+are+easily+distracted+in+short+and+long+contexts%2E+In+Yaser+Al%2DOnaizan%2C+Mohit+Bansa) 455. Eva Sharma, Chen Li, and Lu Wang. [Bigpatent: A large-scale dataset for abstractive and coherent summarization](https://md.research.elamur.ai/free/en/papers/sharma-2019-bigpatent-large-scale-dataset-abstractive/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204–2213, 2019. 456. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. [Self-attention with relative position representations](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/). arXiv preprint arXiv:1803.02155, 2018. 457. Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. Multi-lexsum: Real-world summaries of civil rights lawsuits at multiple granularities. Advances in Neural Information Processing Systems, 35:13158–13173, 2022. [find](https://scholar.google.com/scholar?q=Zejiang+Shen%2C+Kyle+Lo%2C+Lauren+Yu%2C+Nathan+Dahlberg%2C+Margo+Schlanger%2C+and+Doug+Downey%2E+Multi%2Dlexsum%3A+Real%2Dworld+summaries+of+civil+rights+lawsuits+at+multiple+granularities%2E+Advances+in+Neural+Informati) 458. Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531–3539, 2021. [find](https://scholar.google.com/scholar?q=Zhuoran+Shen%2C+Mingyuan+Zhang%2C+Haiyu+Zhao%2C+Shuai+Yi%2C+and+Hongsheng+Li%2E+Efficient+attention%3A+Attention+with+linear+complexities%2E+In+Proceedings+of+the+IEEE%2FCVF+winter+conference+on+applications+of+compu) 459. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. [Flexgen: High-throughput generative inference of large language models with a single gpu](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/). In International Conference on Machine Learning, pages 31094–31116. PMLR, 20… 460. Kaize Shi, Xueyao Sun, Qing Li, and Guandong Xu. Compressing long context for enhancing RAG with amr-based concept distillation. CoRR, abs/2405.03085, 2024. doi: 10.48550/ARX IV.2405.03085. URL https://doi.org/10.48550/arXiv.2405.03085 . — [doi:10.48550/arx](https://doi.org/10.48550/arx) [↗](https://doi.org/10.48550/arXiv.2405.03085) 461. Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Victoria Lin, Noah A. Smith, Luke S. Zettlemoyer, Scott Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. ArXiv, abs/2310.10638, 2023. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Sewon+Min%2C+Maria+Lomeli%2C+Chunting+Zhou%2C+Margaret+Li%2C+Victoria+Lin%2C+Noah+A%2E+Smith%2C+Luke+S%2E+Zettlemoyer%2C+Scott+Yih%2C+and+Mike+Lewis%2E+In%2Dcontext+pretraining%3A+Language+modeling+beyond+document+) 462. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. [REPLUG: Retrieval-augmented black-box language models](https://md.research.elamur.ai/free/en/papers/shi-2023-replug-retrieval-augmented-black-box/). In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of th… 463. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). arXiv preprint arXiv: 2303.11366, 2023. 464. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprin) 465. Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, and Torsten Scholak. Repofusion: Training code models to understand your repository. CoRR, abs/2306.10998, 2023. doi: 10.48550/ARXIV.2306.10998. URL https://doi.org/10.48550/arXiv.230 6.10998 . — [doi:10.48550/arxiv.2306.10998](https://doi.org/10.48550/arxiv.2306.10998) [↗](https://doi.org/10.48550/arXiv.230) 466. Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. Repository-level prompt generation for large language models of code. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23… [find](https://scholar.google.com/scholar?q=Disha+Shrivastava%2C+Hugo+Larochelle%2C+and+Daniel+Tarlow%2E+Repository%2Dlevel+prompt+generation+for+large+language+models+of+code%2E+In+Andreas+Krause%2C+Emma+Brunskill%2C+Kyunghyun+Cho%2C+Barbara+Engelhardt%2C+Sivan) 467. Shuzheng Si, Haozhe Zhao, Gang Chen, Yunshui Li, Kangyang Luo, Chuancheng Lv, Kaikai An, Fanchao Qi, Baobao Chang, and Maosong Sun. Gateau: Selecting influential sample for long context alignment. 2024. [find](https://scholar.google.com/scholar?q=Shuzheng+Si%2C+Haozhe+Zhao%2C+Gang+Chen%2C+Yunshui+Li%2C+Kangyang+Luo%2C+Chuancheng+Lv%2C+Kaikai+An%2C+Fanchao+Qi%2C+Baobao+Chang%2C+and+Maosong+Sun%2E+Gateau%3A+Selecting+influential+sample+for+long+context+alignment%2E+202) 468. Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. Loki: Low-rank keys for efficient sparse attention. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 1669… [find](https://scholar.google.com/scholar?q=Prajwal+Singhania%2C+Siddharth+Singh%2C+Shwai+He%2C+Soheil+Feizi%2C+and+Abhinav+Bhatele%2E+Loki%3A+Low%2Drank+keys+for+efficient+sparse+attention%2E+In+A%2E+Globerson%2C+L%2E+Mackey%2C+D%2E+Belgrave%2C+A%2E+Fan%2C+U%2E+Paquet%2C+J%2E+Tomc) 469. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, June 2023. URL [https://huggingface.co/datasets/cerebras/SlimPajama-627B](https://huggingface.co/datasets/cerebras/SlimPajama-627B) . [↗](https://huggingface.co/datasets/cerebras/SlimPajama-627B) 470. Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking mllms in long context. CoRR, abs/2404.18532, 2024. doi: 10.48550/ARXIV.2404.18532. URL https://doi.org/10.48550/arXiv.2404.18532 . — [doi:10.48550/arxiv.2404.18532](https://doi.org/10.48550/arxiv.2404.18532) [↗](https://doi.org/10.48550/arXiv.2404.18532) 471. Kaiqiang Song, Xiaoyang Wang, Sangwoo Cho, Xiaoman Pan, and Dong Yu. Zebra: Extending context window with layerwise grouped local-global attention, 2023. URL https://arxiv.org/abs/2312.08618 . — [arXiv:2312.08618](https://arxiv.org/abs/2312.08618) [↗](https://arxiv.org/abs/2312.08618) 472. Mingyang Song, Mao Zheng, and Xuan Luo. Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models. Preprint, 2024. [find](https://scholar.google.com/scholar?q=Mingyang+Song%2C+Mao+Zheng%2C+and+Xuan+Luo%2E+Counting%2Dstars%3A+A+multi%2Devidence%2C+position%2Daware%2C+and+scalable+benchmark+for+evaluating+long%2Dcontext+large+language+models%2E+Preprint%2C+2024%2E) 473. Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung-Woo Ha, and Jinwoo Shin. Hierarchical context merging: Better long context understanding for pre-trained llms. arXiv preprint arXiv:2404.10308, 2024. — [arXiv:2404.10308](https://arxiv.org/abs/2404.10308) [find](https://scholar.google.com/scholar?q=Woomin+Song%2C+Seunghyuk+Oh%2C+Sangwoo+Mo%2C+Jaehyung+Kim%2C+Sukmin+Yun%2C+Jung%2DWoo+Ha%2C+and+Jinwoo+Shin%2E+Hierarchical+context+merging%3A+Better+long+context+understanding+for+pre%2Dtrained+llms%2E+arXiv+preprint+arXi) 474. Woomin Song, Jihoon Tack, Sangwoo Mo, Seunghyuk Oh, and Jinwoo Shin. Sparsified state-space models are efficient highway networks. arXiv preprint arXiv:2505.20698, 2025. — [arXiv:2505.20698](https://arxiv.org/abs/2505.20698) [find](https://scholar.google.com/scholar?q=Woomin+Song%2C+Jihoon+Tack%2C+Sangwoo+Mo%2C+Seunghyuk+Oh%2C+and+Jinwoo+Shin%2E+Sparsified+state%2Dspace+models+are+efficient+highway+networks%2E+arXiv+preprint+arXiv%3A2505%2E20698%2C+2025%2E+%E2%80%94+%5BarXiv%3A2505%2E20698%5D%28https%3A%2F%2Fa) 475. Mohammed Sourouri, Tor Gillberg, Scott B Baden, and Xing Cai. Effective multi-gpu communication using multiple cuda streams and threads. In 2014 20th IEEE International Conference on Parallel and Distributed Systems (ICPADS), pages 981–986. IEEE, 2014. 479 spicychat.ai. Spicychat, 2024. URL https://… [find](https://scholar.google.com/scholar?q=Mohammed+Sourouri%2C+Tor+Gillberg%2C+Scott+B+Baden%2C+and+Xing+Cai%2E+Effective+multi%2Dgpu+communication+using+multiple+cuda+streams+and+threads%2E+In+2014+20th+IEEE+International+Conference+on+Parallel+and+Dist) 476. Konrad Staniszewski, Szymon Tworkowski, Sebastian Jaszczur, Yu Zhao, Henryk Michalewski, Łukasz Kuciński, and Piotr Miłoś. Structured packing in llm training improves long context utilization. arXiv preprint arXiv:2312.17296, 2023. — [arXiv:2312.17296](https://arxiv.org/abs/2312.17296) [find](https://scholar.google.com/scholar?q=Konrad+Staniszewski%2C+Szymon+Tworkowski%2C+Sebastian+Jaszczur%2C+Yu+Zhao%2C+Henryk+Michalewski%2C+%C5%81ukasz+Kucin%CC%81ski%2C+and+Piotr+Mi%C5%82os%CC%81%2E+Structured+packing+in+llm+training+improves+long+context+utilization%2E+arXiv) 477. Konrad Staniszewski, Szymon Tworkowski, Yu Zhao, Sebastian Jaszczur, Henryk Michalewski, Lukasz Kuci’nski, and Piotr Milo’s. Structured packing in llm training improves long context utilization. ArXiv, abs/2312.17296, 2023. — [arXiv:2312.17296](https://arxiv.org/abs/2312.17296) [find](https://scholar.google.com/scholar?q=Konrad+Staniszewski%2C+Szymon+Tworkowski%2C+Yu+Zhao%2C+Sebastian+Jaszczur%2C+Henryk+Michalewski%2C+Lukasz+Kuci%E2%80%99nski%2C+and+Piotr+Milo%E2%80%99s%2E+Structured+packing+in+llm+training+improves+long+context+utilization%2E+ArXiv) 478. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. [Asqa: Factoid questions meet long-form answers](https://md.research.elamur.ai/free/en/papers/stelmakh-2022-asqa-factoid-questions-meet-long/). arXiv preprint arXiv:2204.06092, 2022. 479. Jianlin Su. Transformer 10, Aug 2023. URL [https://spaces.ac.cn/archives/9675](https://spaces.ac.cn/archives/9675) . [↗](https://spaces.ac.cn/archives/9675) 480. Jianlin Su. Transformer 12, Aug 2023. URL [https://kexue.fm/archives/9708](https://kexue.fm/archives/9708) . [↗](https://kexue.fm/archives/9708) 481. Jianlin Su. Transformer 17. Online Resource, Mar 2024. URL [https://spaces.ac.cn/a](https://spaces.ac.cn/a) rchives/10040 . [↗](https://spaces.ac.cn/a) 482. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063, 2024. 483. Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. In First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Hanshi+Sun%2C+Zhuoming+Chen%2C+Xinyu+Yang%2C+Yuandong+Tian%2C+and+Beidi+Chen%2E+Triforce%3A+Lossless+acceleration+of+long+sequence+generation+with+hierarchical+speculative+decoding%2E+In+First+Conference+on+Languag) 484. Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. arXiv preprint arXiv: 2305.16653, 2023. [find](https://scholar.google.com/scholar?q=Haotian+Sun%2C+Yuchen+Zhuang%2C+Lingkai+Kong%2C+Bo+Dai%2C+and+Chao+Zhang%2E+Adaplanner%3A+Adaptive+planning+from+feedback+with+language+models%2E+arXiv+preprint+arXiv%3A+2305%2E16653%2C+2023%2E) 485. Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. [Do long-range language models actually use long-range context?](https://md.research.elamur.ai/free/en/papers/sun-2021-do-long-range-language-models/) In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 807–822, 2021. 486. Simeng Sun, Y. Liu, Shuo Wang, Chenguang Zhu, and Mohit Iyyer. Pearl: Prompting large language models to plan and execute actions over long documents. Conference of the European Chapter of the Association for Computational Linguistics, 2023. doi: 10.48550/arXiv.2305.14564. [find](https://scholar.google.com/scholar?q=Simeng+Sun%2C+Y%2E+Liu%2C+Shuo+Wang%2C+Chenguang+Zhu%2C+and+Mohit+Iyyer%2E+Pearl%3A+Prompting+large+language+models+to+plan+and+execute+actions+over+long+documents%2E+Conference+of+the+European+Chapter+of+the+Associa) 487. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. [Is chatgpt good at search? investigating large language models as re-ranking agents](https://md.research.elamur.ai/free/en/papers/sun-2023-chatgpt-good-search-investigating-large/). In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Meth… 488. Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620, 2024. [find](https://scholar.google.com/scholar?q=Yu+Sun%2C+Xinhao+Li%2C+Karan+Dalal%2C+Jiarui+Xu%2C+Arjun+Vikram%2C+Genghan+Zhang%2C+Yann+Dubois%2C+Xinlei+Chen%2C+Xiaolong+Wang%2C+Sanmi+Koyejo%2C+et+al%2E+Learning+to+%28learn+at+test+time%29%3A+Rnns+with+expressive+hidden+stat) 489. Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. [A length-extrapolatable transformer](https://md.research.elamur.ai/free/en/papers/sun-2023-length-extrapolatable-transformer/). arXiv preprint arXiv:2212.10554, 2022. 490. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, 2023. URL https://arxiv.org/abs/2307.08621 . [↗](https://arxiv.org/abs/2307.08621) 491. Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. Advances in Neural Information Processing Systems, 37:7339–7361, 2025. [find](https://scholar.google.com/scholar?q=Yutao+Sun%2C+Li+Dong%2C+Yi+Zhu%2C+Shaohan+Huang%2C+Wenhui+Wang%2C+Shuming+Ma%2C+Quanlu+Zhang%2C+Jianyong+Wang%2C+and+Furu+Wei%2E+You+only+cache+once%3A+Decoder%2Ddecoder+architectures+for+language+models%2E+Advances+in+Neura) 492. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. [Commonsenseqa: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). arXiv preprint arXiv:1811.00937, 2018. 493. Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, et al. Proxyqa: An alternative framework for evaluating long-form text generation with large language models. arXiv preprint arXiv:2401.15042, 2024. [find](https://scholar.google.com/scholar?q=Haochen+Tan%2C+Zhijiang+Guo%2C+Zhan+Shi%2C+Lu+Xu%2C+Zhili+Liu%2C+Yunlong+Feng%2C+Xiaoguang+Li%2C+Yasheng+Wang%2C+Lifeng+Shang%2C+Qun+Liu%2C+et+al%2E+Proxyqa%3A+An+alternative+framework+for+evaluating+long%2Dform+text+generatio) 494. Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, Yifei Bi, Pengjie Gu, Xinrun Wang, Börje F. Karlsson, Bo An, and Zongqing Lu. Towards general computer control: A multimodal agent for red dead redemption II as a case stud… — [doi:10.48550/arxiv.2403.03186](https://doi.org/10.48550/arxiv.2403.03186) [find](https://scholar.google.com/scholar?q=Weihao+Tan%2C+Ziluo+Ding%2C+Wentao+Zhang%2C+Boyu+Li%2C+Bohan+Zhou%2C+Junpeng+Yue%2C+Haochong+Xia%2C+Jiechuan+Jiang%2C+Longtao+Zheng%2C+Xinrun+Xu%2C+Yifei+Bi%2C+Pengjie+Gu%2C+Xinrun+Wang%2C+B%C3%B6rje+F%2E+Karlsson%2C+Bo+An%2C+and+Zongqin) 495. Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and… [find](https://scholar.google.com/scholar?q=Matthew+Tancik%2C+Pratul+Srinivasan%2C+Ben+Mildenhall%2C+Sara+Fridovich%2DKeil%2C+Nithin+Raghavan%2C+Utkarsh+Singhal%2C+Ravi+Ramamoorthi%2C+Jonathan+Barron%2C+and+Ren+Ng%2E+Fourier+features+let+networks+learn+high+freque) 496. Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Razorattention: Efficient kv cache compression through retrieval heads. arXiv preprint arXiv:2407.15891, 2024. — [arXiv:2407.15891](https://arxiv.org/abs/2407.15891) [find](https://scholar.google.com/scholar?q=Hanlin+Tang%2C+Yang+Lin%2C+Jing+Lin%2C+Qingsen+Han%2C+Shikuan+Hong%2C+Yiwu+Yao%2C+and+Gongyi+Wang%2E+Razorattention%3A+Efficient+kv+cache+compression+through+retrieval+heads%2E+arXiv+preprint+arXiv%3A2407%2E15891%2C+2024%2E+%E2%80%94+) 497. Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Danning Ke, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Razorattention: Efficient KV cache compression through retrieval heads. In The Thirteenth International Conference on Learning Representations, 2025. URL [https://openreview.net/forum?id=tkiZQlL04w](https://openreview.net/forum?id=tkiZQlL04w) . [↗](https://openreview.net/forum?id=tkiZQlL04w) 498. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. [QUEST: Query-aware sparsity for efficient long-context LLM inference](https://md.research.elamur.ai/free/en/papers/tang-2024-quest-query-aware-sparsity-efficient/). In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of… 499. Zecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu, and Min Zhang. Logo - long context alignment via efficient preference optimization. ArXiv, abs/2410.18533, 2024. [find](https://scholar.google.com/scholar?q=Zecheng+Tang%2C+Zechen+Sun%2C+Juntao+Li%2C+Qiaoming+Zhu%2C+and+Min+Zhang%2E+Logo+%2D+long+context+alignment+via+efficient+preference+optimization%2E+ArXiv%2C+abs%2F2410%2E18533%2C+2024%2E) 500. Zecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu, and Min Zhang. Logo–long context alignment via efficient preference optimization. arXiv preprint arXiv:2410.18533, 2024. [find](https://scholar.google.com/scholar?q=Zecheng+Tang%2C+Zechen+Sun%2C+Juntao+Li%2C+Qiaoming+Zhu%2C+and+Min+Zhang%2E+Logo%E2%80%93long+context+alignment+via+efficient+preference+optimization%2E+arXiv+preprint+arXiv%3A2410%2E18533%2C+2024%2E) 501. Zecheng Tang, Keyan Zhou, Juntao Li, Baibei Ji, Jianye Hou, and Min Zhang. L-citeeval: Do long-context models truly leverage context for responding?, 2024. [find](https://scholar.google.com/scholar?q=Zecheng+Tang%2C+Keyan+Zhou%2C+Juntao+Li%2C+Baibei+Ji%2C+Jianye+Hou%2C+and+Min+Zhang%2E+L%2Dciteeval%3A+Do+long%2Dcontext+models+truly+leverage+context+for+responding%3F%2C+2024%2E) 502. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. [Long range arena: A benchmark for efficient transformers](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/), 2020. URL https://arxiv.org/abs/2011.04006 . [↗](https://arxiv.org/abs/2011.04006) 503. Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. [Transformer memory as a differentiable search index](https://md.research.elamur.ai/free/en/papers/tay-2022-transformer-memory-differentiable-search-index/). arXiv preprint arXiv: 2202.06991, 2022. 504. Coze Team. Coze. [https://www.coze.cn/](https://www.coze.cn/) , 2025. [↗](https://www.coze.cn/) 505. Dify Team. The innovation engine for genai applications. [https://dify.ai/](https://dify.ai/) , 2025. [↗](https://dify.ai/) 506. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530, 2024. 507. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+et+al%2E+Gemma+2%3A+Improving+open+la) 508. Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. Jamba-1.5: Hybrid transformer-mamba models at scale. arXiv preprint arXiv:2408.12570, 2024. [find](https://scholar.google.com/scholar?q=Jamba+Team%2C+Barak+Lenz%2C+Alan+Arazi%2C+Amir+Bergman%2C+Avshalom+Manevich%2C+Barak+Peleg%2C+Ben+Aviram%2C+Chen+Almagor%2C+Clara+Fridman%2C+Dan+Padnos%2C+et+al%2E+Jamba%2D1%2E5%3A+Hybrid+transformer%2Dmamba+models+at+scale%2E+arXiv) 509. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.[12599, 2025](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/). 510. M-A-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, Kang Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhen-Nan Wei, Chujie Zheng, Kaixin Deng, Shuyue Guo, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, Dehua Ma, Yuansheng Ni, Haoran Que,… [find](https://scholar.google.com/scholar?q=M%2DA%2DP+Team%2C+Xinrun+Du%2C+Yifan+Yao%2C+Kaijing+Ma%2C+Bingli+Wang%2C+Tianyu+Zheng%2C+Kang+Zhu%2C+Minghao+Liu%2C+Yiming+Liang%2C+Xiaolong+Jin%2C+Zhen%2DNan+Wei%2C+Chujie+Zheng%2C+Kaixin+Deng%2C+Shuyue+Guo%2C+Shian+Jia%2C+Sichao+Jiang) 511. Manus Team. Leave it to manus. [https://manus.im/](https://manus.im/) , 2025. [↗](https://manus.im/) 512. OpenAI Team. Computer-using agent. [https://openai.com/index/computer-usi](https://openai.com/index/computer-usi) ng-agent/ , 2025. [↗](https://openai.com/index/computer-usi) 513. OpenAI Team. Introducing deep research. [https://openai.com/index/introducing](https://openai.com/index/introducing) -deep-research/ , 2025. [↗](https://openai.com/index/introducing) 514. Qwen Team. Qwen2.5-vl, January 2025. URL [https://qwenlm.github.io/blog/qwen](https://qwenlm.github.io/blog/qwen) 2.5-vl/ . [↗](https://qwenlm.github.io/blog/qwen) 515. Junfeng Tian, Da Zheng, Yang Cheng, Rui Wang, Colin Zhang, and Debing Zhang. Untie the knots: An efficient data augmentation strategy for long-context pre-training in language models. arXiv preprint arXiv:2409.04774, 2024. [find](https://scholar.google.com/scholar?q=Junfeng+Tian%2C+Da+Zheng%2C+Yang+Cheng%2C+Rui+Wang%2C+Colin+Zhang%2C+and+Debing+Zhang%2E+Untie+the+knots%3A+An+efficient+data+augmentation+strategy+for+long%2Dcontext+pre%2Dtraining+in+language+models%2E+arXiv+preprint+a) 516. Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, Zhong Zhang, Yesai Wu, et al. Distance between relevant information pieces causes bias in long-context llms. arXiv preprint arXiv:2410.14641, 2024. [find](https://scholar.google.com/scholar?q=Runchu+Tian%2C+Yanghao+Li%2C+Yuepeng+Fu%2C+Siyang+Deng%2C+Qinyu+Luo%2C+Cheng+Qian%2C+Shuo+Wang%2C+Xin+Cong%2C+Zhong+Zhang%2C+Yesai+Wu%2C+et+al%2E+Distance+between+relevant+information+pieces+causes+bias+in+long%2Dcontext+llm) 517. Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S. Morcos. [D4: improving LLM pretraining via document de-duplication and diversification](https://md.research.elamur.ai/free/en/papers/tirumala-2023-d4-improving-llm-pretraining-document/). CoRR, abs/2308.12284, 2023. doi: 10.48550/ARXIV.2308.12284. URL https://doi.org/10.48550/arXiv.230 8.12284 . [↗](https://doi.org/10.48550/arXiv.230) 518. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023. 519. Shangqing Tu, Yucheng Wang, Daniel Zhang-Li, Yushi Bai, Jifan Yu, Yuhao Wu, Lei Hou, Huiqin Liu, Zhiyuan Liu, Bin Xu, et al. Longwriter-v: Enabling ultra-long and high-fidelity generation in vision-language models. arXiv preprint arXiv:2502.14834, 2025. — [arXiv:2502.14834](https://arxiv.org/abs/2502.14834) [find](https://scholar.google.com/scholar?q=Shangqing+Tu%2C+Yucheng+Wang%2C+Daniel+Zhang%2DLi%2C+Yushi+Bai%2C+Jifan+Yu%2C+Yuhao+Wu%2C+Lei+Hou%2C+Huiqin+Liu%2C+Zhiyuan+Liu%2C+Bin+Xu%2C+et+al%2E+Longwriter%2Dv%3A+Enabling+ultra%2Dlong+and+high%2Dfidelity+generation+in+vision%2Dla) 520. Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokul Santhanam, James Gabriel, Peter Grasch, and Oncel Tuzel. Fastvlm: Efficient vision encoding for vision language models. CoRR, abs/2412.13303, 2024. doi: 10.48550/A RXIV.2412.13303. URL [https://doi.org…](https://doi.org…) — [doi:10.48550/a](https://doi.org/10.48550/a) [↗](https://doi.org…) 521. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS, pages 5998–6008, 2017. 522. Saranya Venkatraman, Nafis Irtiza Tripto, and Dongwon Lee. Collabstory: Multi-llm collaborative story generation and authorship analysis. arXiv preprint arXiv:2406.12665, 2024. [find](https://scholar.google.com/scholar?q=Saranya+Venkatraman%2C+Nafis+Irtiza+Tripto%2C+and+Dongwon+Lee%2E+Collabstory%3A+Multi%2Dllm+collaborative+story+generation+and+authorship+analysis%2E+arXiv+preprint+arXiv%3A2406%2E12665%2C+2024%2E) 523. Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640, 2024. [find](https://scholar.google.com/scholar?q=Kiran+Vodrahalli%2C+Santiago+Ontanon%2C+Nilesh+Tripuraneni%2C+Kelvin+Xu%2C+Sanil+Jain%2C+Rakesh+Shivanna%2C+Jeffrey+Hui%2C+Nishanth+Dikkala%2C+Mehran+Kazemi%2C+Bahare+Fatemi%2C+et+al%2E+Michelangelo%3A+Long+context+evaluatio) 524. Aaron R. Voelker and Chris Eliasmith. Improving spiking dynamical networks: Accurate delays, higher-order synapses, and time cells. Neural Comput., 30(3), 2018. [find](https://scholar.google.com/scholar?q=Aaron+R%2E+Voelker+and+Chris+Eliasmith%2E+Improving+spiking+dynamical+networks%3A+Accurate+delays%2C+higher%2Dorder+synapses%2C+and+time+cells%2E+Neural+Comput%2E%2C+30%283%29%2C+2018%2E) 525. Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 1288– 1301, Bangkok, Thailand, August 2024. Ass… [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+Javier+Ferrando%2C+and+Christoforos+Nalmpantis%2E+Neurons+in+large+language+models%3A+Dead%2C+n%2Dgram%2C+positional%2E+In+Lun%2DWei+Ku%2C+Andre+Martins%2C+and+Vivek+Srikumar%2C+editors%2C+Findings+of+the+Associ) 526. Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887, 2024. [find](https://scholar.google.com/scholar?q=Roger+Waleffe%2C+Wonmin+Byeon%2C+Duncan+Riach%2C+Brandon+Norick%2C+Vijay+Korthikanti%2C+Tri+Dao%2C+Albert+Gu%2C+Ali+Hatamizadeh%2C+Sudhakar+Singh%2C+Deepak+Narayanan%2C+et+al%2E+An+empirical+study+of+mamba%2Dbased+language+m) 527. Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel Bowman. Squality: Building a long-document summarization dataset the hard way. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1139–1156, 2022. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Richard+Yuanzhe+Pang%2C+Angelica+Chen%2C+Jason+Phang%2C+and+Samuel+Bowman%2E+Squality%3A+Building+a+long%2Ddocument+summarization+dataset+the+hard+way%2E+In+Proceedings+of+the+2022+Conference+on+Empirica) 528. Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. Encoding word order in complex embeddings. arXiv preprint arXiv:1912.12333, 2019. — [arXiv:1912.12333](https://arxiv.org/abs/1912.12333) [find](https://scholar.google.com/scholar?q=Benyou+Wang%2C+Donghao+Zhao%2C+Christina+Lioma%2C+Qiuchi+Li%2C+Peng+Zhang%2C+and+Jakob+Grue+Simonsen%2E+Encoding+word+order+in+complex+embeddings%2E+arXiv+preprint+arXiv%3A1912%2E12333%2C+2019%2E+%E2%80%94+%5BarXiv%3A1912%2E12333%5D%28https) 529. Chong Wang, Jian Zhang, Yebo Feng, Tianlin Li, Weisong Sun, Yang Liu, and Xin Peng. Teaching code llms to use autocompletion tools in repository-level code generation. arXiv preprint arXiv:2401.06391, 2024. — [arXiv:2401.06391](https://arxiv.org/abs/2401.06391) [find](https://scholar.google.com/scholar?q=Chong+Wang%2C+Jian+Zhang%2C+Yebo+Feng%2C+Tianlin+Li%2C+Weisong+Sun%2C+Yang+Liu%2C+and+Xin+Peng%2E+Teaching+code+llms+to+use+autocompletion+tools+in+repository%2Dlevel+code+generation%2E+arXiv+preprint+arXiv%3A2401%2E06391%2C) 530. Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. Ada-leval: Evaluating long-context llms with length-adaptable benchmarks, 2024. [find](https://scholar.google.com/scholar?q=Chonghua+Wang%2C+Haodong+Duan%2C+Songyang+Zhang%2C+Dahua+Lin%2C+and+Kai+Chen%2E+Ada%2Dleval%3A+Evaluating+long%2Dcontext+llms+with+length%2Dadaptable+benchmarks%2C+2024%2E) 531. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: 2305.16291, 2023. [find](https://scholar.google.com/scholar?q=Guanzhi+Wang%2C+Yuqi+Xie%2C+Yunfan+Jiang%2C+Ajay+Mandlekar%2C+Chaowei+Xiao%2C+Yuke+Zhu%2C+Linxi+Fan%2C+and+Anima+Anandkumar%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+arXiv+preprint+arXiv%3A+2) 532. Haonan Wang, Qian Liu, Chao Du, Tongyao Zhu, Cunxiao Du, Kenji Kawaguchi, and Tianyu Pang. When precision meets position: Bfloat16 breaks down rope in long-context training. arXiv preprint arXiv:2411.13476, 2024. — [arXiv:2411.13476](https://arxiv.org/abs/2411.13476) [find](https://scholar.google.com/scholar?q=Haonan+Wang%2C+Qian+Liu%2C+Chao+Du%2C+Tongyao+Zhu%2C+Cunxiao+Du%2C+Kenji+Kawaguchi%2C+and+Tianyu+Pang%2E+When+precision+meets+position%3A+Bfloat16+breaks+down+rope+in+long%2Dcontext+training%2E+arXiv+preprint+arXiv%3A2411%2E) 533. Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking long-context capability of multimodal large language models. CoRR, abs/2406.11230, 2024. doi: 10.48550/ARXIV.2406.11230. URL [https://doi.…](https://doi.…) [↗](https://doi.…) 534. Jiaan Wang, Fandong Meng, Yunlong Liang, and Jie Zhou. Drt-o1: Optimized deep reasoning translation via long chain-of-thought. arXiv preprint arXiv:2412.17498, 2024. — [arXiv:2412.17498](https://arxiv.org/abs/2412.17498) [find](https://scholar.google.com/scholar?q=Jiaan+Wang%2C+Fandong+Meng%2C+Yunlong+Liang%2C+and+Jie+Zhou%2E+Drt%2Do1%3A+Optimized+deep+reasoning+translation+via+long+chain%2Dof%2Dthought%2E+arXiv+preprint+arXiv%3A2412%2E17498%2C+2024%2E+%E2%80%94+%5BarXiv%3A2412%2E17498%5D%28https%3A%2F%2Farxiv) 535. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=NpsVSN6…](https://openreview.net/forum?id=NpsVSN6…) [↗](https://openreview.net/forum?id=NpsVSN6…) 536. Lei Wang, Chengbang Ma, Xueyang Feng, Zeyu Zhang, Hao ran Yang, Jingsen Zhang, Zhi-Yang Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji rong Wen. [A survey on large language model based autonomous agents](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/). Frontiers Comput. Sci., 2023. doi: 10.1007/s11704-024-40231-1. 537. Lei Wang, Shan Dong, Yuhui Xu, Hanze Dong, Yalu Wang, Amrita Saha, Ee-Peng Lim, Caiming Xiong, and Doyen Sahoo. Mathhay: An automated benchmark for long-context mathematical reasoning in llms. arXiv preprint arXiv:2410.04698, 2024. — [arXiv:2410.04698](https://arxiv.org/abs/2410.04698) [find](https://scholar.google.com/scholar?q=Lei+Wang%2C+Shan+Dong%2C+Yuhui+Xu%2C+Hanze+Dong%2C+Yalu+Wang%2C+Amrita+Saha%2C+Ee%2DPeng+Lim%2C+Caiming+Xiong%2C+and+Doyen+Sahoo%2E+Mathhay%3A+An+automated+benchmark+for+long%2Dcontext+mathematical+reasoning+in+llms%2E+arXiv+p) 538. Liang Wang, Nan Yang, and Furu Wei. [Query2doc: Query expansion with large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-query2doc-query-expansion-large-language/). Conference on Empirical Methods in Natural Language Processing, 2023. doi: 10.48550 /arXiv.2303.07678. 539. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. [Improving text embeddings with large language models](https://md.research.elamur.ai/free/en/papers/wang-2024-improving-text-embeddings-large-language/). In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… 540. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Large search model: Redefining search stack in the era of llms. In ACM SIGIR Forum, volume 57, pages 1–16. ACM New York, NY, USA, 2024. [find](https://scholar.google.com/scholar?q=Liang+Wang%2C+Nan+Yang%2C+Xiaolong+Huang%2C+Linjun+Yang%2C+Rangan+Majumder%2C+and+Furu+Wei%2E+Large+search+model%3A+Redefining+search+stack+in+the+era+of+llms%2E+In+ACM+SIGIR+Forum%2C+volume+57%2C+pages+1%E2%80%9316%2E+ACM+New+Yor) 541. Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. [Document-level machine translation with large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-document-level-machine-translation-large/). In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EM… 542. Longyue Wang, Zefeng Du, Wenxiang Jiao, Chenyang Lyu, Jianhui Pang, Leyang Cui, Kaiqiang Song, Derek F. Wong, Shuming Shi, and Zhaopeng Tu. Benchmarking and improving long-text translation with large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Associat… [find](https://scholar.google.com/scholar?q=Longyue+Wang%2C+Zefeng+Du%2C+Wenxiang+Jiao%2C+Chenyang+Lyu%2C+Jianhui+Pang%2C+Leyang+Cui%2C+Kaiqiang+Song%2C+Derek+F%2E+Wong%2C+Shuming+Shi%2C+and+Zhaopeng+Tu%2E+Benchmarking+and+improving+long%2Dtext+translation+with+large+) 543. Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. [Leave no document behind: Benchmarking long-context llms with extended multi-doc qa](https://md.research.elamur.ai/free/en/papers/wang-2024-leave-no-document-behind-benchmarking/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,… 544. [Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world a…](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/) 545. Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020. URL https://arxiv.org/abs/2006.04768 . [↗](https://arxiv.org/abs/2006.04768) 546. Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, and Bang Liu. Resonance rope: Improving context length generalization of large language models. arXiv preprint arXiv:2403.00071, 2024. [find](https://scholar.google.com/scholar?q=Suyuchen+Wang%2C+Ivan+Kobyzev%2C+Peng+Lu%2C+Mehdi+Rezagholizadeh%2C+and+Bang+Liu%2E+Resonance+rope%3A+Improving+context+length+generalization+of+large+language+models%2E+arXiv+preprint+arXiv%3A2403%2E00071%2C+2024%2E) 547. Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, Yibin Liu, Jialong Wu, Shengwei Ding, Long Li, Zhiwei Huang, Xinle Deng, Teng Yu, Gangan Ma, Han Xiao, Zixin Chen, Danjun Xiang, Yunxia Wang, Yuanyuan Zhu, Yi Xiao, Jing Wang… [find](https://scholar.google.com/scholar?q=Tiannan+Wang%2C+Jiamin+Chen%2C+Qingrui+Jia%2C+Shuai+Wang%2C+Ruoyu+Fang%2C+Huilin+Wang%2C+Zhaowei+Gao%2C+Chunzhao+Xie%2C+Chuou+Xu%2C+Jihong+Dai%2C+Yibin+Liu%2C+Jialong+Wu%2C+Shengwei+Ding%2C+Long+Li%2C+Zhiwei+Huang%2C+Xinle+Deng%2C+T) 548. Ting Wang, Chuan Yang, Maoyang Zou, Jiaying Liang, Dong Xiang, Wenjie Yang, Hongyang Wang, and Jia Li. A study of extractive summarization of long documents incorporating local topic and hierarchical information. Scientific Reports, 14(1):10140, 2024. [find](https://scholar.google.com/scholar?q=Ting+Wang%2C+Chuan+Yang%2C+Maoyang+Zou%2C+Jiaying+Liang%2C+Dong+Xiang%2C+Wenjie+Yang%2C+Hongyang+Wang%2C+and+Jia+Li%2E+A+study+of+extractive+summarization+of+long+documents+incorporating+local+topic+and+hierarchical+) 549. Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual C… [find](https://scholar.google.com/scholar?q=Weizhi+Wang%2C+Li+Dong%2C+Hao+Cheng%2C+Xiaodong+Liu%2C+Xifeng+Yan%2C+Jianfeng+Gao%2C+and+Furu+Wei%2E+Augmenting+language+models+with+long%2Dterm+memory%2E+In+Alice+Oh%2C+Tristan+Naumann%2C+Amir+Globerson%2C+Kate+Saenko%2C+Mori) 550. Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal llms to 1000 images efficiently via hybrid architecture. CoRR, abs/2409.02889, 2024. doi: 10.48550/ARXIV.2409.02889. URL https://doi.org/ 10.48550/arXiv.2409.02889 . — [doi:10.48550/arxiv.2409.02889](https://doi.org/10.48550/arxiv.2409.02889) [find](https://scholar.google.com/scholar?q=Xidong+Wang%2C+Dingjie+Song%2C+Shunian+Chen%2C+Chen+Zhang%2C+and+Benyou+Wang%2E+Longllava%3A+Scaling+multi%2Dmodal+llms+to+1000+images+efficiently+via+hybrid+architecture%2E+CoRR%2C+abs%2F2409%2E02889%2C+2024%2E+doi%3A+10%2E48550%2F) 551. Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Armaghan Eshaghi. Beyond the limits: A survey of techniques to extend the context length in large language models. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024,… [find](https://scholar.google.com/scholar?q=Xindi+Wang%2C+Mahsa+Salmani%2C+Parsa+Omidi%2C+Xiangyu+Ren%2C+Mehdi+Rezagholizadeh%2C+and+Armaghan+Eshaghi%2E+Beyond+the+limits%3A+A+survey+of+techniques+to+extend+the+context+length+in+large+language+models%2E+In+Pro) 552. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. [Openhands: An open platform for ai software developers as generalist agents](https://md.research.elamur.ai/free/en/papers/wang-2024-openhands-open-platform-ai/). In The Thirteenth International Conference on Learning Representations, 2024. 553. Yanli Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, et al. Repotransbench: A real-world benchmark for repository-level code translation. arXiv preprint arXiv:2412.17744, 2024. [find](https://scholar.google.com/scholar?q=Yanli+Wang%2C+Yanlin+Wang%2C+Suiquan+Wang%2C+Daya+Guo%2C+Jiachi+Chen%2C+John+Grundy%2C+Xilin+Liu%2C+Yuchi+Ma%2C+Mingzhi+Mao%2C+Hongyu+Zhang%2C+et+al%2E+Repotransbench%3A+A+real%2Dworld+benchmark+for+repository%2Dlevel+code+trans) 554. Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang, and Julian J. McAuley. MEMORYLLM: towards self-updatable large language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, Ju… [find](https://scholar.google.com/scholar?q=Yu+Wang%2C+Yifan+Gao%2C+Xiusi+Chen%2C+Haoming+Jiang%2C+Shiyang+Li%2C+Jingfeng+Yang%2C+Qingyu+Yin%2C+Zheng+Li%2C+Xian+Li%2C+Bing+Yin%2C+Jingbo+Shang%2C+and+Julian+J%2E+McAuley%2E+MEMORYLLM%3A+towards+self%2Dupdatable+large+language) 555. [Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhu Chen, Jie Fu, and Junran Peng. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language…](https://md.research.elamur.ai/free/en/papers/wang-2023-rolellm-benchmarking-eliciting-enhancing-role/) 556. Zhaowei Wang, Wenhao Yu, Xiyu Ren, Jipeng Zhang, Yu Zhao, Rohit Saxena, Liang Cheng, Ginny Wong, Simon See, Pasquale Minervini, et al. Mmlongbench: Benchmarking long-context vision-language models effectively and thoroughly. arXiv preprint arXiv:2505.10610, 2025. — [arXiv:2505.10610](https://arxiv.org/abs/2505.10610) [find](https://scholar.google.com/scholar?q=Zhaowei+Wang%2C+Wenhao+Yu%2C+Xiyu+Ren%2C+Jipeng+Zhang%2C+Yu+Zhao%2C+Rohit+Saxena%2C+Liang+Cheng%2C+Ginny+Wong%2C+Simon+See%2C+Pasquale+Minervini%2C+et+al%2E+Mmlongbench%3A+Benchmarking+long%2Dcontext+vision%2Dlanguage+models+eff) 557. Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for LLM reasoning on long videos. CoRR, abs/2405.19209, 2024. doi: 10.48550/ARXIV.2405.19209. URL https://doi.org/10.48550/arXiv.2405.19209 . — [doi:10.48550/arxiv.2405.19209](https://doi.org/10.48550/arxiv.2405.19209) [↗](https://doi.org/10.48550/arXiv.2405.19209) 558. [Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory…](https://md.research.elamur.ai/free/en/papers/warner-2024-smarter-better-faster-longer-modern/) 559. Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, et al. [Long-form factuality in large language models](https://md.research.elamur.ai/free/en/papers/wei-2024-long-form-factuality-large-language/). arXiv preprint arXiv:2403.18802, 2024. 560. Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding via large language models. In European Conference on Computer Vision, pages 453–470. Springer, 2024. [find](https://scholar.google.com/scholar?q=Yuetian+Weng%2C+Mingfei+Han%2C+Haoyu+He%2C+Xiaojun+Chang%2C+and+Bohan+Zhuang%2E+Longvlm%3A+Efficient+long+video+understanding+via+large+language+models%2E+In+European+Conference+on+Computer+Vision%2C+pages+453%E2%80%93470%2E+S) 561. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. [Towards ai-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). arXiv preprint arXiv:1502.05698, 2015. 566 wiki. Printing press, 2023. URL [https://en.wikipedia.org/wiki/Prin…](https://en.wikipedia.org/wiki/Prin…) [↗](https://en.wikipedia.org/wiki/Prin…) *Updated 24 October 2026*