---
title: "LIFBench — paper by Wu et al. (2025) — summary and ideas"
description: "It defines three scenarios that differ in the granularity of information processed: List, OneDoc and… 58 references, cited by 9 papers here."
authors: ["Xiaodong Wu", "Minhao Wang", "Yichen Liu", "Xiaoming Shi", "He Yan", "Lu Xiangju", "Junmin Zhu", "Wei Zhang"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.803/"
doi: "10.18653/v1/2025.acl-long.803"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2025-lifbench-evaluating-instruction-following-performance/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2025-lifbench-evaluating-instruction-following-performance/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios

# LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios

**Authors:** Xiaodong Wu, Minhao Wang, Yichen Liu, [Xiaoming Shi](https://md.research.elamur.ai/free/en/authors/xiaoming-shi/), He Yan, Lu Xiangju, Junmin Zhu and [Wei Zhang](https://md.research.elamur.ai/free/en/authors/wei-zhang/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.803/) · [PDF](https://aclanthology.org/2025.acl-long.803.pdf) · [DOI 10.18653/v1/2025.acl-long.803](https://doi.org/10.18653/v1/2025.acl-long.803) · [OpenAlex](https://openalex.org/W4412945109)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/wu-2025-lifbench-evaluating-instruction-following-performance/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

LLMs are increasingly given very long inputs, but benchmarks rarely test whether models follow instructions reliably at those lengths. Long-context suites mostly measure task solving. Instruction-following suites use short, fixed-length data and ignore whether behaviour stays consistent across inputs. LIFBench addresses both gaps.

It defines three scenarios that differ in the granularity of information processed: List, OneDoc and MultiDoc. Across these it sets 11 tasks. Each task is built from hand-written templates and grown automatically along three axes: context length, instruction wording (expression) and instruction variables.

Scoring uses LIFEval, which splits each task into small, programmatically checkable rubric points. It needs neither GPT-4-style judges nor human raters. Its outputs include a score-capability map and a stability metric called IFS.

The authors test 20 open and closed LLMs at input lengths from 4k to 128k tokens. All models leave substantial room for improvement, especially as context grows. Stability does not always track raw task performance. The benchmark is scalable and reproducible for probing long-context reliability.

## Key points

- **Dataset:** 2,766 instructions spread over six length intervals (4k, 8k, 16k, 32k, 64k, 128k tokens). Data and code are released at github.com/SheldonWu0327/LIF-Bench-2024.
- **Capabilities measured:** six core abilities (Ori, Num, Spat, Format, Recog, Logic). Format stays fairly steady as inputs lengthen, while Recog drops most sharply.
- **Model size:** larger models do better on Ori, Num and Spat. Format-related skills are handled well even by small models.
- **Stability:** stability rankings diverge from performance rankings. For example, Qwen2.5-72B-Inst. beat all closed-source models on "Expression" stability, and GPT-4 was weaker on instruction variables than Llama-3.1-70B-Inst.
- **Task difficulty:** harder tasks (rubric weight above 10) degrade faster with length than easy ones, so complexity and context length compound each other.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [East China Normal University](https://md.research.elamur.ai/free/en/universities/east-china-normal-university/) — University, Shanghai, China · 50 works · 2,174 citations
- [Shanghai Innovation Institute](https://md.research.elamur.ai/free/en/universities/shanghai-innovation-institute/) — University, Shanghai, China · 24 works · 262 citations

Also on the byline: [iQIYI](https://ror.org/026xvz038).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Lu Xiangju, Junmin Zhu and Wei Zhang. 2025. LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wu2025lifbench,
  title = {{LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios}},
  author = {Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Lu Xiangju, Junmin Zhu and Wei Zhang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.803},
  url = {https://doi.org/10.18653/v1/2025.acl-long.803}
}
```

- [← previous Learning from Diverse Reasoning Paths with Routing and Collaboration](https://md.research.elamur.ai/free/en/papers/lei-2025-learning-diverse-reasoning-paths-routing/)
- [next → MDCure: A Scalable Pipeline for Multi-Document Instruction-Following](https://md.research.elamur.ai/free/en/papers/liu-2025-mdcure-scalable-pipeline-multi-document/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/)

## References

58 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. [L-eval: Instituting standardized evaluation for long context language models](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/). arXiv preprint arXiv:2307.11088.
3. AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1. [find](https://scholar.google.com/scholar?q=AI+Anthropic%2E+2024%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+Claude%2D3+Model+Card%2C+1%2E)
4. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508.
5. Charles E Brown. 1998. Coefficient of variation. In Applied multivariate statistics in geohydrology and related sciences, pages 155–157. Springer. [find](https://scholar.google.com/scholar?q=Charles+E+Brown%2E+1998%2E+Coefficient+of+variation%2E+In+Applied+multivariate+statistics+in+geohydrology+and+related+sciences%2C+pages+155%E2%80%93157%2E+Springer%2E)
6. Tom B Brown. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165.
7. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. [Internlm2 technical report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/). arXiv preprint arXiv:2403.17297.
8. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. [Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/). arXiv preprint arXiv:2402.03216.
9. Kedi Chen, Qin Chen, Jie Zhou, Yishen He, and Liang He. 2024b. Diahalu: A dialogue-level hallucination evaluation benchmark for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 9057–9079. Association for… [find](https://scholar.google.com/scholar?q=Kedi+Chen%2C+Qin+Chen%2C+Jie+Zhou%2C+Yishen+He%2C+and+Liang+He%2E+2024b%2E+Diahalu%3A+A+dialogue%2Dlevel+hallucination+evaluation+benchmark+for+large+language+models%2E+In+Findings+of+the+Association+for+Computational+)
10. Kedi Chen, Zhikai Lei, Fan Zhang, Yinqi Zhang, Qin Chen, Jie Zhou, Liang He, Qipeng Guo, Kai Chen, and Wei Zhang. 2025. Code-driven inductive synthesis: Enhancing reasoning abilities of large language models with sequences. CoRR, abs/2503.13109. — [arXiv:2503.13109](https://arxiv.org/abs/2503.13109) [find](https://scholar.google.com/scholar?q=Kedi+Chen%2C+Zhikai+Lei%2C+Fan+Zhang%2C+Yinqi+Zhang%2C+Qin+Chen%2C+Jie+Zhou%2C+Liang+He%2C+Qipeng+Guo%2C+Kai+Chen%2C+and+Wei+Zhang%2E+2025%2E+Code%2Ddriven+inductive+synthesis%3A+Enhancing+reasoning+abilities+of+large+language)
11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113.
12. Cohere For AI. 2024. c4ai-command-r-08-2024. [find](https://scholar.google.com/scholar?q=Cohere+For+AI%2E+2024%2E+c4ai%2Dcommand%2Dr%2D08%2D2024%2E)
13. Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. 2024. Ticking all the boxes: Generated checklists improve llm evaluation and generation. arXiv preprint arXiv:2410.03608. [find](https://scholar.google.com/scholar?q=Jonathan+Cook%2C+Tim+Rockt%C3%A4schel%2C+Jakob+Foerster%2C+Dennis+Aumiller%2C+and+Alex+Wang%2E+2024%2E+Ticking+all+the+boxes%3A+Generated+checklists+improve+llm+evaluation+and+generation%2E+arXiv+preprint+arXiv%3A2410%2E03608)
14. Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2024. [Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models](https://md.research.elamur.ai/free/en/papers/dong-2024-bamboo-comprehensive-benchmark-evaluating-long/). In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation…
15. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
16. Shawn Gavin, Tuney Zheng, Jiaheng Liu, Quehry Que, Noah Wang, Jian Yang, Chenchen Zhang, Wenhao Huang, Wenhu Chen, and Ge Zhang. 2024. Longins: A challenging long-context instruction-based exam for llms. ArXiv, abs/2406.17588. — [arXiv:2406.17588](https://arxiv.org/abs/2406.17588) [find](https://scholar.google.com/scholar?q=Shawn+Gavin%2C+Tuney+Zheng%2C+Jiaheng+Liu%2C+Quehry+Que%2C+Noah+Wang%2C+Jian+Yang%2C+Chenchen+Zhang%2C+Wenhao+Huang%2C+Wenhu+Chen%2C+and+Ge+Zhang%2E+2024%2E+Longins%3A+A+challenging+long%2Dcontext+instruction%2Dbased+exam+for+ll)
17. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. 2024. [Chatglm: A family of large language models from glm-130b to glm-4 all tools](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/). arXiv preprint arXiv:2406.12793.
18. Greg Kamradt. 2023. LLMTest: Needle in a Haystack. [https://github.com/gkamradt/](https://github.com/gkamradt/) LLMTest_NeedleInAHaystack. [↗](https://github.com/gkamradt/)
19. Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Jiaqing Liang, and Yanghua Xiao. 2024. Can large language models understand real-world complex instructions? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18188–18196. [find](https://scholar.google.com/scholar?q=Qianyu+He%2C+Jie+Zeng%2C+Wenhao+Huang%2C+Lina+Chen%2C+Jin+Xiao%2C+Qianxi+He%2C+Xunzhe+Zhou%2C+Jiaqing+Liang%2C+and+Yanghua+Xiao%2E+2024%2E+Can+large+language+models+understand+real%2Dworld+complex+instructions%3F+In+Proceedi)
20. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. [Unnatural instructions: Tuning language models with (almost) no human labor](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428.
21. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. [Ruler: What’s the real context size of your long-context language models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) arXiv preprint arXiv:2404.06654.
22. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. [Efficient attentions for long document summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/). arXiv preprint arXiv:2104.02112.
23. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2023. [Followbench: A multi-level fine-grained constraints following benchmark for large language models](https://md.research.elamur.ai/free/en/papers/jiang-2024-followbench-multi-level-fine-grained/). arXiv preprint arXiv:2310.20410.
24. Wai-Chung Kwan, Xingshan Zeng, Yufei Wang, Yusen Sun, Liangyou Li, Lifeng Shang, Qun Liu, and Kam-Fai Wong. 2023. M4le: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models. arXiv preprint arXiv:2310.19240. [find](https://scholar.google.com/scholar?q=Wai%2DChung+Kwan%2C+Xingshan+Zeng%2C+Yufei+Wang%2C+Yusen+Sun%2C+Liangyou+Li%2C+Lifeng+Shang%2C+Qun+Liu%2C+and+Kam%2DFai+Wong%2E+2023%2E+M4le%3A+A+multi%2Dability+multi%2Drange+multi%2Dtask+multi%2Ddomain+long%2Dcontext+evaluation+benc)
25. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
26. Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. [Same task, more tokens: the impact of input length on the reasoning performance of large language models](https://md.research.elamur.ai/free/en/papers/levy-2024-same-task-more-tokens-impact/). arXiv preprint arXiv:2402.14848.
27. Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023. [Loogle: Can long-context language models understand long contexts?](https://md.research.elamur.ai/free/en/papers/li-2023-loogle-long-context-language-models/) arXiv preprint arXiv:2311.04939.
28. Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. 2024a. Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963. — [arXiv:2407.11963](https://arxiv.org/abs/2407.11963) [find](https://scholar.google.com/scholar?q=Mo+Li%2C+Songyang+Zhang%2C+Yunxin+Liu%2C+and+Kai+Chen%2E+2024a%2E+Needlebench%3A+Can+llms+do+retrieval+and+reasoning+in+1+million+context+window%3F+arXiv+preprint+arXiv%3A2407%2E11963%2E+%E2%80%94+%5BarXiv%3A2407%2E11963%5D%28https%3A%2F%2Farxi)
29. Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024b. [Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers](https://md.research.elamur.ai/free/en/papers/li-2024-gsm-plus-comprehensive-benchmark-evaluating/). arXiv preprint arXiv:2402.19255.
30. Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024c. [Long-context llms struggle with long in-context learning](https://md.research.elamur.ai/free/en/papers/li-2024-long-context-llms-struggle-long/). arXiv preprint arXiv:2404.02060.
31. Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024a. World model on million-length video and language with blockwise ringattention. ArXiv, abs/2402.08268. [find](https://scholar.google.com/scholar?q=Hao+Liu%2C+Wilson+Yan%2C+Matei+Zaharia%2C+and+Pieter+Abbeel%2E+2024a%2E+World+model+on+million%2Dlength+video+and+language+with+blockwise+ringattention%2E+ArXiv%2C+abs%2F2402%2E08268%2E)
32. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024b. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173.
33. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). In International Conference on Learning Representations (ICLR).
34. J Macqueen. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probability/University of California Press. [find](https://scholar.google.com/scholar?q=J+Macqueen%2E+1967%2E+Some+methods+for+classification+and+analysis+of+multivariate+observations%2E+In+Proceedings+of+5%2Dth+Berkeley+Symposium+on+Mathematical+Statistics+and+Probability%2FUniversity+of+Californ)
35. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.
36. Xuanfan Ni, Hengyi Cai, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, and Piji Li. 2024. Xl 2 bench: A benchmark for extremely long context understanding with long-range dependencies. arXiv preprint arXiv:2404.05446. — [arXiv:2404.05446](https://arxiv.org/abs/2404.05446) [find](https://scholar.google.com/scholar?q=Xuanfan+Ni%2C+Hengyi+Cai%2C+Xiaochi+Wei%2C+Shuaiqiang+Wang%2C+Dawei+Yin%2C+and+Piji+Li%2E+2024%2E+Xl+2+bench%3A+A+benchmark+for+extremely+long+context+understanding+with+long%2Drange+dependencies%2E+arXiv+preprint+arXiv%3A)
37. Joe O’Connor and Jacob Andreas. 2021. [What context features can transformer language models use?](https://md.research.elamur.ai/free/en/papers/connor-2021-what-context-features-can-transformer/) In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 851–86…
38. Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. [Logicbench: Towards systematic evaluation of logical reasoning ability of large language models](https://md.research.elamur.ai/free/en/papers/parmar-2024-logicbench-systematic-evaluation-logical-reasoning/). In Proceedings of the 62nd Annual Meeting of the Association for Computati…
39. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. ArXiv, abs/2309.00071. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+2023%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+ArXiv%2C+abs%2F2309%2E00071%2E)
40. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. [Infobench: Evaluating instruction following ability in large language models](https://md.research.elamur.ai/free/en/papers/qin-2024-infobench-evaluating-instruction-following-ability/). arXiv preprint arXiv:2401.03601.
41. Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, et al. 2024. Hellobench: Evaluating long text generation capabilities of large language models. arXiv preprint arXiv:2409.16191. — [arXiv:2409.16191](https://arxiv.org/abs/2409.16191) [find](https://scholar.google.com/scholar?q=Haoran+Que%2C+Feiyu+Duan%2C+Liqun+He%2C+Yutao+Mou%2C+Wangchunshu+Zhou%2C+Jiaheng+Liu%2C+Wenge+Rong%2C+Zekun+Moore+Wang%2C+Jian+Yang%2C+Ge+Zhang%2C+et+al%2E+2024%2E+Hellobench%3A+Evaluating+long+text+generation+capabilities+of+)
42. Yusuke Sakai, Adam Nohejl, Jiangnan Hang, Hidetaka Kamigaito, and Taro Watanabe. 2024. Toward the evaluation of large language models considering score variance across instruction templates. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 499… [find](https://scholar.google.com/scholar?q=Yusuke+Sakai%2C+Adam+Nohejl%2C+Jiangnan+Hang%2C+Hidetaka+Kamigaito%2C+and+Taro+Watanabe%2E+2024%2E+Toward+the+evaluation+of+large+language+models+considering+score+variance+across+instruction+templates%2E+In+Procee)
43. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. [Zeroscrolls: A zero-shot benchmark for long text understanding](https://md.research.elamur.ai/free/en/papers/shaham-2023-zeroscrolls-zero-shot-benchmark-long/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989.
44. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022. [Scrolls: Standardized comparison over long language sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1200…
45. Orit Shechtman. 2013. The coefficient of variation as an index of measurement reliability. In Methods of clinical epidemiology, pages 39–49. Springer. [find](https://scholar.google.com/scholar?q=Orit+Shechtman%2E+2013%2E+The+coefficient+of+variation+as+an+index+of+measurement+reliability%2E+In+Methods+of+clinical+epidemiology%2C+pages+39%E2%80%9349%2E+Springer%2E)
46. Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, et al. 2024. [Proxyqa: An alternative framework for evaluating long-form text generation with large language models](https://md.research.elamur.ai/free/en/papers/tan-2024-proxyqa-alternative-framework-evaluating-long/). arXiv preprint arXiv:2401.15042.
47. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
48. Zhifang Sui. 2023. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Zhifang+Sui%2E+2023%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2E)
49. Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, et al. 2024. Benchmarking complex instruction-following with multiple constraints composition. arXiv preprint arXiv:2407.03978. — [arXiv:2407.03978](https://arxiv.org/abs/2407.03978) [find](https://scholar.google.com/scholar?q=Bosi+Wen%2C+Pei+Ke%2C+Xiaotao+Gu%2C+Lindong+Wu%2C+Hao+Huang%2C+Jinfeng+Zhou%2C+Wenchuang+Li%2C+Binxin+Hu%2C+Wendy+Gao%2C+Jiaxin+Xu%2C+et+al%2E+2024%2E+Benchmarking+complex+instruction%2Dfollowing+with+multiple+constraints+comp)
50. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+et+al%2E+2024%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2E)
51. Tao Zhang, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, et al. 2024a. [Cfbench: A comprehensive constraints-following benchmark for llms](https://md.research.elamur.ai/free/en/papers/zhang-2025-cfbench-comprehensive-constraints-following-benchmark/). arXiv preprint arXiv:2408.01122.
52. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen Mckeown, and Tatsunori B Hashimoto. 2024b. [Benchmarking large language models for news summarization](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/). Transactions of the Association for Computational Linguistics, 11:39–57.
53. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024c. ∞ bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=Xinrong+Zhang%2C+Yingfa+Chen%2C+Shengding+Hu%2C+Zihang+Xu%2C+Junhao+Chen%2C+Moo+Hao%2C+Xu+Han%2C+Zhen+Thai%2C+Shuo+Wang%2C+Zhiyuan+Liu%2C+et+al%2E+2024c%2E+%E2%88%9E+bench%3A+Extending+long+context+evaluation+beyond+100k+tokens%2E+In+Pr)
54. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021. [Qmsum: A new benchmark for query-based multi-domain meeting summarization](https://md.research.elamur.ai/free/en/papers/zhong-2021-qmsum-new-benchmark-query-based/). arXiv preprint arXiv:2104.05938.
55. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023a. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). arXiv preprint arXiv:2311.07911.
56. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023b. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). arXiv preprint arXiv:2311.07911.
57. Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. 2024. [Leave no document behind: Benchmarking long-context llms with extended multi-doc qa](https://md.research.elamur.ai/free/en/papers/wang-2024-leave-no-document-behind-benchmarking/). arXiv preprint arXiv:2406.17419.
58. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Zefan+Cai%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and)

*Updated 30 October 2026*