---
title: "KCIF: Knowledge-Conditioned Instruction — Murthy 2026"
description: "Standard LLM benchmarks usually test knowledge and reasoning separately from instruction following."
authors: ["Rudra Murthy", "Praveen Venkateswaran", "Prince Kumar", "Danish Contractor"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.339/"
doi: "10.63317/2vmfhqzwvdkw"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/murthy-2026-kcif-knowledge-conditioned-instruction-following/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/murthy-2026-kcif-knowledge-conditioned-instruction-following/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/3/) / KCIF: Knowledge-Conditioned Instruction Following

# KCIF: Knowledge-Conditioned Instruction Following

**Authors:** [Rudra Murthy](https://md.research.elamur.ai/free/authors/rudra-murthy/), Praveen Venkateswaran, Prince Kumar, Danish Contractor · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.339/) · [PDF](https://aclanthology.org/2026.lrec-1.339.pdf) · [DOI 10.63317/2vmfhqzwvdkw](https://doi.org/10.63317/2vmfhqzwvdkw) · [OpenAlex](https://openalex.org/W7160115305)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14) · 2 in our library, of the 140673 papers whose bibliographies we read

## Summary
Standard LLM benchmarks usually test knowledge and reasoning separately from instruction following. In real use, however, people often combine the two, for example by asking a model to answer a question and then transform that answer. The authors build KCIF to measure this combination. They take five multiple-choice or binary benchmarks (MMLUPro, MathQA, Winogrande, BoolQ, PIQA) and attach simple instructions that act on the answer or on the candidate options. These include case changes, numeric edits and formatting, list sorting, and "distractor" instructions that should leave the answer unchanged. Because the answer space is fixed, scoring uses exact match without any LLM judge, and most errors can be attributed automatically to either knowledge failures or instruction-following failures. Across 19 models from 1B to 405B parameters, every model loses substantial accuracy on these easy compositions. The losses are largest for smaller models. The work argues that knowledge and instruction following should be evaluated together. The benchmark, code and results are publicly released.

## Key points
- The benchmark samples 1,500 items from each of five source datasets. It defines 13 instruction types grouped into 5 categories and comes in Full and Lite versions.
- Asking for the answer text instead of the option label alone reduces exact match by about 20% on average, even for GPT-4o. Adding no-effect distractor instructions lowers scores by a further 5–30%.
- Large and frontier models lose roughly 40–50% of their performance on the composed tasks. Small and medium models sometimes lose more than 80%.
- Within the same model family, siblings that score similarly on the original tasks can diverge sharply, with the smaller model degrading more.
- About 80% of errors are classified automatically. Code is available at github.com/IBM/KCIF and data at ibm.biz/KCIFData.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{murthy2026kcif,
  title = {{KCIF: Knowledge-Conditioned Instruction Following}},
  author = {Murthy, Rudra and Venkateswaran, Praveen and Kumar, Prince and Contractor, Danish},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2vmfhqzwvdkw},
  url = {https://doi.org/10.63317/2vmfhqzwvdkw}
}
```

- [← previous K-CoT：基于关键词思维链提示的中文排比句生成研究](https://md.research.elamur.ai/free/papers/zhong-2026-k-cot-chinese/)
- [next → KinyCOMET: Automatic Evaluation of Machine Translation Systems for Kinyarwanda–English](https://md.research.elamur.ai/free/papers/mazimpaka-2026-kinycomet-automatic-evaluation-machine-translation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/3/)

## References

32 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computation… [find](https://scholar.google.com/scholar?q=Aida+Amini%2C+Saadia+Gabriel%2C+Shanchuan+Lin%2C+Rik+Koncel%2DKedziorski%2C+Yejin+Choi%2C+and+Hannaneh+Hajishirzi%2E+2019%2E+MathQA%3A+Towards+interpretable+math+word+problem+solving+with+operation%2Dbased+formalisms%2E+In) [our card](https://md.research.elamur.ai/free/papers/amini-2019-mathqa-interpretable-math-word-problem/)
2. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Yonatan+Bisk%2C+Rowan+Zellers%2C+Ronan+Le+Bras%2C+Jianfeng+Gao%2C+and+Yejin+Choi%2E+2020%2E+Piqa%3A+Reasoning+about+physical+commonsense+in+natural+language%2E+In+Thirty%2DFourth+AAAI+Conference+on+Artificial+Intellige) [our card](https://md.research.elamur.ai/free/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
3. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:… [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+2019%2E+BoolQ%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+In+Proceedings+of) [our card](https://md.research.elamur.ai/free/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
4. Wanyong Feng, Jaewook Lee, Hunter McNichols, Alexander Scarlatos, Digory Smith, Simon Woodhead, Nancy Ornelas, and Andrew Lan. 2024. Exploring automated distractor generation for math multiple-choice questions via large language models. In Findings of the Association for Computational Linguistics: N… [find](https://scholar.google.com/scholar?q=Wanyong+Feng%2C+Jaewook+Lee%2C+Hunter+McNichols%2C+Alexander+Scarlatos%2C+Digory+Smith%2C+Simon+Woodhead%2C+Nancy+Ornelas%2C+and+Andrew+Lan%2E+2024%2E+Exploring+automated+distractor+generation+for+math+multiple%2Dchoice+)
5. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
6. Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Adam+Fisch%2C+and+Danqi+Chen%2E+2020%2E+Making+pre%2Dtrained+language+models+better+few%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2012%2E15723%2E) [our card](https://md.research.elamur.ai/free/papers/gao-2021-making-pre-trained-language-models/)
7. Keno Harada, Yudai Yamazaki, Masachika Taniguchi, Takeshi Kojima, Yusuke Iwasawa, and Yutaka Matsuo. 2025. Curse of instructions: Large language models cannot follow multiple instructions at once. [find](https://scholar.google.com/scholar?q=Keno+Harada%2C+Yudai+Yamazaki%2C+Masachika+Taniguchi%2C+Takeshi+Kojima%2C+Yusuke+Iwasawa%2C+and+Yutaka+Matsuo%2E+2025%2E+Curse+of+instructions%3A+Large+language+models+cannot+follow+multiple+instructions+at+once%2E)
8. Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. 2024. From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. arXiv preprint arXiv:2404.15846. [find](https://scholar.google.com/scholar?q=Qianyu+He%2C+Jie+Zeng%2C+Qianxi+He%2C+Jiaqing+Liang%2C+and+Yanghua+Xiao%2E+2024%2E+From+complex+to+simple%3A+Enhancing+multi%2Dconstraint+complex+instruction+following+ability+of+large+language+models%2E+arXiv+preprint)
9. Jie Huang and Kevin Chen-Chuan Chang. 2022. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403. [find](https://scholar.google.com/scholar?q=Jie+Huang+and+Kevin+Chen%2DChuan+Chang%2E+2022%2E+Towards+reasoning+in+large+language+models%3A+A+survey%2E+arXiv+preprint+arXiv%3A2212%2E10403%2E) [our card](https://md.research.elamur.ai/free/papers/huang-2023-reasoning-large-language-models-survey/)
10. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024. FollowBench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computa… [find](https://scholar.google.com/scholar?q=Yuxin+Jiang%2C+Yufei+Wang%2C+Xingshan+Zeng%2C+Wanjun+Zhong%2C+Liangyou+Li%2C+Fei+Mi%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Qun+Liu%2C+and+Wei+Wang%2E+2024%2E+FollowBench%3A+A+multi%2Dlevel+fine%2Dgrained+constraints+following+benchmark)
11. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022a. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213. [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+Shane+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+2022a%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+Advances+in+neural+information+processing+systems%2C+35%3A22199%E2%80%93222) [our card](https://md.research.elamur.ai/free/papers/kojima-2022-large-language-models-zero-shot/)
12. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022b. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+%28Shane%29+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+2022b%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+) [our card](https://md.research.elamur.ai/free/papers/kojima-2022-large-language-models-zero-shot/)
13. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Zhuohan+Li%2C+Siyuan+Zhuang%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Cody+Hao+Yu%2C+Joseph+E%2E+Gonzalez%2C+Hao+Zhang%2C+and+Ion+Stoica%2E+2023%2E+Efficient+memory+management+for+large+language+model+serving+with+p)
14. Shiyang Li, Jun Yan, Hai Wang, Zheng Tang, Xiang Ren, Vijay Srinivasan, and Hongxia Jin. 2024. Instruction-following evaluation through verbalizer manipulation. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3678–3692, Mexico City, Mexico. Association for Computation… [find](https://scholar.google.com/scholar?q=Shiyang+Li%2C+Jun+Yan%2C+Hai+Wang%2C+Zheng+Tang%2C+Xiang+Ren%2C+Vijay+Srinivasan%2C+and+Hongxia+Jin%2E+2024%2E+Instruction%2Dfollowing+evaluation+through+verbalizer+manipulation%2E+In+Findings+of+the+Association+for+Comp)
15. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+2022%2E+TruthfulQA%3A+Measuring+how+models+mimic+human+falsehoods%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+) [our card](https://md.research.elamur.ai/free/papers/lin-2022-truthfulqa-measuring-how-models-mimic/)
16. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Tim+Rockt%C3%A4schel%2C+Sebastian+Riedel%2C+Patrick+Lewis%2C+Anton+Bakhtin%2C+Yuxiang+Wu%2C+and+Alexander+Miller%2E+2019%2E+Language+models+as+knowledge+bases%3F+In+Proceedings+of+the+2019+Conference+on+Emp) [our card](https://md.research.elamur.ai/free/papers/petroni-2019-language-models-as-knowledge-bases/)
17. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. Infobench: Evaluating instruction following ability in large language models. [find](https://scholar.google.com/scholar?q=Yiwei+Qin%2C+Kaiqiang+Song%2C+Yebowen+Hu%2C+Wenlin+Yao%2C+Sangwoo+Cho%2C+Xiaoyang+Wang%2C+Xuansheng+Wu%2C+Fei+Liu%2C+Pengfei+Liu%2C+and+Dong+Yu%2E+2024%2E+Infobench%3A+Evaluating+instruction+following+ability+in+large+langua) [our card](https://md.research.elamur.ai/free/papers/qin-2024-infobench-evaluating-instruction-following-ability/)
18. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Langu)
19. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+an+adversarial+winograd+schema+challenge+at+scale%2E+Commun%2E+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
20. Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Mehran Kazemi, Najoung Kim, and He He. 2023. Testing the general deductive reasoning capacity of large language models using OOD examples. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Abulhair+Saparov%2C+Richard+Yuanzhe+Pang%2C+Vishakh+Padmakumar%2C+Nitish+Joshi%2C+Mehran+Kazemi%2C+Najoung+Kim%2C+and+He+He%2E+2023%2E+Testing+the+general+deductive+reasoning+capacity+of+large+language+models+using+O)
21. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. [find](https://scholar.google.com/scholar?q=Freda+Shi%2C+Xinyun+Chen%2C+Kanishka+Misra%2C+Nathan+Scales%2C+David+Dohan%2C+Ed+Chi%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2023%2E+Large+language+models+can+be+easily+distracted+by+irrelevant+context%2E) [our card](https://md.research.elamur.ai/free/papers/shi-2023-large-language-models-easily-distracted/)
22. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. [find](https://scholar.google.com/scholar?q=Xuezhi+Wang%2C+Jason+Wei%2C+Dale+Schuurmans%2C+Quoc+Le%2C+Ed+Chi%2C+Sharan+Narang%2C+Aakanksha+Chowdhery%2C+and+Denny+Zhou%2E+2022%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+arXiv+prepr) [our card](https://md.research.elamur.ai/free/papers/wang-2022-self-consistency-improves-chain-thought/)
23. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark… [find](https://scholar.google.com/scholar?q=Yubo+Wang%2C+Xueguang+Ma%2C+Ge+Zhang%2C+Yuansheng+Ni%2C+Abhranil+Chandra%2C+Shiguang+Guo%2C+Weiming+Ren%2C+Aaran+Arulraj%2C+Xuan+He%2C+Ziyan+Jiang%2C+Tianle+Li%2C+Max+Ku%2C+Kai+Wang%2C+Alex+Zhuang%2C+Rongqi+Fan%2C+Xiang+Yue%2C+and+W) [our card](https://md.research.elamur.ai/free/papers/wang-2024-mmlu-pro-more-robust-challenging/)
24. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+brian+ichter%2C+Fei+Xia%2C+Ed+H%2E+Chi%2C+Quoc+V+Le%2C+and+Denny+Zhou%2E+2022%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+In+Adv) [our card](https://md.research.elamur.ai/free/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
25. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2024. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, R… [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Brian+Ichter%2C+Fei+Xia%2C+Ed+H%2E+Chi%2C+Quoc+V%2E+Le%2C+and+Denny+Zhou%2E+2024%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+In+Pr) [our card](https://md.research.elamur.ai/free/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
26. Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxing Xu, et al. 2024. Benchmarking complex instruction-following with multiple constraints composition. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Bench… [find](https://scholar.google.com/scholar?q=Bosi+Wen%2C+Pei+Ke%2C+Xiaotao+Gu%2C+Lindong+Wu%2C+Hao+Huang%2C+Jinfeng+Zhou%2C+Wenchuang+Li%2C+Binxin+Hu%2C+Wendy+Gao%2C+Jiaxing+Xu%2C+et+al%2E+2024%2E+Benchmarking+complex+instruction%2Dfollowing+with+multiple+constraints+com)
27. Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. 2024. Fofo: A benchmark to evaluate llms’ format-following capability. arXiv preprint arXiv:2402.18667. [find](https://scholar.google.com/scholar?q=Congying+Xia%2C+Chen+Xing%2C+Jiangshu+Du%2C+Xinyi+Yang%2C+Yihao+Feng%2C+Ran+Xu%2C+Wenpeng+Yin%2C+and+Caiming+Xiong%2E+2024%2E+Fofo%3A+A+benchmark+to+evaluate+llms%E2%80%99+format%2Dfollowing+capability%2E+arXiv+preprint+arXiv%3A2402%2E1)
28. Mengge Xue, Zhenyu Hu, Liqun Liu, Kuo Liao, Shuang Li, Honglin Han, Meng Zhao, and Chengguo Yin. 2024. Strengthened symbol binding makes large language models reliable multiple-choice selectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=Mengge+Xue%2C+Zhenyu+Hu%2C+Liqun+Liu%2C+Kuo+Liao%2C+Shuang+Li%2C+Honglin+Han%2C+Meng+Zhao%2C+and+Chengguo+Yin%2E+2024%2E+Strengthened+symbol+binding+makes+large+language+models+reliable+multiple%2Dchoice+selectors%2E+In+Pr)
29. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Jiatong+Yu%2C+Tianyu+Gao%2C+Yu+Meng%2C+Tanya+Goyal%2C+and+Danqi+Chen%2E+2024%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E+In+International+Conference+on+Learning+Represen)
30. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595– 46623. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+2023%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+A)
31. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023a. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representatio… [find](https://scholar.google.com/scholar?q=Denny+Zhou%2C+Nathanael+Sch%C3%A4rli%2C+Le+Hou%2C+Jason+Wei%2C+Nathan+Scales%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Claire+Cui%2C+Olivier+Bousquet%2C+Quoc+V+Le%2C+and+Ed+H%2E+Chi%2E+2023a%2E+Least%2Dto%2Dmost+prompting+enables+complex+re) [our card](https://md.research.elamur.ai/free/papers/al-2022-least-most-prompting-enables-complex/)
32. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023b. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. [find](https://scholar.google.com/scholar?q=Jeffrey+Zhou%2C+Tianjian+Lu%2C+Swaroop+Mishra%2C+Siddhartha+Brahma%2C+Sujoy+Basu%2C+Yi+Luan%2C+Denny+Zhou%2C+and+Le+Hou%2E+2023b%2E+Instruction%2Dfollowing+evaluation+for+large+language+models%2E+arXiv+preprint+arXiv%3A2311%2E) [our card](https://md.research.elamur.ai/free/papers/zhou-2023-instruction-following-evaluation-large-language/)