---
title: "FollowBench — paper by Jiang et al. (2024) — summary and ideas"
description: "FollowBench addresses a gap in how LLMs are evaluated. 54 references, cited by 69 papers here."
authors: ["Yuxin Jiang", "Yufei Wang", "Xingshan Zeng", "Wanjun Zhong", "Liangyou Li", "Fei Mi", "Lifeng Shang", "Xin Jiang", "Qun Liu", "Wei Wang"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.257/"
doi: "10.18653/v1/2024.acl-long.257"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jiang-2024-followbench-multi-level-fine-grained/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jiang-2024-followbench-multi-level-fine-grained/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models

# FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models

**Authors:** [Yuxin Jiang](https://md.research.elamur.ai/free/en/authors/yuxin-jiang/), [Yufei Wang](https://md.research.elamur.ai/free/en/authors/yufei-wang/), [Xingshan Zeng](https://md.research.elamur.ai/free/en/authors/xingshan-zeng/), [Wanjun Zhong](https://md.research.elamur.ai/free/en/authors/wanjun-zhong/), [Liangyou Li](https://md.research.elamur.ai/free/en/authors/liangyou-li/), [Fei Mi](https://md.research.elamur.ai/free/en/authors/fei-mi/), [Lifeng Shang](https://md.research.elamur.ai/free/en/authors/lifeng-shang/), [Xin Jiang](https://md.research.elamur.ai/free/en/authors/xin-jiang/), [Qun Liu](https://md.research.elamur.ai/free/en/authors/qun-liu/) and [Wei Wang](https://md.research.elamur.ai/free/en/authors/wei-wang/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.257/) · [PDF](https://aclanthology.org/2024.acl-long.257.pdf) · [DOI 10.18653/v1/2024.acl-long.257](https://doi.org/10.18653/v1/2024.acl-long.257) · [OpenAlex](https://openalex.org/W4402672064)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [69](https://md.research.elamur.ai/free/en/papers/jiang-2024-followbench-multi-level-fine-grained/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

FollowBench addresses a gap in how LLMs are evaluated. Most existing benchmarks judge the overall quality of a response but do not check whether the model actually obeyed the explicit requirements in a prompt. The authors built a benchmark of 820 curated instructions drawn from over 50 NLP tasks. Across these instructions, five kinds of constraint are represented, dealing respectively with Example, Format, Content, Style and Situation. Difficulty is controlled by a Multi-level mechanism: starting from a simple prompt, one constraint is added at each step, up to five. This shows the exact level at which a model breaks down. Closed-ended items are scored with rules. Open-ended items are judged by a strong LLM, which receives the full constraint-evolution path of the instruction so it can check each requirement separately. Humans verified both the data and the evaluation. Across 13 models, accuracy fell sharply as constraints accumulated, and GPT-4 and GPT-3.5 clearly outperformed open-source systems. The work matters because instruction following turns out to be a skill distinct from knowledge, reasoning or general response quality.

## Key points

- The benchmark has 820 instructions spanning 50+ tasks, organised into five constraint types at levels 1–5. Data and code are available at github.com/YJiangcm/FollowBench.
- The evaluation is hybrid: rule-based for closed-ended items, and LLM-judge-based for open-ended ones, with the judge prompted with the instruction's evolution history. The paper defines three metrics, including HSR.
- Even GPT-4 and GPT-3.5 reliably satisfied only about three consecutive constraints on average. Situation and Example constraints proved the hardest categories.
- Rankings on FollowBench diverge from those on AlpacaEval, MMLU and BBH. For example:
  - WizardLM-13B-V1.2 beats GPT-3.5 on response quality but trails it on constraint following.
  - Vicuna-V1.5 outperforms LLaMA2-Chat on knowledge and reasoning yet does worse on constraint following.
- Failure consistency (the share of cases where failing at one level means failing at all higher levels) was 42.2% for GPT-4-Preview-1106, compared with 57.3–64.0% for WizardLM-13B-V1.2, Vicuna-7B-V1.5 and ChatGLM3-6B. A moderate sampling temperature worked better than greedy decoding.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 58 works · 23,058 citations
- [The Hong Kong University of Science and Technology (Guangzhou)](https://md.research.elamur.ai/free/en/universities/the-hong-kong-university-of-science-and-technology-guangzhou/) — University, Guangzhou, China · 7 works · 1,026 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu and Wei Wang. 2024. FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jiang2024followbench,
  title = {{FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models}},
  author = {Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu and Wei Wang},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.257},
  url = {https://doi.org/10.18653/v1/2024.acl-long.257}
}
```

- [← previous BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer](https://md.research.elamur.ai/free/en/papers/asai-2024-buffet-benchmarking-large-language-models/)
- [next → Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment](https://md.research.elamur.ai/free/en/papers/raina-2024-is-llm-as-judge-robust/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

54 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. arXiv.org submitters. 2023. arxiv dataset. [find](https://scholar.google.com/scholar?q=arXiv%2Eorg+submitters%2E+2023%2E+arxiv+dataset%2E)
2. Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya… [find](https://scholar.google.com/scholar?q=Stephen+Bach%2C+Victor+Sanh%2C+Zheng+Xin+Yong%2C+Albert+Webson%2C+Colin+Raffel%2C+Nihal+V%2E+Nayak%2C+Abheesht+Sharma%2C+Taewoon+Kim%2C+M+Saiful+Bari%2C+Thibault+Fevry%2C+Zaid+Alyafeai%2C+Manan+Dey%2C+Andrea+Santilli%2C+Zhiqing+)
3. [Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P…](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+2022%2E+Training+a+helpful+and+harmless+assistant+with+rein)
5. Baichuan. 2023. [Baichuan 2: Open large-scale language models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/). arXiv preprint arXiv:2309.10305.
6. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. [Wit3: Web inventory of transcribed and translated talks](https://md.research.elamur.ai/free/en/papers/cettolo-2012-wit3-web-inventory-transcribed-translated/). In Proceedings of the Conference of European Association for Machine Translation (EAMT), pages 261–268.
8. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). arXiv preprint arXiv:2307.03109.
9. Howard Chen, Huihan Li, Danqi Chen, and Karthik Narasimhan. 2022. Controllable text generation with language constraints. arXiv preprint arXiv:2212.10466. — [arXiv:2212.10466](https://arxiv.org/abs/2212.10466) [find](https://scholar.google.com/scholar?q=Howard+Chen%2C+Huihan+Li%2C+Danqi+Chen%2C+and+Karthik+Narasimhan%2E+2022%2E+Controllable+text+generation+with+language+constraints%2E+arXiv+preprint+arXiv%3A2212%2E10466%2E+%E2%80%94+%5BarXiv%3A2212%2E10466%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F22)
10. [Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai…](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
11. Pengyu Cheng and Ruineng Li. 2022. Replacing language model for style transfer. arXiv preprint arXiv:2211.07343. [find](https://scholar.google.com/scholar?q=Pengyu+Cheng+and+Ruineng+Li%2E+2022%2E+Replacing+language+model+for+style+transfer%2E+arXiv+preprint+arXiv%3A2211%2E07343%2E)
12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
13. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. [Glm: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335…
14. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+2018%2E+Hierarchical+neural+story+generation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29)
15. Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic research. Blog post, April, 1. [find](https://scholar.google.com/scholar?q=Xinyang+Geng%2C+Arnav+Gudibande%2C+Hao+Liu%2C+Eric+Wallace%2C+Pieter+Abbeel%2C+Sergey+Levine%2C+and+Dawn+Song%2E+2023%2E+Koala%3A+A+dialogue+model+for+academic+research%2E+Blog+post%2C+April%2C+1%2E)
16. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. [Chatgpt outperforms crowd-workers for text-annotation tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/). arXiv preprint arXiv:2303.15056.
17. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bogdan+Gliwa%2C+Iwona+Mochol%2C+Maciej+Biesek%2C+and+Aleksander+Wawer%2E+2019%2E+SAMSum+corpus%3A+A+human%2Dannotated+dialogue+dataset+for+abstractive+summarization%2E+In+Proceedings+of+the+2nd+Workshop+on+New+Fronti)
18. David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consortium, Philadelphia, 4(1):34. [find](https://scholar.google.com/scholar?q=David+Graff%2C+Junbo+Kong%2C+Ke+Chen%2C+and+Kazuaki+Maeda%2E+2003%2E+English+gigaword%2E+Linguistic+Data+Consortium%2C+Philadelphia%2C+4%281%29%3A34%2E)
19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Proceedings of the International Conference on Learning Representations (ICLR).
20. F Huang, H Kwak, and J An. 2023. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. arxiv. [find](https://scholar.google.com/scholar?q=F+Huang%2C+H+Kwak%2C+and+J+An%2E+2023%2E+Is+chatgpt+better+than+human+annotators%3F+potential+and+limitations+of+chatgpt+in+explaining+implicit+hate+speech%2E+arxiv%2E)
21. Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial distillation of proprietary large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 3134–3154. Association for C… [find](https://scholar.google.com/scholar?q=Yuxin+Jiang%2C+Chunkit+Chan%2C+Mingyang+Chen%2C+and+Wei+Wang%2E+2023%2E+Lion%3A+Adversarial+distillation+of+proprietary+large+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural)
22. Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang, Lifeng Shang, Ruiming Tang, Qun Liu, and Wei Wang. 2024. Learning to edit: Aligning llms with knowledge editing. CoRR, abs/2402.11905. [find](https://scholar.google.com/scholar?q=Yuxin+Jiang%2C+Yufei+Wang%2C+Chuhan+Wu%2C+Wanjun+Zhong%2C+Xingshan+Zeng%2C+Jiahui+Gao%2C+Liangyou+Li%2C+Xin+Jiang%2C+Lifeng+Shang%2C+Ruiming+Tang%2C+Qun+Liu%2C+and+Wei+Wang%2E+2024%2E+Learning+to+edit%3A+Aligning+llms+with+knowl)
23. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32. [find](https://scholar.google.com/scholar?q=Sumith+Kulal%2C+Panupong+Pasupat%2C+Kartik+Chandra%2C+Mina+Lee%2C+Oded+Padon%2C+Alex+Aiken%2C+and+Percy+S+Liang%2E+2019%2E+Spoc%3A+Search%2Dbased+pseudocode+to+code%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2E)
24. Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2024. [Mt-eval: A multi-turn capabilities evaluation benchmark for large language models](https://md.research.elamur.ai/free/en/papers/kwan-2024-mt-eval-multi-turn-capabilities/). CoRR, abs/2401.16745.
25. Sha Li, Heng Ji, and Jiawei Han. 2021. [Document-level event argument extraction by conditional generation](https://md.research.elamur.ai/free/en/papers/li-2021-document-level-event-argument-extraction/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages…
26. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/](https://github.com/) tatsu-lab/alpaca_eval. [↗](https://github.com/)
27. Pierre Lison and Jörg Tiedemann. 2016. [OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles](https://md.research.elamur.ai/free/en/papers/lison-2016-opensubtitles2016-extracting-large-parallel-corpora/). In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929, Portorož, Slovenia. European Language Resources Association (EL…
28. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. [Agentbench: Evaluating llms as agents](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/). arXiv preprint arXiv:2308.03688.
29. George A. Miller. 1992. [WordNet: A lexical database for English](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992.
30. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470–3487, Dublin, Irelan… [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+th)
31. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Associati… [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+Allen%2E+2016%2E+A+corpus+and+cloze+evaluation+for+deeper+understanding+of+commo)
32. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Caglar+Gulcehre%2C+Bing+Xiang%2C+et+al%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2E)
33. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+In+Proceedings+of+the+2018)
34. Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1797%E2%80%931807%2C+Brussels%2C+Belgium%2E+Association+for+Computational+Linguistics%2E)
35. Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, I… [find](https://scholar.google.com/scholar?q=Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Ferrer%2C+Moya+Chen%2C+Guillem+Cucurull%2C+David+Esiobu%2C+Jude+Fernandes%2C+Jeremy+Fu%2C+Wenyin+Fu%2C+Brian+Fuller%2C+Cynthia+Gao%2C+Vedanuj+Goswami%2C+Naman+Goyal%2C+Ant)
36. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. [The E2E dataset: New challenges for end-to-end generation](https://md.research.elamur.ai/free/en/papers/novikova-2017-e2e-dataset-new-challenges-end/). In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 201–206, Saarbrücken, Germany. Association for Computational Linguistics.
37. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
38. CoRR, TB OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. OpenAI. [find](https://scholar.google.com/scholar?q=CoRR%2C+TB+OpenAI%2E+2022%2E+Chatgpt%3A+Optimizing+language+models+for+dialogue%2E+OpenAI%2E)
39. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
40. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. [Multitask prompted training enables zero-shot task generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/). arXiv preprint arXiv:2110.08207.
41. Murray Shanahan, Kyle McDonell, and Laria Reynolds. 2023. [Role-play with large language models](https://md.research.elamur.ai/free/en/papers/shanahan-2023-role-play-large-language-models/). arXiv preprint arXiv:2305.16367.
42. Eric Michael Smith, Diana Gonzalez-Rico, Emily Dinan, and Y-Lan Boureau. 2020. Controlling style in generated dialogue. arXiv preprint arXiv:2009.10855. — [arXiv:2009.10855](https://arxiv.org/abs/2009.10855) [find](https://scholar.google.com/scholar?q=Eric+Michael+Smith%2C+Diana+Gonzalez%2DRico%2C+Emily+Dinan%2C+and+Y%2DLan+Boureau%2E+2020%2E+Controlling+style+in+generated+dialogue%2E+arXiv+preprint+arXiv%3A2009%2E10855%2E+%E2%80%94+%5BarXiv%3A2009%2E10855%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2009)
43. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. 2022. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261.
44. Xiangru Tang, Yiming Zong, Yilun Zhao, Arman Cohan, and Mark Gerstein. 2023. Struc-bench: Are large language models really good at generating complex structured data? arXiv preprint arXiv:2309.08963. — [arXiv:2309.08963](https://arxiv.org/abs/2309.08963) [find](https://scholar.google.com/scholar?q=Xiangru+Tang%2C+Yiming+Zong%2C+Yilun+Zhao%2C+Arman+Cohan%2C+and+Mark+Gerstein%2E+2023%2E+Struc%2Dbench%3A+Are+large+language+models+really+good+at+generating+complex+structured+data%3F+arXiv+preprint+arXiv%3A2309%2E08963%2E+)
45. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Lrec, volume 2012, pages 2214– 2218. Citeseer. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+opus%2E+In+Lrec%2C+volume+2012%2C+pages+2214%E2%80%93+2218%2E+Citeseer%2E)
46. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. [Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/). In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147.
47. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Alicia Tsai, Shereen Oraby, Vittorio Perera, Jiun-Yu Kao, Yuheng Du, Anjali Narayan-Chen, Tagyoung Chung, and Dilek Hakkani-Tur. 2021. Style control for… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Alicia+Tsai%2C+Shereen+Oraby%2C+Vittorio+Perera%2C+Jiun%2DYu+)
48. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85. [find](https://scholar.google.com/scholar?q=Denny+Vrandec%CC%8Cic%CC%81+and+Markus+Kr%C3%B6tzsch%2E+2014%2E+Wikidata%3A+a+free+collaborative+knowledgebase%2E+Communications+of+the+ACM%2C+57%2810%29%3A78%E2%80%9385%2E)
49. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023a. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023a%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Pro)
50. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023b. [Aligning large language models with human: A survey](https://md.research.elamur.ai/free/en/papers/wang-2023-aligning-large-language-models-human/). arXiv preprint arXiv:2307.12966.
51. Zekun Wang, Ge Zhang, Kexin Yang, Ning Shi, Wangchunshu Zhou, Shaochun Hao, Guangzheng Xiong, Yizhi Li, Mong Yuan Sim, Xiuying Chen, et al. 2023c. Interactive natural language processing. arXiv preprint arXiv:2305.13246. — [arXiv:2305.13246](https://arxiv.org/abs/2305.13246) [find](https://scholar.google.com/scholar?q=Zekun+Wang%2C+Ge+Zhang%2C+Kexin+Yang%2C+Ning+Shi%2C+Wangchunshu+Zhou%2C+Shaochun+Hao%2C+Guangzheng+Xiong%2C+Yizhi+Li%2C+Mong+Yuan+Sim%2C+Xiuying+Chen%2C+et+al%2E+2023c%2E+Interactive+natural+language+processing%2E+arXiv+prepri)
52. Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. 2020. [Learning from task descriptions](https://md.research.elamur.ai/free/en/papers/weller-2020-learning-task-descriptions/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1361–1375, Online. Association for Computational Linguistics.
53. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. number of constraints. We use accuracy as the evaluation metric. [find](https://scholar.google.com/scholar?q=Can+Xu%2C+Qingfeng+Sun%2C+Kai+Zheng%2C+Xiubo+Geng%2C+Pu+Zhao%2C+Jiazhan+Feng%2C+Chongyang+Tao%2C+and+Daxin+Jiang%2E+2023%2E+Wizardlm%3A+Empowering+large+language+models+to+follow+complex+instructions%2E+number+of+constrain)
54. Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. 2023. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129. [find](https://scholar.google.com/scholar?q=Sherry+Yang%2C+Ofir+Nachum%2C+Yilun+Du%2C+Jason+Wei%2C+Pieter+Abbeel%2C+and+Dale+Schuurmans%2E+2023%2E+Foundation+models+for+decision+making%3A+Problems%2C+methods%2C+and+opportunities%2E+arXiv+preprint+arXiv%3A2303%2E04129%2E)

*Updated 24 October 2026*