---
title: "Aligning Instruction Tasks Unlocks Large Language Models as"
description: "Instruction-tuned LLMs perform well on many NLP tasks without examples, yet they still… 8.6 points. 36 references, cited by 52 papers here."
authors: ["Kai Zhang", "Bernal Jimenez Gutierrez", "Yu Su"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: ACL 2023"
source: "https://aclanthology.org/2023.findings-acl.50/"
doi: "10.18653/v1/2023.findings-acl.50"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2023-aligning-instruction-tasks-unlocks-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2023-aligning-instruction-tasks-unlocks-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors

# Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors

**Authors:** [Kai Zhang](https://md.research.elamur.ai/free/en/authors/kai-zhang/), [Bernal Jimenez Gutierrez](https://md.research.elamur.ai/free/en/authors/bernal-jimenez-gutierrez/) and [Yu Su](https://md.research.elamur.ai/free/en/authors/yu-su/) · **Year:** 2023 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2023](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2023](https://md.research.elamur.ai/free/en/venues/findings/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-acl.50/) · [PDF](https://aclanthology.org/2023.findings-acl.50.pdf) · [DOI 10.18653/v1/2023.findings-acl.50](https://doi.org/10.18653/v1/2023.findings-acl.50) · [OpenAlex](https://openalex.org/W4385569985)

**Citations:** 66 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/zhang-2023-aligning-instruction-tasks-unlocks-large/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Instruction-tuned LLMs perform well on many NLP tasks without examples, yet they still lose to compact models on relation extraction (RE), the task of naming how two entities in a sentence are connected. The authors attribute this to how rarely RE appears in instruction-tuning collections: under 1% of tasks in Super-NaturalInstructions. Multiple-choice question answering, by contrast, makes up roughly 12–15%. Their framework, QA4RE, therefore recasts RE as multiple-choice QA. The sentence serves as context, and each candidate relation becomes an answer option built from existing entity-filled templates, with type constraints applied where possible. The model only has to output the option letter. Tests covered six instruction-tuned models from the GPT-3.5 and FLAN-T5 families on four benchmarks. QA4RE beat the plain RE prompt and, for the first time, let LLMs clearly surpass the previous zero-shot state of the art built on small NLI-based models. The study suggests that rare tasks can be reframed as common instruction formats.

## Key points
- Benchmarks were TACRED, RETACRED, TACREV and SemEval. Models were GPT-3.5 variants, including text-davinci-003, and FLAN-T5 from Small (80M) to XXL (11B).
- Averaged across the four datasets, F1 rose by 8.2 points for text-davinci-003 (47.6 → 55.8) and 8.6 points for FLAN-T5 XXL (46.5 → 55.1).
- Applying the NLI-style formulation directly to text-davinci-003 performed worst, with an average F1 of 35.0. Causes were a poorly suited 0.5 no-relation threshold, per-relation scoring that magnifies model bias, and the expense of many inference calls.
- Gains held across FLAN-T5 sizes (+4.2 to +8.6 F1), and the larger FLAN-T5 models improved more. Most GPT-3.5 models gained less, and the older text-davinci-001 did not benefit.
- The authors also reported robustness to prompt variations and positive few-shot results. Code and data are released at github.com/OSU-NLP-Group/QA4RE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The Ohio State University](https://md.research.elamur.ai/free/en/universities/the-ohio-state-university/) — University, Columbus, United States · 57 works · 17,793 citations

Also on the byline: [Ohio Supercomputer Center](https://ror.org/01apna436).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Kai Zhang, Bernal Jimenez Gutierrez and Yu Su. 2023. Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2023aligning,
  title = {{Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors}},
  author = {Kai Zhang, Bernal Jimenez Gutierrez and Yu Su},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.50},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.50}
}
```

- [← previous UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://md.research.elamur.ai/free/en/papers/masry-2023-unichart-universal-vision-language-pretrained/)
- [next → MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/juraska-2023-metricx-23-google-submission-wmt/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

36 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. [TACRED revisited: A thorough evaluation of the TACRED relation extraction task](https://md.research.elamur.ai/free/en/papers/alt-2020-tacred-revisited-thorough-evaluation-tacred/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1558– 1569, Online. Association for Computational Linguist…
2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
3. Ohio Supercomputer Center. 1987. Ohio supercomputer center. [find](https://scholar.google.com/scholar?q=Ohio+Supercomputer+Center%2E+1987%2E+Ohio+supercomputer+center%2E)
4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mik… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
5. Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 202… [find](https://scholar.google.com/scholar?q=Xiang+Chen%2C+Ningyu+Zhang%2C+Xin+Xie%2C+Shumin+Deng%2C+Yunzhi+Yao%2C+Chuanqi+Tan%2C+Fei+Huang%2C+Luo+Si%2C+and+Huajun+Chen%2E+2022%2E+Knowprompt%3A+Knowledge%2Daware+prompt%2Dtuning+with+synergistic+optimization+for+relation+)
6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
7. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+Xiny)
8. Geli Fei and Bing Liu. 2016. Breaking the closed world assumption in text classification. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 506–514. Th… [find](https://scholar.google.com/scholar?q=Geli+Fei+and+Bing+Liu%2E+2016%2E+Breaking+the+closed+world+assumption+in+text+classification%2E+In+NAACL+HLT+2016%2C+The+2016+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Ling)
9. Hao Fu, Yao; Peng and Tushar Khot. 2022. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu’s Notion. [find](https://scholar.google.com/scholar?q=Hao+Fu%2C+Yao%3B+Peng+and+Tushar+Khot%2E+2022%2E+How+does+gpt+obtain+its+ability%3F+tracing+emergent+abilities+of+language+models+to+their+sources%2E+Yao+Fu%E2%80%99s+Notion%2E)
10. Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. Fewrel 2.0: Towards more challenging few-shot relation classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural L… [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Xu+Han%2C+Hao+Zhu%2C+Zhiyuan+Liu%2C+Peng+Li%2C+Maosong+Sun%2C+and+Jie+Zhou%2E+2019%2E+Fewrel+2%2E0%3A+Towards+more+challenging+few%2Dshot+relation+classification%2E+In+Proceedings+of+the+2019+Conference+on+Empi)
11. Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2022. Ptr: Prompt tuning with rules for text classification. AI Open, 3:182–192. [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Weilin+Zhao%2C+Ning+Ding%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2022%2E+Ptr%3A+Prompt+tuning+with+rules+for+text+classification%2E+AI+Open%2C+3%3A182%E2%80%93192%2E)
12. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. [Deberta: decoding-enhanced bert with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
13. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. [SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals](https://md.research.elamur.ai/free/en/papers/hendrickx-2010-semeval-2010-task-8-multi/). In Proceedings of the 5th International Wo…
14. Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O’Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. 2022. [OPT-IML: scaling lang…](https://md.research.elamur.ai/free/en/papers/iyer-2022-opt-iml-scaling-language-model/)
15. Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. [Thinking about GPT-3 in-context learning for biomedical IE? think again](https://md.research.elamur.ai/free/en/papers/gutierrez-2022-thinking-about-gpt-3-context/). In Findings of EMNLP, pages 4497–4512, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
17. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R’e, Diana Acosta-Navas, Drew A. Hudson, E. Z… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
18. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. [What makes good in-context examples for gpt-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ACL 2022, Dublin, Ir…
19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+C)
20. Keming Lu, I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma, and Muhao Chen. 2022. Summarization as indirect supervision for relation extraction. In Findings of EMNLP, pages 6575–6594, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Keming+Lu%2C+I%2DHung+Hsu%2C+Wenxuan+Zhou%2C+Mingyu+Derek+Ma%2C+and+Muhao+Chen%2E+2022%2E+Summarization+as+indirect+supervision+for+relation+extraction%2E+In+Findings+of+EMNLP%2C+pages+6575%E2%80%936594%2C+Abu+Dhabi%2C+United+Arab)
21. Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022a. [Noisy channel language model prompting for few-shot text classification](https://md.research.elamur.ai/free/en/papers/min-2021-noisy-channel-language-model-prompting/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27,…
22. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022b. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791–2809, Seattle, United States.… [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+2022b%2E+MetaICL%3A+Learning+to+learn+in+context%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+f)
23. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Tr… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
24. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. [True Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/).
25. Ofer Sabo, Yanai Elazar, Yoav Goldberg, and Ido Dagan. 2021. Revisiting few-shot relation classification: Evaluation data and classification schemes. Trans. Assoc. Comput. Linguistics, 9:691–706. [find](https://scholar.google.com/scholar?q=Ofer+Sabo%2C+Yanai+Elazar%2C+Yoav+Goldberg%2C+and+Ido+Dagan%2E+2021%2E+Revisiting+few%2Dshot+relation+classification%3A+Evaluation+data+and+classification+schemes%2E+Trans%2E+Assoc%2E+Comput%2E+Linguistics%2C+9%3A691%E2%80%93706%2E)
26. Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, and Eneko Agirre. 2021. [Label verbalization and entailment for effective zero and few-shot relation extraction](https://md.research.elamur.ai/free/en/papers/sainz-2021-label-verbalization-entailment-effective-zero/). In Proceedings of EMNLP, pages 1199–1212, Online and Punta Cana, Dominican Republic. Association for Computational Linguist…
27. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan C… — [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
28. Lei Shu, Hu Xu, and Bing Liu. 2017. DOC: deep open classification of text documents. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2911– 2916. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lei+Shu%2C+Hu+Xu%2C+and+Bing+Liu%2E+2017%2E+DOC%3A+deep+open+classification+of+text+documents%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+EMNLP+2017%2C+Copenhagen%2C+)
29. George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. 2021. Re-tacred: Addressing shortcomings of the TACRED dataset. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleve… [find](https://scholar.google.com/scholar?q=George+Stoica%2C+Emmanouil+Antonios+Platanios%2C+and+Barnab%C3%A1s+P%C3%B3czos%2E+2021%2E+Re%2Dtacred%3A+Addressing+shortcomings+of+the+TACRED+dataset%2E+In+Thirty%2DFifth+AAAI+Conference+on+Artificial+Intelligence%2C+AAAI+2021%2C)
30. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Super-naturalinstructions:generalization via declarative instructions on 1600+ tasks. In EMNLP. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+et+al%2E+2022%2E+Super%2Dnaturalinst)
31. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenRevie… [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y%2E+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M%2E+Dai%2C+and+Quoc+V%2E+Le%2E+2022a%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+In+The+Tenth+Internat)
32. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022b. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). CoRR, abs/2201.11903.
33. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
34. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. [PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/). In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328–11339. PMLR.
35. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. [OPT: open pre-t…](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
36. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017,… [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Victor+Zhong%2C+Danqi+Chen%2C+Gabor+Angeli%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Position%2Daware+attention+and+supervised+data+improve+slot+filling%2E+In+Proceedings+of+the+2017+Conference+on+Empir)

*Updated 24 October 2026*