---
title: "When does In-context Learning Fall Short and Why? A Study on"
description: "The authors study \"specification-heavy\" tasks, where annotation guidelines… 33.7 F1 on ACE 2005. 121 references, cited by 37 papers here."
authors: ["Hao Peng", "Xiaozhi Wang", "Jianhui Chen", "Weikai Li", "Yunjia Qi", "Zimu Wang", "Zhili Wu", "Kaisheng Zeng", "Bin Xu", "Lei Hou", "Juanzi Li"]
year: 2023
venue: "arXiv:2311.08993"
source: "https://arxiv.org/abs/2311.08993"
doi: "10.48550/arxiv.2311.08993"
arxiv: "2311.08993"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/peng-2023-context-learning-fall-short-why/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/peng-2023-context-learning-fall-short-why/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks

# When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks

**Authors:** [Hao Peng](https://md.research.elamur.ai/free/en/authors/hao-peng/), [Xiaozhi Wang](https://md.research.elamur.ai/free/en/authors/xiaozhi-wang/), Jianhui Chen, Weikai Li, [Yunjia Qi](https://md.research.elamur.ai/free/en/authors/yunjia-qi/), [Zimu Wang](https://md.research.elamur.ai/free/en/authors/zimu-wang/), Zhili Wu, Kaisheng Zeng, [Bin Xu](https://md.research.elamur.ai/free/en/authors/bin-xu/), [Lei Hou](https://md.research.elamur.ai/free/en/authors/lei-hou/) and [Juanzi Li](https://md.research.elamur.ai/free/en/authors/juanzi-li/) · **Year:** 2023 · **Venue:** arXiv:2311.08993 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.08993) · [PDF](https://arxiv.org/pdf/2311.08993) · [DOI 10.48550/arxiv.2311.08993](https://doi.org/10.48550/arxiv.2311.08993) · [arXiv 2311.08993](https://arxiv.org/abs/2311.08993) · [OpenAlex](https://openalex.org/W4388748297)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [37](https://md.research.elamur.ai/free/en/papers/peng-2023-context-learning-fall-short-why/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models are mostly used through in-context learning (ICL), but its limits are poorly mapped. The authors study "specification-heavy" tasks, where annotation guidelines are long and intricate enough that a person needs hours of training. Classic information extraction is a typical case: the ACE 2005 event guideline runs to 77 pages. They assemble 18 such tasks across six categories and test six LLMs, ranging from FLAN-UL2 to GPT-4. ICL scores usually stay below half of fine-tuned state-of-the-art systems.

Error analysis and targeted experiments point to three causes:
- Models read context superficially, relying on prior knowledge or missing key words.
- Their interpretation of label schemas diverges from human definitions.
- They handle long inputs poorly, including long task specifications.

Fine-tuning FLAN-UL2 brings results close to SoTA, and gains grow with model size. The weakness therefore lies in current alignment methods rather than in the models themselves. A small instruction-tuning set built for these tasks lifts FLAN-UL2's ICL to the level of text-davinci-003. This matters for anyone extracting structured data through prompting.

## Key points
- **Benchmark:** 18 tasks in six groups.
  - NER: CoNLL 2003, ACE 2005, FewNERD
  - RE: TACRED, SemEval, FewRel 2.0, DocRED
  - ED: ACE 2005, MAVEN, RichERE
  - EAE: ACE 2005, RichERE
  - Event relations: MATRES, MAVEN-Causal/Subevent/Temporal
  - Sentiment: SST-5, GoEmotions
- **Models evaluated:** FLAN-UL2, Alpaca, Vicuna, ChatGPT, text-davinci-003, GPT-4.
- **Gap to SoTA:** GPT-4, the strongest model, still trails badly. It gets 33.7 F1 on ACE 2005 event detection versus a SoTA of 73.5, and 25.2 on TACRED versus 76.8. Open models often score near zero on relation and event tasks.
- **Typical error examples:** event types are confused, for instance BUSINESS predicted as TRANSACTION. Models also keep their original answer after small edits that change the gold label.
- **Fine-tuning and alignment:** fine-tuning FLAN-UL2 (20B) reaches results mostly comparable to SoTA, and fine-tuning shows positive scaling with model size. Instruction tuning used 10 hand-written instructions per task, augmented by shuffling the label schema.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [University of California, Los Angeles](https://md.research.elamur.ai/free/en/universities/university-of-california-los-angeles/) — University, Los Angeles, United States · 87 works · 54,961 citations
- [Xi’an Jiaotong-Liverpool University](https://md.research.elamur.ai/free/en/universities/xi-an-jiaotong-liverpool-university/) — University, Suzhou, China · 10 works · 319 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Hao Peng, Xiaozhi Wang, Jianhui Chen, Weikai Li, Yunjia Qi, Zimu Wang, Zhili Wu, Kaisheng Zeng, Bin Xu, Lei Hou and Juanzi Li. 2023. When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks. arXiv:2311.08993.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{peng2023when,
  title = {{When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks}},
  author = {Hao Peng, Xiaozhi Wang, Jianhui Chen, Weikai Li, Yunjia Qi, Zimu Wang, Zhili Wu, Kaisheng Zeng, Bin Xu, Lei Hou and Juanzi Li},
  year = {2023},
  doi = {10.48550/arxiv.2311.08993},
  eprint = {2311.08993},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.08993}
}
```

- [← previous The ROOTS Search Tool: Data Transparency for LLMs](https://md.research.elamur.ai/free/en/papers/piktus-2023-roots-search-tool-data-transparency/)
- [next → xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval](https://md.research.elamur.ai/free/en/papers/khan-2023-xcodeeval-large-scale-multilingual-multitask/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

121 references, 63 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. [PaLM 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023. URL [https://arxiv.org/pdf/2305.10403](https://arxiv.org/pdf/2305.10403). pdf. [↗](https://arxiv.org/pdf/2305.10403)
2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. URL [https://arxiv.org/pdf/…](https://arxiv.org/pdf/…) [↗](https://arxiv.org/pdf/…)
3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional AI: Harmlessness from AI feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022b. URL [https://arxiv.org/pdf/2212.08073.pdf](https://arxiv.org/pdf/2212.08073.pdf). [↗](https://arxiv.org/pdf/2212.08073.pdf)
4. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of — [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/) [https://www.ldc.upenn.edu/](https://www.ldc.upenn.edu/) https://github.com/facebookresearch/llama/blob/main/LICENSE 4 https://github.com/face… [↗](https://www.ldc.upenn.edu/)
5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Walker Christopher, Strassel Stephanie, Medero Julie, and Maeda Kazuaki. ACE 2005 multilingual training corpus, 2005. URL [https://catalog.ldc.upenn.edu/LDC2006T06](https://catalog.ldc.upenn.edu/LDC2006T06). [↗](https://catalog.ldc.upenn.edu/LDC2006T06)
7. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. URL [https://arxiv.org/pdf/2210](https://arxiv.org/pdf/2210). 11416.pdf. [↗](https://arxiv.org/pdf/2210)
8. Amir DN Cohen, Shachar Rosenman, and Yoav Goldberg. Relation classification as two-way span-prediction. arXiv preprint arXiv:2010.04829, 2020. URL [https://arxiv.org/pdf/](https://arxiv.org/pdf/) 2010.04829.pdf. — [arXiv:2010.04829](https://arxiv.org/abs/2010.04829) [↗](https://arxiv.org/pdf/)
9. Linguistic Data Consortium. ACE (Automatic Content Extraction) English annotation guidelines for events. Version, 5(4), 2005. URL [https://www.ldc.upenn.edu/sites/www.ldc](https://www.ldc.upenn.edu/sites/www.ldc). upenn.edu/files/english-events-guidelines-v5.4.3.pdf. [↗](https://www.ldc.upenn.edu/sites/www.ldc)
10. Xuan-Quy Dao and Ngoc-Bich Le. Investigating the effectiveness of chatgpt in mathematical reasoning and problem solving: Evidence from the vietnamese national high school graduation examination. arXiv preprint arXiv:2306.06331, 2023. URL [https://arxiv.org/pdf/](https://arxiv.org/pdf/) 2306.06331.pdf. [↗](https://arxiv.org/pdf/)
11. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. GoEmotions: A dataset of fine-grained emotions. In Proceedings of ACL, pp. 4040– 4054, 2020. URL https://aclanthology.org/2020.acl-main.372.pdf. [↗](https://aclanthology.org/2020.acl-main.372.pdf)
12. Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. [Few-NERD: A few-shot named entity recognition dataset](https://md.research.elamur.ai/free/en/papers/ding-2021-few-nerd-few-shot-named/). In Proceedings of ACL-IJCNLP, pp. 3198–3213, 2021. URL https://aclanthology.org/2021.acl-long. 248.pdf. [↗](https://aclanthology.org/2021.acl-long)
13. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06904, 2022. URL [https://arxiv.org/pdf/2203…](https://arxiv.org/pdf/2203…) — [arXiv:2203.06904](https://arxiv.org/abs/2203.06904) [↗](https://arxiv.org/pdf/2203…)
14. Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. [RAFT: Reward ranked finetuning for generative foundation model alignment](https://md.research.elamur.ai/free/en/papers/dong-2023-raft-reward-ranked-finetuning-generative/). arXiv preprint arXiv:2304.06767, 2023. URL [https://arxiv.org/pdf/2304.06767](https://arxiv.org/pdf/2304.06767). pdf. [↗](https://arxiv.org/pdf/2304.06767)
15. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.[00234, 2022](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/). URL [https://arxiv.org/pdf/2301.00234.pdf](https://arxiv.org/pdf/2301.00234.pdf). [↗](https://arxiv.org/pdf/2301.00234.pdf)
16. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. [Mathematical capabilities of ChatGPT](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/). arXiv preprint arXiv:2301.13867, 2023. URL [https://arxiv.org/pdf/2301](https://arxiv.org/pdf/2301). 13867.pdf. [↗](https://arxiv.org/pdf/2301)
17. Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411–437, 2020. URL [https://arxiv.org/pdf/2001.09768.pdf](https://arxiv.org/pdf/2001.09768.pdf). — [arXiv:2001.09768](https://arxiv.org/abs/2001.09768) [↗](https://arxiv.org/pdf/2001.09768.pdf)
18. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. [Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/). arXiv preprint arXiv:2209.07858, 2022. URL https://ar…
19. Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. [Exploring the feasibility of ChatGPT for event extraction](https://md.research.elamur.ai/free/en/papers/gao-2023-exploring-feasibility-chatgpt-event-extraction/). arXiv preprint arXiv:2303.03836, 2023a. URL [https://arxiv.org/pdf/](https://arxiv.org/pdf/) 2303.03836.pdf. [↗](https://arxiv.org/pdf/)
20. Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. FewRel 2.0: Towards more challenging few-shot relation classification. In Proceedings of EMNLP-IJCNLP, pp. 6250–6255, 2019. URL https://aclanthology.org/D19-1649.pdf. [↗](https://aclanthology.org/D19-1649.pdf)
21. Yuan Gao, Ruili Wang, and Feng Hou. How to design translation prompts for ChatGPT: An empirical study. arXiv preprint arXiv:2304.02182, 2023b. URL [https://arxiv.org/pdf/](https://arxiv.org/pdf/) 2304.02182.pdf. — [arXiv:2304.02182](https://arxiv.org/abs/2304.02182) [↗](https://arxiv.org/pdf/)
22. Siddhant Garg, Thuy Vu, and Alessandro Moschitti. TANDA: Transfer and adapt pre-trained transformer models for answer sentence selection. In Proceedings of AAAI, pp. 7780–7788, 2020. URL [https://arxiv.org/pdf/1911.04118.pdf](https://arxiv.org/pdf/1911.04118.pdf). — [arXiv:1911.04118](https://arxiv.org/abs/1911.04118) [↗](https://arxiv.org/pdf/1911.04118.pdf)
23. Lise Getoor and Ben Taskar. Global inference for entity and relation identification via linear programming formulation, 2007. URL [https://cogcomp.seas.upenn.edu/papers/](https://cogcomp.seas.upenn.edu/papers/) RothYi07.pdf. [↗](https://cogcomp.seas.upenn.edu/papers/)
24. Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan. Is information extraction solved by ChatGPT? An analysis of performance, evaluation criteria, robustness and errors. arXiv preprint arXiv:2305.14450, 2023. URL [https://arxiv.org/pdf/2305](https://arxiv.org/pdf/2305). 14450.pdf. — [arXiv:2305.14450](https://arxiv.org/abs/2305.14450) [↗](https://arxiv.org/pdf/2305)
25. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. [Towards a unified view of parameter-efficient transfer learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/). In Proceedings of ICLR, 2021. URL [https://openreview.net/pdf?id=0RDcd5Axok](https://openreview.net/pdf?id=0RDcd5Axok). [↗](https://openreview.net/pdf?id=0RDcd5Axok)
26. Franz A Heinsen. An algorithm for routing vectors in sequences. arXiv preprint arXiv:2211.11754, 2022. URL [https://arxiv.org/pdf/2211.11754.pdf](https://arxiv.org/pdf/2211.11754.pdf). — [arXiv:2211.11754](https://arxiv.org/abs/2211.11754) [↗](https://arxiv.org/pdf/2211.11754.pdf)
27. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. [SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals](https://md.research.elamur.ai/free/en/papers/hendrickx-2010-semeval-2010-task-8-multi/).
28. ACL 2010, pp. 33, 2010. URL https://aclanthology.org/S10-1006.pdf. — [SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals](https://md.research.elamur.ai/free/en/papers/hendrickx-2010-semeval-2010-task-8-multi/) [↗](https://aclanthology.org/S10-1006.pdf)
29. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. [Aligning AI with shared human values](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-aligning-ai-shared-human-values/). In Proceedings of ICLR, 2020. URL [https://openreview.net/pdf?id=dNy_RKzJacY](https://openreview.net/pdf?id=dNy_RKzJacY). [↗](https://openreview.net/pdf?id=dNy_RKzJacY)
30. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Proceedings of NeurIPS Datasets and Benchmarks Track (Round 2), 2021. URL [https://openreview.net/pdf?id=7Bywt2mQsCe](https://openreview.net/pdf?id=7Bywt2mQsCe). [↗](https://openreview.net/pdf?id=7Bywt2mQsCe)
31. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of ICML, pp. 2790–2799, 2019. URL [http://proceedings.mlr.press/v97/houlsby19a/houlsby19a.pdf](http://proceedings.mlr.press/v97/houlsby19a/houlsby19a.pdf). [↗](http://proceedings.mlr.press/v97/houlsby19a/houlsby19a.pdf)
32. Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. arXiv preprint arXiv:2308.00675, 2023. URL [https://arxiv.org/pdf/](https://arxiv.org/pdf/) 2308.00675.pdf. [↗](https://arxiv.org/pdf/)
33. Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. [OPT-IML: Scaling language model instruction meta learning through the lens of generalization](https://md.research.elamur.ai/free/en/papers/iyer-2022-opt-iml-scaling-language-model/). arXiv preprint arXiv:2212.12017, 2022. URL https:…
34. Wenxiang Jiao, Wenxuan Wang, JT Huang, Xing Wang, and ZP Tu. [Is ChatGPT a good translator? Yes with gpt-4 as the engine](https://md.research.elamur.ai/free/en/papers/jiao-2023-chatgpt-good-translator-yes-gpt/). arXiv preprint arXiv:2301.08745, 2023. URL [https://arxiv.org/pdf/2301.08745.pdf](https://arxiv.org/pdf/2301.08745.pdf). [↗](https://arxiv.org/pdf/2301.08745.pdf)
35. Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. [Thinking about GPT-3 in-context learning for biomedical IE? think again](https://md.research.elamur.ai/free/en/papers/gutierrez-2022-thinking-about-gpt-3-context/). In Findings of EMNLP, pp. 4497–4512, 2022. doi: 10.18653/v1/2022.findings-emnlp.329. URL https://aclanthology.org/2022.findi… [↗](https://aclanthology.org/2022.findi…)
36. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. [Challenges and applications of large language models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/). arXiv preprint arXiv:2307.10169, 2023. URL [https://arxiv.org/pdf/2307.10169.pdf](https://arxiv.org/pdf/2307.10169.pdf). [↗](https://arxiv.org/pdf/2307.10169.pdf)
37. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. Alignment of language agents. arXiv preprint arXiv:2103.14659, 2021. URL [https://arxiv.org/pdf/2103.14659.pdf](https://arxiv.org/pdf/2103.14659.pdf). [↗](https://arxiv.org/pdf/2103.14659.pdf)
38. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. [ChatGPT: Jack of all trades, master of none](https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/). Information Fusion, pp. 101861, 2023. URL [https://arxiv.org/](https://arxiv.org/) pdf/2302.10724.pdf. [↗](https://arxiv.org/)
39. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction. arXiv preprint arXiv:1811.07871, 2018. URL [https://arxiv.org/pdf/1811.07871.pdf](https://arxiv.org/pdf/1811.07871.pdf). [↗](https://arxiv.org/pdf/1811.07871.pdf)
40. Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. Evaluating ChatGPT’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness. arXiv preprint arXiv:2304.11633, 2023a. URL [https://arxiv.org/pdf/2304.11633.pdf](https://arxiv.org/pdf/2304.11633.pdf). [↗](https://arxiv.org/pdf/2304.11633.pdf)
41. Jing Li, Shanshan Feng, and Billy Chiu. Few-shot relation extraction with dual graph neural network interaction. IEEE Transactions on Neural Networks and Learning Systems, 2023b. URL [https://ieeexplore.ieee.org/document/10143375](https://ieeexplore.ieee.org/document/10143375). [↗](https://ieeexplore.ieee.org/document/10143375)
42. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. RAIN: Your language models can align themselves without finetuning. arXiv preprint arXiv:2309.07124, 2023c. URL [https://arxiv.org/pdf/2309.07124.pdf](https://arxiv.org/pdf/2309.07124.pdf). — [arXiv:2309.07124](https://arxiv.org/abs/2309.07124) [↗](https://arxiv.org/pdf/2309.07124.pdf)
43. Zhongyang Li, Xiao Ding, and Ting Liu. Story ending prediction by transferable BERT. In Proceedings of IJCAI, pp. 1800–1806, 2019. URL [https://www.ijcai.org/Proceedings/](https://www.ijcai.org/Proceedings/) 2019/0249.pdf. [↗](https://www.ijcai.org/Proceedings/)
44. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arxiv:2211.09110, 2022. URL [https://arxiv.org/pdf/2211](https://arxiv.org/pdf/2211). 09110.pdf. [↗](https://arxiv.org/pdf/2211)
45. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. URL — [Let's Verify Step by Step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/) [https://arxiv.org/pdf/2305.20050.pdf](https://arxiv.org/pdf/2305.20050.pdf). [↗](https://arxiv.org/pdf/2305.20050.pdf)
46. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. [Evaluating the logical reasoning ability of ChatGPT and GPT-4](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-logical-reasoning-ability-chatgpt/). arXiv preprint arXiv:2304.03439, 2023a. URL [https://arxiv.org/pdf/2304.03439.pdf](https://arxiv.org/pdf/2304.03439.pdf). [↗](https://arxiv.org/pdf/2304.03439.pdf)
47. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). Advances in Neural Information Processing Systems, 35:1950–1965, 2022. URL [https://proceedings.neurips.cc/pape…](https://proceedings.neurips.cc/pape…) [↗](https://proceedings.neurips.cc/pape…)
48. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). arXiv preprint arXiv:2307.03172, 2023b. URL [https://arxiv.org/pdf/2307.03172.pdf](https://arxiv.org/pdf/2307.03172.pdf). [↗](https://arxiv.org/pdf/2307.03172.pdf)
49. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. [Entity-based knowledge conflicts in question answering](https://md.research.elamur.ai/free/en/papers/longpre-2021-entity-based-knowledge-conflicts-question/). In Proceedings of EMNLP, pp. 7052– 7063, 2021. URL https://aclanthology.org/2021.emnlp-main.565.pdf. [↗](https://aclanthology.org/2021.emnlp-main.565.pdf)
50. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/). arXiv preprint arXiv:2301.13688, 2023. URL [https://arxiv.org/](https://arxiv.org/) pdf/2301.13688.pdf. [↗](https://arxiv.org/)
51. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of ICLR, 2018. URL [https://openreview.net/pdf?id=Bkg6RiCqY7](https://openreview.net/pdf?id=Bkg6RiCqY7). [↗](https://openreview.net/pdf?id=Bkg6RiCqY7)
52. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. [Chameleon: Plug-and-play compositional reasoning with large language models](https://md.research.elamur.ai/free/en/papers/lu-2023-chameleon-plug-play-compositional-reasoning/). arXiv preprint arXiv:2304.09842, 2023. URL [https://arxiv.org/pdf/2304](https://arxiv.org/pdf/2304). 09842.pdf. [↗](https://arxiv.org/pdf/2304)
53. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). In Proceedings of ACL, pp. 8086–8098, 2022. URL https://aclanthology.org/2022.acl-long. 556.pdf. [↗](https://aclanthology.org/2022.acl-long)
54. Youmi Ma, An Wang, and Naoaki Okazaki. DREEAM: Guiding attention with evidence for improving document-level relation extraction. In Proceedings of EACL, pp. 1963–1975, 2023. URL https://aclanthology.org/2023.eacl-main.145.pdf. [↗](https://aclanthology.org/2023.eacl-main.145.pdf)
55. Yubo Ma, Zehao Wang, Yixin Cao, Mukai Li, Meiqi Chen, Kun Wang, and Jing Shao. [Prompt for Extraction? PAIE: Prompting argument interaction for event argument extraction](https://md.research.elamur.ai/free/en/papers/ma-2022-prompt-extraction-paie-prompting-argument/). In Proceedings of ACL, pp. 6759–6774, 2022. URL https://aclanthology.org/2022.acl-long. 466.pdf. [↗](https://aclanthology.org/2022.acl-long)
56. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of ACL, pp. 9802–9822, 2023. URL https://aclanthology.org/2023.acl-long.546.pdf. [↗](https://aclanthology.org/2023.acl-long.546.pdf)
57. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) In Proceedings of EMNLP, pp. 11048–11064, 2022. URL https://aclanthology.org/ 2022.emnlp-main.759.pdf. [↗](https://aclanthology.org/)
58. Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing instructional prompts to GPTk’s language. In Findings of ACL, pp. 589–612, 2022a. [URL https://aclanthology.org/2022.findings-acl.50.pdf](https://md.research.elamur.ai/free/en/papers/mishra-2022-reframing-instructional-prompts-gptk-s/). [↗](https://aclanthology.org/2022.findings-acl.50.pdf)
59. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of ACL, pp. 3470–3487, 2022b. URL https://aclanthology.org/2022.acl-long.244.pdf. [↗](https://aclanthology.org/2022.acl-long.244.pdf)
60. Qiang Ning, Hao Wu, and Dan Roth. A multi-axis annotation scheme for event temporal relations. In Proceedings of ACL, pp. 1318–1328, 2018. URL https://aclanthology.org/ P18-1122.pdf. [↗](https://aclanthology.org/)
61. OpenAI. Introducing ChatGPT, 2022. URL [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
62. OpenAI. Improving mathematical reasoning with process supervision, 2023a. URL [https://openai.com/research/](https://openai.com/research/) improving-mathematical-reasoning-with-process-supervision. [↗](https://openai.com/research/)
63. OpenAI. Introducing superalignment, 2023b. introducing-superalignment. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Introducing+superalignment%2C+2023b%2E+introducing%2Dsuperalignment%2E)
64. URL — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) [https://openai.com/blog/R](https://openai.com/blog/R) OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023c. URL https://arxiv.org/pdf/2303.08774.pdf. [↗](https://openai.com/blog/R)
65. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022. URL http…
66. Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning” learns” in-context: Disentangling task recognition and task learning. arXiv preprint arXiv:2305.[09731, 2023](https://md.research.elamur.ai/free/en/papers/pan-2023-what-context-learning-learns-context/). URL [https://arxiv.org/pdf/2305.09731.pdf](https://arxiv.org/pdf/2305.09731.pdf). [↗](https://arxiv.org/pdf/2305.09731.pdf)
67. Sachin Pawar, Girish K Palshikar, and Pushpak Bhattacharyya. Relation extraction: A survey. arXiv preprint arXiv:1712.05191, 2017. URL [https://arxiv.org/pdf/1712.05191.pdf](https://arxiv.org/pdf/1712.05191.pdf). — [arXiv:1712.05191](https://arxiv.org/abs/1712.05191) [↗](https://arxiv.org/pdf/1712.05191.pdf)
68. Hao Peng, Xiaozhi Wang, Feng Yao, Zimu Wang, Chuzhao Zhu, Kaisheng Zeng, Lei Hou, and Juanzi Li. OmniEvent: A comprehensive, fair, and easy-to-use toolkit for event understanding. arXiv preprint arXiv:2309.14258, 2023. URL [https://arxiv.org/pdf/2309.14258](https://arxiv.org/pdf/2309.14258). pdf. [↗](https://arxiv.org/pdf/2309.14258)
69. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. [Red teaming language models with language models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/). In Proceedings of EMNLP, pp. 3419–3448, 2022. URL https://aclanthology.org/ 2022.emnlp-main.225.pdf. [↗](https://aclanthology.org/)
70. Jason Phang, Thibault Févry, and Samuel R Bowman. [Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/). arXiv preprint arXiv:1811.01088, 2018. URL [https://arxiv.org/pdf/1811.01088.pdf](https://arxiv.org/pdf/1811.01088.pdf). [↗](https://arxiv.org/pdf/1811.01088.pdf)
71. Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. URL [https://arxiv.org/pdf/2108.12409.pdf](https://arxiv.org/pdf/2108.12409.pdf). [↗](https://arxiv.org/pdf/2108.12409.pdf)
72. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. [Is ChatGPT a general-purpose natural language processing task solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/) arXiv preprint arXiv:2302.06476, 2023. URL [https://arxiv.org/pdf/2302.06476.pdf](https://arxiv.org/pdf/2302.06476.pdf). [↗](https://arxiv.org/pdf/2302.06476.pdf)
73. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. [Pre-trained models for natural language processing: A survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/). Science China Technological Sciences, 63(10): 1872–1897, 2020. URL https://arxiv.org/abs/2003.08271. [↗](https://arxiv.org/abs/2003.08271)
74. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551, 2020. URL [https://jmlr.org/papers…](https://jmlr.org/papers…) [↗](https://jmlr.org/papers…)
75. Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1):2, 2019. URL [https://arxiv.org/pdf/](https://arxiv.org/pdf/) 1910.01708.pdf. — [arXiv:1910.01708](https://arxiv.org/abs/1910.01708) [↗](https://arxiv.org/pdf/)
76. Erik Tjong Kim Sang and Fien De Meulder. [Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/). In Proceedings of CoNLL, pp. 142–147, 2003. URL https://aclanthology.org/W03-0419.pdf. [↗](https://aclanthology.org/W03-0419.pdf)
77. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. [SCROLLS: Standardized CompaRison over long language sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/). In Proceedings of EMNLP, pp. 12007–12021, 2022. doi: 10.18653/v1/2022. emnlp-main.823. URL https://…
78. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. [HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/). arXiv preprint arXiv:2303.17580, 2023. URL [https://arxiv.org/pdf/2303.17580.pdf](https://arxiv.org/pdf/2303.17580.pdf). [↗](https://arxiv.org/pdf/2303.17580.pdf)
79. Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Danqi Chen, and He He. Measuring inductive biases of in-context learning with underspecified demonstrations. In Proceedings of ACL, pp. 11289–11310, 2023. doi: 10.18653/v1/2023.acl-long.632. URL https://aclanthology.org/2023.acl-long.632. [↗](https://aclanthology.org/2023.acl-long.632)
80. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of EMNLP, pp. 1631–1642, 2013. URL https://aclanthology.org/D13-1170.pdf. [↗](https://aclanthology.org/D13-1170.pdf)
81. Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. [From light to Rich ERE: Annotation of entities, relations, and events](https://md.research.elamur.ai/free/en/papers/song-2015-light-rich-ere-annotation-entities/). In Proceedings of the 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Repres…
82. Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-Tail: How knowledgeable are large language models (LLM)? Aka will LLMs replace knowledge graphs? arXiv preprint arXiv:2308.10168, 2023a. URL [https://arxiv.org/pdf/2308.10168.pdf](https://arxiv.org/pdf/2308.10168.pdf). [↗](https://arxiv.org/pdf/2308.10168.pdf)
83. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. [Principle-driven self-alignment of language models from scratch with minimal human supervision](https://md.research.elamur.ai/free/en/papers/sun-2023-principle-driven-self-alignment-language/). arXiv preprint arXiv:2305.03047, 2023b. URL [https://arxiv.org/pdf/2305.03047.pdf](https://arxiv.org/pdf/2305.03047.pdf). [↗](https://arxiv.org/pdf/2305.03047.pdf)
84. Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprint arXiv:2102.02503, 2021. URL [https://arxiv.org/pdf/2102.02503.pdf](https://arxiv.org/pdf/2102.02503.pdf). [↗](https://arxiv.org/pdf/2102.02503.pdf)
85. Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. Evaluation of ChatGPT as a question answering system for answering complex questions. arXiv preprint arXiv:2303.07992, 2023. URL [https://arxiv.org/pdf/2303.07992.pdf](https://arxiv.org/pdf/2303.07992.pdf). — [arXiv:2303.07992](https://arxiv.org/abs/2303.07992) [↗](https://arxiv.org/pdf/2303.07992.pdf)
86. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpaca: A strong, replicable instruction-following model, 2023. URL [https://crfm.stanford.edu/2023/03/13/alpaca.html](https://crfm.stanford.edu/2023/03/13/alpaca.html). [↗](https://crfm.stanford.edu/2023/03/13/alpaca.html)
87. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. [UL2: Unifying language learning paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/). In Proceedings of ICLR, 2022. URL [https://openreview.net/pdf?id=](https://openreview.net/pdf?id=) 6ruVLB727MC. [↗](https://openreview.net/pdf?id=)
88. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023. URL [https://arxiv.org/pdf/2302.13971.…](https://arxiv.org/pdf/2302.13971.…) [↗](https://arxiv.org/pdf/2302.13971.…)
89. Minh Van Nguyen, Bonan Min, Franck Dernoncourt, and Thien Nguyen. Joint extraction of entities, relations, and events via modeling inter-instance and inter-label dependencies. In Proceedings of NAACL-HLT, pp. 4363–4374, 2022. URL https://aclanthology.org/2022. naacl-main.324.pdf. [↗](https://aclanthology.org/2022)
90. Vicuna. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/). [↗](https://lmsys.org/blog/2023-03-30-vicuna/)
91. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. ACE 2005 multilingual training corpus. Linguistic Data Consortium, 57, 2006. URL [https://catalog.ldc.upenn.edu/LDC2006T06](https://catalog.ldc.upenn.edu/LDC2006T06). [↗](https://catalog.ldc.upenn.edu/LDC2006T06)
92. Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. DeepStruct: Pretraining of language models for structure prediction. In Findings of ACL, pp. 803–823, 2022a. URL https://aclanthology.org/2022.findings-acl.67v2.pdf. [↗](https://aclanthology.org/2022.findings-acl.67v2.pdf)
93. Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. [MAVEN: A massive general domain event detection dataset](https://md.research.elamur.ai/free/en/papers/wang-2020-maven-massive-general-domain-event/). In Proceedings of EMNLP, pp. 1652–1671, 2020. URL https://aclanthology.org/2020. emnlp-main.129.pdf. [↗](https://aclanthology.org/2020)
94. Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, et al. [MAVEN-ERE: A unified large-scale dataset for event coreference, temporal, causal, and subevent relation extraction](https://md.research.elamur.ai/free/en/papers/wang-2022-maven-ere-unified-large-scale/). arXiv preprint arXiv:2211.07342, 2022b. URL [https://arxiv.org/pdf/…](https://arxiv.org/pdf/…) [↗](https://arxiv.org/pdf/…)
95. Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. [Automated concatenation of embeddings for structured prediction](https://md.research.elamur.ai/free/en/papers/wang-2021-automated-concatenation-embeddings-structured-prediction/). In Proceedings of ACL-IJCNLP, pp. 2643–2660, 2021a. URL https://aclanthology.org/2021.acl-long. 206.pdf. [↗](https://aclanthology.org/2021.acl-long)
96. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022c. URL [https://arxiv.org/pdf/2212.10560](https://arxiv.org/pdf/2212.10560). pdf. [↗](https://arxiv.org/pdf/2212.10560)
97. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-Naturalinstructions: Generalization via declarative instructions on 1600+ tasks. In Proceedings of EMNLP, pp. 5085–51… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+et+al%2E+Super%2DNaturalinstructio)
98. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. [Aligning large language models with human: A survey](https://md.research.elamur.ai/free/en/papers/wang-2023-aligning-large-language-models-human/). arXiv preprint arXiv:2307.12966, 2023. URL [https://arxiv.org/pdf/2307.12966.pdf](https://arxiv.org/pdf/2307.12966.pdf). [↗](https://arxiv.org/pdf/2307.12966.pdf)
99. Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li, and Jie Zhou. [CLEVE: Contrastive pre-training for event extraction](https://md.research.elamur.ai/free/en/papers/wang-2021-cleve-contrastive-pre-training-event/). In Proceedings of ACL-IJCNLP, pp. 6283–6297, 2021b. URL https://aclanthology.org/2021.acl-long.491.pdf. [↗](https://aclanthology.org/2021.acl-long.491.pdf)
100. Albert Webson and Ellie Pavlick. [Do prompt-based models really understand the meaning of their prompts?](https://md.research.elamur.ai/free/en/papers/webson-2022-do-prompt-based-models-really/) In Proceedings of NAACL-HLT, pp. 2300–2344, 2022. URL https://aclanthology.org/2022.naacl-main.167.pdf. [↗](https://aclanthology.org/2022.naacl-main.167.pdf)
101. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In Proceedings of ICLR, 2021. URL [https://openreview.net/pdf?id=gEZrGCozdqR](https://openreview.net/pdf?id=gEZrGCozdqR). [↗](https://openreview.net/pdf?id=gEZrGCozdqR)
102. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/). arXiv preprint arXiv:2303.03846, 2023. URL [https://arxiv.org/pdf/2303.03846.pdf](https://arxiv.org/pdf/2303.03846.pdf). [↗](https://arxiv.org/pdf/2303.03846.pdf)
103. Lilian Weng. Prompt engineering. lilianweng.github.io, Mar 2023. URL [https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/](https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/). [↗](https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/)
104. Chenxi Whitehouse, Monojit Choudhury, and Alham Fikri Aji. LLM-powered data augmentation for enhanced crosslingual performance. arXiv preprint arXiv:2305.14288, 2023. URL [https://arxiv.org/pdf/2305.14288.pdf](https://arxiv.org/pdf/2305.14288.pdf). — [arXiv:2305.14288](https://arxiv.org/abs/2305.14288) [↗](https://arxiv.org/pdf/2305.14288.pdf)
105. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
106. Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge conflicts. arXiv preprint arXiv:2305.[13300, 2023](https://md.research.elamur.ai/free/en/papers/xie-2023-adaptive-chameleon-stubborn-sloth-revealing/). URL [https://arxiv.org/pdf/2305.13300.pdf](https://arxiv.org/pdf/2305.13300.pdf). [↗](https://arxiv.org/pdf/2305.13300.pdf)
107. Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reasoners? A comprehensive evaluation from deductive, inductive and abductive views. arXiv preprint arXiv:2306.[09841, 2023a](https://md.research.elamur.ai/free/en/papers/xu-2023-large-language-models-really-good/). URL [https://arxiv.org/](https://arxiv.org/) pdf/2306.09841.pdf. [↗](https://arxiv.org/)
108. Xin Xu, Yuqi Zhu, Xiaohan Wang, and Ningyu Zhang. How to unleash the power of large language models for few-shot relation extraction? arXiv preprint arXiv:2305.01555, 2023b. URL [https://arxiv.org/pdf/2305.01555.pdf](https://arxiv.org/pdf/2305.01555.pdf). [↗](https://arxiv.org/pdf/2305.01555.pdf)
109. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of ACL, pp. 764–777, 2019. URL https://aclanthology.org/P19-1074.pdf. [↗](https://aclanthology.org/P19-1074.pdf)
110. Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al. KoLA: Carefully benchmarking world knowledge of large language models. arXiv preprint arXiv:2306.09296, 2023a. URL [https://arxiv.org/](https://arxiv.org/) pdf/2306.09296.pdf. — [arXiv:2306.09296](https://arxiv.org/abs/2306.09296) [↗](https://arxiv.org/)
111. Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. [Large language model as attributed training data generator: A tale of diversity and bias](https://md.research.elamur.ai/free/en/papers/yu-2023-large-language-model-attributed-training/). arXiv preprint arXiv:2306.15895, 2023b. URL [https://arxiv.org/pdf/2306](https://arxiv.org/pdf/2306). 15895.pdf. [↗](https://arxiv.org/pdf/2306)
112. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. [MAmmoTH: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653, 2023. URL [https://arxiv.org/pdf/2309.05653.pdf](https://arxiv.org/pdf/2309.05653.pdf). [↗](https://arxiv.org/pdf/2309.05653.pdf)
113. Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. [Word-level textual adversarial attacking as combinatorial optimization](https://md.research.elamur.ai/free/en/papers/zang-2020-word-level-textual-adversarial-attacking/). In Proceedings of ACL, pp. 6066–6080, 2020. URL https://aclanthology.org/2020.acl-main. 540.pdf. [↗](https://aclanthology.org/2020.acl-main)
114. Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon. Optimizing bi-encoder for named entity recognition via contrastive learning. arXiv preprint arXiv:2208.14565, 2022. URL [https://arxiv.org/pdf/2208.14565.pdf](https://arxiv.org/pdf/2208.14565.pdf). [↗](https://arxiv.org/pdf/2208.14565.pdf)
115. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. Position-aware attention and supervised data improve slot filling. In Proceedings of EMNLP, 2017. URL https://aclanthology.org/D17-1004.pdf. [↗](https://aclanthology.org/D17-1004.pdf)
116. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In Proceedings of ICML, pp. 12697–12706, 2021. URL [http://proceedings.mlr.press/v139/zhao21c/zhao21c.pdf](http://proceedings.mlr.press/v139/zhao21c/zhao21c.pdf). [↗](http://proceedings.mlr.press/v139/zhao21c/zhao21c.pdf)
117. Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795, 2023. URL [https://arxiv.org/pdf/2305.14795.pdf](https://arxiv.org/pdf/2305.14795.pdf). [↗](https://arxiv.org/pdf/2305.14795.pdf)
118. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. LIMA: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023a. URL [https://arxiv.org/pdf/2305.11206.pdf](https://arxiv.org/pdf/2305.11206.pdf). [↗](https://arxiv.org/pdf/2305.11206.pdf)
119. Jie Zhou, Shenpo Dong, Hongkui Tu, Xiaodong Wang, and Yong Dou. RSGT: Relational structure guided temporal relation extraction. In Proceedings of COLING, pp. 2001–2010, 2022. URL https://aclanthology.org/2022.coling-1.174.pdf. [↗](https://aclanthology.org/2022.coling-1.174.pdf)
120. Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. [Context-faithful prompting for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-context-faithful-prompting-large-language/). arXiv preprint arXiv:2303.11315, 2023b. URL [https://arxiv.org/pdf/2303.11315.pdf](https://arxiv.org/pdf/2303.11315.pdf). [↗](https://arxiv.org/pdf/2303.11315.pdf)
121. Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, et al. Efficiently measuring the cognitive ability of llms: An adaptive testing perspective. arXiv preprint arXiv:2306.10512, 2023. URL [https://arxiv.org/pdf/2306.10512.pdf](https://arxiv.org/pdf/2306.10512.pdf). [↗](https://arxiv.org/pdf/2306.10512.pdf)

*Updated 29 October 2026*