---
title: "GoLLIE: Annotation Guidelines improve Zero-Shot — Sainz 2023"
description: "Instruction-tuned LLMs handle many unseen tasks well, but in Information Extraction (IE) they still lag behind task-specific systems."
authors: ["Oscar Sainz", "Iker Garcia-Ferrero", "Rodrigo Agerri", "Oier Lopez de Lacalle", "German Rigau", "Eneko Agirre"]
year: 2023
venue: "arXiv:2310.03668"
source: "https://arxiv.org/abs/2310.03668"
doi: "10.48550/arxiv.2310.03668"
arxiv: "2310.03668"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sainz-2023-gollie-annotation-guidelines-improve-zero/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sainz-2023-gollie-annotation-guidelines-improve-zero/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction

# GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction

**Authors:** [Oscar Sainz](https://md.research.elamur.ai/free/en/authors/oscar-sainz/), [Iker Garcia-Ferrero](https://md.research.elamur.ai/free/en/authors/iker-garcia-ferrero/), [Rodrigo Agerri](https://md.research.elamur.ai/free/en/authors/rodrigo-agerri/), [Oier Lopez de Lacalle](https://md.research.elamur.ai/free/en/authors/oier-lopez-de-lacalle/), [German Rigau](https://md.research.elamur.ai/free/en/authors/german-rigau/), [Eneko Agirre](https://md.research.elamur.ai/free/en/authors/eneko-agirre/) · **Year:** 2023 · **Venue:** arXiv:2310.03668 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.03668) · [PDF](https://arxiv.org/pdf/2310.03668) · [DOI 10.48550/arxiv.2310.03668](https://doi.org/10.48550/arxiv.2310.03668) · [arXiv 2310.03668](https://arxiv.org/abs/2310.03668) · [OpenAlex](https://openalex.org/W4387430794)

**Citations:** 17 citations (openalex-snapshot · 2026-10-14) · [97](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Instruction-tuned LLMs handle many unseen tasks well, but in Information Extraction (IE) they still lag behind task-specific systems. One reason is that a label name alone is ambiguous, since schemas define the same category differently. For example, ACE05 counts pronouns as persons, while CoNLL03 does not. When off-the-shelf models are given annotation guidelines, they tend to fall back on their prior knowledge instead.

The authors present GoLLIE, a Code-LLaMA-based model fine-tuned on a small set of established IE datasets so that it learns to actually use guidelines. Every task is written as Python code:
- labels become dataclasses,
- guideline definitions become docstrings,
- typical candidate spans appear as comments.

The model then generates a list of class instances, which can be executed directly to get structured output. Regularization during training pushes the model to rely on the guideline text.

On unseen schemas, GoLLIE beats earlier zero-shot IE methods, including the previous SoTA. Ablations show that detailed guidelines drive much of the gain. This suggests a new domain could be handled by writing guidelines rather than annotating a corpus.

## Key points
- **Tasks covered:** a single code-style format handles NER, event extraction (EE), EAE, RE and SF through per-task templates. Adding a dataset only requires a label list and guidelines, usually reused from the original dataset authors.
- **Guideline preparation:** TACRED's lengthy guidelines were condensed by hand. Where no guidelines were published, experts derived them from development-split examples.
- **Motivating baseline:** gpt-3.5-turbo prompted with guidelines reaches only about 20 F1 on the Music and Politics domains of out-of-domain zero-shot NER.
- **Limitations:**
  - **Code overhead:** the share of input characters spent on code structure is 0.43 for the terse OntoNotes 5 Person definition but only 0.08 for the detailed HarveyNER Point guideline.
  - **Label count:** experiments use datasets with fewer than 20 labels. Schemas with hundreds of labels, such as FIGER, may exceed the context window; splitting labels across several prompts is proposed as a workaround.
- **Data contamination and release:** contamination is addressed by comparing GoLLIE against a no-guideline Baseline that shares the same backbone. Code, data and models are released at github.com/hitz-zentroa/GoLLIE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

97 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 94 not yet.

- [Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models](https://md.research.elamur.ai/free/en/papers/min-2024-synergetic-event-understanding-collaborative-approach/)
- [Contrastively Pre-trained Event Embeddings with Schema-free LLM Annotations](https://md.research.elamur.ai/free/en/papers/mtumbuka-2026-contrastively-pre-trained-event-embeddings/)
- [GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer](https://md.research.elamur.ai/free/en/papers/zaratiana-2024-gliner-generalist-model-named-entity/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sainz2023gollie,
  title = {{GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction}},
  author = {Sainz, Oscar and Garcia-Ferrero, Iker and Agerri, Rodrigo and Lacalle, Oier Lopez de and Rigau, German and Agirre, Eneko},
  year = {2023},
  doi = {10.48550/arxiv.2310.03668},
  eprint = {2310.03668},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.03668}
}
```

- [← previous GlotLID: Language Identification for Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/kargaran-2023-glotlid-language-identification-low-resource/)
- [next → Gorilla: Large Language Model Connected with Massive APIs](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

84 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregress… [find](https://scholar.google.com/scholar?q=Sidney+Black%2C+Stella+Biderman%2C+Eric+Hallahan%2C+Quentin+Anthony%2C+Leo+Gao%2C+Laurence+Golding%2C+Horace+He%2C+Connor+Leahy%2C+Kyle+McDonell%2C+Jason+Phang%2C+Michael+Pieler%2C+Usvsn+Sai+Prashanth%2C+Shivanshu+Purohit%2C+L) [our card](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
2. Terra Blevins, Hila Gonen, and Luke Zettlemoyer. Prompting language models for linguistic structure. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, C… [find](https://scholar.google.com/scholar?q=Terra+Blevins%2C+Hila+Gonen%2C+and+Luke+Zettlemoyer%2E+Prompting+language+models+for+linguistic+structure%2E+In+Anna+Rogers%2C+Jordan+L%2E+Boyd%2DGraber%2C+and+Naoaki+Okazaki+%28eds%2E%29%2C+Proceedings+of+the+61st+Annual+Me)
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
4. Pei Chen, Haotian Xu, Cheng Zhang, and Ruihong Huang. Crossroads, buildings and neighborhoods: A dataset for fine-grained location recognition. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the… [find](https://scholar.google.com/scholar?q=Pei+Chen%2C+Haotian+Xu%2C+Cheng+Zhang%2C+and+Ruihong+Huang%2E+Crossroads%2C+buildings+and+neighborhoods%3A+A+dataset+for+fine%2Dgrained+location+recognition%2E+In+Marine+Carpuat%2C+Marie%2DCatherine+de+Marneffe%2C+and+Ivan)
5. Yi Chen, Haiyun Jiang, Lemao Liu, Shuming Shi, Chuang Fan, Min Yang, and Ruifeng Xu. An empirical study on multiple information sources for zero-shot fine-grained entity typing. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on… [find](https://scholar.google.com/scholar?q=Yi+Chen%2C+Haiyun+Jiang%2C+Lemao+Liu%2C+Shuming+Shi%2C+Chuang+Fan%2C+Min+Yang%2C+and+Ruifeng+Xu%2E+An+empirical+study+on+multiple+information+sources+for+zero%2Dshot+fine%2Dgrained+entity+typing%2E+In+Marie%2DFrancine+Moen)
6. Zekai Chen, Mariann Micsinai Balan, and Kevin Brown. Language models are few-shot learners for prognostic prediction. CoRR, abs/2302.12692, 2023. doi: 10.48550/arXiv.2302.12692. URL https://doi.org/10.48550/arXiv.2302.12692. — [doi:10.48550/arxiv.2302.12692](https://doi.org/10.48550/arxiv.2302.12692) [↗](https://doi.org/10.48550/arXiv.2302.12692)
7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+Parker+Schuh%2C+Kensen+Shi%2C+Sasha+Tsvyashc)
8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+Xiny)
9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Unsupervised+cross%2Dlingual+re)
10. Leon Derczynski, Kalina Bontcheva, and Ian Roberts. Broad Twitter corpus: A diverse named entity recognition resource. In Yuji Matsumoto and Rashmi Prasad (eds.), Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 1169–1179, Osaka, Japan… [find](https://scholar.google.com/scholar?q=Leon+Derczynski%2C+Kalina+Bontcheva%2C+and+Ian+Roberts%2E+Broad+Twitter+corpus%3A+A+diverse+named+entity+recognition+resource%2E+In+Yuji+Matsumoto+and+Rashmi+Prasad+%28eds%2E%29%2C+Proceedings+of+COLING+2016%2C+the+26th+)
11. The COLING 2016 Organizing Committee. URL https://aclanthology.org/C16-1111. [↗](https://aclanthology.org/C16-1111)
12. Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 140–147, Copenhagen, D… [find](https://scholar.google.com/scholar?q=Leon+Derczynski%2C+Eric+Nichols%2C+Marieke+van+Erp%2C+and+Nut+Limsopatham%2E+Results+of+the+WNUT2017+shared+task+on+novel+and+emerging+entity+recognition%2E+In+Leon+Derczynski%2C+Wei+Xu%2C+Alan+Ritter%2C+and+Tim+Bald) [our card](https://md.research.elamur.ai/free/en/papers/derczynski-2017-results-wnut2017-shared-task-novel/)
13. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. CoRR, abs/2305.14314, 2023. doi: 10.48550/arXiv.2305.14314. URL https://doi.org/10.48550/arXiv.2305.14314. [↗](https://doi.org/10.48550/arXiv.2305.14314)
14. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), P… [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Maarten+Sap%2C+Ana+Marasovic%CC%81%2C+William+Agnew%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+clean+cr)
15. Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. Multi-sentence argument linking. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8057–8077, Online, July… [find](https://scholar.google.com/scholar?q=Seth+Ebner%2C+Patrick+Xia%2C+Ryan+Culkin%2C+Kyle+Rawlins%2C+and+Benjamin+Van+Durme%2E+Multi%2Dsentence+argument+linking%2E+In+Dan+Jurafsky%2C+Joyce+Chai%2C+Natalie+Schluter%2C+and+Joel+Tetreault+%28eds%2E%29%2C+Proceedings+of+th) [our card](https://md.research.elamur.ai/free/en/papers/ebner-2020-multi-sentence-argument-linking/)
16. Hermenegildo Fabregat, Juan Martı́nez-Romo, and Lourdes Araujo. Overview of the DIANN task: Disability annotation task. In Paolo Rosso, Julio Gonzalo, Raquel Martı́nez, Soto Montalvo, and Jorge Carrillo de Albornoz (eds.), Proceedings of the Third Workshop on Evaluation of Human Language Technologie… [find](https://scholar.google.com/scholar?q=Hermenegildo+Fabregat%2C+Juan+Mart%C4%B1%CC%81nez%2DRomo%2C+and+Lourdes+Araujo%2E+Overview+of+the+DIANN+task%3A+Disability+annotation+task%2E+In+Paolo+Rosso%2C+Julio+Gonzalo%2C+Raquel+Mart%C4%B1%CC%81nez%2C+Soto+Montalvo%2C+and+Jorge+Carril)
17. CEUR-WS.org, 2018. URL [https://ceur-ws.org/Vol-2150/overview-diann-task.pdf](https://ceur-ws.org/Vol-2150/overview-diann-task.pdf). [↗](https://ceur-ws.org/Vol-2150/overview-diann-task.pdf)
18. Besnik Fetahu, Sudipta Kar, Zhiyu Chen, Oleg Rokhlenko, and Shervin Malmasi. Semeval-2023 task 2: Fine-grained multilingual named entity recognition (multiconer 2). In Atul Kr. Ojha, A. Seza Dogruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, and Elisa Sartori (eds.), Proceedin… [find](https://scholar.google.com/scholar?q=Besnik+Fetahu%2C+Sudipta+Kar%2C+Zhiyu+Chen%2C+Oleg+Rokhlenko%2C+and+Shervin+Malmasi%2E+Semeval%2D2023+task+2%3A+Fine%2Dgrained+multilingual+named+entity+recognition+%28multiconer+2%29%2E+In+Atul+Kr%2E+Ojha%2C+A%2E+Seza+Dogruo%CC%88z%2C)
19. Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. [find](https://scholar.google.com/scholar?q=Pengcheng+He%2C+Jianfeng+Gao%2C+and+Weizhu+Chen%2E+Debertav3%3A+Improving+deberta+using+electra%2Dstyle+pre%2Dtraining+with+gradient%2Ddisentangled+embedding+sharing%2E+In+The+Eleventh+International+Conference+on+Lea) [our card](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/)
20. OpenReview.net, 2023. URL [https://openreview.net/pdf?id=sE7-XhLxHA](https://openreview.net/pdf?id=sE7-XhLxHA). [↗](https://openreview.net/pdf?id=sE7-XhLxHA)
21. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. [find](https://scholar.google.com/scholar?q=Edward+J%2E+Hu%2C+Yelong+Shen%2C+Phillip+Wallis%2C+Zeyuan+Allen%2DZhu%2C+Yuanzhi+Li%2C+Shean+Wang%2C+Lu+Wang%2C+and+Weizhu+Chen%2E+Lora%3A+Low%2Drank+adaptation+of+large+language+models%2E+In+The+Tenth+International+Conference)
22. OpenReview.net, 2022. URL [https://openreview.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9). [↗](https://openreview.net/forum?id=nZeVKeeFYf9)
23. Rezarta Islamaj Doğan and Zhiyong Lu. An improved corpus of disease mentions in PubMed citations. In Kevin B. Cohen, Dina Demner-Fushman, Sophia Ananiadou, Bonnie Webber, Jun’ichi Tsujii, and John Pestian (eds.), BioNLP: Proceedings of the 2012 Workshop on Biomedical Natural Language Processing, pp… [find](https://scholar.google.com/scholar?q=Rezarta+Islamaj+Dog%CC%86an+and+Zhiyong+Lu%2E+An+improved+corpus+of+disease+mentions+in+PubMed+citations%2E+In+Kevin+B%2E+Cohen%2C+Dina+Demner%2DFushman%2C+Sophia+Ananiadou%2C+Bonnie+Webber%2C+Jun%E2%80%99ichi+Tsujii%2C+and+John+Pe)
24. Veysel Kocaman and David Talby. Biomedical named entity recognition at scale. In Alberto Del Bimbo, Rita Cucchiara, Stan Sclaroff, Giovanni Maria Farinella, Tao Mei, Marco Bertini, Hugo Jair Escalante, and Roberto Vezzani (eds.), Pattern Recognition. ICPR International Workshops and Challenges, pp.… [find](https://scholar.google.com/scholar?q=Veysel+Kocaman+and+David+Talby%2E+Biomedical+named+entity+recognition+at+scale%2E+In+Alberto+Del+Bimbo%2C+Rita+Cucchiara%2C+Stan+Sclaroff%2C+Giovanni+Maria+Farinella%2C+Tao+Mei%2C+Marco+Bertini%2C+Hugo+Jair+Escalante)
25. Aman Kumar and Binil Starly. ”fabner”: information extraction from manufacturing process science domain literature using named entity recognition. J. Intell. Manuf., 33(8):2393– 2407, 2022. doi: 10.1007/s10845-021-01807-x. URL https://doi.org/10.1007/ s10845-021-01807-x. — [doi:10.1007/s10845-021-01807-x](https://doi.org/10.1007/s10845-021-01807-x) [find](https://scholar.google.com/scholar?q=Aman+Kumar+and+Binil+Starly%2E+%E2%80%9Dfabner%E2%80%9D%3A+information+extraction+from+manufacturing+process+science+domain+literature+using+named+entity+recognition%2E+J%2E+Intell%2E+Manuf%2E%2C+33%288%29%3A2393%E2%80%93+2407%2C+2022%2E+doi%3A+10%2E10)
26. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Roger Levy and Lucia Specia (eds.), Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. 333–342, Vancouver, Canada, August 2017. Associat… [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Minjoon+Seo%2C+Eunsol+Choi%2C+and+Luke+Zettlemoyer%2E+Zero%2Dshot+relation+extraction+via+reading+comprehension%2E+In+Roger+Levy+and+Lucia+Specia+%28eds%2E%29%2C+Proceedings+of+the+21st+Conference+on+Computa)
27. Peng Li, Tianxiang Sun, Qiong Tang, Hang Yan, Yuanbin Wu, Xuanjing Huang, and Xipeng Qiu. CodeIE: Large code generation models are better few-shot information extractors. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Comp… [find](https://scholar.google.com/scholar?q=Peng+Li%2C+Tianxiang+Sun%2C+Qiong+Tang%2C+Hang+Yan%2C+Yuanbin+Wu%2C+Xuanjing+Huang%2C+and+Xipeng+Qiu%2E+CodeIE%3A+Large+code+generation+models+are+better+few%2Dshot+information+extractors%2E+In+Anna+Rogers%2C+Jordan+Boyd%2DG)
28. Sha Li, Heng Ji, and Jiawei Han. Document-level event argument extraction by conditional generation. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference… [find](https://scholar.google.com/scholar?q=Sha+Li%2C+Heng+Ji%2C+and+Jiawei+Han%2E+Document%2Dlevel+event+argument+extraction+by+conditional+generation%2E+In+Kristina+Toutanova%2C+Anna+Rumshisky%2C+Luke+Zettlemoyer%2C+Dilek+Hakkani%2DTur%2C+Iz+Beltagy%2C+Steven+Beth) [our card](https://md.research.elamur.ai/free/en/papers/li-2021-document-level-event-argument-extraction/)
29. Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. A joint neural model for information extraction with global features. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7999–8009, Online… [find](https://scholar.google.com/scholar?q=Ying+Lin%2C+Heng+Ji%2C+Fei+Huang%2C+and+Lingfei+Wu%2E+A+joint+neural+model+for+information+extraction+with+global+features%2E+In+Dan+Jurafsky%2C+Joyce+Chai%2C+Natalie+Schluter%2C+and+Joel+Tetreault+%28eds%2E%29%2C+Proceeding) [our card](https://md.research.elamur.ai/free/en/papers/lin-2020-joint-neural-model-information-extraction/)
30. Xiao Ling and Daniel S. Weld. Fine-grained entity recognition. In Jörg Hoffmann and Bart Selman (eds.), Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, July 22-26, 2012, Toronto, Ontario, Canada, pp. 94–100. [find](https://scholar.google.com/scholar?q=Xiao+Ling+and+Daniel+S%2E+Weld%2E+Fine%2Dgrained+entity+recognition%2E+In+Jo%CC%88rg+Hoffmann+and+Bart+Selman+%28eds%2E%29%2C+Proceedings+of+the+Twenty%2DSixth+AAAI+Conference+on+Artificial+Intelligence%2C+July+22%2D26%2C+2012%2C+T)
31. AAAI Press, 2012. doi: 10.1609/AAAI.V26I1. 8122. URL https://doi.org/10.1609/aaai.v26i1.8122. — [doi:10.1609/aaai.v26i1](https://doi.org/10.1609/aaai.v26i1) [↗](https://doi.org/10.1609/aaai.v26i1.8122)
32. Jingjing Liu, Panupong Pasupat, Scott Cyphers, and James R. Glass. Asgard: A portable architecture for multilingual dialogue systems. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May 26-31, 2013, pp. 8386–8390. [find](https://scholar.google.com/scholar?q=Jingjing+Liu%2C+Panupong+Pasupat%2C+Scott+Cyphers%2C+and+James+R%2E+Glass%2E+Asgard%3A+A+portable+architecture+for+multilingual+dialogue+systems%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+P)
33. IEEE, 2013. doi: 10.1109/ICASSP.2013.6639301. URL https://doi.org/10.1109/ ICASSP.2013.6639301. — [doi:10.1109/icassp.2013.6639301](https://doi.org/10.1109/icassp.2013.6639301) [find](https://scholar.google.com/scholar?q=IEEE%2C+2013%2E+doi%3A+10%2E1109%2FICASSP%2E2013%2E6639301%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F10%2E1109%2F+ICASSP%2E2013%2E6639301%2E+%E2%80%94+%5Bdoi%3A10%2E1109%2Ficassp%2E2013%2E6639301%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1109%2Ficassp%2E2013%2E6639301%29)
34. Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. Crossner: Evaluating cross-domain named entity recognition. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artif… [find](https://scholar.google.com/scholar?q=Zihan+Liu%2C+Yan+Xu%2C+Tiezheng+Yu%2C+Wenliang+Dai%2C+Ziwei+Ji%2C+Samuel+Cahyawijaya%2C+Andrea+Madotto%2C+and+Pascale+Fung%2E+Crossner%3A+Evaluating+cross%2Ddomain+named+entity+recognition%2E+In+Thirty%2DFifth+AAAI+Conferenc)
35. AAAI Press, 2021a. URL [https://ojs.aaai.org/index.php/AAAI/](https://ojs.aaai.org/index.php/AAAI/) article/view/17587. [↗](https://ojs.aaai.org/index.php/AAAI/)
36. Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. Crossner: Evaluating cross-domain named entity recognition. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artif… [find](https://scholar.google.com/scholar?q=Zihan+Liu%2C+Yan+Xu%2C+Tiezheng+Yu%2C+Wenliang+Dai%2C+Ziwei+Ji%2C+Samuel+Cahyawijaya%2C+Andrea+Madotto%2C+and+Pascale+Fung%2E+Crossner%3A+Evaluating+cross%2Ddomain+named+entity+recognition%2E+In+Thirty%2DFifth+AAAI+Conferenc)
37. AAAI Press, 2021b. doi: 10.1609/aaai.v35i15.17587. URL https://doi.org/10.1609/ aaai.v35i15.17587. — [doi:10.1609/aaai.v35i15.17587](https://doi.org/10.1609/aaai.v35i15.17587) [find](https://scholar.google.com/scholar?q=AAAI+Press%2C+2021b%2E+doi%3A+10%2E1609%2Faaai%2Ev35i15%2E17587%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F10%2E1609%2F+aaai%2Ev35i15%2E17587%2E+%E2%80%94+%5Bdoi%3A10%2E1609%2Faaai%2Ev35i15%2E17587%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1609%2Faaai%2Ev35i15%2E17587%29)
38. Jie Lou, Yaojie Lu, Dai Dai, Wei Jia, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. Universal information extraction as unified semantic matching. In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference… [find](https://scholar.google.com/scholar?q=Jie+Lou%2C+Yaojie+Lu%2C+Dai+Dai%2C+Wei+Jia%2C+Hongyu+Lin%2C+Xianpei+Han%2C+Le+Sun%2C+and+Hua+Wu%2E+Universal+information+extraction+as+unified+semantic+matching%2E+In+Brian+Williams%2C+Yiling+Chen%2C+and+Jennifer+Neville+%28)
39. AAAI Press, 2023. doi: 10.1609/aaai.v37i11.26563. URL https://doi.org/10.1609/aaai.v37i11.26563. — [doi:10.1609/aaai.v37i11.26563](https://doi.org/10.1609/aaai.v37i11.26563) [↗](https://doi.org/10.1609/aaai.v37i11.26563)
40. Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. Unified structure generation for universal information extraction. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational L… [find](https://scholar.google.com/scholar?q=Yaojie+Lu%2C+Qing+Liu%2C+Dai+Dai%2C+Xinyan+Xiao%2C+Hongyu+Lin%2C+Xianpei+Han%2C+Le+Sun%2C+and+Hua+Wu%2E+Unified+structure+generation+for+universal+information+extraction%2E+In+Smaranda+Muresan%2C+Preslav+Nakov%2C+and+Aline)
41. Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. Unified structure generation for universal information extraction. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational L… [find](https://scholar.google.com/scholar?q=Yaojie+Lu%2C+Qing+Liu%2C+Dai+Dai%2C+Xinyan+Xiao%2C+Hongyu+Lin%2C+Xianpei+Han%2C+Le+Sun%2C+and+Hua+Wu%2E+Unified+structure+generation+for+universal+information+extraction%2E+In+Smaranda+Muresan%2C+Preslav+Nakov%2C+and+Aline)
42. Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 157–165, Dublin, Ireland, May… [find](https://scholar.google.com/scholar?q=Inbal+Magar+and+Roy+Schwartz%2E+Data+contamination%3A+From+memorization+to+exploitation%2E+In+Smaranda+Muresan%2C+Preslav+Nakov%2C+and+Aline+Villavicencio+%28eds%2E%29%2C+Proceedings+of+the+60th+Annual+Meeting+of+the+A)
43. Bernardo Magnini, Begoña Altuna, Alberto Lavelli, Manuela Speranza, and Roberto Zanoli. The E3C project: European clinical case corpus. In Jon Alkorta, Itziar Gonzalez-Dios, Aitziber Atutxa, Koldo Gojenola, Eugenio Martı́nez-Cámara, Álvaro Rodrigo, and Paloma Martı́nez (eds.), Proceedings of the… [find](https://scholar.google.com/scholar?q=Bernardo+Magnini%2C+Begon%CC%83a+Altuna%2C+Alberto+Lavelli%2C+Manuela+Speranza%2C+and+Roberto+Zanoli%2E+The+E3C+project%3A+European+clinical+case+corpus%2E+In+Jon+Alkorta%2C+Itziar+Gonzalez%2DDios%2C+Aitziber+Atutxa%2C+Koldo+Go)
44. CEUR-WS.org, 2021. URL [https://ceur-ws.org/Vol-2968/paper5.pdf](https://ceur-ws.org/Vol-2968/paper5.pdf). [↗](https://ceur-ws.org/Vol-2968/paper5.pdf)
45. Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Comput. Surv., 56(2), sep 2023. ISSN 0360-0300. doi: 10.1145/36059… [find](https://scholar.google.com/scholar?q=Bonan+Min%2C+Hayley+Ross%2C+Elior+Sulem%2C+Amir+Pouran+Ben+Veyseh%2C+Thien+Huu+Nguyen%2C+Oscar+Sainz%2C+Eneko+Agirre%2C+Ilana+Heintz%2C+and+Dan+Roth%2E+Recent+advances+in+natural+language+processing+via+large+pre%2Dtrain) [our card](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
46. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel… [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Thomas+Wang%2C+Lintang+Sutawika%2C+Adam+Roberts%2C+Stella+Biderman%2C+Teven+Le+Scao%2C+M%2E+Saiful+Bari%2C+Sheng+Shen%2C+Zheng+Xin+Yong%2C+Hailey+Schoelkopf%2C+Xiangru+Tang%2C+Dragomir+Radev%2C+Alham+Fikr) [our card](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
47. Rasha Obeidat, Xiaoli Fern, Hamed Shahbazi, and Prasad Tadepalli. Description-based zero-shot fine-grained entity typing. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hum… [find](https://scholar.google.com/scholar?q=Rasha+Obeidat%2C+Xiaoli+Fern%2C+Hamed+Shahbazi%2C+and+Prasad+Tadepalli%2E+Description%2Dbased+zero%2Dshot+fine%2Dgrained+entity+typing%2E+In+Jill+Burstein%2C+Christy+Doran%2C+and+Thamar+Solorio+%28eds%2E%29%2C+Proceedings+of+the)
48. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. CoRR, abs/2306.01116, 2023. doi… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+The+refinedweb+dataset+for+falcon+)
49. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. Towards robust linguistic analysis using OntoNotes. In Julia Hockenmaier and Sebastian Riedel (eds.), Proceedings of the Seventeenth Conference on Computational Natural Lan… [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Hwee+Tou+Ng%2C+Anders+Bjo%CC%88rkelund%2C+Olga+Uryupina%2C+Yuchen+Zhang%2C+and+Zhi+Zhong%2E+Towards+robust+linguistic+analysis+using+OntoNotes%2E+In+Julia+Hockenmaier)
50. Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+The+Tenth+International+Conference+on+Learning+Representations%2C+I)
51. OpenReview.net, 2022. URL [https://openreview.net/forum?id=R8sQPpGCv0](https://openreview.net/forum?id=R8sQPpGCv0). [↗](https://openreview.net/forum?id=R8sQPpGCv0)
52. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
53. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020. URL [http://jmlr.org/](http://jmlr.org/) papers/v21/20-074.html. [↗](http://jmlr.org/)
54. Lev Ratinov and Dan Roth. Design challenges and misconceptions in named entity recognition. In Suzanne Stevenson and Xavier Carreras (eds.), Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pp. 147–155, Boulder, Colorado, June 2009. Association for Co… [find](https://scholar.google.com/scholar?q=Lev+Ratinov+and+Dan+Roth%2E+Design+challenges+and+misconceptions+in+named+entity+recognition%2E+In+Suzanne+Stevenson+and+Xavier+Carreras+%28eds%2E%29%2C+Proceedings+of+the+Thirteenth+Conference+on+Computational+N) [our card](https://md.research.elamur.ai/free/en/papers/ratinov-2009-design-challenges-misconceptions-named-entity/)
55. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Fai… [find](https://scholar.google.com/scholar?q=Baptiste+Rozie%CC%80re%2C+Jonas+Gehring%2C+Fabian+Gloeckle%2C+Sten+Sootla%2C+Itai+Gat%2C+Xiaoqing+Ellen+Tan%2C+Yossi+Adi%2C+Jingyu+Liu%2C+Tal+Remez%2C+Je%CC%81re%CC%81my+Rapin%2C+Artyom+Kozhevnikov%2C+Ivan+Evtimov%2C+Joanna+Bitton%2C+Manish+) [our card](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
56. Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, and Eneko Agirre. Label verbalization and entailment for effective zero and few-shot relation extraction. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empiri… [find](https://scholar.google.com/scholar?q=Oscar+Sainz%2C+Oier+Lopez+de+Lacalle%2C+Gorka+Labaka%2C+Ander+Barrena%2C+and+Eneko+Agirre%2E+Label+verbalization+and+entailment+for+effective+zero+and+few%2Dshot+relation+extraction%2E+In+Marie%2DFrancine+Moens%2C+Xuan)
57. Oscar Sainz, Itziar Gonzalez-Dios, Oier Lopez de Lacalle, Bonan Min, and Eneko Agirre. Textual entailment for event argument extraction: Zero- and few-shot with multi-source learning. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for… [find](https://scholar.google.com/scholar?q=Oscar+Sainz%2C+Itziar+Gonzalez%2DDios%2C+Oier+Lopez+de+Lacalle%2C+Bonan+Min%2C+and+Eneko+Agirre%2E+Textual+entailment+for+event+argument+extraction%3A+Zero%2D+and+few%2Dshot+with+multi%2Dsource+learning%2E+In+Marine+Carpua)
58. Oscar Sainz, Haoling Qiu, Oier Lopez de Lacalle, Eneko Agirre, and Bonan Min. ZS4IE: A toolkit for zero-shot information extraction with simple verbalizations. In Hannaneh Hajishirzi, Qiang Ning, and Avi Sil (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association… [find](https://scholar.google.com/scholar?q=Oscar+Sainz%2C+Haoling+Qiu%2C+Oier+Lopez+de+Lacalle%2C+Eneko+Agirre%2C+and+Bonan+Min%2E+ZS4IE%3A+A+toolkit+for+zero%2Dshot+information+extraction+with+simple+verbalizations%2E+In+Hannaneh+Hajishirzi%2C+Qiang+Ning%2C+and+)
59. Oscar Sainz, Jon Campos, Iker Garcı́a-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Oscar+Sainz%2C+Jon+Campos%2C+Iker+Garc%C4%B1%CC%81a%2DFerrero%2C+Julen+Etxaniz%2C+Oier+Lopez+de+Lacalle%2C+and+Eneko+Agirre%2E+NLP+evaluation+in+trouble%3A+On+the+need+to+measure+LLM+data+contamination+for+each+benchmark%2E+In+H) [our card](https://md.research.elamur.ai/free/en/papers/sainz-2023-nlp-evaluation-trouble-need-measure/)
60. Oscar Sainz, Jon Ander Campos, Iker Garcı́a-Ferrero, Julen Etxaniz, and Eneko Agirre. Did chatgpt cheat on your test?, Jun 2023b. URL [https://hitz-zentroa.github.io/](https://hitz-zentroa.github.io/) lm-contamination/blog/. [↗](https://hitz-zentroa.github.io/)
61. Taneeya Satyapanich, Francis Ferraro, and Tim Finin. Casie: Extracting cybersecurity event information from text. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8749– 8757, Apr. 2020. doi: 10.1609/aaai.v34i05.6401. URL [https://ojs.aaai.org/index](https://ojs.aaai.org/index). php/AAAI/article/view/6401. — [doi:10.1609/aaai.v34i05.6401](https://doi.org/10.1609/aaai.v34i05.6401) [↗](https://ojs.aaai.org/index)
62. Teven Le Scao and Alexander M. Rush. How many data points is a prompt worth? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American C… [find](https://scholar.google.com/scholar?q=Teven+Le+Scao+and+Alexander+M%2E+Rush%2E+How+many+data+points+is+a+prompt+worth%3F+In+Kristina+Toutanova%2C+Anna+Rumshisky%2C+Luke+Zettlemoyer%2C+Dilek+Hakkani%2DTu%CC%88r%2C+Iz+Beltagy%2C+Steven+Bethard%2C+Ryan+Cotterell%2C+Ta) [our card](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/)
63. Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main V… [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Schu%CC%88tze%2E+Exploiting+cloze%2Dquestions+for+few%2Dshot+text+classification+and+natural+language+inference%2E+In+Paola+Merlo%2C+Jo%CC%88rg+Tiedemann%2C+and+Reut+Tsarfaty+%28eds%2E%29%2C+Proceedings+of+) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)
64. Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Schärli, Aakanksha Chowdhery, Philip Andrew Mansfield, Blaise Agüera y Arc… [find](https://scholar.google.com/scholar?q=Karan+Singhal%2C+Shekoofeh+Azizi%2C+Tao+Tu%2C+S%2E+Sara+Mahdavi%2C+Jason+Wei%2C+Hyung+Won+Chung%2C+Nathan+Scales%2C+Ajay+Kumar+Tanwani%2C+Heather+Cole%2DLewis%2C+Stephen+Pfohl%2C+Perry+Payne%2C+Martin+Seneviratne%2C+Paul+Gamble%2C) [our card](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)
65. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, pp. 1–9, 2023. [find](https://scholar.google.com/scholar?q=Karan+Singhal%2C+Shekoofeh+Azizi%2C+Tao+Tu%2C+S+Sara+Mahdavi%2C+Jason+Wei%2C+Hyung+Won+Chung%2C+Nathan+Scales%2C+Ajay+Tanwani%2C+Heather+Cole%2DLewis%2C+Stephen+Pfohl%2C+et+al%2E+Large+language+models+encode+clinical+knowled) [our card](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)
66. Zeqi Tan, Shen Huang, Zixia Jia, Jiong Cai, Yinghui Li, Weiming Lu, Yueting Zhuang, Kewei Tu, Pengjun Xie, and Fei Huang. DAMO-NLP at semeval-2023 task 2: A unified retrieval-augmented system for multilingual named entity recognition. In Atul Kr. Ojha, A. Seza Dogruöz, Giovanni Da San Martino, Hari… [find](https://scholar.google.com/scholar?q=Zeqi+Tan%2C+Shen+Huang%2C+Zixia+Jia%2C+Jiong+Cai%2C+Yinghui+Li%2C+Weiming+Lu%2C+Yueting+Zhuang%2C+Kewei+Tu%2C+Pengjun+Xie%2C+and+Fei+Huang%2E+DAMO%2DNLP+at+semeval%2D2023+task+2%3A+A+unified+retrieval%2Daugmented+system+for+mult)
67. Simone Tedeschi and Roberto Navigli. MultiNERD: A multilingual, multi-genre and fine-grained dataset for named entity recognition (and disambiguation). In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACL… [find](https://scholar.google.com/scholar?q=Simone+Tedeschi+and+Roberto+Navigli%2E+MultiNERD%3A+A+multilingual%2C+multi%2Dgenre+and+fine%2Dgrained+dataset+for+named+entity+recognition+%28and+disambiguation%29%2E+In+Marine+Carpuat%2C+Marie%2DCatherine+de+Marneffe%2C+)
68. Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142–147, 2003. URL https://aclanthology.org/W03-0419. [↗](https://aclanthology.org/W03-0419)
69. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timothe%CC%81e+Lacroix%2C+Baptiste+Rozie%CC%80re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+Aure%CC%81lien+Rodriguez%2C+Armand+Joulin%2C+Ed)
70. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton%2DFe)
71. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57:45, 2006. URL [https://catalog.ldc.upenn.edu/LDC2006T06](https://catalog.ldc.upenn.edu/LDC2006T06). [↗](https://catalog.ldc.upenn.edu/LDC2006T06)
72. Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021. [find](https://scholar.google.com/scholar?q=Ben+Wang+and+Aran+Komatsuzaki%2E+Gpt%2Dj%2D6b%3A+A+6+billion+parameter+autoregressive+language+model%2C+2021%2E)
73. Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jihua Kang, Jingsheng Yang, Siyuan Li, and Chunsai Du. Instructuie: Multi-task instruction tuning for unified information extraction. CoRR, abs/2304.08085, 2023a. doi: 10.48550/arXiv.2304.0… [find](https://scholar.google.com/scholar?q=Xiao+Wang%2C+Weikang+Zhou%2C+Can+Zu%2C+Han+Xia%2C+Tianze+Chen%2C+Yuansen+Zhang%2C+Rui+Zheng%2C+Junjie+Ye%2C+Qi+Zhang%2C+Tao+Gui%2C+Jihua+Kang%2C+Jingsheng+Yang%2C+Siyuan+Li%2C+and+Chunsai+Du%2E+Instructuie%3A+Multi%2Dtask+instructio) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-instructuie-multi-task-instruction-tuning/)
74. Xingyao Wang, Sha Li, and Heng Ji. Code4Struct: Code generation for few-shot event structure prediction. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3640–3663, Toro… [find](https://scholar.google.com/scholar?q=Xingyao+Wang%2C+Sha+Li%2C+and+Heng+Ji%2E+Code4Struct%3A+Code+generation+for+few%2Dshot+event+structure+prediction%2E+In+Anna+Rogers%2C+Jordan+Boyd%2DGraber%2C+and+Naoaki+Okazaki+%28eds%2E%29%2C+Proceedings+of+the+61st+Annual+M)
75. Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. Improving named entity recognition by external context retrieving and cooperative learning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Assoc… [find](https://scholar.google.com/scholar?q=Xinyu+Wang%2C+Yong+Jiang%2C+Nguyen+Bach%2C+Tao+Wang%2C+Zhongqiang+Huang%2C+Fei+Huang%2C+and+Kewei+Tu%2E+Improving+named+entity+recognition+by+external+context+retrieving+and+cooperative+learning%2E+In+Chengqing+Zong%2C)
76. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Ku… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Atharva+Naik%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Anjana+Arunkumar%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karaman)
77. Chih-Hsuan Wei, Yifan Peng, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Jiao Li, Thomas C Wiegers, and Zhiyong Lu. Assessing the state of the art in biomedical relation extraction: overview of the biocreative v chemical-disease relation (cdr) task. Database, 2016, 2016. [find](https://scholar.google.com/scholar?q=Chih%2DHsuan+Wei%2C+Yifan+Peng%2C+Robert+Leaman%2C+Allan+Peter+Davis%2C+Carolyn+J+Mattingly%2C+Jiao+Li%2C+Thomas+C+Wiegers%2C+and+Zhiyong+Lu%2E+Assessing+the+state+of+the+art+in+biomedical+relation+extraction%3A+overview)
78. Renzo M. Rivera Zabala, Paloma Martinez, and Isabel Segura-Bedmar. A hybrid bi-lstm-crf model to recognition of disabilities from biomedical texts. In Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages, 2018. URL [https://ceur-ws.org/Vol-2150/DIANN_p…](https://ceur-ws.org/Vol-2150/DIANN_p…) [↗](https://ceur-ws.org/Vol-2150/DIANN_p…)
79. Mozhi Zhang, Hang Yan, Yaqian Zhou, and Xipeng Qiu. Promptner: A prompting method for few-shot named entity recognition via k nearest neighbor search. CoRR, abs/2305.12217, 2023a. doi: 10.48550/arXiv.2305.12217. URL https://doi.org/10.48550/arXiv.2305.12217. — [doi:10.48550/arxiv.2305.12217](https://doi.org/10.48550/arxiv.2305.12217) [↗](https://doi.org/10.48550/arXiv.2305.12217)
80. Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon. Optimizing bi-encoder for named entity recognition via contrastive learning. In The Eleventh International Conference on Learning Representations, 2023b. URL [https://openreview.net/forum?id=9EAQVEINuum](https://openreview.net/forum?id=9EAQVEINuum). [↗](https://openreview.net/forum?id=9EAQVEINuum)
81. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: open pre-trained… [find](https://scholar.google.com/scholar?q=Susan+Zhang%2C+Stephen+Roller%2C+Naman+Goyal%2C+Mikel+Artetxe%2C+Moya+Chen%2C+Shuohui+Chen%2C+Christopher+Dewan%2C+Mona+T%2E+Diab%2C+Xian+Li%2C+Xi+Victoria+Lin%2C+Todor+Mihaylov%2C+Myle+Ott%2C+Sam+Shleifer%2C+Kurt+Shuster%2C+Danie)
82. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position-aware attention and supervised data improve slot filling. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp.… [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Victor+Zhong%2C+Danqi+Chen%2C+Gabor+Angeli%2C+and+Christopher+D%2E+Manning%2E+Position%2Daware+attention+and+supervised+data+improve+slot+filling%2E+In+Martha+Palmer%2C+Rebecca+Hwa%2C+and+Sebastian+Riedel+)
83. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. CoRR, abs/2306.05685, 2023. doi: 10.48550/arXiv.2306.05685. URL… [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+P%2E+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+Judging+llm%2Das%2Da%2Djud)
84. Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. Universalner: Targeted distillation from large language models for open named entity recognition. CoRR, abs/2308.03279, 2023. doi: 10.48550/arXiv.2308.03279. URL https://doi.org/10.48550/arXiv. 2308.03279. [↗](https://doi.org/10.48550/arXiv)