---
title: "A Herd of Language Models Makes a Better Zero-shot Annotator"
description: "To handle this trade-off, the authors propose MARY, a label model built on an encoder such as RoBERTa-base. 41 references."
authors: ["Seiji Shimizu", "Shoko Wakamiya", "Eiji Aramaki"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.599/"
doi: "10.18653/v1/2026.findings-acl.599"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shimizu-2026-herd-language-models-makes-better/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shimizu-2026-herd-language-models-makes-better/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/) / A Herd of Language Models Makes a Better Zero-shot Annotator for Clinical Named Entity Recognition

# A Herd of Language Models Makes a Better Zero-shot Annotator for Clinical Named Entity Recognition

**Authors:** [Seiji Shimizu](https://md.research.elamur.ai/free/en/authors/seiji-shimizu/), [Shoko Wakamiya](https://md.research.elamur.ai/free/en/authors/shoko-wakamiya/) and [Eiji Aramaki](https://md.research.elamur.ai/free/en/authors/eiji-aramaki/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.599/) · [PDF](https://aclanthology.org/2026.findings-acl.599.pdf) · [DOI 10.18653/v1/2026.findings-acl.599](https://doi.org/10.18653/v1/2026.findings-acl.599) · [OpenAlex](https://openalex.org/W7166829097)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Manual annotation of clinical notes is expensive and constrained by data-sharing rules. Single LLMs used as zero-shot annotators still perform poorly on clinical named entity recognition. The authors instead combine the outputs of three different LLMs: a general-purpose model, a medically adapted model and an NER-specialized model.

Entities found by most of the models tend to be correct, but they miss many true mentions. Entities found by just one model raise recall but contain a lot of noise. To handle this trade-off, the authors propose MARY, a label model built on an encoder such as RoBERTa-base. It is trained in three steps:
- domain-adaptive masked-LM pre-training on the unlabeled notes;
- contrastive learning, weighted by how many models agree on an entity, that pulls entity embeddings toward class centroids computed from unanimously extracted entities;
- self-training.

As a result, a minority-only entity is kept only when its context resembles that of high-agreement entities. On tasks drawn from i2b2 and n2c2, MARY outperforms both individual LLMs and label-model baselines such as majority voting. It also greatly reduces the manual labeling needed to train downstream NER models.

## Key points
- Evaluation tasks come from i2b2 2010, i2b2 2014 and n2c2 2018. The LLM herd pairs Llama-3.3-70B-instruct with smaller 7B/8B medical and NER-specialized models.
- Combined with zero-shot refinement methods, MARY improves F1 by 8.6 points on average over vanilla Llama-3.3-70B-instruct zero-shot annotation.
- In the active learning experiments, 20 samples were labeled per round over five rounds (100 in total). Vanilla AL needed about 2.6–3.4× more labeled samples to reach the level MARY+AL achieved with 100.
- Unlike union-based LLM ensembling, MARY explicitly filters noise among entities extracted by only one model, using contextual similarity.
- Limitations noted by the authors:
  - fixed hyperparameters, since no validation data exists in the zero-shot setting;
  - higher inference cost from running several LLMs;
  - testing mainly on a single three-model configuration (alternative combinations were explored, but the general + medical + NER-specialized configuration worked best).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nara Institute of Science and Technology](https://md.research.elamur.ai/free/en/universities/nara-institute-of-science-and-technology/) — University, Ikoma, Japan · 46 works · 4,565 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 234 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 231 works

🧭 Topics (1): [🏷️ Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/) · 200 works

## Cite

Cite as: Seiji Shimizu, Shoko Wakamiya and Eiji Aramaki. 2026. A Herd of Language Models Makes a Better Zero-shot Annotator for Clinical Named Entity Recognition. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shimizu2026herd,
  title = {{A Herd of Language Models Makes a Better Zero-shot Annotator for Clinical Named Entity Recognition}},
  author = {Seiji Shimizu, Shoko Wakamiya and Eiji Aramaki},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.599},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.599}
}
```

- [← previous A Generative Grammar Underlying the Voynich Manuscript, the Pastiche Hypothesis: Evidence from Large Language Models](https://md.research.elamur.ai/free/en/papers/turenne-2026-generative-grammar-underlying-voynich-manuscript/)
- [next → A hybrid approach to predicting the age group of authors of texts written in Portuguese](https://md.research.elamur.ai/free/en/papers/ribeiro-2026-uma-abordagem-hibrida-para-predicao/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/)

## References

41 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. [Large language models are few-shot clinical information extractors](https://md.research.elamur.ai/free/en/papers/agrawal-2022-large-language-models-are-few/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1998–2022, Abu Dhabi, United Arab Emirates. Associati…
2. Reza Averly and Xia Ning. 2025. Entity decomposition with filtering: A zero-shot clinical named entity recognition framework. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pap… [find](https://scholar.google.com/scholar?q=Reza+Averly+and+Xia+Ning%2E+2025%2E+Entity+decomposition+with+filtering%3A+A+zero%2Dshot+clinical+named+entity+recognition+framework%2E+In+Proceedings+of+the+2025+Conference+of+the+Nations+of+the+Americas+Chapt)
3. Alberto Mario Ceballos-Arroyo, Monica Munnangi, Jiuding Sun, Karen Zhang, Jered McInerney, Byron C. Wallace, and Silvio Amir. 2024. Open (clinical) LLMs are sensitive to instruction phrasings. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 50–71, Bangkok, Thaila… [find](https://scholar.google.com/scholar?q=Alberto+Mario+Ceballos%2DArroyo%2C+Monica+Munnangi%2C+Jiuding+Sun%2C+Karen+Zhang%2C+Jered+McInerney%2C+Byron+C%2E+Wallace%2C+and+Silvio+Amir%2E+2024%2E+Open+%28clinical%29+LLMs+are+sensitive+to+instruction+phrasings%2E+In+Proc)
4. Wendy W Chapman, Prakash M Nadkarni, Lynette Hirschman, Leonard W D’Avolio, Guergana K Savova, and Ozlem Uzuner. 2011. Overcoming barriers to nlp for clinical text: the role of shared tasks and the need for additional creative solutions. Journal of the American Medical Informatics Association, 18(5)… [find](https://scholar.google.com/scholar?q=Wendy+W+Chapman%2C+Prakash+M+Nadkarni%2C+Lynette+Hirschman%2C+Leonard+W+D%E2%80%99Avolio%2C+Guergana+K+Savova%2C+and+Ozlem+Uzuner%2E+2011%2E+Overcoming+barriers+to+nlp+for+clinical+text%3A+the+role+of+shared+tasks+and+the+ne)
5. Qingyu Chen, Yan Hu, Xueqing Peng, Qianqian Xie, Qiao Jin, Aidan Gilson, Maxwell B. Singer, Xuguang Ai, Po-Ting Lai, Zhizheng Wang, Vipina K. Keloth, Kalpana Raja, Jimin Huang, Huan He, Fongci Lin, Jingcheng Du, Rui Zhang, W. Jim Zheng, Ron A. Adelman, and 2 others. 2025a. Benchmarking large languag… [find](https://scholar.google.com/scholar?q=Qingyu+Chen%2C+Yan+Hu%2C+Xueqing+Peng%2C+Qianqian+Xie%2C+Qiao+Jin%2C+Aidan+Gilson%2C+Maxwell+B%2E+Singer%2C+Xuguang+Ai%2C+Po%2DTing+Lai%2C+Zhizheng+Wang%2C+Vipina+K%2E+Keloth%2C+Kalpana+Raja%2C+Jimin+Huang%2C+Huan+He%2C+Fongci+Lin%2C+Ji)
6. Zhijun Chen, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Ming Li, Likang Xiao, Dingqi Yang, Yikun Ban, Hailong Sun, and Philip S Yu. 2025b. Harnessing Multiple Large Language Models: A Survey on LLM Ensemble. Preprint, arXiv:2502.18036. — [arXiv:2502.18036](https://arxiv.org/abs/2502.18036) [find](https://scholar.google.com/scholar?q=Zhijun+Chen%2C+Jingzheng+Li%2C+Pengpeng+Chen%2C+Zhuoran+Li%2C+Kai+Sun%2C+Yuankai+Luo%2C+Qianren+Mao%2C+Ming+Li%2C+Likang+Xiao%2C+Dingqi+Yang%2C+Yikun+Ban%2C+Hailong+Sun%2C+and+Philip+S+Yu%2E+2025b%2E+Harnessing+Multiple+Large+La)
7. Clément Christophe, Praveenkumar Kanithi, Prateek Munjal, Tathagata Raha, Nasir Hayat, Ronnie Rajan, Ahmed Al Mahrooqi, Avani Gupta, Muhammad Umar Salman, Marco AF Pimentel, Shadab Khan, and Boulbaba Ben Amor. 2024. Med42– evaluating fine-tuning strategies for medical LLMs: full-parameter vs. parame… [find](https://scholar.google.com/scholar?q=Cl%C3%A9ment+Christophe%2C+Praveenkumar+Kanithi%2C+Prateek+Munjal%2C+Tathagata+Raha%2C+Nasir+Hayat%2C+Ronnie+Rajan%2C+Ahmed+Al+Mahrooqi%2C+Avani+Gupta%2C+Muhammad+Umar+Salman%2C+Marco+AF+Pimentel%2C+Shadab+Khan%2C+and+Boulbaba+)
8. Marika Cusick, Prakash Adekkanattu, Thomas R. Campion, Evan T. Sholle, Annie Myers, Samprit Banerjee, George Alexopoulos, Yanshan Wang, and Jyotishman Pathak. 2021. Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation. Journal of Psychia… [find](https://scholar.google.com/scholar?q=Marika+Cusick%2C+Prakash+Adekkanattu%2C+Thomas+R%2E+Campion%2C+Evan+T%2E+Sholle%2C+Annie+Myers%2C+Samprit+Banerjee%2C+George+Alexopoulos%2C+Yanshan+Wang%2C+and+Jyotishman+Pathak%2E+2021%2E+Using+weak+supervision+and+deep+lea)
9. Surabhi Datta and Kirk Roberts. 2023. Weakly supervised spatial relation extraction from radiology reports. JAMIA Open, 6(2):ooad027. [find](https://scholar.google.com/scholar?q=Surabhi+Datta+and+Kirk+Roberts%2E+2023%2E+Weakly+supervised+spatial+relation+extraction+from+radiology+reports%2E+JAMIA+Open%2C+6%282%29%3Aooad027%2E)
10. Rodrigo del Moral-González, Helena Gómez-Adorno, and Orlando Ramos-Flores. 2025. Comparative analysis of generative LLMs for labeling entities in clinical notes. Genomics & Informatics, 23(1):3. [find](https://scholar.google.com/scholar?q=Rodrigo+del+Moral%2DGonz%C3%A1lez%2C+Helena+G%C3%B3mez%2DAdorno%2C+and+Orlando+Ramos%2DFlores%2E+2025%2E+Comparative+analysis+of+generative+LLMs+for+labeling+entities+in+clinical+notes%2E+Genomics+%26+Informatics%2C+23%281%29%3A3%2E)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
12. Zhuojun Ding, Wei Wei, and Chenghao Fan. 2025. Selecting and merging: Towards adaptable and scalable named entity recognition with large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9869–9886, Vienna, Austr… [find](https://scholar.google.com/scholar?q=Zhuojun+Ding%2C+Wei+Wei%2C+and+Chenghao+Fan%2E+2025%2E+Selecting+and+merging%3A+Towards+adaptable+and+scalable+named+entity+recognition+with+large+language+models%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+t)
13. Felix J Dorfner, Amin Dada, Felix Busch, Marcus R Makowski, Tianyu Han, Daniel Truhn, Jens Kleesiek, Madhumita Sushil, Jacqueline Lammert, Lisa C Adams, and Keno K Bressem. 2024. Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data. Preprint, arXiv:24… — [arXiv:2408.13833](https://arxiv.org/abs/2408.13833) [find](https://scholar.google.com/scholar?q=Felix+J+Dorfner%2C+Amin+Dada%2C+Felix+Busch%2C+Marcus+R+Makowski%2C+Tianyu+Han%2C+Daniel+Truhn%2C+Jens+Kleesiek%2C+Madhumita+Sushil%2C+Jacqueline+Lammert%2C+Lisa+C+Adams%2C+and+Keno+K+Bressem%2E+2024%2E+Biomedical+Large+Lang)
14. David Fraile Navarro, Kiran Ijaz, Dana Rezazadegan, Hania Rahimi-Ardabili, Mark Dras, Enrico Coiera, and Shlomo Berkovsky. 2023. Clinical named entity recognition and relation extraction using natural language processing of medical free text: A systematic review. International Journal of Medical Inf… [find](https://scholar.google.com/scholar?q=David+Fraile+Navarro%2C+Kiran+Ijaz%2C+Dana+Rezazadegan%2C+Hania+Rahimi%2DArdabili%2C+Mark+Dras%2C+Enrico+Coiera%2C+and+Shlomo+Berkovsky%2E+2023%2E+Clinical+named+entity+recognition+and+relation+extraction+using+natural)
15. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
16. Sam Henry, Kevin Buchan, Michele Filannino, Amber Stubbs, and Özlem Uzuner. 2020. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. Journal of the American Medical Informatics Association, 27:3–12. [find](https://scholar.google.com/scholar?q=Sam+Henry%2C+Kevin+Buchan%2C+Michele+Filannino%2C+Amber+Stubbs%2C+and+%C3%96zlem+Uzuner%2E+2020%2E+2018+n2c2+shared+task+on+adverse+drug+events+and+medication+extraction+in+electronic+health+records%2E+Journal+of+the+Am)
17. Enshuo Hsu and Kirk Roberts. 2025. Leveraging large language models for knowledge-free weak supervision in clinical natural language processing. Scientific Reports, 15(1):8241. [find](https://scholar.google.com/scholar?q=Enshuo+Hsu+and+Kirk+Roberts%2E+2025%2E+Leveraging+large+language+models+for+knowledge%2Dfree+weak+supervision+in+clinical+natural+language+processing%2E+Scientific+Reports%2C+15%281%29%3A8241%2E)
18. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-Rank Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In Proceedings of the 10th International Conference on Learning Representations (ICLR), pages 1–13.
19. Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, Kirk Roberts, and Hua Xu. 2024. Improving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informatics… [find](https://scholar.google.com/scholar?q=Yan+Hu%2C+Qingyu+Chen%2C+Jingcheng+Du%2C+Xueqing+Peng%2C+Vipina+Kuttichi+Keloth%2C+Xu+Zuo%2C+Yujia+Zhou%2C+Zehan+Li%2C+Xiaoqian+Jiang%2C+Zhiyong+Lu%2C+Kirk+Roberts%2C+and+Hua+Xu%2E+2024%2E+Improving+large+language+models+for+c)
20. Yan Hu, Xu Zuo, Yujia Zhou, Xueqing Peng, Jimin Huang, Vipina K Keloth, Vincent J Zhang, Ruey-Ling Weng, Qingyu Chen, Xiaoqian Jiang, Kirk E Roberts, and Hua Xu. 2026. Information extraction from clinical notes: Are we ready to switch to large language models? Journal of the American Medical Informa… [find](https://scholar.google.com/scholar?q=Yan+Hu%2C+Xu+Zuo%2C+Yujia+Zhou%2C+Xueqing+Peng%2C+Jimin+Huang%2C+Vipina+K+Keloth%2C+Vincent+J+Zhang%2C+Ruey%2DLing+Weng%2C+Qingyu+Chen%2C+Xiaoqian+Jiang%2C+Kirk+E+Roberts%2C+and+Hua+Xu%2E+2026%2E+Information+extraction+from+clin)
21. Yizhu Jiao, Ming Zhong, Sha Li, Ruining Zhao, Siru Ouyang, Heng Ji, and Jiawei Han. 2023. Instruct and extract: Instruction tuning for on-demand information extraction. In Proceedings of the 2023 [find](https://scholar.google.com/scholar?q=Yizhu+Jiao%2C+Ming+Zhong%2C+Sha+Li%2C+Ruining+Zhao%2C+Siru+Ouyang%2C+Heng+Ji%2C+and+Jiawei+Han%2E+2023%2E+Instruct+and+extract%3A+Instruction+tuning+for+on%2Ddemand+information+extraction%2E+In+Proceedings+of+the+2023)
22. Conference on Empirical Methods in Natural Language Processing, pages 10030–10051, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+10030%E2%80%9310051%2C+Singapore%2E+Association+for+Computational+Linguistics%2E)
23. Veysel Kocaman, Fu-Yuan Cheng, Julio Bonis, Ganesh Raut, Prem Timsina, David Talby, and Arash Kia. 2025. Exploring named entity recognition potential and the value of tailored natural language processing pipelines for radiology, pathology, and progress notes in clinical decision support: Quantitativ… [find](https://scholar.google.com/scholar?q=Veysel+Kocaman%2C+Fu%2DYuan+Cheng%2C+Julio+Bonis%2C+Ganesh+Raut%2C+Prem+Timsina%2C+David+Talby%2C+and+Arash+Kia%2E+2025%2E+Exploring+named+entity+recognition+potential+and+the+value+of+tailored+natural+language+process)
24. Egoitz Laparra, Steven Bethard, and Timothy A Miller. 2020. Rethinking domain adaptation for machine learning over clinical language. JAMIA Open, 3(2):146–150. [find](https://scholar.google.com/scholar?q=Egoitz+Laparra%2C+Steven+Bethard%2C+and+Timothy+A+Miller%2E+2020%2E+Rethinking+domain+adaptation+for+machine+learning+over+clinical+language%2E+JAMIA+Open%2C+3%282%29%3A146%E2%80%93150%2E)
25. Yinghao Li, Pranav Shetty, Lucas Liu, Chao Zhang, and Le Song. 2021. BERTifying the hidden Markov model for multi-source weakly supervised named entity recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference… [find](https://scholar.google.com/scholar?q=Yinghao+Li%2C+Pranav+Shetty%2C+Lucas+Liu%2C+Chao+Zhang%2C+and+Le+Song%2E+2021%2E+BERTifying+the+hidden+Markov+model+for+multi%2Dsource+weakly+supervised+named+entity+recognition%2E+In+Proceedings+of+the+59th+Annual+M)
26. Zhiyi Li, Shengjie Zhang, Yujie Song, and Jungyeul Park. 2023. Extrinsic factors affecting the accuracy of biomedical NER. Preprint, arXiv:2305.18152. — [arXiv:2305.18152](https://arxiv.org/abs/2305.18152) [find](https://scholar.google.com/scholar?q=Zhiyi+Li%2C+Shengjie+Zhang%2C+Yujie+Song%2C+and+Jungyeul+Park%2E+2023%2E+Extrinsic+factors+affecting+the+accuracy+of+biomedical+NER%2E+Preprint%2C+arXiv%3A2305%2E18152%2E+%E2%80%94+%5BarXiv%3A2305%2E18152%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E1)
27. Pierre Lison, Jeremy Barnes, Aliaksandr Hubin, and Samia Touileb. 2020. Named entity recognition without labelled data: A weak supervision approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1518– 1533, Online. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Pierre+Lison%2C+Jeremy+Barnes%2C+Aliaksandr+Hubin%2C+and+Samia+Touileb%2E+2020%2E+Named+entity+recognition+without+labelled+data%3A+A+weak+supervision+approach%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+As)
28. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. [What makes good in-context examples for GPT-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114…
29. Ivan Lopez, Akshay Swaminathan, Karthik Vedula, Sanjana Narayanan, Fateme Nateghi Haredasht, Stephen P. Ma, April S. Liang, Steven Tate, Manoj Maddali, Robert Joseph Gallo, Nigam H. Shah, and Jonathan H. Chen. 2025. Clinical entity augmented retrieval for clinical information extraction. npj Digital… [find](https://scholar.google.com/scholar?q=Ivan+Lopez%2C+Akshay+Swaminathan%2C+Karthik+Vedula%2C+Sanjana+Narayanan%2C+Fateme+Nateghi+Haredasht%2C+Stephen+P%2E+Ma%2C+April+S%2E+Liang%2C+Steven+Tate%2C+Manoj+Maddali%2C+Robert+Joseph+Gallo%2C+Nigam+H%2E+Shah%2C+and+Jonathan)
30. Meng Lu, Yuzhang Xie, Zhenyu Bi, Shuxiang Cao, and Xuan Wang. 2025. CROSSAGENTIE: Cross-type and cross-task multi-agent LLM collaboration for zero-shot information extraction. In Findings of the Association for Computational Linguistics: ACL 2025, pages 13953–13977, Vienna, Austria. Association for… [find](https://scholar.google.com/scholar?q=Meng+Lu%2C+Yuzhang+Xie%2C+Zhenyu+Bi%2C+Shuxiang+Cao%2C+and+Xuan+Wang%2E+2025%2E+CROSSAGENTIE%3A+Cross%2Dtype+and+cross%2Dtask+multi%2Dagent+LLM+collaboration+for+zero%2Dshot+information+extraction%2E+In+Findings+of+the+Assoc)
31. Marco Naguib, Xavier Tannier, and Aurélie Névéol. 2024. Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6829–6852, Miami, Florida, USA. A… [find](https://scholar.google.com/scholar?q=Marco+Naguib%2C+Xavier+Tannier%2C+and+Aur%C3%A9lie+N%C3%A9v%C3%A9ol%2E+2024%2E+Few%2Dshot+clinical+entity+recognition+in+English%2C+French+and+Spanish%3A+masked+language+models+outperform+generative+model+prompting%2E+In+Findings+o)
32. Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: rapid training data creation with weak supervision. Proc. VLDB Endow., 11(3):269–282. [find](https://scholar.google.com/scholar?q=Alexander+Ratner%2C+Stephen+H%2E+Bach%2C+Henry+Ehrenberg%2C+Jason+Fries%2C+Sen+Wu%2C+and+Christopher+R%C3%A9%2E+2017%2E+Snorkel%3A+rapid+training+data+creation+with+weak+supervision%2E+Proc%2E+VLDB+Endow%2E%2C+11%283%29%3A269%E2%80%93282%2E)
33. Sonish Sivarajkumar, Mark Kelley, Alyssa Samolyk-Mazzanti, Shyam Visweswaran, and Yanshan Wang. 2024. An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: Algorithm development and validation study. JMIR Medical Informatics, 12:… [find](https://scholar.google.com/scholar?q=Sonish+Sivarajkumar%2C+Mark+Kelley%2C+Alyssa+Samolyk%2DMazzanti%2C+Shyam+Visweswaran%2C+and+Yanshan+Wang%2E+2024%2E+An+empirical+evaluation+of+prompting+strategies+for+large+language+models+in+zero%2Dshot+clinical+na)
34. Ryan Smith, Jason A. Fries, Braden Hancock, and Stephen H. Bach. 2024. Language models in the loop: Incorporating prompting into weak supervision. ACM / IMS J. Data Sci., 1(2). [find](https://scholar.google.com/scholar?q=Ryan+Smith%2C+Jason+A%2E+Fries%2C+Braden+Hancock%2C+and+Stephen+H%2E+Bach%2E+2024%2E+Language+models+in+the+loop%3A+Incorporating+prompting+into+weak+supervision%2E+ACM+%2F+IMS+J%2E+Data+Sci%2E%2C+1%282%29%2E)
35. Amber Stubbs and Özlem Uzuner. 2015. Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus. Journal of Biomedical Informatics, 58:S20–S29. Supplement: Proceedings of the 2014 i2b2/UTHealth Shared-Tasks and Workshop on Challenges in Natural Language Processi… [find](https://scholar.google.com/scholar?q=Amber+Stubbs+and+%C3%96zlem+Uzuner%2E+2015%2E+Annotating+longitudinal+clinical+narratives+for+de%2Didentification%3A+The+2014+i2b2%2FUTHealth+corpus%2E+Journal+of+Biomedical+Informatics%2C+58%3AS20%E2%80%93S29%2E+Supplement%3A+Procee)
36. Jinyan Su, Peilin Yu, Jieyu Zhang, and Stephen H. Bach. 2023. Leveraging large language models for structure learning in prompted weak supervision. In Proceedings of the 2023 IEEE International Conference on Big Data (BigData), pages 875–884. [find](https://scholar.google.com/scholar?q=Jinyan+Su%2C+Peilin+Yu%2C+Jieyu+Zhang%2C+and+Stephen+H%2E+Bach%2E+2023%2E+Leveraging+large+language+models+for+structure+learning+in+prompted+weak+supervision%2E+In+Proceedings+of+the+2023+IEEE+International+Confer)
37. Xin Su, Yiyun Zhao, and Steven Bethard. 2022. A comparison of strategies for source-free domain adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8352–8367, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xin+Su%2C+Yiyun+Zhao%2C+and+Steven+Bethard%2E+2022%2E+A+comparison+of+strategies+for+source%2Dfree+domain+adaptation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28)
38. Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552–556. [find](https://scholar.google.com/scholar?q=%C3%96zlem+Uzuner%2C+Brett+R+South%2C+Shuying+Shen%2C+and+Scott+L+DuVall%2E+2011%2E+2010+i2b2%2Fva+challenge+on+concepts%2C+assertions%2C+and+relations+in+clinical+text%2E+Journal+of+the+American+Medical+Informatics+Associa)
39. Han Wang, Wesley Lok Kin Yeung, Qin Xiang Ng, Angeline Tung, Joey Ai Meng Tay, Davin Ryanpu-tra, Marcus Eng Hock Ong, Mengling Feng, and Shalini Arulanandam. 2021. A weakly-supervised named entity recognition machine learning approach for emergency medical services clinical audit. International Jour… [find](https://scholar.google.com/scholar?q=Han+Wang%2C+Wesley+Lok+Kin+Yeung%2C+Qin+Xiang+Ng%2C+Angeline+Tung%2C+Joey+Ai+Meng+Tay%2C+Davin+Ryanpu%2Dtra%2C+Marcus+Eng+Hock+Ong%2C+Mengling+Feng%2C+and+Shalini+Arulanandam%2E+2021%2E+A+weakly%2Dsupervised+named+entity+rec)
40. Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, Guoyin Wang, and Chen Guo. 2025. [GPT-NER: Named entity recognition via large language models](https://md.research.elamur.ai/free/en/papers/wang-2025-gpt-ner-named-entity-recognition/). In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4257–4275, Albuquerque, New Mexico. Assoc…
41. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Atharva+Naik%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Anjana+Arunkumar%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karaman)

*Updated 1 November 2026*