---
title: "A Domain-Specific Curated Benchmark for Entity and"
description: "The authors present GutBrainIE, a benchmark of 1,647 PubMed titles and abstracts. 55 references, cited by 1 paper here."
authors: ["Marco Martinelli", "Stefano Marchesin", "Vanessa Bonato", "Giorgio Di Nunzio", "Nicola Ferro", "Ornella Irrera", "Laura Menotti", "Federica Vezzani", "Gianmaria Silvello"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.301/"
doi: "10.18653/v1/2026.findings-eacl.301"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/martinelli-2026-domain-specific-curated-benchmark-entity/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/martinelli-2026-domain-specific-curated-benchmark-entity/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) / A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extraction

# A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extraction

**Authors:** Marco Martinelli, Stefano Marchesin, Vanessa Bonato, [Giorgio Di Nunzio](https://md.research.elamur.ai/free/en/authors/giorgio-maria-di-nunzio/), Nicola Ferro, Ornella Irrera, Laura Menotti, Federica Vezzani and Gianmaria Silvello · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.301/) · [PDF](https://aclanthology.org/2026.findings-eacl.301.pdf) · [DOI 10.18653/v1/2026.findings-eacl.301](https://doi.org/10.18653/v1/2026.findings-eacl.301) · [OpenAlex](https://openalex.org/W7140074262)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Biomedical information extraction (IE) resources tend to cover few entity or relation types, work at sentence level, and depend on noisy distant supervision. This is a particular problem for the fast-growing gut-brain axis literature, which links gut microbiota to neurological and psychiatric disorders. The authors present GutBrainIE, a benchmark of 1,647 PubMed titles and abstracts. Experts, trained lay annotators and automatic methods annotated the documents, and the annotations are grouped into quality tiers (Platinum, Gold, Silver, Bronze). The schema is unusually detailed. It has 13 entity types, 17 overloaded relation predicates and concept links to standard vocabularies. Annotators used GLiNER pre-annotations in the MetaTron platform. The benchmark defines four tasks: NER, NEL, mention-level RE (M-RE) and concept-level RE (C-RE). It also ships baselines, evaluation scripts and Codabench leaderboards with a hidden test set. In a public evaluation campaign, NER proved largely solvable even in this specialised domain, while RE remained clearly harder. GutBrainIE matters as a reproducible, demanding testbed for domains with specialised terminology and relations that span sentences.

## Key points
- **Corpus construction:** two PubMed query rounds in 2024 returned 1,662 documents. After removing duplicates and low-relevance items, 1,647 remained.
- **Schema:** 13 entity types (including microbiome, dietary supplement and statistical technique) and 17 relation predicates, which combine into 55 distinct relation triples. Mentions are linked to 6 biomedical vocabularies, plus a custom ontology for terms that have no match.
- **Annotation team and support:** 7 experts and 40 terminography master's students did the work. GLiNER pre-annotations (NuNER Zero checkpoint, temperature 0.8) reduced annotation time from about 30 minutes to 10–15 minutes per document.
- **Inter-annotator agreement:** Fleiss' κ was 0.89 for NER and Cohen's κ was 0.88. RE agreement was much lower, at 0.43. An expert revision pass then corrected or removed 204 entities and 135 relations.
- **Evaluation campaign:** the NER and M-RE tasks featured in an international campaign with 17 teams and nearly 200 submissions. Data and code are on Zenodo (record 16,845,409). Planned extensions include Alzheimer's and Multiple Sclerosis documents and manual revision of the Silver and Bronze tiers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Padua](https://md.research.elamur.ai/free/en/universities/university-of-padua/) — University, Padua, Italy · 18 works · 4,260 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 234 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 231 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 167 works

## Cite

Cite as: Marco Martinelli, Stefano Marchesin, Vanessa Bonato, Giorgio Di Nunzio, Nicola Ferro, Ornella Irrera, Laura Menotti, Federica Vezzani and Gianmaria Silvello. 2026. A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extraction. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{martinelli2026domain,
  title = {{A Domain-Specific Curated Benchmark for Entity and Document-Level Relation Extraction}},
  author = {Marco Martinelli, Stefano Marchesin, Vanessa Bonato, Giorgio {Di Nunzio}, Nicola Ferro, Ornella Irrera, Laura Menotti, Federica Vezzani and Gianmaria Silvello},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.301},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.301}
}
```

- [← previous A Context-aware Framework for Translation-mediated Conversations](https://md.research.elamur.ai/free/en/papers/pombal-2026-context-aware-framework-translation-mediated/)
- [next → A Goal Without a Plan Is Just a Wish: Efficient and Effective Global Planner Training for Long-Horizon Agent Tasks](https://md.research.elamur.ai/free/en/papers/si-2026-goal-without-plan-is-just/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/)

## References

55 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Khaled Alomar, Halil Ibrahim Aysel, and Xiaohao Cai. 2023. Data augmentation in classification and segmentation: A survey and new strategies. Journal of Imaging, 9(2):46. [find](https://scholar.google.com/scholar?q=Khaled+Alomar%2C+Halil+Ibrahim+Aysel%2C+and+Xiaohao+Cai%2E+2023%2E+Data+augmentation+in+classification+and+segmentation%3A+A+survey+and+new+strategies%2E+Journal+of+Imaging%2C+9%282%29%3A46%2E)
2. Omar Alonso and Gary Marchionini. 2019. The Practice of Crowdsourcing. Morgan & Claypool Publishers. [find](https://scholar.google.com/scholar?q=Omar+Alonso+and+Gary+Marchionini%2E+2019%2E+The+Practice+of+Crowdsourcing%2E+Morgan+%26+Claypool+Publishers%2E)
3. Sultan Alrowili and Vijay Shanker. 2021. BioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 221–227, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sultan+Alrowili+and+Vijay+Shanker%2E+2021%2E+BioM%2DTransformers%3A+Building+Large+Biomedical+Language+Models+with+BERT%2C+ALBERT+and+ELECTRA%2E+In+Proceedings+of+the+20th+Workshop+on+Biomedical+Language+Processi)
4. Saadullah Amin, Katherine Ann Dunfield, Anna Vechkaeva, and Guenter Neumann. 2020. A data-driven approach for noise reduction in distantly supervised biomedical relation extraction. arXiv preprint arXiv:2005.12565. [find](https://scholar.google.com/scholar?q=Saadullah+Amin%2C+Katherine+Ann+Dunfield%2C+Anna+Vechkaeva%2C+and+Guenter+Neumann%2E+2020%2E+A+data%2Ddriven+approach+for+noise+reduction+in+distantly+supervised+biomedical+relation+extraction%2E+arXiv+preprint+arX)
5. Saadullah Amin, Pasquale Minervini, David Chang, Pontus Stenetorp, and Günter Neumann. 2022. Meddistant19: towards an accurate benchmark for broad-coverage biomedical relation extraction. arXiv preprint arXiv:2204.04779. [find](https://scholar.google.com/scholar?q=Saadullah+Amin%2C+Pasquale+Minervini%2C+David+Chang%2C+Pontus+Stenetorp%2C+and+G%C3%BCnter+Neumann%2E+2022%2E+Meddistant19%3A+towards+an+accurate+benchmark+for+broad%2Dcoverage+biomedical+relation+extraction%2E+arXiv+prepri)
6. Jeremy Appleton. 2018. The gut-brain axis: influence of microbiota on mood and mental health. Integrative Medicine: A Clinician’s Journal, 17(4):28. [find](https://scholar.google.com/scholar?q=Jeremy+Appleton%2E+2018%2E+The+gut%2Dbrain+axis%3A+influence+of+microbiota+on+mood+and+mental+health%2E+Integrative+Medicine%3A+A+Clinician%E2%80%99s+Journal%2C+17%284%29%3A28%2E)
7. Olivier Bodenreider. 2004. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic acids research, 32(suppl_1):D267– D270. [find](https://scholar.google.com/scholar?q=Olivier+Bodenreider%2E+2004%2E+The+unified+medical+language+system+%28UMLS%29%3A+integrating+biomedical+terminology%2E+Nucleic+acids+research%2C+32%28suppl%5F1%29%3AD267%E2%80%93+D270%2E)
8. Sergei Bogdanov, Alexandre Constantin, Timothée Bernard, Benoit Crabbé, and Etienne Bernard. 2024. NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data. [find](https://scholar.google.com/scholar?q=Sergei+Bogdanov%2C+Alexandre+Constantin%2C+Timoth%C3%A9e+Bernard%2C+Benoit+Crabb%C3%A9%2C+and+Etienne+Bernard%2E+2024%2E+NuNER%3A+Entity+Recognition+Encoder+Pre%2Dtraining+via+LLM%2DAnnotated+Data%2E)
9. Marilia Carabotti, Annunziata Scirocco, Maria Antonietta Maselli, and Carola Severi. 2015. The gut-brain axis: interactions between enteric microbiota, central and enteric nervous systems. Annals of gastroenterology: quarterly publication of the Hellenic Society of Gastroenterology, 28(2):203. [find](https://scholar.google.com/scholar?q=Marilia+Carabotti%2C+Annunziata+Scirocco%2C+Maria+Antonietta+Maselli%2C+and+Carola+Severi%2E+2015%2E+The+gut%2Dbrain+axis%3A+interactions+between+enteric+microbiota%2C+central+and+enteric+nervous+systems%2E+Annals+of+g)
10. Yukun Chen, Thomas A Lasko, Qiaozhu Mei, Joshua C Denny, and Hua Xu. 2015. A study of active learning methods for named entity recognition in clinical text. Journal of biomedical informatics, 58:11–18. [find](https://scholar.google.com/scholar?q=Yukun+Chen%2C+Thomas+A+Lasko%2C+Qiaozhu+Mei%2C+Joshua+C+Denny%2C+and+Hua+Xu%2E+2015%2E+A+study+of+active+learning+methods+for+named+entity+recognition+in+clinical+text%2E+Journal+of+biomedical+informatics%2C+58%3A11%E2%80%9318)
11. Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. HacRED: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of the Association for Computational Linguistics: ACL-IJCNLP… [find](https://scholar.google.com/scholar?q=Qiao+Cheng%2C+Juntao+Liu%2C+Xiaoye+Qu%2C+Jin+Zhao%2C+Jiaqing+Liang%2C+Zhefeng+Wang%2C+Baoxing+Huai%2C+Nicholas+Jing+Yuan%2C+and+Yanghua+Xiao%2E+2021%2E+HacRED%3A+A+large%2Dscale+relation+extraction+dataset+toward+hard+cases+)
12. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+psychological+measurement%2C+20%281%29%3A37%E2%80%9346%2E)
13. Nigel Collier, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Jin-Dong Kim. 2004. [Introduction to the bio-entity recognition task at JNLPBA](https://md.research.elamur.ai/free/en/papers/collier-2004-introduction-bio-entity-recognition-task/). In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP), pages 73–78.
14. Gamal Crichton, Sampo Pyysalo, Billy Chiu, and Anna Korhonen. 2017. A neural network multi-task learning approach to biomedical named entity recognition. BMC bioinformatics, 18:1–14. [find](https://scholar.google.com/scholar?q=Gamal+Crichton%2C+Sampo+Pyysalo%2C+Billy+Chiu%2C+and+Anna+Korhonen%2E+2017%2E+A+neural+network+multi%2Dtask+learning+approach+to+biomedical+named+entity+recognition%2E+BMC+bioinformatics%2C+18%3A1%E2%80%9314%2E)
15. John F Cryan, Kenneth J O’Riordan, Kiran Sandhu, Veronica Peterson, and Timothy G Dinan. 2020. The gut microbiome in neurological disorders. The Lancet Neurology, 19(2):179–194. [find](https://scholar.google.com/scholar?q=John+F+Cryan%2C+Kenneth+J+O%E2%80%99Riordan%2C+Kiran+Sandhu%2C+Veronica+Peterson%2C+and+Timothy+G+Dinan%2E+2020%2E+The+gut+microbiome+in+neurological+disorders%2E+The+Lancet+Neurology%2C+19%282%29%3A179%E2%80%93194%2E)
16. Kartik Detroja, C.K. Bhensdadia, and Brijesh S. Bhatt. 2023. A survey on Relation Extraction. Intelligent Systems with Applications, 19:200244. [find](https://scholar.google.com/scholar?q=Kartik+Detroja%2C+C%2EK%2E+Bhensdadia%2C+and+Brijesh+S%2E+Bhatt%2E+2023%2E+A+survey+on+Relation+Extraction%2E+Intelligent+Systems+with+Applications%2C+19%3A200244%2E)
17. Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. NCBI disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10. [find](https://scholar.google.com/scholar?q=Rezarta+Islamaj+Dog%CC%86an%2C+Robert+Leaman%2C+and+Zhiyong+Lu%2E+2014%2E+NCBI+disease+corpus%3A+a+resource+for+disease+name+recognition+and+concept+normalization%2E+Journal+of+biomedical+informatics%2C+47%3A1%E2%80%9310%2E)
18. Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. 2019. Hypergraph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3558–3565. [find](https://scholar.google.com/scholar?q=Yifan+Feng%2C+Haoxuan+You%2C+Zizhao+Zhang%2C+Rongrong+Ji%2C+and+Yue+Gao%2E+2019%2E+Hypergraph+neural+networks%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+volume+33%2C+pages+3558%E2%80%933565%2E)
19. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [find](https://scholar.google.com/scholar?q=Joseph+L+Fleiss%2E+1971%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+bulletin%2C+76%285%29%3A378%2E)
20. Kuzman Ganchev, Fernando Pereira, Mark Mandel, Steven Carroll, and Peter White. 2007. Semi-automated named entity annotation. In Proceedings of the linguistic annotation workshop, pages 53–56. [find](https://scholar.google.com/scholar?q=Kuzman+Ganchev%2C+Fernando+Pereira%2C+Mark+Mandel%2C+Steven+Carroll%2C+and+Peter+White%2E+2007%2E+Semi%2Dautomated+named+entity+annotation%2E+In+Proceedings+of+the+linguistic+annotation+workshop%2C+pages+53%E2%80%9356%2E)
21. Shivani Ghaisas, Joshua Maher, and Anumantha Kan-thasamy. 2016. Gut microbiome in health and disease: Linking the microbiome–gut–brain axis and environmental factors in the pathogenesis of systemic and neurodegenerative diseases. Pharmacology & therapeutics, 158:52–62. [find](https://scholar.google.com/scholar?q=Shivani+Ghaisas%2C+Joshua+Maher%2C+and+Anumantha+Kan%2Dthasamy%2E+2016%2E+Gut+microbiome+in+health+and+disease%3A+Linking+the+microbiome%E2%80%93gut%E2%80%93brain+axis+and+environmental+factors+in+the+pathogenesis+of+systemic+an)
22. Sreyan Ghosh, Utkarsh Tyagi, Manan Suri, Sonal Kumar, S Ramaneswaran, and Dinesh Manocha. 2023. Aclm: A selective-denoising based generative data augmentation approach for low-resource complex ner. arXiv preprint arXiv:2306.00928. [find](https://scholar.google.com/scholar?q=Sreyan+Ghosh%2C+Utkarsh+Tyagi%2C+Manan+Suri%2C+Sonal+Kumar%2C+S+Ramaneswaran%2C+and+Dinesh+Manocha%2E+2023%2E+Aclm%3A+A+selective%2Ddenoising+based+generative+data+augmentation+approach+for+low%2Dresource+complex+ner%2E+ar)
23. Robert Greinacher and Franziska Horn. 2018. The DAL-PHI annotation framework & how its pre-annotations can improve annotator efficiency. arXiv preprint arXiv:1808.05558. — [arXiv:1808.05558](https://arxiv.org/abs/1808.05558) [find](https://scholar.google.com/scholar?q=Robert+Greinacher+and+Franziska+Horn%2E+2018%2E+The+DAL%2DPHI+annotation+framework+%26+how+its+pre%2Dannotations+can+improve+annotator+efficiency%2E+arXiv+preprint+arXiv%3A1808%2E05558%2E+%E2%80%94+%5BarXiv%3A1808%2E05558%5D%28https%3A%2F%2Fa)
24. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. [Domain-specific language model pretraining for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/). ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23.
25. Yue Guo and Yi Yang. 2024. Improving weak-to-strong generalization with reliability-aware alignment. arXiv preprint arXiv:2406.19032. — [arXiv:2406.19032](https://arxiv.org/abs/2406.19032) [find](https://scholar.google.com/scholar?q=Yue+Guo+and+Yi+Yang%2E+2024%2E+Improving+weak%2Dto%2Dstrong+generalization+with+reliability%2Daware+alignment%2E+arXiv+preprint+arXiv%3A2406%2E19032%2E+%E2%80%94+%5BarXiv%3A2406%2E19032%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2406%2E19032%29)
26. Gibong Hong, Veronica Hindle, Nadine M Veasley, Hannah D Holscher, and Halil Kilicoglu. 2025. DiMB-RE: mining the scientific literature for diet-microbiome associations. Journal of the American Medical Informatics Association, 32(6):998–1006. [find](https://scholar.google.com/scholar?q=Gibong+Hong%2C+Veronica+Hindle%2C+Nadine+M+Veasley%2C+Hannah+D+Holscher%2C+and+Halil+Kilicoglu%2E+2025%2E+DiMB%2DRE%3A+mining+the+scientific+literature+for+diet%2Dmicrobiome+associations%2E+Journal+of+the+American+Medica)
27. Karim M Ibrahim, Elena V Epure, Geoffroy Peeters, and Gael Richard. 2020. Confidence-based weighted loss for multi-label classification with missing labels. In Proceedings of the 2020 International Conference on Multimedia Retrieval, pages 291–295. [find](https://scholar.google.com/scholar?q=Karim+M+Ibrahim%2C+Elena+V+Epure%2C+Geoffroy+Peeters%2C+and+Gael+Richard%2E+2020%2E+Confidence%2Dbased+weighted+loss+for+multi%2Dlabel+classification+with+missing+labels%2E+In+Proceedings+of+the+2020+International+Co)
28. Ornella Irrera, Stefano Marchesin, and Gianmaria Silvello. 2024. MetaTron: advancing biomedical annotation empowering relation annotation and collaboration. BMC bioinformatics, 25(1):112. [find](https://scholar.google.com/scholar?q=Ornella+Irrera%2C+Stefano+Marchesin%2C+and+Gianmaria+Silvello%2E+2024%2E+MetaTron%3A+advancing+biomedical+annotation+empowering+relation+annotation+and+collaboration%2E+BMC+bioinformatics%2C+25%281%29%3A112%2E)
29. Rezarta Islamaj, Chih-Hsuan Wei, David Cissel, Nicholas Miliaras, Olga Printseva, Oleg Rodionov, Keiko Sekiya, Janice Ward, and Zhiyong Lu. 2021. NLM-Gene, a richly annotated gold standard dataset for gene entities that addresses ambiguity and multi-species gene recognition. Journal of biomedical in… [find](https://scholar.google.com/scholar?q=Rezarta+Islamaj%2C+Chih%2DHsuan+Wei%2C+David+Cissel%2C+Nicholas+Miliaras%2C+Olga+Printseva%2C+Oleg+Rodionov%2C+Keiko+Sekiya%2C+Janice+Ward%2C+and+Zhiyong+Lu%2E+2021%2E+NLM%2DGene%2C+a+richly+annotated+gold+standard+dataset+for)
30. Peter D Karp. 2016. How much does curation cost? Database, 2016:baw110. [find](https://scholar.google.com/scholar?q=Peter+D+Karp%2E+2016%2E+How+much+does+curation+cost%3F+Database%2C+2016%3Abaw110%2E)
31. Jung-Jae Kim, Xu Han, Vivian K Lee, and Dietrich Rebholz Schuhmann. 2013. GRO Task: Populating the Gene Regulation Ontology with events and relations. In Proceedings of the BioNLP Shared Task 2013 Workshop, pages 50–57. [find](https://scholar.google.com/scholar?q=Jung%2DJae+Kim%2C+Xu+Han%2C+Vivian+K+Lee%2C+and+Dietrich+Rebholz+Schuhmann%2E+2013%2E+GRO+Task%3A+Populating+the+Gene+Regulation+Ontology+with+events+and+relations%2E+In+Proceedings+of+the+BioNLP+Shared+Task+2013+Wor)
32. Martin Krallinger, Obdulia Rabal, Florian Leitner, Miguel Vazquez, David Salgado, Zhiyong Lu, Robert Leaman, Yanan Lu, Donghong Ji, Daniel M Lowe, et al. 2015. The CHEMDNER corpus of chemicals and drugs and its annotation principles. Journal of cheminformatics, 7(Suppl 1):S2. [find](https://scholar.google.com/scholar?q=Martin+Krallinger%2C+Obdulia+Rabal%2C+Florian+Leitner%2C+Miguel+Vazquez%2C+David+Salgado%2C+Zhiyong+Lu%2C+Robert+Leaman%2C+Yanan+Lu%2C+Donghong+Ji%2C+Daniel+M+Lowe%2C+et+al%2E+2015%2E+The+CHEMDNER+corpus+of+chemicals+and+dru)
33. Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. BioASQ-QA: A manually curated corpus for Biomedical Question Answering. Scientific Data, 10(1):170. [find](https://scholar.google.com/scholar?q=Anastasia+Krithara%2C+Anastasios+Nentidis%2C+Konstantinos+Bougiatiotis%2C+and+Georgios+Paliouras%2E+2023%2E+BioASQ%2DQA%3A+A+manually+curated+corpus+for+Biomedical+Question+Answering%2E+Scientific+Data%2C+10%281%29%3A170%2E)
34. Sang Gyu Kwak and Jong Hae Kim. 2017. Central limit theorem: the cornerstone of modern statistics. Korean journal of anesthesiology, 70(2):144. [find](https://scholar.google.com/scholar?q=Sang+Gyu+Kwak+and+Jong+Hae+Kim%2E+2017%2E+Central+limit+theorem%3A+the+cornerstone+of+modern+statistics%2E+Korean+journal+of+anesthesiology%2C+70%282%29%3A144%2E)
35. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. [BioBERT: pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/). arXiv preprint arXiv:1901.08746.
36. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. [find](https://scholar.google.com/scholar?q=Jiao+Li%2C+Yueping+Sun%2C+Robin+J+Johnson%2C+Daniela+Sci%2Daky%2C+Chih%2DHsuan+Wei%2C+Robert+Leaman%2C+Allan+Peter+Davis%2C+Carolyn+J+Mattingly%2C+Thomas+C+Wiegers%2C+and+Zhiyong+Lu%2E+2016%2E+BioCreative+V+CDR+task+corpus%3A+a+)
37. Database, 2016. knowledge graphs. Journal of Artificial Intelligence for Medical Sciences, 1(3):30–42. [find](https://scholar.google.com/scholar?q=Database%2C+2016%2E+knowledge+graphs%2E+Journal+of+Artificial+Intelligence+for+Medical+Sciences%2C+1%283%29%3A30%E2%80%9342%2E)
38. Ling Luo, Po-Ting Lai, Chih-Hsuan Wei, Cecilia N Arighi, and Zhiyong Lu. 2022. BioRED: a rich biomedical relation extraction dataset. Briefings in Bioinformatics, 23(5):bbac282. [find](https://scholar.google.com/scholar?q=Ling+Luo%2C+Po%2DTing+Lai%2C+Chih%2DHsuan+Wei%2C+Cecilia+N+Arighi%2C+and+Zhiyong+Lu%2E+2022%2E+BioRED%3A+a+rich+biomedical+relation+extraction+dataset%2E+Briefings+in+Bioinformatics%2C+23%285%29%3Abbac282%2E)
39. M Martinelli, G Silvello, V Bonato, GM Di Nunzio, N Ferro, O Irrera, S Marchesin, L Menotti, F Vezzani, et al. 2025. Overview of GutBrainIE@ CLEF 2025: Gut-Brain Interplay Information Extraction. In CLEF 2025 Working Notes, volume 4038, pages 65–98. [find](https://scholar.google.com/scholar?q=M+Martinelli%2C+G+Silvello%2C+V+Bonato%2C+GM+Di+Nunzio%2C+N+Ferro%2C+O+Irrera%2C+S+Marchesin%2C+L+Menotti%2C+F+Vezzani%2C+et+al%2E+2025%2E+Overview+of+GutBrainIE%40+CLEF+2025%3A+Gut%2DBrain+Interplay+Information+Extraction%2E+In+C)
40. Marco Martinelli, Stefano Marchesin, and Gianmaria Silvello. 2026. Efficient and Reliable Estimation of Named Entity Linking Quality: A Case Study on GutBrainIE. [find](https://scholar.google.com/scholar?q=Marco+Martinelli%2C+Stefano+Marchesin%2C+and+Gianmaria+Silvello%2E+2026%2E+Efficient+and+Reliable+Estimation+of+Named+Entity+Linking+Quality%3A+A+Case+Study+on+GutBrainIE%2E)
41. Marie Mikulová, Milan Straka, Jan Štěpánek, Barbora Štěpánková, and Jan Hajič. 2023. Quality and efficiency of manual annotation: Pre-annotation bias. arXiv preprint arXiv:2306.09307. [find](https://scholar.google.com/scholar?q=Marie+Mikulov%C3%A1%2C+Milan+Straka%2C+Jan+%C5%A0te%CC%8Cp%C3%A1nek%2C+Barbora+%C5%A0te%CC%8Cp%C3%A1nkov%C3%A1%2C+and+Jan+Hajic%CC%8C%2E+2023%2E+Quality+and+efficiency+of+manual+annotation%3A+Pre%2Dannotation+bias%2E+arXiv+preprint+arXiv%3A2306%2E09307%2E)
42. Anastasios Nentidis, Georgios Katsimpras, Anastasia Krithara, Martin Krallinger, Miguel Rodríguez-Ortega, Eduard Rodriguez-López, Natalia Loukachevitch, Andrey Sakhovskiy, Elena Tutubalina, Dimitris Dimitriadis, et al. 2025. Overview of bioasq 2025: The thirteenth bioasq challenge on large-scale bio… [find](https://scholar.google.com/scholar?q=Anastasios+Nentidis%2C+Georgios+Katsimpras%2C+Anastasia+Krithara%2C+Martin+Krallinger%2C+Miguel+Rodr%C3%ADguez%2DOrtega%2C+Eduard+Rodriguez%2DL%C3%B3pez%2C+Natalia+Loukachevitch%2C+Andrey+Sakhovskiy%2C+Elena+Tutubalina%2C+Dimitris+D)
43. Yesol Park, Gyujin Son, and Mina Rho. 2024. Biomedical flat and nested named entity recognition: Methods, challenges, and advances. APPLIED SCIENCES-BASEL, 14(20):1–23. [find](https://scholar.google.com/scholar?q=Yesol+Park%2C+Gyujin+Son%2C+and+Mina+Rho%2E+2024%2E+Biomedical+flat+and+nested+named+entity+recognition%3A+Methods%2C+challenges%2C+and+advances%2E+APPLIED+SCIENCES%2DBASEL%2C+14%2820%29%3A1%E2%80%9323%2E)
44. Sampo Pyysalo, Tomoko Ohta, and Sophia Ananiadou. 2013. Overview of the cancer genetics (cg) task of bionlp shared task 2013. In Proceedings of the BioNLP Shared Task 2013 Workshop, pages 58–66. [find](https://scholar.google.com/scholar?q=Sampo+Pyysalo%2C+Tomoko+Ohta%2C+and+Sophia+Ananiadou%2E+2013%2E+Overview+of+the+cancer+genetics+%28cg%29+task+of+bionlp+shared+task+2013%2E+In+Proceedings+of+the+BioNLP+Shared+Task+2013+Workshop%2C+pages+58%E2%80%9366%2E)
45. Peng Su, Gang Li, Cathy Wu, and K Vijay-Shanker. 2019. Using distant supervision to augment manually annotated data for relation extraction. PloS one, 14(7):e0216913. [find](https://scholar.google.com/scholar?q=Peng+Su%2C+Gang+Li%2C+Cathy+Wu%2C+and+K+Vijay%2DShanker%2E+2019%2E+Using+distant+supervision+to+augment+manually+annotated+data+for+relation+extraction%2E+PloS+one%2C+14%287%29%3Ae0216913%2E)
46. Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022. [Revisiting DocRED–Addressing the False Negative Problem in Relation Extraction](https://md.research.elamur.ai/free/en/papers/tan-2022-revisiting-docred-addressing-false-negative/). arXiv preprint arXiv:2205.12696.
47. Ying Lin, Liyuan Liu, Heng Ji, Dong Yu, and Jiawei Han. 2019. Reliability-aware dynamic feature composition for name tagging. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 165–174. [find](https://scholar.google.com/scholar?q=Ying+Lin%2C+Liyuan+Liu%2C+Heng+Ji%2C+Dong+Yu%2C+and+Jiawei+Han%2E+2019%2E+Reliability%2Daware+dynamic+feature+composition+for+name+tagging%2E+In+Proceedings+of+the+57th+annual+meeting+of+the+association+for+computati)
48. Bui Duc Tho, Minh-Tien Nguyen, Dung Tien Le, Lin-Lung Ying, Shumpei Inoue, and Tri-Thanh Nguyen. 2024. Improving biomedical Named Entity Recognition with additional external contexts. Journal of Biomedical Informatics, 156:104674. [find](https://scholar.google.com/scholar?q=Bui+Duc+Tho%2C+Minh%2DTien+Nguyen%2C+Dung+Tien+Le%2C+Lin%2DLung+Ying%2C+Shumpei+Inoue%2C+and+Tri%2DThanh+Nguyen%2E+2024%2E+Improving+biomedical+Named+Entity+Recognition+with+additional+external+contexts%2E+Journal+of+Biome)
49. Ting Liu, Xueli Pan, Xu Wang, K Anton Feenstra, Jaap Heringa, and Zhisheng Huang. 2021. Exploring the microbiota-gut-brain axis for mental disorders with Erik M Van Mulligen, Annie Fourrier-Reglat, David Gurwitz, Mariam Molokhia, Ainhoa Nieto, Gianluca Trifiro, Jan A Kors, and Laura I Furlong. 2012.… [find](https://scholar.google.com/scholar?q=Ting+Liu%2C+Xueli+Pan%2C+Xu+Wang%2C+K+Anton+Feenstra%2C+Jaap+Heringa%2C+and+Zhisheng+Huang%2E+2021%2E+Exploring+the+microbiota%2Dgut%2Dbrain+axis+for+mental+disorders+with+Erik+M+Van+Mulligen%2C+Annie+Fourrier%2DReglat%2C+Da)
50. Ruili Wang, Feng Hou, Steven F Cahan, Li Chen, Xiaoyun Jia, and Wanting Ji. 2022. Fine-grained entity typing with a type taxonomy: a systematic review. IEEE Transactions on Knowledge and Data Engineering, 35(5):4794–4812. [find](https://scholar.google.com/scholar?q=Ruili+Wang%2C+Feng+Hou%2C+Steven+F+Cahan%2C+Li+Chen%2C+Xiaoyun+Jia%2C+and+Wanting+Ji%2E+2022%2E+Fine%2Dgrained+entity+typing+with+a+type+taxonomy%3A+a+systematic+review%2E+IEEE+Transactions+on+Knowledge+and+Data+Engineer)
51. Xuan Wang, Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitnik, Jingbo Shang, Curtis Langlotz, and Jiawei Han. 2019. Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics, 35(10):1745–1752. [find](https://scholar.google.com/scholar?q=Xuan+Wang%2C+Yu+Zhang%2C+Xiang+Ren%2C+Yuhao+Zhang%2C+Marinka+Zitnik%2C+Jingbo+Shang%2C+Curtis+Langlotz%2C+and+Jiawei+Han%2E+2019%2E+Cross%2Dtype+biomedical+named+entity+recognition+with+deep+multi%2Dtask+learning%2E+Bioinfor)
52. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 764–777, Flor… [find](https://scholar.google.com/scholar?q=Yuan+Yao%2C+Deming+Ye%2C+Peng+Li%2C+Xu+Han%2C+Yankai+Lin%2C+Zhenghao+Liu%2C+Zhiyuan+Liu%2C+Lixin+Huang%2C+Jie+Zhou%2C+and+Maosong+Sun%2E+2019%2E+DocRED%3A+A+large%2Dscale+document%2Dlevel+relation+extraction+dataset%2E+In+Proceedi)
53. Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. [LinkBERT: Pretraining Language Models with Document Links](https://md.research.elamur.ai/free/en/papers/yasunaga-2022-linkbert-pretraining-language-models-document/). In Association for Computational Linguistics (ACL).
54. Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. 2024. [GLiNER: Generalist model for named entity recognition using bidirectional transformer](https://md.research.elamur.ai/free/en/papers/zaratiana-2024-gliner-generalist-model-named-entity/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
55. Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-level relation extraction with adaptive thresholding and localized context pooling. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14612–14620. [find](https://scholar.google.com/scholar?q=Wenxuan+Zhou%2C+Kevin+Huang%2C+Tengyu+Ma%2C+and+Jing+Huang%2E+2021%2E+Document%2Dlevel+relation+extraction+with+adaptive+thresholding+and+localized+context+pooling%2E+In+Proceedings+of+the+AAAI+conference+on+artifi)

*Updated 31 October 2026*