---
title: "PharmaQA.IT — paper by Zeinalipour et al. (2026)"
description: "The authors present PharmaQA.IT, an extractive QA benchmark derived from RCPs in the Silver portion of… 0.679 EM. 17 references."
authors: ["Kamyar Zeinalipour", "Andrea Zugarini", "Asya Zanollo", "Leonardo Rigutini"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)"
source: "https://aclanthology.org/2026.eacl-industry.70/"
doi: "10.18653/v1/2026.eacl-industry.70"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zeinalipour-2026-pharmaqa-it-italian-dataset-q/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zeinalipour-2026-pharmaqa-it-italian-dataset-q/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/44/) / PharmaQA.IT: an Italian dataset for Q&A in the pharmaceutical domain

# PharmaQA.IT: an Italian dataset for Q&A in the pharmaceutical domain

**Authors:** [Kamyar Zeinalipour](https://md.research.elamur.ai/free/en/authors/kamyar-zeinalipour/), Andrea Zugarini, Asya Zanollo and Leonardo Rigutini · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-industry.70/) · [PDF](https://aclanthology.org/2026.eacl-industry.70.pdf) · [DOI 10.18653/v1/2026.eacl-industry.70](https://doi.org/10.18653/v1/2026.eacl-industry.70) · [OpenAlex](https://openalex.org/W7140088886)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LLM-based medical question answering needs test sets grounded in authoritative documents, but almost all such benchmarks are in English. In Italy, the official product summaries (RCP) published by AIFA are the reference source on medicines, yet no QA resource had been built on them. The authors present PharmaQA.IT, an extractive QA benchmark derived from RCPs in the Silver portion of PharmaER.IT. Their semi-automatic pipeline turns leaflet pages into images and selects representative pages by content type. The multimodal model Qwen3-VL-235B-A22B then proposes question–answer pairs in simple or complex variants, each time adopting a randomly drawn pharmaceutical professional persona. Native-Italian linguists then accepted or rejected each pair against comprehensibility, completeness, relevance and grounding criteria. The resulting pairs cover dosage, contraindications, interactions, warnings and pharmacological properties, with structured JSON answers and evidence. Under a standard RAG baseline with varied chunking, strong open models get close to proprietary systems. Data, code and prompts are released on Hugging Face, filling a gap in Italian biomedical NLP.

## Key points
- Source material: 1,077 RCP leaflets of at least 10 pages from the PharmaER.IT Silver corpus. The final dataset holds 861 validated QA pairs and is intended mainly for evaluation, not training.
- Generation used Qwen3-VL-235B-A22B on page images. Complex prompt variants were chosen with 25% probability, and validation was a binary accept/discard decision by independent annotators.
- The retrieval baseline uses multilingual-e5-base with top-k = 10. Retrieval hit@10 rises with chunk size: 0.436 at 64–16, 0.511 at 128–32, 0.580 at 256–64 and 0.615 at 512–128.
- Among closed-source models at 512–128 chunking, DeepSeek-Chat scored best (ROUGE-L F1 0.679, EM 0.432), followed by GPT-4.1 (0.636 / 0.367). Gemini-2.5-Flash was lowest (0.329 / 0.005).
- Stated limitations:
  - The benchmark is text-only, since figures and tables were ignored.
  - Answers are single-document factoids, with no multi-hop or unanswerable questions.
  - The annotator pool was small.
  - Questions may carry stylistic bias from the generating model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Siena](https://md.research.elamur.ai/free/en/universities/university-of-siena/) — University, Siena, Italy · 10 works · 198 citations

Also on the byline: [Istituto Universitario di Studi Superiori di Pavia](https://ror.org/0290wsh42).

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 263 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 260 works

## Cite

Cite as: Kamyar Zeinalipour, Andrea Zugarini, Asya Zanollo and Leonardo Rigutini. 2026. PharmaQA.IT: an Italian dataset for Q&A in the pharmaceutical domain. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zeinalipour2026pharmaqa,
  title = {{PharmaQA.IT: an Italian dataset for Q\&A in the pharmaceutical domain}},
  author = {Kamyar Zeinalipour, Andrea Zugarini, Asya Zanollo and Leonardo Rigutini},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.70},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.70}
}
```

- [← previous PFW at SemEval-2026 Task 6: Multi-Seed DeBERTa Ensembles for Political Response Clarity and Evasion Classification](https://md.research.elamur.ai/free/en/papers/tamsal-2026-pfw-semeval-2026-task-6/)
- [next → Phase 1 Implementation of LLM-generated Discharge Summaries showing high Adoption in a Dutch Academic Hospital](https://md.research.elamur.ai/free/en/papers/nadalini-2026-phase-1-implementation-llm-generated/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/44/)

## References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Alves and Luisa Coheur. 2022. Bootstrapped distant supervision for named entity recognition. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Daniel+Alves+and+Luisa+Coheur%2E+2022%2E+Bootstrapped+distant+supervision+for+named+entity+recognition%2E+In+Proceedings+of+LREC%2E)
2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020a. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th 5 Villanova: [https://www.opencup.gov.it/portale/](https://www.opencup.gov.it/portale/) web/opencup/home/progetto/-/cup/B29J24000850005 6 RESPIRA: https://www.opencup.gov.it/portale/ we… [↗](https://www.opencup.gov.it/portale/)
3. Annual Meeting of the Association for Computational Linguistics, pages 4623–4637. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Holger Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020b. Retrieval-augmented generation for knowledge… — [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/)
4. J. Li and 1 others. 2016. Biocreative v cdr task corpus. In Proceedings of BioCreative. [find](https://scholar.google.com/scholar?q=J%2E+Li+and+1+others%2E+2016%2E+Biocreative+v+cdr+task+corpus%2E+In+Proceedings+of+BioCreative%2E)
5. Bernardo Magnini and 1 others. 2006. I-cab: the italian content annotation bank. In Proceedings of CLiC-it. [find](https://scholar.google.com/scholar?q=Bernardo+Magnini+and+1+others%2E+2006%2E+I%2Dcab%3A+the+italian+content+annotation+bank%2E+In+Proceedings+of+CLiC%2Dit%2E)
6. Telmo Menezes and Benjamin Roth. 2019. Distant supervision for ner: A systematic study. arXiv. [find](https://scholar.google.com/scholar?q=Telmo+Menezes+and+Benjamin+Roth%2E+2019%2E+Distant+supervision+for+ner%3A+A+systematic+study%2E+arXiv%2E)
7. Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. [emrqa: A large corpus for question answering on electronic medical records](https://md.research.elamur.ai/free/en/papers/pampari-2018-emrqa-large-corpus-question-answering/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2357–2368.
8. Andrea Zugarini, Kamyar Zeinalipour, Surya Sai Kadali, Marco Maggini, Marco Gori, and Leonardo Rigutini. 2024b. Clue-instruct: Text-based clue generation for educational crossword puzzles. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and… [find](https://scholar.google.com/scholar?q=Andrea+Zugarini%2C+Kamyar+Zeinalipour%2C+Surya+Sai+Kadali%2C+Marco+Maggini%2C+Marco+Gori%2C+and+Leonardo+Rigutini%2E+2024b%2E+Clue%2Dinstruct%3A+Text%2Dbased+clue+generation+for+educational+crossword+puzzles%2E+In+Proceedi)
9. Chris Quirk and 1 others. 2016. Distant supervision for clinical ner. Journal of Biomedical Informatics. [find](https://scholar.google.com/scholar?q=Chris+Quirk+and+1+others%2E+2016%2E+Distant+supervision+for+clinical+ner%2E+Journal+of+Biomedical+Informatics%2E)
10. N. Ringland and 1 others. 2019. Nne: A distantly supervised named entity dataset. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=N%2E+Ringland+and+1+others%2E+2019%2E+Nne%3A+A+distantly+supervised+named+entity+dataset%2E+In+Proceedings+of+ACL%2E)
11. Simone Tedeschi and Roberto Navigli. 2022. MultiN-ERD: A multilingual, multi-genre and fine-grained dataset for named entity recognition (and disambiguation). In Findings of the Association for Computational Linguistics: NAACL 2022, pages 801–812, Seattle, United States. Association for Computationa… [find](https://scholar.google.com/scholar?q=Simone+Tedeschi+and+Roberto+Navigli%2E+2022%2E+MultiN%2DERD%3A+A+multilingual%2C+multi%2Dgenre+and+fine%2Dgrained+dataset+for+named+entity+recognition+%28and+disambiguation%29%2E+In+Findings+of+the+Association+for+Comput)
12. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, and 1 others. 2015a. An overview of the bioasq large-scale biomedical semantic indexing and question… [find](https://scholar.google.com/scholar?q=George+Tsatsaronis%2C+Georgios+Balikas%2C+Prodromos+Malakasiotis%2C+Ioannis+Partalas%2C+Matthias+Zschunke%2C+Michael+R+Alvers%2C+Dirk+Weissenborn%2C+Anastasia+Krithara%2C+Sergios+Petridis%2C+Dimitris+Polychronopoulos%2C+)
13. George Tsatsaronis and 1 others. 2015b. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics. [find](https://scholar.google.com/scholar?q=George+Tsatsaronis+and+1+others%2E+2015b%2E+An+overview+of+the+bioasq+large%2Dscale+biomedical+semantic+indexing+and+question+answering+competition%2E+BMC+Bioinformatics%2E)
14. Jiawei Zhou, Yu Su, Yijia Wang, Junghyun Chung, Chen Li, Huan Chen, and Xiang Ren. 2023. Universalner: A universal toolkit for cross-domain and multilingual named entity recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Jiawei+Zhou%2C+Yu+Su%2C+Yijia+Wang%2C+Junghyun+Chung%2C+Chen+Li%2C+Huan+Chen%2C+and+Xiang+Ren%2E+2023%2E+Universalner%3A+A+universal+toolkit+for+cross%2Ddomain+and+multilingual+named+entity+recognition%2E+In+Proceedings+of)
15. Andrea Zugarini and Leonardo Rigutini. 2025a. Pharmaer.it: An italian dataset for entity recognition in the pharmaceutical domain. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), pages 1171–1180. CEUR Workshop Proceedings. [find](https://scholar.google.com/scholar?q=Andrea+Zugarini+and+Leonardo+Rigutini%2E+2025a%2E+Pharmaer%2Eit%3A+An+italian+dataset+for+entity+recognition+in+the+pharmaceutical+domain%2E+In+Proceedings+of+the+Eleventh+Italian+Conference+on+Computational+Li)
16. Andrea Zugarini and Leonardo Rigutini. 2025b. Pharmaer.it: an italian dataset for named entity recognition in the pharmaceutical domain. In Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025)., Cagliari, Italy. CEUR Workshop Proceedings. [find](https://scholar.google.com/scholar?q=Andrea+Zugarini+and+Leonardo+Rigutini%2E+2025b%2E+Pharmaer%2Eit%3A+an+italian+dataset+for+named+entity+recognition+in+the+pharmaceutical+domain%2E+In+Eleventh+Italian+Conference+on+Computational+Linguistics+%28CL)
17. Andrea Zugarini, Kamyar Zeinalipour, Achille Fusco, and Asya Zanollo. 2024a. ECWCA - educational CrossWord clues answering: A CALAMITA challenge. In Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024), pages 1239–1244, Pisa, Italy. CEUR Workshop Proceedings. [find](https://scholar.google.com/scholar?q=Andrea+Zugarini%2C+Kamyar+Zeinalipour%2C+Achille+Fusco%2C+and+Asya+Zanollo%2E+2024a%2E+ECWCA+%2D+educational+CrossWord+clues+answering%3A+A+CALAMITA+challenge%2E+In+Proceedings+of+the+Tenth+Italian+Conference+on+Comp)

*Updated 2 November 2026*