---
title: "Evaluating Large Language Models for Medical Named Entity"
description: "First, they build what they describe as the first manually annotated Urdu medical NER dataset: 2,057… 15 references, cited by 1 paper here."
authors: ["Bushra Nasim", "Kinza Latif", "Muhammad Zohair", "Muhammad Hassan Asif", "Zarmeen Nasim"]
year: 2026
venue: "Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)"
source: "https://aclanthology.org/2026.chipsal-1.5/"
doi: "10.63317/5iobohdgnfh4"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nasim-2026-evaluating-large-language-models-medical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nasim-2026-evaluating-large-language-models-medical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/) / Evaluating Large Language Models for Medical Named Entity Recognition in Urdu: A Benchmark Study

# Evaluating Large Language Models for Medical Named Entity Recognition in Urdu: A Benchmark Study

**Authors:** Bushra Nasim, Kinza Latif, Muhammad Zohair, Muhammad Hassan Asif and Zarmeen Nasim · **Year:** 2026 · **Venue:** [Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)](https://md.research.elamur.ai/free/en/venues/chipsal/) ([workshop on Challenges in Processing South Asian Languages 2026](https://md.research.elamur.ai/free/en/venues/chipsal/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.chipsal-1.5/) · [PDF](https://aclanthology.org/2026.chipsal-1.5.pdf) · [DOI 10.63317/5iobohdgnfh4](https://doi.org/10.63317/5iobohdgnfh4) · [OpenAlex](https://openalex.org/W7166536125)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/nasim-2026-evaluating-large-language-models-medical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Urdu is spoken by roughly 300 million people but has very few labeled resources for pulling medical concepts out of text. It has also been unclear how well general-purpose LLMs handle such extraction without any adaptation. The authors address both gaps. First, they build what they describe as the first manually annotated Urdu medical NER dataset: 2,057 health-related items from a public Urdu news headline collection. Annotators with medical-science training tagged these in LabelBox with five entity types: Disease, Symptom, Medication, Treatment and Body Part. Second, they test ChatGPT-4o and LLaMA 3.2 zero-shot, giving each a structured prompt through its web interface, and score the outputs with precision, recall and F1. Both models produce many incorrect extractions, so precision is weak while recall is only middling. Overall micro-F1 stays below 0.19 for each model, and Treatment mentions are almost never identified correctly. The work offers a first baseline and dataset for Urdu clinical NLP. The authors argue that fine-tuning, few-shot learning, transfer learning and better prompt design are needed for usable performance.

## Key points
- **Dataset:** 2,057 Urdu news items (headlines with brief summaries) drawn from the public Urdu-News-Headline-Dataset on GitHub, labeled in LabelBox across five medical entity categories.
- **Annotation quality:** Two annotators independently labeled the first 1,000 items, reaching an inter-annotator F1 of 0.91 and 90.56% exact agreement. Conflicts were settled through group discussion, and a single annotator labeled the remaining 1,057 items.
- **Overall results:** Micro-averaged F1 was 0.187 for ChatGPT-4o and 0.183 for LLaMA 3.2, so the two models performed about equally.
- **Per-category results:** Disease was the best category (F1 0.35 for ChatGPT-4o, 0.33 for LLaMA 3.2). Treatment was the worst (F1 0.011 for ChatGPT-4o, 0.036 for LLaMA 3.2).
- **Setup:** Pure zero-shot evaluation via web interfaces, with no fine-tuning or parameter-efficient methods. The paper appeared at CHiPSAL 2026 (co-located with LREC).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [Dow University of Health Sciences](https://ror.org/01h85hm56) · [Aga Khan University](https://ror.org/03gd0dm95).

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 234 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 231 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🏷️ Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/) · 200 works

## Cite

Cite as: Bushra Nasim, Kinza Latif, Muhammad Zohair, Muhammad Hassan Asif and Zarmeen Nasim. 2026. Evaluating Large Language Models for Medical Named Entity Recognition in Urdu: A Benchmark Study. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nasim2026evaluating,
  title = {{Evaluating Large Language Models for Medical Named Entity Recognition in Urdu: A Benchmark Study}},
  author = {Bushra Nasim, Kinza Latif, Muhammad Zohair, Muhammad Hassan Asif and Zarmeen Nasim},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/5iobohdgnfh4},
  url = {https://doi.org/10.63317/5iobohdgnfh4}
}
```

- [← previous Evaluating Encoder- and LLM-Based Approaches for Robust Indirect Personal Identifier Detection](https://md.research.elamur.ai/free/en/papers/otto-2026-evaluating-encoder-llm-based-approaches/)
- [next → Evaluating Linguistic Knowledge of LLMs in Tamil: The ILAKKANAM Benchmark](https://md.research.elamur.ai/free/en/papers/varsha-2026-evaluating-linguistic-knowledge-llms-tamil/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/)

## References

15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmed, A., Huang, D., & Arafat, S. Y. (2024). Enriching Urdu NER with BERT Embedding, Data Augmentation, and Hybrid Encoder-CNN Architecture. ACM Transactions on Asian and Low-Resource Language Information Processing, 23(4). https://doi.org/10.1145/3648362 — [doi:10.1145/3648362](https://doi.org/10.1145/3648362) [↗](https://doi.org/10.1145/3648362)
2. Alsentzer, E., Murphy, J. R., Boag, W., Weng, W.-H., Jin, D., Naumann, T., & McDermott, M. B. A. (2019). [Publicly Available Clinical BERT Embeddings](https://md.research.elamur.ai/free/en/papers/alsentzer-2019-publicly-available-clinical-bert-embeddings/). https://arxiv.org/abs/1904.03323v3 [↗](https://arxiv.org/abs/1904.03323v3)
3. Anam, R., Anwar, M. W., Jamal, M. H., Bajwa, U. I., De la Torre Diez, I., Alvarado, E. S., Flores, E. S., & Ashraf, I. (2024). [A deep learning approach for Named Entity Recognition in Urdu language](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). PLOS ONE, 19(3), e0300725. https://doi.org/10.1371/JOURNAL.PONE. 0300725 Anil, R., Dai, A. M., Firat,… [↗](https://doi.org/10.1371/JOURNAL.PONE)
4. Coleman, J., Krishnamachari, B., Iskarous, K., & Rosales, R. (2024). [LLM-Assisted Rule Based Machine Translation for Low/No-Resource Languages](https://md.research.elamur.ai/free/en/papers/coleman-2024-llm-assisted-rule-based-machine/). 67–87. https://doi.org/10.18653/v1/2024.america snlp-1.9 Deng, X., Bashlovkina, V., Han, F., Baumgartner, S., & Bendersky, M. (2023). LLMs to the Moon? Redd… [↗](https://doi.org/10.18653/v1/2024.america)
5. ACM Web Conference 2023 -Companion of the World Wide Web Conference, WWW 2023, 1014–1019. https://doi.org/10.1145/3543873.3587605 — [doi:10.1145/3543873.3587605](https://doi.org/10.1145/3543873.3587605) [↗](https://doi.org/10.1145/3543873.3587605)
6. Elshin, D., Karpachev, N., Gruzdev, B., Golovanov, I., Ivanov, G., Antonov, A., Skachkov, N., Latypova, E., Layner, V., Enikeeva, E., Popov, D., Chekashev, A., Negodin, V., Frantsuzova, V., Chernyshev, A., & Denisov, K. (2024). From General LLM to Translation: How We Dramatically Improve Translation… [find](https://scholar.google.com/scholar?q=Elshin%2C+D%2E%2C+Karpachev%2C+N%2E%2C+Gruzdev%2C+B%2E%2C+Golovanov%2C+I%2E%2C+Ivanov%2C+G%2E%2C+Antonov%2C+A%2E%2C+Skachkov%2C+N%2E%2C+Latypova%2C+E%2E%2C+Layner%2C+V%2E%2C+Enikeeva%2C+E%2E%2C+Popov%2C+D%2E%2C+Chekashev%2C+A%2E%2C+Negodin%2C+V%2E%2C+Frantsuzova%2C+V%2E%2C+Chernyshev)
7. Iwakami, Y., Takuma, H., & Iwashita, M. (2024). Complementary Method for NER by Extracting Proper Nouns from Images when Utilizing LLM. 2024 IEEE/ACIS 9th International Conference on Big Data, Cloud Computing, and Data Science (BCD), 233–238. https://doi.org/10.1109/BCD61269.2024.1 0743114 Jahan, I.… — [doi:10.1109/bcd61269.2024.1](https://doi.org/10.1109/bcd61269.2024.1) [↗](https://doi.org/10.1109/BCD61269.2024.1)
8. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016). [Neural Architectures for Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/lample-2016-neural-architectures-named-entity-recognition/). 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conferenc…
9. Leaman, R., Khare, R., & Lu, Z. (2015). Challenges in clinical natural language processing for automated disorder normalization. Journal of Biomedical Informatics, 57, 28–37. https://doi.org/10.1016/J.JBI.2015.07.010 — [doi:10.1016/j.jbi.2015.07.010](https://doi.org/10.1016/j.jbi.2015.07.010) [↗](https://doi.org/10.1016/J.JBI.2015.07.010)
10. Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020a). [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/). Bioinformatics, 36(4), 1234–1240. https://doi.org/10.1093/BIOINFORMATIC S/BTZ682 Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So,… [↗](https://doi.org/10.1093/BIOINFORMATIC)
11. Nasim, Z., Abidi, S., & Haider, S. (2020). Modeling POS Tagging for the Urdu Language. 2020 International Conference on Emerging Trends in Smart Technologies, ICETST 2020. https://doi.org/10.1109/ICETST49965.20 20.9080721 Nasim, Z., & Haider, S. (2022). Automatic Labeling of Clusters for a Low-Resou… — [doi:10.1109/icetst49965.20](https://doi.org/10.1109/icetst49965.20) [↗](https://doi.org/10.1109/ICETST49965.20)
12. Rohanian, O., Nouriborji, M., Kouchaki, S., Nooralahzadeh, F., Clifton, L., & Clifton, D. A. (2024). Exploring the effectiveness of instruction tuning in biomedical language processing. Artificial Intelligence in Medicine, 158, 103007. https://doi.org/10.1016/J.ARTMED.2024. 103007 Wang, Z., Zhou, Q.… — [doi:10.1016/j.artmed.2024](https://doi.org/10.1016/j.artmed.2024) [↗](https://doi.org/10.1016/J.ARTMED.2024)
13. Zhang, M., & Li, J. (2021). A commentary of GPT-3 in MIT Technology Review 2021. Fundamental Research, 1(6), 831–833. https://doi.org/10.1016/J.FMRE.2021.11. 011 Zhao, Y., Wang, C., & Fu, G. (2012). A CRF Sequence Labeling Approach to Chinese Punctuation Prediction. Pacific Asia Conference on Langua… — [doi:10.1016/j.fmre.2021.11](https://doi.org/10.1016/j.fmre.2021.11) [↗](https://doi.org/10.1016/J.FMRE.2021.11)
14. Zhu, Z., Zhao, Q., Li, J., Ge, Y., Ding, X., Gu, T., Zou, J., Lv, S., Wang, S., & Yang, J. J. (2024). Comparative Analysis of Large Language Models in Chinese Medical Named Entity Recognition. [find](https://scholar.google.com/scholar?q=Zhu%2C+Z%2E%2C+Zhao%2C+Q%2E%2C+Li%2C+J%2E%2C+Ge%2C+Y%2E%2C+Ding%2C+X%2E%2C+Gu%2C+T%2E%2C+Zou%2C+J%2E%2C+Lv%2C+S%2E%2C+Wang%2C+S%2E%2C+%26+Yang%2C+J%2E+J%2E+%282024%29%2E+Comparative+Analysis+of+Large+Language+Models+in+Chinese+Medical+Named+Entity+Recognition%2E)
15. Bioengineering 2024, Vol. 11, Page 982, 11(10), 982. https://doi.org/10.3390/BIOENGINEERIN G11100982 — [doi:10.3390/bioengineerin](https://doi.org/10.3390/bioengineerin) [↗](https://doi.org/10.3390/BIOENGINEERIN)

*Updated 31 October 2026*