---
title: "LingVarBench — paper by Mohammadi et al. (2026)"
description: "The authors from Infinitus Systems propose LingVarBench, a synthetic data pipeline and benchmark specification with three… 24 references."
authors: ["Seyedali Mohammadi", "Manas Paldhe", "Amit Chhabra", "Youngseo Son", "Vishal Seshagiri"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)"
source: "https://aclanthology.org/2026.eacl-industry.42/"
doi: "10.18653/v1/2026.eacl-industry.42"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mohammadi-2026-lingvarbench-benchmarking-llms-entity-recognitions/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mohammadi-2026-lingvarbench-benchmarking-llms-entity-recognitions/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/) / LingVarBench: Benchmarking LLMs on Entity Recognitions and Linguistic Verbalization Patterns in Phone-Call Transcripts

# LingVarBench: Benchmarking LLMs on Entity Recognitions and Linguistic Verbalization Patterns in Phone-Call Transcripts

**Authors:** [Seyedali Mohammadi](https://md.research.elamur.ai/free/en/authors/seyedali-mohammadi/), [Manas Paldhe](https://md.research.elamur.ai/free/en/authors/manas-paldhe/), Amit Chhabra, [Youngseo Son](https://md.research.elamur.ai/free/en/authors/youngseo-son/) and Vishal Seshagiri · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-industry.42/) · [PDF](https://aclanthology.org/2026.eacl-industry.42.pdf) · [DOI 10.18653/v1/2026.eacl-industry.42](https://doi.org/10.18653/v1/2026.eacl-industry.42) · [OpenAlex](https://openalex.org/W7140077301)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Extracting structured fields such as names, birth dates or ZIP codes from phone-call transcripts is difficult. Callers hesitate, correct themselves and read values in many different ways. In healthcare, privacy rules (HIPAA) also make real annotated calls scarce and costly. The authors from Infinitus Systems propose LingVarBench, a synthetic data pipeline and benchmark specification with three stages:
- an LLM proposes plausible entity values from a field description;
- a second step renders each value as spoken-style text according to curated linguistic verbalization patterns (LVPs), such as digit-by-digit reading, grouping, repetitions, fillers and self-corrections;
- a consistency checker removes transcripts that do not clearly encode the intended value.

The filtered data is then used by DSPy's SIMBA optimizer to build extraction prompts automatically, without touching protected health information. On real production calls, these prompts beat zero-shot prompting. They reach roughly the level of prompts that humans tuned on real transcripts, with F1 around 94–95 for ZIP, DOB and name. This makes the approach especially useful before a product has collected any real data.

## Key points
- Six entities are evaluated: ZIP code, name, DOB, pain rating, respiratory issues and hearing issues. The last three are subjective questionnaire items, where optimized prompts clearly improve over zero-shot.
- Three LLMs are used throughout: GPT 4, Gemini 2.0 Flash and Gemini 2.5 Pro. The human-written baseline prompts were tuned specifically for Gemini 2.0 Flash.
- The pipeline is a composition G(d, v) = C(T(V(d), v)), combining a Value Generator, a Transcript Generator and a Consistency Checker. Recursive balancing ensures even coverage of value–variation pairs.
- Cosine similarity between real and synthetic transcripts was measured with gemini-embedding-001 and text-embedding-3-large. Similarity rises as the LVPs of synthetic and real data align more closely (Match > Super > Sub > Null).
- Neither real transcripts nor the synthetic dataset are released; only the entity schemas, LVP inventory and generation/validation protocol are documented. The framework currently covers only direct answers, not refusals or off-topic replies.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🏷️ Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/) · 214 works

## Cite

Cite as: Seyedali Mohammadi, Manas Paldhe, Amit Chhabra, Youngseo Son and Vishal Seshagiri. 2026. LingVarBench: Benchmarking LLMs on Entity Recognitions and Linguistic Verbalization Patterns in Phone-Call Transcripts. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mohammadi2026lingvarbench,
  title = {{LingVarBench: Benchmarking LLMs on Entity Recognitions and Linguistic Verbalization Patterns in Phone-Call Transcripts}},
  author = {Seyedali Mohammadi, Manas Paldhe, Amit Chhabra, Youngseo Son and Vishal Seshagiri},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.42},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.42}
}
```

- [← previous Linguistics to LLMs: Teaching with and about Chatbots](https://md.research.elamur.ai/free/en/papers/pado-2026-linguistics-llms-teaching-about-chatbots/)
- [next → Link Prediction over N-ary Knowledge Graphs: A Survey](https://md.research.elamur.ai/free/en/papers/wei-2026-link-prediction-over/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/39/)

## References

24 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Oluwatomisin Arokodare, Hayden Wimmer, and Jie Du. 2025. Clinical Text Summarization using NLP Pre-trained Language Models: A Case Study of MIMIC-IV-Notes. Journal of Information Systems Applied Research and Analytics, 18(1):17–31. [find](https://scholar.google.com/scholar?q=Oluwatomisin+Arokodare%2C+Hayden+Wimmer%2C+and+Jie+Du%2E+2025%2E+Clinical+Text+Summarization+using+NLP+Pre%2Dtrained+Language+Models%3A+A+Case+Study+of+MIMIC%2DIV%2DNotes%2E+Journal+of+Information+Systems+Applied+Resea)
2. Augnito. 2024. Voice ai in healthcare: Statistics and trends for 2024. [https://augnito.ai/resources/](https://augnito.ai/resources/) stats-on-voice-ai-in-healthcare/. [↗](https://augnito.ai/resources/)
3. Jianzhu Bao, Rui Wang, Yasheng Wang, Aixin Sun, Yitong Li, Fei Mi, and Ruifeng Xu. 2023. A synthetic data generation framework for grounded dialogues. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10866–10882, Toronto, Canad… [find](https://scholar.google.com/scholar?q=Jianzhu+Bao%2C+Rui+Wang%2C+Yasheng+Wang%2C+Aixin+Sun%2C+Yitong+Li%2C+Fei+Mi%2C+and+Ruifeng+Xu%2E+2023%2E+A+synthetic+data+generation+framework+for+grounded+dialogues%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+)
4. Tom B Brown, Benjamin Mann, Nick Ryder, and et al. 2020a. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877–1901.
5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Saxena, Amanda Bosma, and 1 others. 2020b. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curr…
6. Sam Davidson, Jordan Hosier, Yu Zhou, and Vijay Gurbani. 2021. Improved named entity recognition for noisy call center transcripts. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 361–370, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sam+Davidson%2C+Jordan+Hosier%2C+Yu+Zhou%2C+and+Vijay+Gurbani%2E+2021%2E+Improved+named+entity+recognition+for+noisy+call+center+transcripts%2E+In+Proceedings+of+the+Seventh+Workshop+on+Noisy+User%2Dgenerated+Text+)
7. Steffen Eger, Abdelrahman Youssef, and Iryna Gurevych. 2020. Is it time to swish? comparing deep learning activation functions across nlp tasks. In Proceedings of EMNLP, pages 4263–4273. [find](https://scholar.google.com/scholar?q=Steffen+Eger%2C+Abdelrahman+Youssef%2C+and+Iryna+Gurevych%2E+2020%2E+Is+it+time+to+swish%3F+comparing+deep+learning+activation+functions+across+nlp+tasks%2E+In+Proceedings+of+EMNLP%2C+pages+4263%E2%80%934273%2E)
8. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. [Generative adversarial nets](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-generative-adversarial-nets/). In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
9. Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari, and Mohammad Norouzi. 2022. [Generate, annotate, and learn: NLP with synthetic text](https://md.research.elamur.ai/free/en/papers/he-2022-generate-annotate-learn-nlp-synthetic/). Transactions of the Association for Computational Linguistics, 10:826–842.
10. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, and 1 others. 2024. [Dspy: Compiling declarative language model calls into self-improving pipelines](https://md.research.elamur.ai/free/en/papers/khattab-2023-dspy-compiling-declarative-language-model/). In The Twelfth International Conference o…
11. Hyunjae Kim and Jaewoo Kang. 2022. How do your biomedical named entity recognition models generalize to novel entities? IEEE Access, 10:31513–31523. [find](https://scholar.google.com/scholar?q=Hyunjae+Kim+and+Jaewoo+Kang%2E+2022%2E+How+do+your+biomedical+named+entity+recognition+models+generalize+to+novel+entities%3F+IEEE+Access%2C+10%3A31513%E2%80%9331523%2E)
12. Liu Liu, Zhaoyuan Wu, Yirong Wu, Yuxia Wang, Rui Yao, Xin Li, Jiani Hu, Lixia Ruan, and Yi Zhou. 2025. Using natural language processing to extract information from clinical text in electronic medical records for populating clinical registries: a systematic review. Journal of the American Medical In… [find](https://scholar.google.com/scholar?q=Liu+Liu%2C+Zhaoyuan+Wu%2C+Yirong+Wu%2C+Yuxia+Wang%2C+Rui+Yao%2C+Xin+Li%2C+Jiani+Hu%2C+Lixia+Ruan%2C+and+Yi+Zhou%2E+2025%2E+Using+natural+language+processing+to+extract+information+from+clinical+text+in+electronic+medical)
13. Foley & Lardner LLP. 2024. Hipaa compliance for ai in digital health: What privacy officers need to know. Overview of HIPAA requirements for AI systems handling PHI in digital health. [find](https://scholar.google.com/scholar?q=Foley+%26+Lardner+LLP%2E+2024%2E+Hipaa+compliance+for+ai+in+digital+health%3A+What+privacy+officers+need+to+know%2E+Overview+of+HIPAA+requirements+for+AI+systems+handling+PHI+in+digital+health%2E)
14. Seyedali Mohammadi, Bhaskara Hanuma Vedula, Hemank Lamba, Edward Raff, Ponnurangam Kumaraguru, Francis Ferraro, and Manas Gaur. 2025. Do llms adhere to label definitions? examining their receptivity to external label definitions. In Proceedings of the 2025 Conference on Empirical Methods in Natural… [find](https://scholar.google.com/scholar?q=Seyedali+Mohammadi%2C+Bhaskara+Hanuma+Vedula%2C+Hemank+Lamba%2C+Edward+Raff%2C+Ponnurangam+Kumaraguru%2C+Francis+Ferraro%2C+and+Manas+Gaur%2E+2025%2E+Do+llms+adhere+to+label+definitions%3F+examining+their+receptivity+t)
15. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744.
16. Grand View Research. 2024. Ai voice agents in healthcare market \| industry report, 2030. [https://www.grandviewresearch.com/industry-analysis/](https://www.grandviewresearch.com/industry-analysis/) ai-voice-agents-healthcare-market-report. Market size estimated at $468M in 2024, projected CAGR of 37.79% through 2030. [↗](https://www.grandviewresearch.com/industry-analysis/)
17. German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez. 2016. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3234–3243. [find](https://scholar.google.com/scholar?q=German+Ros%2C+Laura+Sellart%2C+Joanna+Materzynska%2C+David+Vazquez%2C+and+Antonio+M%2E+Lopez%2E+2016%2E+The+synthia+dataset%3A+A+large+collection+of+synthetic+images+for+semantic+segmentation+of+urban+scenes%2E+In+2016)
18. Andrew Rosenberg, Yu Zhang, Bhuvana Ramabhadran, Ye Jia, Pedro Moreno, Yonghui Wu, and Zelin Wu. 2019. Speech recognition with augmented synthesized speech. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 996–1002. [find](https://scholar.google.com/scholar?q=Andrew+Rosenberg%2C+Yu+Zhang%2C+Bhuvana+Ramabhadran%2C+Ye+Jia%2C+Pedro+Moreno%2C+Yonghui+Wu%2C+and+Zelin+Wu%2E+2019%2E+Speech+recognition+with+augmented+synthesized+speech%2E+In+2019+IEEE+Automatic+Speech+Recognition+a)
19. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. [Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/). In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147, Edmonton, Canada. Association for Computational Lingui…
20. Yuxin Wang, Duanyu Feng, Yongfu Dai, Zhengyu Chen, Jimin Huang, Sophia Ananiadou, Qianqian Xie, and Hao Wang. 2024. Harmonic: Harnessing llms for tabular data synthesis and privacy protection. Advances in Neural Information Processing Systems, 37:100196–100212. [find](https://scholar.google.com/scholar?q=Yuxin+Wang%2C+Duanyu+Feng%2C+Yongfu+Dai%2C+Zhengyu+Chen%2C+Jimin+Huang%2C+Sophia+Ananiadou%2C+Qianqian+Xie%2C+and+Hao+Wang%2E+2024%2E+Harmonic%3A+Harnessing+llms+for+tabular+data+synthesis+and+privacy+protection%2E+Advance)
21. Jason Wei, Xuezhi Wang, Dale Schuurmans, and et al. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24824–24837.
22. Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2023. [Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts](https://md.research.elamur.ai/free/en/papers/xie-2023-adaptive-chameleon-stubborn-sloth-revealing/). In The Twelfth International Conference on Learning Representations.
23. Xiao Zhan, Noura Abdi, William Seymour, and Jose Such. 2024. Healthcare voice ai assistants: Factors influencing trust and intention to use. Proc. ACM Hum.-Comput. Interact., 8(CSCW1). [find](https://scholar.google.com/scholar?q=Xiao+Zhan%2C+Noura+Abdi%2C+William+Seymour%2C+and+Jose+Such%2E+2024%2E+Healthcare+voice+ai+assistants%3A+Factors+influencing+trust+and+intention+to+use%2E+Proc%2E+ACM+Hum%2E%2DComput%2E+Interact%2E%2C+8%28CSCW1%29%2E)
24. Jian Zhang, Junyi Guo, Junyi Yuan, Huanda Lu, Yanlin Zhou, Fangyu Wu, Qiufeng Wang, and Dongming Lu. 2025. Llm-driven completeness and consistency evaluation for cultural heritage data augmentation in cross-modal retrieval. In Proceedings of the 2025 Conference on Empirical Methods in Natural Langua… [find](https://scholar.google.com/scholar?q=Jian+Zhang%2C+Junyi+Guo%2C+Junyi+Yuan%2C+Huanda+Lu%2C+Yanlin+Zhou%2C+Fangyu+Wu%2C+Qiufeng+Wang%2C+and+Dongming+Lu%2E+2025%2E+Llm%2Ddriven+completeness+and+consistency+evaluation+for+cultural+heritage+data+augmentation+in)

*Updated 2 November 2026*