---
title: "Lakefront AI Ramblers at MEDIQA-SYNUR 2026: Hybrid"
description: "The paper describes a system for the MEDIQA-SYNUR 2026 shared task. 0.711 F1. 24 references, cited by 1 paper here."
authors: ["Michael T. Saban", "Arsalan Yaghoubi", "Behnaz Eslami", "Samie Tootooni", "Dmitriy Dligach"]
year: 2026
venue: "Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026"
source: "https://aclanthology.org/2026.clinicalnlp-1.23/"
doi: "10.63317/2jfca3zi8p4o"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/saban-2026-lakefront-ai-ramblers-mediqa-synur/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/saban-2026-lakefront-ai-ramblers-mediqa-synur/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/) / Lakefront AI Ramblers at MEDIQA-SYNUR 2026: Hybrid Retrieval and LLM Verification for Open-Source Schema-Guided Clinical Information Extraction

# Lakefront AI Ramblers at MEDIQA-SYNUR 2026: Hybrid Retrieval and LLM Verification for Open-Source Schema-Guided Clinical Information Extraction

**Authors:** Michael T. Saban, Arsalan Yaghoubi, Behnaz Eslami, Samie Tootooni and [Dmitriy Dligach](https://md.research.elamur.ai/free/en/authors/dmitriy-dligach/) · **Year:** 2026 · **Venue:** [Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/clinicalnlp/) ([Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/clinicalnlp/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.clinicalnlp-1.23/) · [PDF](https://aclanthology.org/2026.clinicalnlp-1.23.pdf) · [DOI 10.63317/2jfca3zi8p4o](https://doi.org/10.63317/2jfca3zi8p4o) · [OpenAlex](https://openalex.org/W7166546167)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/saban-2026-lakefront-ai-ramblers-mediqa-synur/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes a system for the MEDIQA-SYNUR 2026 shared task. The task turns synthetic nursing dictations into structured JSON observations tied to a fixed clinical schema of 193 concepts. Scoring uses strict exact matching. The authors start from the official baseline, which segments transcripts, retrieves schema items and extracts with an LLM, and add three extensions:
- **Hybrid retrieval:** a weighted blend of dense embedding scores and BM25 scores to pick candidate concepts for each segment.
- **Fine-tuning:** LoRA-based supervised fine-tuning of open-weight Llama models, trained on transcript-level targets.
- **Post-processing:** deterministic schema validation, optional multi-run voting, and a separate LLM verifier that keeps or drops each extracted observation.

Hybrid retrieval brought schema coverage close to complete. Verification gave the largest F1 gains, especially for smaller fine-tuned models. On the test set, the best open-source submission reached 0.711 F1, while GPT-4o reached 0.791. The work shows that open models can come reasonably close to closed ones for privacy-sensitive clinical extraction. It also shows that post-processing gains on development data do not reliably carry over to test data.

## Key points
- The dataset has 122 training, 101 development and 199 test transcripts. Transcripts are split at each "[Clinician]" tag.
- Retrieval combines BAAI/bge-base-en-v1.5 with BM25 (α=0.6), keeping the top 60 schema items per segment. This gives recall@60 of 0.994 on the development set.
- The extractors were GPT-4o, GPT-4o-mini, Llama-3-8B-Instruct and Llama-3.3-70B-Instruct. The 70B model was fine-tuned with 4-bit QLoRA, and all training ran on a single NVIDIA H200.
- The best test submission was Llama-3.3-70B-Instruct-SFT with union voting and GPT-4o-mini verification, at 0.711 F1. After the competition, Llama-3-8B-Instruct-SFT trained on train + dev reached 0.723 F1.
- On the development set, verification by Llama-3.3-70B raised Llama-3-8B-SFT to 0.776 F1. GPT-4o (about 0.84 F1 on dev) did not benefit from post-processing.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Loyola University Chicago](https://md.research.elamur.ai/free/en/universities/loyola-university-chicago/) — University, Chicago, United States · 2 works · 1 citation

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 263 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 260 works

🧭 Topics (2): [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works · [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 175 works

## Cite

Cite as: Michael T. Saban, Arsalan Yaghoubi, Behnaz Eslami, Samie Tootooni and Dmitriy Dligach. 2026. Lakefront AI Ramblers at MEDIQA-SYNUR 2026: Hybrid Retrieval and LLM Verification for Open-Source Schema-Guided Clinical Information Extraction. Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{saban2026lakefront,
  title = {{Lakefront AI Ramblers at MEDIQA-SYNUR 2026: Hybrid Retrieval and LLM Verification for Open-Source Schema-Guided Clinical Information Extraction}},
  author = {Michael T. Saban, Arsalan Yaghoubi, Behnaz Eslami, Samie Tootooni and Dmitriy Dligach},
  booktitle = {Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2jfca3zi8p4o},
  url = {https://doi.org/10.63317/2jfca3zi8p4o}
}
```

- [← previous Label-Consistent Data Generation for Aspect-Based Sentiment Analysis Using LLM Agents](https://md.research.elamur.ai/free/en/papers/monfared-2026-label-consistent-data-generation-aspect/)
- [next → Lakksh at SemEval-2026 Task 11(1 & 2): Neuro-Symbolic Decomposition to Mitigate Content Bias in Syllogistic Reasoning](https://md.research.elamur.ai/free/en/papers/sharma-2026-lakksh-semeval-2026-task-11/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/)

## References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., & Anadkat, S. (2023). [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Bernal, J., & Mazo, C. (2022). Transparency of artificial intelligence in healthcare: insights from professionals in computing and healthcare worldwide. Applied Sciences, 12(20), 10228. [find](https://scholar.google.com/scholar?q=Bernal%2C+J%2E%2C+%26+Mazo%2C+C%2E+%282022%29%2E+Transparency+of+artificial+intelligence+in+healthcare%3A+insights+from+professionals+in+computing+and+healthcare+worldwide%2E+Applied+Sciences%2C+12%2820%29%2C+10228%2E)
3. Bruch, S., Gai, S., & Ingber, A. (2023). An analysis of fusion functions for hybrid retrieval. ACM Transactions on Information Systems, 42(1), 1-35. [find](https://scholar.google.com/scholar?q=Bruch%2C+S%2E%2C+Gai%2C+S%2E%2C+%26+Ingber%2C+A%2E+%282023%29%2E+An+analysis+of+fusion+functions+for+hybrid+retrieval%2E+ACM+Transactions+on+Information+Systems%2C+42%281%29%2C+1%2D35%2E)
4. Chen, X., Lakhotia, K., Oguz, B., Gupta, A., Lewis, P., Peshterliev, S., Mehdad, Y., Gupta, S., & Yih, W.-t. (2022). Salient phrase aware dense retrieval: can a dense retriever imitate a sparse one? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 250-262. [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Lakhotia%2C+K%2E%2C+Oguz%2C+B%2E%2C+Gupta%2C+A%2E%2C+Lewis%2C+P%2E%2C+Peshterliev%2C+S%2E%2C+Mehdad%2C+Y%2E%2C+Gupta%2C+S%2E%2C+%26+Yih%2C+W%2E%2Dt%2E+%282022%29%2E+Salient+phrase+aware+dense+retrieval%3A+can+a+dense+retriever+imitate+a+sparse+one%3F+I)
5. Corbeil, J.-P., Abacha, A. B., Michalopoulos, G., Swazinna, P., Del-Agua, M., Tremblay, J., Daniel, A. J., Bader, C., Cho, K., & Krishnan, P. (2025). Empowering healthcare practitioners with language models: Structuring speech transcripts in two real-world clinical applications. In Proceedings of th… [find](https://scholar.google.com/scholar?q=Corbeil%2C+J%2E%2DP%2E%2C+Abacha%2C+A%2E+B%2E%2C+Michalopoulos%2C+G%2E%2C+Swazinna%2C+P%2E%2C+Del%2DAgua%2C+M%2E%2C+Tremblay%2C+J%2E%2C+Daniel%2C+A%2E+J%2E%2C+Bader%2C+C%2E%2C+Cho%2C+K%2E%2C+%26+Krishnan%2C+P%2E+%282025%29%2E+Empowering+healthcare+practitioners+with+language+)
6. Demsash, A. W., Kassie, S. Y., Dubale, A. T., Chereka, A. A., Ngusie, H. S., Hunde, M. K., Emanu, M. D., Shibabaw, A. A., & Walle, A. D. (2023). Health professionals’ routine practice documentation and its associated factors in a resource-limited setting: a cross-sectional study. BMJ health & care i… [find](https://scholar.google.com/scholar?q=Demsash%2C+A%2E+W%2E%2C+Kassie%2C+S%2E+Y%2E%2C+Dubale%2C+A%2E+T%2E%2C+Chereka%2C+A%2E+A%2E%2C+Ngusie%2C+H%2E+S%2E%2C+Hunde%2C+M%2E+K%2E%2C+Emanu%2C+M%2E+D%2E%2C+Shibabaw%2C+A%2E+A%2E%2C+%26+Walle%2C+A%2E+D%2E+%282023%29%2E+Health+professionals%E2%80%99+routine+practice+documentation+an)
7. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). Advances in neural information processing systems, 36, 10088-10115.
8. Fu, S., Chen, D., He, H., Liu, S., Moon, S., Peterson, K. J., Shen, F., Wang, L., Wang, Y., & Wen, A. (2020). Clinical concept extraction: a methodology review. Journal of biomedical informatics, 109, 103526. [find](https://scholar.google.com/scholar?q=Fu%2C+S%2E%2C+Chen%2C+D%2E%2C+He%2C+H%2E%2C+Liu%2C+S%2E%2C+Moon%2C+S%2E%2C+Peterson%2C+K%2E+J%2E%2C+Shen%2C+F%2E%2C+Wang%2C+L%2E%2C+Wang%2C+Y%2E%2C+%26+Wen%2C+A%2E+%282020%29%2E+Clinical+concept+extraction%3A+a+methodology+review%2E+Journal+of+biomedical+informatics%2C+109%2C)
9. Gao, L., Dai, Z., & Callan, J. (2021). [COIL: Revisit exact lexical match in information retrieval with contextualized inverted list](https://md.research.elamur.ai/free/en/papers/gao-2021-coil-revisit-exact-lexical-match/). arXiv preprint arXiv:2104.07186.
10. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., & Vaughan, A. (2024). [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
11. Hein, D., Christie, A., Holcomb, M., Xie, B., Jain, A., Vento, J., Rakheja, N., Shakur, A. H., Christley, S., & Cowell, L. G. (2025). Prompts to table: specification and iterative refinement for clinical information extraction with large language models. medRxiv. [find](https://scholar.google.com/scholar?q=Hein%2C+D%2E%2C+Christie%2C+A%2E%2C+Holcomb%2C+M%2E%2C+Xie%2C+B%2E%2C+Jain%2C+A%2E%2C+Vento%2C+J%2E%2C+Rakheja%2C+N%2E%2C+Shakur%2C+A%2E+H%2E%2C+Christley%2C+S%2E%2C+%26+Cowell%2C+L%2E+G%2E+%282025%29%2E+Prompts+to+table%3A+specification+and+iterative+refinement+for+clini)
12. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations (ICLR), pages.
13. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
14. Lin, S.-C., & Lin, J. (2023). A dense representation framework for lexical and semantic matching. ACM Transactions on Information Systems, 41(4), 1-29. [find](https://scholar.google.com/scholar?q=Lin%2C+S%2E%2DC%2E%2C+%26+Lin%2C+J%2E+%282023%29%2E+A+dense+representation+framework+for+lexical+and+semantic+matching%2E+ACM+Transactions+on+Information+Systems%2C+41%284%29%2C+1%2D29%2E)
15. Mandikal, P., & Mooney, R. (2024). Sparse meets dense: A hybrid approach to enhance scientific document retrieval. arXiv preprint arXiv:2401.04055. — [arXiv:2401.04055](https://arxiv.org/abs/2401.04055) [find](https://scholar.google.com/scholar?q=Mandikal%2C+P%2E%2C+%26+Mooney%2C+R%2E+%282024%29%2E+Sparse+meets+dense%3A+A+hybrid+approach+to+enhance+scientific+document+retrieval%2E+arXiv+preprint+arXiv%3A2401%2E04055%2E+%E2%80%94+%5BarXiv%3A2401%2E04055%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2401%2E0405)
16. Michalopoulos, G., Corbeil, J.-P., Bader, C., Bodenstab, N., & Ben Abacha, A. (2026). [Overview of the MEDIQA-SYNUR 2026 Shared Task on Observation Extraction from Nurse Dictations](https://md.research.elamur.ai/free/en/papers/michalopoulos-2026-overview-mediqa-synur-2026-shared/). In Proceedings of the 8th Clinical Natural Language Processing Workshop, ClinicalNLP@LREC 2026, pages, Palma, Mallorca,…
17. Riedemann, L., Labonne, M., & Gilbert, S. (2024). The path forward for large language models in medicine is open. npj Digital Medicine, 7(1), 339. [find](https://scholar.google.com/scholar?q=Riedemann%2C+L%2E%2C+Labonne%2C+M%2E%2C+%26+Gilbert%2C+S%2E+%282024%29%2E+The+path+forward+for+large+language+models+in+medicine+is+open%2E+npj+Digital+Medicine%2C+7%281%29%2C+339%2E)
18. Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4), 333-389. [find](https://scholar.google.com/scholar?q=Robertson%2C+S%2E%2C+%26+Zaragoza%2C+H%2E+%282009%29%2E+The+probabilistic+relevance+framework%3A+BM25+and+beyond%2E+Foundations+and+Trends+in+Information+Retrieval%2C+3%284%29%2C+333%2D389%2E)
19. Sezgin, E., Hussain, S.-A., Rust, S., & Huang, Y. (2023). Extracting medical information from free-text and unstructured patient-generated health data using natural language processing methods: feasibility study with real-world data. JMIR Formative Research, 7, e43014. [find](https://scholar.google.com/scholar?q=Sezgin%2C+E%2E%2C+Hussain%2C+S%2E%2DA%2E%2C+Rust%2C+S%2E%2C+%26+Huang%2C+Y%2E+%282023%29%2E+Extracting+medical+information+from+free%2Dtext+and+unstructured+patient%2Dgenerated+health+data+using+natural+language+processing+methods%3A+feasib)
20. Song, K., Tan, X., Qin, T., Lu, J., & Liu, T.-Y. (2020). [MPNet: Masked and permuted pre-training for language understanding](https://md.research.elamur.ai/free/en/papers/song-2020-mpnet-masked-permuted-pre-training/). Advances in neural information processing systems, 33, 16857-16867.
21. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020). MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33, 5776-5788. [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+Wei%2C+F%2E%2C+Dong%2C+L%2E%2C+Bao%2C+H%2E%2C+Yang%2C+N%2E%2C+%26+Zhou%2C+M%2E+%282020%29%2E+MiniLM%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+Advances+in+neural+information+pr)
22. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022). [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
23. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., & Funtowicz, M. (2020). [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, p…
24. Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2024). [C-Pack: Packed resources for general chinese embeddings](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/). In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641-649.

*Updated 31 October 2026*