---
title: "Semantic Parsing for Evaluating Large — VERGNETTE 2026"
description: "Off-the-shelf LLMs struggle to produce formal meaning representations."
authors: ["Rémi DE VERGNETTE", "Maxime Amblard"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.765/"
doi: "10.63317/4hqtuv5e47pw"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/vergnette-2026-semantic-parsing-evaluating-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vergnette-2026-semantic-parsing-evaluating-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/) / Semantic Parsing for Evaluating Large Language Models: Separating Linguistic Abilities with YARN

# Semantic Parsing for Evaluating Large Language Models: Separating Linguistic Abilities with YARN

**Authors:** [Rémi DE VERGNETTE](https://md.research.elamur.ai/free/en/authors/remi-de-vergnette/), [Maxime Amblard](https://md.research.elamur.ai/free/en/authors/maxime-amblard/) · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.765/) · [PDF](https://aclanthology.org/2026.lrec-1.765.pdf) · [DOI 10.63317/4hqtuv5e47pw](https://doi.org/10.63317/4hqtuv5e47pw) · [OpenAlex](https://openalex.org/W7160145043)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Off-the-shelf LLMs struggle to produce formal meaning representations. Earlier evaluations usually use one notation, often AMR, and a single aggregate graph-matching score. This makes it unclear whether errors come from the format or from particular semantic phenomena, and well-known formalisms risk data contamination. The authors address this by having LLMs parse sentences into Yarn, a layered formalism derived from AMR. Yarn separates predicate–argument structure from feature layers such as tense, aspect, modality, negation and quantification. Outputs are scored with SmatchY, an extension of Smatch that can restrict scoring to chosen element types or layers. The experiments vary three factors:
- inference mode (standard vs. reasoning)
- linearization (a JSON graph encoding vs. a logic-style CFG notation)
- whether partial semantic parses are given as auxiliary hints

No configuration wins consistently, and results depend heavily on format and supervision. Extra semantic hints help some models but hurt others. Tense and negation are recovered more reliably than quantification. The approach shows which aspects of meaning LLMs capture and can guide the design of LLM-based semantic parsing pipelines.

## Key points
- Reference data: 100 Yarn annotations of short sentences from the Parallel Universal Dependencies (PUD) English treebank (UD 1.0). The sentences average 7.22 words and range from 3 to 10 words. Five additional human-annotated sentences serve as few-shot examples.
- The annotations are released at github.com/YARN-World and can be browsed on semantics.grew.fr with GREW queries.
- SmatchY aligns atomic elements of the gold and candidate graphs, then computes precision, recall and F1. Scoring can be filtered by predicates, arguments, feature modifiers or specific layers. One example is SmatchY-Fol, which covers predicates, arguments, negation and quantification.
- SmatchY is purely structural. Logically equivalent but structurally different annotations, such as reordered commuting quantifiers, count as mismatches, so the score is a conservative lower bound.
- The two linearizations contrast familiarity with novelty. JSON is common in pretraining data, while the CFG notation is more compact, shows operator scope explicitly and is unfamiliar to LLMs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · [Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{vergnette2026semantic,
  title = {{Semantic Parsing for Evaluating Large Language Models: Separating Linguistic Abilities with YARN}},
  author = {VERGNETTE, Rémi DE and Amblard, Maxime},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4hqtuv5e47pw},
  url = {https://doi.org/10.63317/4hqtuv5e47pw}
}
```

- [← previous Semantic Information: A Difference That Makes a Difference](https://md.research.elamur.ai/free/en/papers/philipp-2026-semantic-information-difference-that-makes/)
- [next → SemBench: A Universal Semantic Framework for LLM Evaluation](https://md.research.elamur.ai/free/en/papers/zubillaga-2026-sembench-universal-semantic-framework-llm/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/)

## References

29 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discou… [find](https://scholar.google.com/scholar?q=Laura+Banarescu%2C+Claire+Bonial%2C+Shu+Cai%2C+Madalina+Georgescu%2C+Kira+Griffitt%2C+Ulf+Hermjakob%2C+Kevin+Knight%2C+Philipp+Koehn%2C+Martha+Palmer%2C+and+Nathan+Schneider%2E+2013%2E+Abstract+Meaning+Representation+for+s) [our card](https://md.research.elamur.ai/free/en/papers/banarescu-2013-abstract-meaning-representation-sembanking/)
2. Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2023. Prompting language models for linguistic structure. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6649– 6663, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Terra+Blevins%2C+Hila+Gonen%2C+and+Luke+Zettlemoyer%2E+2023%2E+Prompting+language+models+for+linguistic+structure%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28V)
3. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shu+Cai+and+Kevin+Knight%2E+2013%2E+Smatch%3A+an+evaluation+metric+for+semantic+feature+structures%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Shor)
4. Aaron Conklin, Michael Frank, and Ellie Pavlick. 2023. What do llms know about linguistic theory? arXiv preprint arXiv:2310.01672. — [arXiv:2310.01672](https://arxiv.org/abs/2310.01672) [find](https://scholar.google.com/scholar?q=Aaron+Conklin%2C+Michael+Frank%2C+and+Ellie+Pavlick%2E+2023%2E+What+do+llms+know+about+linguistic+theory%3F+arXiv+preprint+arXiv%3A2310%2E01672%2E+%E2%80%94+%5BarXiv%3A2310%2E01672%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E01672%29)
5. Rémi de Vergnette, Maxime Amblard, and Bruno Guillaume. 2025. Evaluation framework for layered meaning representation. In Proceedings of the Sixth International Workshop on Designing Meaning Representations, pages 38–48, Prague, Czechia. Association for Computational Lingustics. [find](https://scholar.google.com/scholar?q=R%C3%A9mi+de+Vergnette%2C+Maxime+Amblard%2C+and+Bruno+Guillaume%2E+2025%2E+Evaluation+framework+for+layered+meaning+representation%2E+In+Proceedings+of+the+Sixth+International+Workshop+on+Designing+Meaning+Represent)
6. Yanai Elazar, Atticus Geiger, Ellie Pavlick, and Yoav Goldberg. 2021. Measuring and improving compositional generalization in language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Atticus+Geiger%2C+Ellie+Pavlick%2C+and+Yoav+Goldberg%2E+2021%2E+Measuring+and+improving+compositional+generalization+in+language+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Associat)
7. Allyson Ettinger. 2020. What bert is not: Lessons from a new suite of psycholinguistic diagnostics. Transactions of the Association for Computational Linguistics, 8:34–48. [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2E+2020%2E+What+bert+is+not%3A+Lessons+from+a+new+suite+of+psycholinguistic+diagnostics%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A34%E2%80%9348%2E)
8. Allyson Ettinger, Jena Hwang, Valentina Pyatkin, Chandra Bhagavatula, and Yejin Choi. 2023. “you are an expert linguistic annotator”: Limits of LLMs as analyzers of Abstract Meaning Representation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8250–8263, Singapore.… [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2C+Jena+Hwang%2C+Valentina+Pyatkin%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2023%2E+%E2%80%9Cyou+are+an+expert+linguistic+annotator%E2%80%9D%3A+Limits+of+LLMs+as+analyzers+of+Abstract+Meaning+Representation%2E+In)
9. Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A Discriminative Graph-Based Parser for the Abstract Meaning Representation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426–1436, Balti… [find](https://scholar.google.com/scholar?q=Jeffrey+Flanigan%2C+Sam+Thomson%2C+Jaime+Carbonell%2C+Chris+Dyer%2C+and+Noah+A%2E+Smith%2E+2014%2E+A+Discriminative+Graph%2DBased+Parser+for+the+Abstract+Meaning+Representation%2E+In+Proceedings+of+the+52nd+Annual+Meet)
10. Gottlob Frege. 1967. Begriffsschrift, a formula language, modeled upon that of arithmetic, for pure thought. In Jean van Heijenoort, editor, From Frege to Gödel: A Source Book in Mathematical Logic, 1879–1931, pages 1–82. Harvard University Press. Originally published in 1879. [find](https://scholar.google.com/scholar?q=Gottlob+Frege%2E+1967%2E+Begriffsschrift%2C+a+formula+language%2C+modeled+upon+that+of+arithmetic%2C+for+pure+thought%2E+In+Jean+van+Heijenoort%2C+editor%2C+From+Frege+to+G%C3%B6del%3A+A+Source+Book+in+Mathematical+Logic%2C+1)
11. Bruno Guillaume, Guillame Bonfante, Paul Masson, Mathieu Morey, and Guy Perrier. 2012. Grew : un outil de réécriture de graphes pour le TAL (Grew: a graph rewriting tool for NLP) [in French]. In Proceedings of the Joint Conference JEP-TALN-RECITAL 2012, volume 5: Software Demonstrations, pages 1–2,… [find](https://scholar.google.com/scholar?q=Bruno+Guillaume%2C+Guillame+Bonfante%2C+Paul+Masson%2C+Mathieu+Morey%2C+and+Guy+Perrier%2E+2012%2E+Grew+%3A+un+outil+de+r%C3%A9%C3%A9criture+de+graphes+pour+le+TAL+%28Grew%3A+a+graph+rewriting+tool+for+NLP%29+%5Bin+French%5D%2E+In+Proce)
12. Aishwarya Kamath and Rajarshi Das. 2018. A survey on semantic parsing. arXiv preprint arXiv:1812.00978. — [arXiv:1812.00978](https://arxiv.org/abs/1812.00978) [find](https://scholar.google.com/scholar?q=Aishwarya+Kamath+and+Rajarshi+Das%2E+2018%2E+A+survey+on+semantic+parsing%2E+arXiv+preprint+arXiv%3A1812%2E00978%2E+%E2%80%94+%5BarXiv%3A1812%2E00978%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1812%2E00978%29)
13. Paul Kingsbury and Martha Palmer. 2002. From TreeBank to PropBank. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC’02), Las Palmas, Canary Islands - Spain. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Paul+Kingsbury+and+Martha+Palmer%2E+2002%2E+From+TreeBank+to+PropBank%2E+In+Proceedings+of+the+Third+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9902%29%2C+Las+Palmas%2C+Canary+Islands+%2D+Spa) [our card](https://md.research.elamur.ai/free/en/papers/kingsbury-2002-treebank-propbank/)
14. Percy Liang. 2016. Learning executable semantic parsers for natural language understanding. Commun. ACM, 59(9):68–76. [find](https://scholar.google.com/scholar?q=Percy+Liang%2E+2016%2E+Learning+executable+semantic+parsers+for+natural+language+understanding%2E+Commun%2E+ACM%2C+59%289%29%3A68%E2%80%9376%2E)
15. Meta AI. 2025. The llama 4 herd: The beginning of a new era of natively multimodal intelligence. Meta AI Blog. [find](https://scholar.google.com/scholar?q=Meta+AI%2E+2025%2E+The+llama+4+herd%3A+The+beginning+of+a+new+era+of+natively+multimodal+intelligence%2E+Meta+AI+Blog%2E)
16. Mistral AI. 2023. Mixtral of experts. Blog post. [find](https://scholar.google.com/scholar?q=Mistral+AI%2E+2023%2E+Mixtral+of+experts%2E+Blog+post%2E)
17. Siyana Pavlova. 2025. Tools and methods for semantically annotated corpora. Ph.D. thesis, Université de Lorraine. [find](https://scholar.google.com/scholar?q=Siyana+Pavlova%2E+2025%2E+Tools+and+methods+for+semantically+annotated+corpora%2E+Ph%2ED%2E+thesis%2C+Universit%C3%A9+de+Lorraine%2E)
18. Siyana Pavlova, Maxime Amblard, and Bruno Guillaume. 2024. YARN is All You Knit: Encoding Multiple Semantic Phenomena with Layers. In Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024, pages 66–76, Torino, Italia. ELRA and ICCL. [find](https://scholar.google.com/scholar?q=Siyana+Pavlova%2C+Maxime+Amblard%2C+and+Bruno+Guillaume%2E+2024%2E+YARN+is+All+You+Knit%3A+Encoding+Multiple+Semantic+Phenomena+with+Layers%2E+In+Proceedings+of+the+Fifth+International+Workshop+on+Designing+Meani)
19. Phillip Schneider, Manuel Klettner, Kristiina Jokinen, Elena Simperl, and Florian Matthes. 2024. Evaluating large language models in semantic parsing for conversational question answering over knowledge graphs. [find](https://scholar.google.com/scholar?q=Phillip+Schneider%2C+Manuel+Klettner%2C+Kristiina+Jokinen%2C+Elena+Simperl%2C+and+Florian+Matthes%2E+2024%2E+Evaluating+large+language+models+in+semantic+parsing+for+conversational+question+answering+over+knowled)
20. Wesley Scivetti, Melissa Torgbi, Austin Blodgett, Mollie Shichman, Taylor Hudson, Claire Bonial, and Harish Tayyar Madabushi. 2025. Beyond memorization: Assessing semantic generalization in large language models using phrasal constructions. [find](https://scholar.google.com/scholar?q=Wesley+Scivetti%2C+Melissa+Torgbi%2C+Austin+Blodgett%2C+Mollie+Shichman%2C+Taylor+Hudson%2C+Claire+Bonial%2C+and+Harish+Tayyar+Madabushi%2E+2025%2E+Beyond+memorization%3A+Assessing+semantic+generalization+in+large+lang)
21. Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2025. A survey on employing large language models for text-to-sql tasks. ACM Comput. Surv., 58(2). [find](https://scholar.google.com/scholar?q=Liang+Shi%2C+Zhengju+Tang%2C+Nan+Zhang%2C+Xiaotong+Zhang%2C+and+Zhi+Yang%2E+2025%2E+A+survey+on+employing+large+language+models+for+text%2Dto%2Dsql+tasks%2E+ACM+Comput%2E+Surv%2E%2C+58%282%29%2E)
22. John F. Sowa. 1984. Conceptual Structures: Information Processing in Mind and Machine. Addison-Wesley, Reading, MA. [find](https://scholar.google.com/scholar?q=John+F%2E+Sowa%2E+1984%2E+Conceptual+Structures%3A+Information+Processing+in+Mind+and+Machine%2E+Addison%2DWesley%2C+Reading%2C+MA%2E)
23. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Alicia+Parrish%2C+Haokun+Liu%2C+Anhad+Mohananey%2C+Wei+Peng%2C+Sheng%2DFu+Wang%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+Blimp%3A+The+benchmark+of+linguistic+minimal+pairs+for+english%2E+Transactions+of+the+Assoc)
24. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2E)
25. Shira Wein and Juri Opitz. 2024. A survey of AMR applications. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6856–6875, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shira+Wein+and+Juri+Opitz%2E+2024%2E+A+survey+of+AMR+applications%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+6856%E2%80%936875%2C+Miami%2C+Florida%2C+USA%2E+Associat)
26. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Anfeng+Li%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Gao%2C+Chengen+Huang%2C+Chenxu+Lv%2C+Chujie+Zheng%2C+Dayiheng+Liu%2C+Fan+Zhou%2C+Fei+Huang%2C+Feng+Hu%2C+Hao+Ge%2C+Haoran+Wei%2C+Hua) [our card](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/)
27. Yu Yuan, Lili Zhao, Kai Zhang, Guangting Zheng, and Qi Liu. 2024. Do LLMs overcome shortcut learning? an evaluation of shortcut challenges in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12188–12200, Miami, Florida, USA. Asso… [find](https://scholar.google.com/scholar?q=Yu+Yuan%2C+Lili+Zhao%2C+Kai+Zhang%2C+Guangting+Zheng%2C+and+Qi+Liu%2E+2024%2E+Do+LLMs+overcome+shortcut+learning%3F+an+evaluation+of+shortcut+challenges+in+large+language+models%2E+In+Proceedings+of+the+2024+Conferen)
28. Zeyu Zhan, E. Haihong, and Min Song. 2025. Leveraging large language models for enhanced text-to-sql parsing. IEEE Access. 7. [find](https://scholar.google.com/scholar?q=Zeyu+Zhan%2C+E%2E+Haihong%2C+and+Min+Song%2E+2025%2E+Leveraging+large+language+models+for+enhanced+text%2Dto%2Dsql+parsing%2E+IEEE+Access%2E+7%2E)
29. Language Resource References Nivre, Joakim and Bosco, Cristina and Choi, Jinho and de Marneffe, Marie-Catherine and Dozat, Timothy and Farkas, Richárd and Foster, Jennifer and Ginter, Filip and Goldberg, Yoav and Hajič, Jan and Kanerva, Jenna and Laippala, Veronika and Lenci, Alessandro and Lynn, Te… [find](https://scholar.google.com/scholar?q=Language+Resource+References+Nivre%2C+Joakim+and+Bosco%2C+Cristina+and+Choi%2C+Jinho+and+de+Marneffe%2C+Marie%2DCatherine+and+Dozat%2C+Timothy+and+Farkas%2C+Rich%C3%A1rd+and+Foster%2C+Jennifer+and+Ginter%2C+Filip+and+Goldbe)