--- title: "Do LLMs Use Relational Structure in — Okonkwo 2026" description: "KG-grounded dialogue systems usually assume that LLMs use the typed relations linking entities, not just the entities." authors: ["Dominic Okonkwo", "Ismailcem Budak Arpinar"] year: 2026 venue: "Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue" source: "https://aclanthology.org/2026.sigdial-1.26/" doi: "10.18653/v1/2026.sigdial-1.26" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/okonkwo-2026-do-llms-use-relational-structure/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/okonkwo-2026-do-llms-use-relational-structure/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) / Do LLMs Use Relational Structure in KG-Grounded Dialogue? Evidence from Entity and Relation Perturbation # Do LLMs Use Relational Structure in KG-Grounded Dialogue? Evidence from Entity and Relation Perturbation **Authors:** Dominic Okonkwo, Ismailcem Budak Arpinar · **Year:** 2026 · **Venue:** Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.sigdial-1.26/) · [PDF](https://aclanthology.org/2026.sigdial-1.26.pdf) · [DOI 10.18653/v1/2026.sigdial-1.26](https://doi.org/10.18653/v1/2026.sigdial-1.26) · [OpenAlex](https://openalex.org/W7213995504) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) ## Summary KG-grounded dialogue systems usually assume that LLMs use the typed relations linking entities, not just the entities. The authors test this with a controlled perturbation design. Each turn is generated three ways: with the correct triple, with the same entity pair but a randomly swapped relation label, and with entities only. They define the Relation Sensitivity Score (RSS), one minus ROUGE-L between the intact and shuffled outputs, and compare it with how far outputs drift when relations are dropped entirely. Six zero-shot models from three families are tested on OpenDialKG and on WoW-KG, a new corpus that adds Wikidata triples to Wizard of Wikipedia. In every setting, swapping a relation changes responses less than deleting relations does. This suggests relation labels contribute much less than assumed. Two factors drive the effect: the model's prior knowledge of familiar entities, and reactions to the wording of the wrong label. Larger models do not reduce either problem. Human judgments agree with these findings. The authors recommend relation-perturbed baselines as a standard evaluation check. ## Key points - **Models and data:** GPT-4o, GPT-4o-mini, Llama-3.1-8B/70B and Qwen2.5-7B/32B were tested on two datasets. - OpenDialKG uses Freebase with 175 relation types; 3,216 test turns remain after filtering. - WoW-KG was built with spaCy NER and SPARQL queries to Wikidata. It has 8,715 wizard turns, 95.4% KG coverage and 219 relation types. - **Main result:** RSS fell below the entity-only divergence in all 12 model–dataset pairs (Wilcoxon p \< 0.001). RSS-to-divergence ratios ranged from 0.821 to 0.978. - **Entity salience:** Better-known head entities made models less sensitive to the relation. The Spearman correlation was ρ = −0.108 on OpenDialKG and −0.071 on WoW-KG, and the pattern held for all six models. - **Model scale:** Larger models scored no lower on RSS. - Llama-3.1-70B scored 0.613 versus 0.563 for the 8B model on OpenDialKG. - Qwen2.5-32B scored 0.580 versus 0.556 for the 7B model, and had the highest ratio overall (0.978). - **Human evaluation:** Three blind raters judged 100 GPT-4o turns from OpenDialKG. - In 53% of turns the raters favored the answer grounded in the correct KG triple, while the shuffled-relation answer won in only 8%. - They could not identify the corrupted response in 58% of turns overall, rising to 95% for zero-RSS turns. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{okonkwo2026do, title = {{Do LLMs Use Relational Structure in KG-Grounded Dialogue? Evidence from Entity and Relation Perturbation}}, author = {Okonkwo, Dominic and Arpinar, Ismailcem Budak}, booktitle = {Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue}, year = {2026}, doi = {10.18653/v1/2026.sigdial-1.26}, url = {https://doi.org/10.18653/v1/2026.sigdial-1.26} } ``` - [← previous Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset](https://md.research.elamur.ai/free/en/papers/plum-2026-do-llms-judge-distantly-supervised/) - [next → Do Multimodal LLMs Understand Order? Measuring the Fragility of Multimodal Reasoning under Input Order Perturbations](https://md.research.elamur.ai/free/en/papers/wei-2026-do-multimodal-llms-understand-order/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) ## References 6 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, page 1247–1250, New York, NY, USA.… [find](https://scholar.google.com/scholar?q=Kurt+Bollacker%2C+Colin+Evans%2C+Praveen+Paritosh%2C+Tim+Sturge%2C+and+Jamie+Taylor%2E+2008%2E+Freebase%3A+a+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+Proceedings+of+the+2008+ACM+SI) 2. Alexandru Coca, Bo-Hsiang Tseng, Jinghong Chen, Weizhe Lin, Weixuan Zhang, Tisha Anders, and Bill Byrne. 2023. Grounding description-driven dialogue state trackers with knowledge-seeking turns. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 4… [find](https://scholar.google.com/scholar?q=Alexandru+Coca%2C+Bo%2DHsiang+Tseng%2C+Jinghong+Chen%2C+Weizhe+Lin%2C+Weixuan+Zhang%2C+Tisha+Anders%2C+and+Bill+Byrne%2E+2023%2E+Grounding+description%2Ddriven+dialogue+state+trackers+with+knowledge%2Dseeking+turns%2E+In+Pro) 3. Nelson Filipe Costa and Leila Kosseim. 2025. Multilingual implicit discourse relation recognition with multi-label hierarchical learning. pages 48–61. [find](https://scholar.google.com/scholar?q=Nelson+Filipe+Costa+and+Leila+Kosseim%2E+2025%2E+Multilingual+implicit+discourse+relation+recognition+with+multi%2Dlabel+hierarchical+learning%2E+pages+48%E2%80%9361%2E) 4. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Emily+Dinan%2C+Stephen+Roller%2C+Kurt+Shuster%2C+Angela+Fan%2C+Michael+Auli%2C+and+Jason+Weston%2E+2019%2E+Wizard+of+wikipedia%3A+Knowledge%2Dpowered+conversational+agents%2E+In+International+Conference+on+Learning+Repre) 5. Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 5110–5117. AAAI Press. [find](https://scholar.google.com/scholar?q=Marjan+Ghazvininejad%2C+Chris+Brockett%2C+Ming%2DWei+Chang%2C+Bill+Dolan%2C+Jianfeng+Gao%2C+Wen%2Dtau+Yih%2C+and+Michel+Galley%2E+2018%2E+A+knowledge%2Dgrounded+neural+conversation+model%2E+In+Proceedings+of+the+Thirty%2DSecon) 6. He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1766–1776, Vancouver, Ca… [find](https://scholar.google.com/scholar?q=He+He%2C+Anusha+Balakrishnan%2C+Mihail+Eric%2C+and+Percy+Liang%2E+2017%2E+Learning+symmetric+collaborative+dialogue+agents+with+dynamic+knowledge+graph+embeddings%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+t)