---
title: "Verification and Refinement of Natural Language Explanations"
description: "The authors propose Explanation-Refiner, a neuro-symbolic loop that pairs LLMs with a theorem prover. 49 references, cited by 39 papers here."
authors: ["Xin Quan", "Marco Valentino", "Louise A. Dennis", "Andre Freitas"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.172/"
doi: "10.18653/v1/2024.emnlp-main.172"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/quan-2024-verification-refinement-natural-language-explanations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/quan-2024-verification-refinement-natural-language-explanations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving

# Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving

**Authors:** Xin Quan, [Marco Valentino](https://md.research.elamur.ai/free/en/authors/marco-valentino/), Louise A. Dennis and [Andre Freitas](https://md.research.elamur.ai/free/en/authors/andre-freitas/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.172/) · [PDF](https://aclanthology.org/2024.emnlp-main.172.pdf) · [DOI 10.18653/v1/2024.emnlp-main.172](https://doi.org/10.18653/v1/2024.emnlp-main.172) · [OpenAlex](https://openalex.org/W4404782636)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [39](https://md.research.elamur.ai/free/en/papers/quan-2024-verification-refinement-natural-language-explanations/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Checking whether a natural language explanation actually supports an NLI conclusion is difficult. Crowd-sourced explanation corpora are costly to build and often contain gaps or logical mistakes. Text-generation metrics also miss properties such as logical soundness.

The authors propose Explanation-Refiner, a neuro-symbolic loop that pairs LLMs with a theorem prover. An LLM translates the premise, hypothesis and explanatory facts into an Isabelle/HOL theory. It uses Neo-Davidsonian event semantics combined with First-Order Logic to preserve as much sentence meaning as possible. The prover first reports syntax problems, which the LLM repairs over several rounds. The LLM then sketches a proof strategy and writes formal proof steps. If the proof breaks, the failing step and the axioms involved go back to the LLM, which rewrites the explanation in a new iteration.

Experiments on e-SNLI, QASC and WorldTree show that this feedback substantially increases the share of logically valid human-written explanations. The work matters because it offers an automatic, formally grounded way to audit and repair explanation datasets. It also serves as a benchmark for LLM reasoning and autoformalisation.

## Key points
- With GPT-4, logical validity of explanations rose from 36% to 84% on e-SNLI, from 12% to 55% on QASC, and from 2% to 37% on WorldTree.
- Prover feedback cut autoformalisation syntax errors by an average of 68.67%, 62.31% and 55.17% across the three datasets.
- Five LLMs were evaluated, including GPT-4, GPT-3.5, Mistral and LLama models. Closed models clearly beat open-source ones, and all struggled as explanations grew from 1 up to 16 sentences.
- An explanation counts as valid when the premise plus the explanation entails the hypothesis. A refined explanation is accepted only if its facts are consistent, complementary and non-redundant.
- Code and data are released at github.com/neuro-symbolic-ai/explanation_refinement (EMNLP 2024).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Manchester](https://md.research.elamur.ai/free/en/universities/university-of-manchester/) — University, Manchester, United Kingdom · 87 works · 44,938 citations
- [Idiap Research Institute](https://md.research.elamur.ai/free/en/labs/idiap-research-institute/) — Research institute, Martigny-Combe, Switzerland · 32 works · 1,954 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Xin Quan, Marco Valentino, Louise A. Dennis and Andre Freitas. 2024. Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{quan2024verification,
  title = {{Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving}},
  author = {Xin Quan, Marco Valentino, Louise A. Dennis and Andre Freitas},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.172},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.172}
}
```

- [← previous MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks](https://md.research.elamur.ai/free/en/papers/zhang-2024-mlcopilot-unleashing-power-large-language/)
- [next → FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2024-fanoutqa-multi-hop-multi-document/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

49 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Afra Feyza Akyurek, Ekin Akyurek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, and Niket Tandon. 2023. [RL4F: Generating natural language feedback with reinforcement learning for repairing model outputs](https://md.research.elamur.ai/free/en/papers/akyurek-2023-rl4f-generating-natural-language-feedback/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vo…
2. Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. 2023. [Faithfulness tests for natural language explanations](https://md.research.elamur.ai/free/en/papers/atanasova-2023-faithfulness-tests-natural-language-explanations/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), page…
3. Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, and Greg Durrett. 2022. Natural language deduction through search over statement compositions. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4871–4883, Abu Dhabi, United Arab Emirates. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Kaj+Bostrom%2C+Zayne+Sprague%2C+Swarat+Chaudhuri%2C+and+Greg+Durrett%2E+2022%2E+Natural+language+deduction+through+search+over+statement+compositions%2E+In+Findings+of+the+Association+for+Computational+Linguistic)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
5. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4157– 4165, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4157%E2%80%93+4165%2C+Online%2E+Association+for+Computational+Linguistics%2E)
6. Chun Sik Chan, Huanqi Kong, and Liang Guanqing. 2022. A comparative study of faithfulness metrics for model interpretability methods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5029–5038, Dublin, Ireland. Association for… [find](https://scholar.google.com/scholar?q=Chun+Sik+Chan%2C+Huanqi+Kong%2C+and+Liang+Guanqing%2E+2022%2E+A+comparative+study+of+faithfulness+metrics+for+model+interpretability+methods%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+C)
7. Qianglong Chen, Feng Ji, Xiangji Zeng, Feng-Lin Li, Ji Zhang, Haiqing Chen, and Yin Zhang. 2021. [KACE: Generating knowledge aware contrastive explanations for natural language inference](https://md.research.elamur.ai/free/en/papers/chen-2021-kace-generating-knowledge-aware-contrastive/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati…
8. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. Preprint, arXiv:2304.05128. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2023%2E+Teaching+large+language+models+to+self%2Ddebug%2E+Preprint%2C+arXiv%3A2304%2E05128%2E)
9. Garett Cunningham, Razvan Bunescu, and David Juedes. 2022. Towards autoformalization of mathematics and code correctness: Experiments with elementary proofs. In Proceedings of the 1st Workshop on Mathematical Natural Language Processing (MathNLP), pages 25–32, Abu Dhabi, United Arab Emirates (Hybrid… [find](https://scholar.google.com/scholar?q=Garett+Cunningham%2C+Razvan+Bunescu%2C+and+David+Juedes%2E+2022%2E+Towards+autoformalization+of+mathematics+and+code+correctness%3A+Experiments+with+elementary+proofs%2E+In+Proceedings+of+the+1st+Workshop+on+Math)
10. Dhairya Dalal, Marco Valentino, André Freitas, and Paul Buitelaar. 2024. Inference to the best explanation in large language models. Preprint, arXiv:2402.10767. [find](https://scholar.google.com/scholar?q=Dhairya+Dalal%2C+Marco+Valentino%2C+Andr%C3%A9+Freitas%2C+and+Paul+Buitelaar%2E+2024%2E+Inference+to+the+best+explanation+in+large+language+models%2E+Preprint%2C+arXiv%3A2402%2E10767%2E)
11. Yanai Elazar, Hongming Zhang, Yoav Goldberg, and Dan Roth. 2021. Back to square one: Artifact detection, training and commonsense disentanglement in the Winograd schema. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10486–10500, Online and Punta Can… [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Hongming+Zhang%2C+Yoav+Goldberg%2C+and+Dan+Roth%2E+2021%2E+Back+to+square+one%3A+Artifact+detection%2C+training+and+commonsense+disentanglement+in+the+Winograd+schema%2E+In+Proceedings+of+the+2021+Con)
12. Emily First, Markus N. Rabe, Talia Ringer, and Yuriy Brun. 2023. Baldur: Whole-proof generation and repair with large language models. Preprint, arXiv:2303.04910. [find](https://scholar.google.com/scholar?q=Emily+First%2C+Markus+N%2E+Rabe%2C+Talia+Ringer%2C+and+Yuriy+Brun%2E+2023%2E+Baldur%3A+Whole%2Dproof+generation+and+repair+with+large+language+models%2E+Preprint%2C+arXiv%3A2303%2E04910%2E)
13. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. [e-snli: Natural language inference with natural language explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/). In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
14. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. [Critic: Large language models can self-correct with tool-interactive critiquing](https://md.research.elamur.ai/free/en/papers/gou-2023-critic-large-language-models-self/). Preprint, arXiv:2305.11738.
15. Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. 2020. Make up your mind! adversarial generation of inconsistent natural language explanations. In Peter Jansen, Elizabeth Wainwright, Steven Marmorstein, and Clayton Morrison. 2018. WorldTree: A corpu… [find](https://scholar.google.com/scholar?q=Oana%2DMaria+Camburu%2C+Brendan+Shillingford%2C+Pasquale+Minervini%2C+Thomas+Lukasiewicz%2C+and+Phil+Blunsom%2E+2020%2E+Make+up+your+mind%21+adversarial+generation+of+inconsistent+natural+language+explanations%2E+In+Pe)
16. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
17. Albert Qiaochu Jiang, Sean Welleck, Jin Peng Zhou, Wenda Li, Jiacheng Liu, Mateja Jamnik, Timoth’ee Lacroix, Yuhuai Wu, and Guillaume Lample. 2023. [Draft, Sketch, and Prove: Guiding formal theorem provers with informal proofs](https://md.research.elamur.ai/free/en/papers/jiang-2022-draft-sketch-prove-guiding-formal/). In International Conference on Learning Representations.
18. Dongwei Jiang, Marcio Fonseca, and Shay B. Cohen. 2024b. Leanreasoner: Boosting complex logical reasoning with lean. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Dongwei+Jiang%2C+Marcio+Fonseca%2C+and+Shay+B%2E+Cohen%2E+2024b%2E+Leanreasoner%3A+Boosting+complex+logical+reasoning+with+lean%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapter+of+the+Associat)
19. Tushar Khot, Peter Clark, Michal Guerquin, Peter Alexander Jansen, and Ashish Sabharwal. 2019. QASC: A dataset for question answering via sentence composition. In AAAI. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Peter+Clark%2C+Michal+Guerquin%2C+Peter+Alexander+Jansen%2C+and+Ashish+Sabharwal%2E+2019%2E+QASC%3A+A+dataset+for+question+answering+via+sentence+composition%2E+In+AAAI%2E)
20. Sawan Kumar and Partha Talukdar. 2020. [NILE : Natural language inference with faithful natural language explanations](https://md.research.elamur.ai/free/en/papers/kumar-2020-nile-natural-language-inference-faithful/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8730–8742, Online. Association for Computational Linguistics.
21. Haochen Liu, Joseph Thekinen, Sinem Mollaoglu, Da Tang, Ji Yang, Youlong Cheng, Hui Liu, and Jiliang Tang. 2022. Toward annotator group bias in crowdsourcing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1797–1806, Dublin,… [find](https://scholar.google.com/scholar?q=Haochen+Liu%2C+Joseph+Thekinen%2C+Sinem+Mollaoglu%2C+Da+Tang%2C+Ji+Yang%2C+Youlong+Cheng%2C+Hui+Liu%2C+and+Jiliang+Tang%2E+2022%2E+Toward+annotator+group+bias+in+crowdsourcing%2E+In+Proceedings+of+the+60th+Annual+Meeting)
22. Mistral AI. 2024. [https://docs.mistral.ai/](https://docs.mistral.ai/). [↗](https://docs.mistral.ai/)
23. Deepak Nathani, David Wang, Liangming Pan, and William Wang. 2023. MAF: Multi-aspect feedback for improving reasoning in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6591–6616, Singapore. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Deepak+Nathani%2C+David+Wang%2C+Liangming+Pan%2C+and+William+Wang%2E+2023%2E+MAF%3A+Multi%2Daspect+feedback+for+improving+reasoning+in+large+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Metho)
24. Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong, Xiaokai Wei, Henghui Zhu, Xinchi Chen, Peng Xu, Zhiheng Huang, Andrew Arnold, and Dan Roth. 2022. Entailment tree explanations via iterative retrieval-generation reasoner. In Findings of the Association for Computational Linguistics: NAACL 2022,… [find](https://scholar.google.com/scholar?q=Danilo+Neves+Ribeiro%2C+Shen+Wang%2C+Xiaofei+Ma%2C+Rui+Dong%2C+Xiaokai+Wei%2C+Henghui+Zhu%2C+Xinchi+Chen%2C+Peng+Xu%2C+Zhiheng+Huang%2C+Andrew+Arnold%2C+and+Dan+Roth%2E+2022%2E+Entailment+tree+explanations+via+iterative+retr)
25. Tobias Nipkow, Markus Wenzel, and Lawrence C Paulson. 2002. Isabelle/HOL: a proof assistant for higher-order logic. Springer. [find](https://scholar.google.com/scholar?q=Tobias+Nipkow%2C+Markus+Wenzel%2C+and+Lawrence+C+Paulson%2E+2002%2E+Isabelle%2FHOL%3A+a+proof+assistant+for+higher%2Dorder+logic%2E+Springer%2E)
26. Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, and Roger Levy. 2023. [LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers](https://md.research.elamur.ai/free/en/papers/olausson-2023-linc-neurosymbolic-approach-logical-reasoning/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan…
27. Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024a. Is self-repair a silver bullet for code generation? In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Theo+X%2E+Olausson%2C+Jeevana+Priya+Inala%2C+Chenglong+Wang%2C+Jianfeng+Gao%2C+and+Armando+Solar%2DLezama%2E+2024a%2E+Is+self%2Drepair+a+silver+bullet+for+code+generation%3F+In+International+Conference+on+Learning+Repres)
28. Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024b. Is self-repair a silver bullet for code generation? Preprint, arXiv:2306.09896. [find](https://scholar.google.com/scholar?q=Theo+X%2E+Olausson%2C+Jeevana+Priya+Inala%2C+Chenglong+Wang%2C+Jianfeng+Gao%2C+and+Armando+Solar%2DLezama%2E+2024b%2E+Is+self%2Drepair+a+silver+bullet+for+code+generation%3F+Preprint%2C+arXiv%3A2306%2E09896%2E)
29. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
30. CoRR, Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023a. [Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning](https://md.research.elamur.ai/free/en/papers/pan-2023-logic-lm-empowering-large-language/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3806–3824, Singapore. Association for Computational…
31. Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023b. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. Preprint, arXiv:2308.03188. [find](https://scholar.google.com/scholar?q=Liangming+Pan%2C+Michael+Saxon%2C+Wenda+Xu%2C+Deepak+Nathani%2C+Xinyi+Wang%2C+and+William+Yang+Wang%2E+2023b%2E+Automatically+correcting+large+language+models%3A+Surveying+the+landscape+of+diverse+self%2Dcorrection+str)
32. Terence Parsons. 1990. Events in the semantics of english: A study in subatomic semantics. [find](https://scholar.google.com/scholar?q=Terence+Parsons%2E+1990%2E+Events+in+the+semantics+of+english%3A+A+study+in+subatomic+semantics%2E)
33. Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2024. [REFINER: Reasoning feedback on intermediate representations](https://md.research.elamur.ai/free/en/papers/paul-2023-refiner-reasoning-feedback-intermediate-representations/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1:…
34. Lawrence Charles Paulson. 1999. A generic tableau prover and its integration with isabelle. J. Univers. Comput. Sci., 5:73–87. [find](https://scholar.google.com/scholar?q=Lawrence+Charles+Paulson%2E+1999%2E+A+generic+tableau+prover+and+its+integration+with+isabelle%2E+J%2E+Univers%2E+Comput%2E+Sci%2E%2C+5%3A73%E2%80%9387%2E)
35. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. [Check your facts and try again: Improving large language models with external knowledge and automated feedback](https://md.research.elamur.ai/free/en/papers/peng-2023-check-facts-try-again-improving/). Preprint, arXiv:2302.12813.
36. Xin Quan, Marco Valentino, Louise Dennis, and Andre Freitas. 2024. Enhancing ethical explanations of large language models through iterative symbolic refinement. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pa… [find](https://scholar.google.com/scholar?q=Xin+Quan%2C+Marco+Valentino%2C+Louise+Dennis%2C+and+Andre+Freitas%2E+2024%2E+Enhancing+ethical+explanations+of+large+language+models+through+iterative+symbolic+refinement%2E+In+Proceedings+of+the+18th+Conference+)
37. Boris Shminke. 2022. Python client for isabelle server. Preprint, arXiv:2212.11173. — [arXiv:2212.11173](https://arxiv.org/abs/2212.11173) [find](https://scholar.google.com/scholar?q=Boris+Shminke%2E+2022%2E+Python+client+for+isabelle+server%2E+Preprint%2C+arXiv%3A2212%2E11173%2E+%E2%80%94+%5BarXiv%3A2212%2E11173%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2212%2E11173%29)
38. Mokanarangan Thayaparan, Marco Valentino, and André Freitas. 2021. Explainable inference over grounding-abstract chains for science questions. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1–12, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mokanarangan+Thayaparan%2C+Marco+Valentino%2C+and+Andr%C3%A9+Freitas%2E+2021%2E+Explainable+inference+over+grounding%2Dabstract+chains+for+science+questions%2E+In+Findings+of+the+Association+for+Computational+Linguist)
39. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
40. Marco Valentino, Ian Pratt-Hartmann, and André Freitas. 2021. [Do natural language explanations represent valid logical arguments? verifying entailment in explainable NLI gold standards](https://md.research.elamur.ai/free/en/papers/valentino-2021-do-natural-language-explanations-represent/). In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 76–86, Groningen, The…
41. Marco Valentino, Mokanarangan Thayaparan, Deborah Ferreira, and André Freitas. 2022a. Hybrid autoregressive inference for scalable multi-hop explanation regeneration. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11403–11411. [find](https://scholar.google.com/scholar?q=Marco+Valentino%2C+Mokanarangan+Thayaparan%2C+Deborah+Ferreira%2C+and+Andr%C3%A9+Freitas%2E+2022a%2E+Hybrid+autoregressive+inference+for+scalable+multi%2Dhop+explanation+regeneration%2E+Proceedings+of+the+AAAI+Conferenc)
42. Marco Valentino, Mokanarangan Thayaparan, and André Freitas. 2022b. Case-based abductive natural language inference. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1556–1568, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marco+Valentino%2C+Mokanarangan+Thayaparan%2C+and+Andr%C3%A9+Freitas%2E+2022b%2E+Case%2Dbased+abductive+natural+language+inference%2E+In+Proceedings+of+the+29th+International+Conference+on+Computational+Linguistics%2C+p)
43. Nathaniel Weir, Peter Clark, and Benjamin Van Durme. 2023. Nellie: A neuro-symbolic inference engine for grounded, compositional, and explainable reasoning. Preprint, arXiv:2209.07662. — [arXiv:2209.07662](https://arxiv.org/abs/2209.07662) [find](https://scholar.google.com/scholar?q=Nathaniel+Weir%2C+Peter+Clark%2C+and+Benjamin+Van+Durme%2E+2023%2E+Nellie%3A+A+neuro%2Dsymbolic+inference+engine+for+grounded%2C+compositional%2C+and+explainable+reasoning%2E+Preprint%2C+arXiv%3A2209%2E07662%2E+%E2%80%94+%5BarXiv%3A2209%2E0)
44. Sarah Wiegreffe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). [find](https://scholar.google.com/scholar?q=Sarah+Wiegreffe+and+Ana+Marasovic%2E+2021%2E+Teach+me+to+explain%3A+A+review+of+datasets+for+explainable+natural+language+processing%2E+In+Thirty%2Dfifth+Conference+on+Neural+Information+Processing+Systems+Data)
45. Yuhuai Wu, Albert Qiaochu Jiang, Wenda Li, Markus Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. 2022. [Autoformalization with large language models](https://md.research.elamur.ai/free/en/papers/wu-2022-autoformalization-large-language-models/). In Advances in Neural Information Processing Systems, volume 35, pages 32353–32368. Curran Associates, Inc.
46. Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, and Ashish Sabharwal. 2023. [Improving language models via plug-and-play retrieval feedback](https://md.research.elamur.ai/free/en/papers/yu-2023-improving-language-models-plug-play/). Preprint, arXiv:2305.14002.
47. Li Yuan, Yi Cai, Haopeng Ren, and Jiexin Wang. 2024. A logical pattern memory pre-trained model for entailment tree generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 759–772, Torino, Italia.… [find](https://scholar.google.com/scholar?q=Li+Yuan%2C+Yi+Cai%2C+Haopeng+Ren%2C+and+Jiexin+Wang%2E+2024%2E+A+logical+pattern+memory+pre%2Dtrained+model+for+entailment+tree+generation%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Computation)
48. Wenting Zhao, Justin Chiu, Claire Cardie, and Alexander Rush. 2023. Abductive commonsense reasoning exploiting mutually exclusive explanations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14883–14896, Toronto, Canada. Asso… [find](https://scholar.google.com/scholar?q=Wenting+Zhao%2C+Justin+Chiu%2C+Claire+Cardie%2C+and+Alexander+Rush%2E+2023%2E+Abductive+commonsense+reasoning+exploiting+mutually+exclusive+explanations%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Associa)
49. Xinyan Zhao and V.G.Vinod Vydiswaran. 2021. Lirex: Augmenting language inference with relevant explanations. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):14532–14539. [find](https://scholar.google.com/scholar?q=Xinyan+Zhao+and+V%2EG%2EVinod+Vydiswaran%2E+2021%2E+Lirex%3A+Augmenting+language+inference+with+relevant+explanations%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+35%2816%29%3A14532%E2%80%9314539%2E)

*Updated 30 October 2026*