---
title: "Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark"
description: "The authors introduce BEGIN, a benchmark for meta-evaluating such metrics. 78 references, cited by 52 papers here."
authors: ["Nouha Dziri", "Hannah Rashkin", "Tal Linzen", "David Reitter"]
year: 2022
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2022.tacl-1.62/"
doi: "10.1162/tacl_a_00506"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dziri-2022-evaluating-attribution-dialogue-systems-begin/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dziri-2022-evaluating-attribution-dialogue-systems-begin/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark

# Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark

**Authors:** [Nouha Dziri](https://md.research.elamur.ai/free/en/authors/nouha-dziri/), [Hannah Rashkin](https://md.research.elamur.ai/free/en/authors/hannah-rashkin/), [Tal Linzen](https://md.research.elamur.ai/free/en/authors/tal-linzen/) and [David Reitter](https://md.research.elamur.ai/free/en/authors/david-reitter/) · **Year:** 2022 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2022](https://md.research.elamur.ai/free/en/venues/tacl/2022/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.tacl-1.62/) · [PDF](https://aclanthology.org/2022.tacl-1.62.pdf) · [DOI 10.1162/tacl\_a\_00506](https://doi.org/10.1162/tacl_a_00506) · [OpenAlex](https://openalex.org/W4296711106)

**Citations:** 47 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/dziri-2022-evaluating-attribution-dialogue-systems-begin/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Dialogue systems that draw on external knowledge and are built on large language models often produce fluent replies that the supplied evidence does not support. Building systems that avoid this requires automatic metrics that can reliably detect it. The authors introduce BEGIN, a benchmark for meta-evaluating such metrics. It contains roughly 12k responses from four neural dialogue models, each trained on three knowledge-grounded corpora. Human annotators labelled each response as fully attributable to the background knowledge, not fully attributable, or generic. The generic class covers replies like admissions of uncertainty, which should not be penalised as harshly as fabricated content. The authors use BEGIN to test eight existing metrics. These cover word-overlap, embedding-based, QA-based, and NLI-based approaches, plus a classifier the authors trained on adversarially generated data. Every metric leans heavily on lexical overlap. The metrics confuse abstractive but faithful replies with extractive but unfaithful ones, handle generic replies poorly, and degrade when the knowledge passage is long. The benchmark exposes weaknesses that small correlation studies conceal, and it is released publicly to support more robust attribution metrics.

## Key points
- **Data and models:** BEGIN has about 12k annotated turns generated by GPT2-base, T5-base, DoHA (BART with two-view attention), and CTRL-DIALOG (T5 with control tokens). Each model was trained on Wizard of Wikipedia (WoW), CMU-DoG, and TopicalChat, giving 12 model–corpus combinations.
- **Inputs:** Models received a knowledge snippet concatenated with only the previous dialogue turn.
- **Label distribution:** Table 1 reports the three categories as roughly 23.5% fully attributable, 70.1% not attributable, and 6.4% generic.
- **Metrics evaluated:** The assessed metrics include BERTScore (with DeBERTa-xl-MNLI), BLEURT-20, BARTScore, QA-based and NLI-based scorers, and an adversarially trained classifier.
- **Availability:** The resource is available at github.com/google/BEGIN-dataset. The paper is published under a CC-BY 4.0 license (TACL vol. 10, 2022).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 136 works · 86,390 citations
- [University of Alberta](https://md.research.elamur.ai/free/en/universities/university-of-alberta/) — University, Edmonton, Canada · 23 works · 2,655 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works

## Cite

Cite as: Nouha Dziri, Hannah Rashkin, Tal Linzen and David Reitter. 2022. Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{dziri2022evaluating,
  title = {{Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark}},
  author = {Nouha Dziri, Hannah Rashkin, Tal Linzen and David Reitter},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00506},
  url = {https://doi.org/10.1162/tacl_a_00506}
}
```

- [← previous UniTE: Unified Translation Evaluation](https://md.research.elamur.ai/free/en/papers/wan-2022-unite-unified-translation-evaluation/)
- [next → Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets](https://md.research.elamur.ai/free/en/papers/wu-2022-generating-data-mitigate-spurious-correlations/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

78 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. ◦ Generic: This utterance is uninformative (too bland or not specific enough to be sharing any new information) [find](https://scholar.google.com/scholar?q=%E2%97%A6+Generic%3A+This+utterance+is+uninformative+%28too+bland+or+not+specific+enough+to+be+sharing+any+new+information%29)
2. Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. [Towards a human-like open-domain chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/). CoRR (arXiv preprint), abs/2001.09977.
3. ◦ Contains any unsupported Information: This utterance is sharing information that cannot be fully verified by the document. It may include false information, unverifiable information, and personal stories/opinions. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000. A neural probabilistic la… [find](https://scholar.google.com/scholar?q=%E2%97%A6+Contains+any+unsupported+Information%3A+This+utterance+is+sharing+information+that+cannot+be+fully+verified+by+the+document%2E+It+may+include+false+information%2C+unverifiable+information%2C+and+personal+st)
4. ◦ All information is fully supported by the document: This utterance contains only information that is fully supported by the document. B Implementations GPT2, T5 We implement these models using the TensorFlow Huggingface Transformers library (Wolf et al., 2020). During training, we use the Adam opt… [find](https://scholar.google.com/scholar?q=%E2%97%A6+All+information+is+fully+supported+by+the+document%3A+This+utterance+contains+only+information+that+is+fully+supported+by+the+document%2E+B+Implementations+GPT2%2C+T5+We+implement+these+models+using+the+T)
5. CTRL- DIALOG We reproduce the results from (Rashkin et al., 2021b), following the training details in that paper. [find](https://scholar.google.com/scholar?q=CTRL%2D+DIALOG+We+reproduce+the+results+from+%28Rashkin+et+al%2E%2C+2021b%29%2C+following+the+training+details+in+that+paper%2E)
6. DoHA We use the code and the pre-trained model on CMU-D O G that are publicly available by the authors at their Github account. 7 For W O W and T OPICAL C HAT , we follow closely the authors’ [https://bit.ly/3bBup2M](https://bit.ly/3bBup2M). [↗](https://bit.ly/3bBup2M)
7. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. [Re-evaluating evaluation in text summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359, Online. Association for Computational Linguist…
8. Harry Bunt, Volha Petukhova, Emer Gilmartin, Catherine Pelachaud, Alex Fang, Simon Keizer, and Laurent Prévot. 2020. [The ISO standard for dialogue act annotation, second edition](https://md.research.elamur.ai/free/en/papers/bunt-2020-iso-standard-dialogue-act-annotation/). In Proceedings of the 12th Language Resources and Evaluation Conference, pages 549–558, Marseille, France. European Lang…
9. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai .v32i1.11912 — [doi:10.1609/aaai](https://doi.org/10.1609/aaai) [↗](https://doi.org/10.1609/aaai)
10. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. [QuAC: Question answering in context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, Brussels, Belgium. Association for Computa…
11. Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. [Handling divergent reference texts when evaluating table-to-text generation](https://md.research.elamur.ai/free/en/papers/dhingra-2019-handling-divergent-reference-texts-when/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4884–4895, Florence, Ita…
12. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6–9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Emily+Dinan%2C+Stephen+Roller%2C+Kurt+Shuster%2C+Angela+Fan%2C+Michael+Auli%2C+and+Jason+Weston%2E+2019%2E+Wizard+of+wikipedia%3A+Knowledge%2Dpowered+conversational+agents%2E+In+7th+International+Conference+on+Learning+R)
13. Esin Durmus, He He, and Mona Diab. 2020. [FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization](https://md.research.elamur.ai/free/en/papers/durmus-2020-feqa-question-answering-evaluation-framework/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055–5070, Online. Association for Computational Lingui…
14. Esin Durmus, Faisal Ladhak, and Tatsunori Hashimoto. 2022. Spurious correlations in reference-free evaluation of text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1443–1454, Dublin, Ireland. Association for Comp… [find](https://scholar.google.com/scholar?q=Esin+Durmus%2C+Faisal+Ladhak%2C+and+Tatsunori+Hashimoto%2E+2022%2E+Spurious+correlations+in+reference%2Dfree+evaluation+of+text+generation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Compu)
15. Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar Zaiane. 2019. Evaluating coherence in dialogue systems using entailment. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short… [find](https://scholar.google.com/scholar?q=Nouha+Dziri%2C+Ehsan+Kamalloo%2C+Kory+Mathewson%2C+and+Osmar+Zaiane%2E+2019%2E+Evaluating+coherence+in+dialogue+systems+using+entailment%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+t)
16. Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. 2022a. [Faithdial: A faithful benchmark for information-seeking dialogue](https://md.research.elamur.ai/free/en/papers/dziri-2022-faithdial-faithful-benchmark-information-seeking/). CoRR (arXiv preprint), abs/2204.10757.
17. Nouha Dziri, Andrea Madotto, Osmar Zaı̈ane, and Avishek Joey Bose. 2021. [Neural path hunter: Reducing hallucination in dialogue systems via path grounding](https://md.research.elamur.ai/free/en/papers/dziri-2021-neural-path-hunter-reducing-hallucination/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2197–2214, Online and Punta Cana, Dominican Rep…
18. Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaı̈ane, and Siva Reddy. 2022b. [On the origin of hallucinations in conversational models: Is it the datasets or the models?](https://md.research.elamur.ai/free/en/papers/dziri-2022-origin-hallucinations-conversational-models-is/) CoRR (arXiv preprint), abs/2204.07931.
19. Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summ-Eval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. https://doi.org/10.1162/tacl a 00373 — [doi:10.1162/tacl](https://doi.org/10.1162/tacl) [↗](https://doi.org/10.1162/tacl)
20. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. [Ranking generated summaries by correctness: An interesting but challenging application for natural language inference](https://md.research.elamur.ai/free/en/papers/falke-2019-ranking-generated-summaries-correctness-interesting/). In Proceedings of the 57th Annual Meeting of the Association for Computational Lingui…
21. Dan Flickinger, Emily M. Bender, and Stephan Oepen. 2014. ERG semantic documentation. Accessed on 2020-08-25. [find](https://scholar.google.com/scholar?q=Dan+Flickinger%2C+Emily+M%2E+Bender%2C+and+Stephan+Oepen%2E+2014%2E+ERG+semantic+documentation%2E+Accessed+on+2020%2D08%2D25%2E)
22. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. [Experts, errors, and context: A large-scale study of human evaluation for machine translation](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/). Transactions of the Association for Computational Linguistics, 9:1460–1474. https://doi.org/10.1162 /…
23. Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2021. [GO FIGURE: A meta evaluation of factuality in summarization](https://md.research.elamur.ai/free/en/papers/gabriel-2021-go-figure-meta-evaluation-factuality/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 478–487, Online. Association for Computational Linguistics. [https://doi.o…](https://doi.o…) [↗](https://doi.o…)
24. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori… [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Tosin+Adewumi%2C+Karmanya+Aggarwal%2C+Pawan+Sasanka+Ammanamanchi%2C+Anuoluwapo+Aremu%2C+Antoine+Bosselut%2C+Khyathi+Raghavi+Chandu%2C+Miruna%2DAdriana+Clinciu%2C+Dipanjan+Das%2C+Kaustubh+Dhole%2C+Wany)
25. Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. CoRR (arXiv preprint), abs/2202.06935. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Elizabeth+Clark%2C+and+Thibault+Sellam%2E+2022%2E+Repairing+the+cracked+foundation%3A+A+survey+of+obstacles+in+evaluation+practices+for+generated+text%2E+CoRR+%28arXiv+preprint%29%2C+abs%2F2202%2E0693)
26. Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019. Topical-Chat: Towards knowledge-grounded open-domain conversations. In Proceedings of Interspeech 2019, pages 1891–1895. https://doi.org/10 .21437/Inte… [find](https://scholar.google.com/scholar?q=Karthik+Gopalakrishnan%2C+Behnam+Hedayatnia%2C+Qinlang+Chen%2C+Anna+Gottardi%2C+Sanjeev+Kwatra%2C+Anu+Venkatesh%2C+Raefer+Gabriel%2C+and+Dilek+Hakkani%2DTu%CC%88r%2E+2019%2E+Topical%2DChat%3A+Towards+knowledge%2Dgrounded+open%2Ddomai)
27. Paul Grice. 1989. Studies in the Way of Words. Harvard University Press. [find](https://scholar.google.com/scholar?q=Paul+Grice%2E+1989%2E+Studies+in+the+Way+of+Words%2E+Harvard+University+Press%2E)
28. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. [Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa…
29. Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. [DialFact: A benchmark for fact-checking in dialogue](https://md.research.elamur.ai/free/en/papers/gupta-2022-dialfact-benchmark-fact-checking-dialogue/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3785–3801, Dublin, Ireland. Association for Computational…
30. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. [DeBERTa: Decoding-enhanced BERT with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In International Conference on Learning Representations.
31. Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. SpaCy: Industrial-strength natural language processing in Python. 10.5281/zenodo.1212303. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal%2C+Ines+Montani%2C+Sofie+Van+Landeghem%2C+and+Adriane+Boyd%2E+2020%2E+SpaCy%3A+Industrial%2Dstrength+natural+language+processing+in+Python%2E+10%2E5281%2Fzenodo%2E1212303%2E)
32. Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. [Q 2 : Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering](https://md.research.elamur.ai/free/en/papers/honovich-2021-q-2-evaluating-factual-consistency/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,…
33. Mladan Jovanović, Marcos Baez, and Fabio Casati. 2021. Chatbots as conversational healthcare services. IEEE Internet Computing, 25(3):44–51. https://doi.org/10.1109/MIC.2020.3037151 [↗](https://doi.org/10.1109/MIC.2020.3037151)
34. Daniel Kang and Tatsunori B. Hashimoto. 2020. [Improved natural language generation via loss truncation](https://md.research.elamur.ai/free/en/papers/kang-2020-improved-natural-language-generation-loss/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 718–731, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .acl-… [↗](https://doi.org/10.18653/v1/2020)
35. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. [CTRL: A conditional transformer language model for controllable generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/). CoRR (arXiv preprint), abs/1909.05858.
36. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+3rd+International+Conference+on+Learning+Representations%2C+ICLR+2015%2C+San+Diego%2C+CA%2C+USA%2C+May+7%E2%80%939%2C+2015%2C+Conference)
37. Ekaterina Kochmar, Dung Do Vu, Robert Belfer, Varun Gupta, Iulian Vlad Serban, and Joelle Pineau. 2021. Automated data-driven generation of personalized pedagogical interventions in intelligent tutoring systems. International Journal of Artificial Intelligence in Education, pages 1–27. [https://doi.o…](https://doi.o…) [↗](https://doi.o…)
38. Liliana Laranjo, Adam G. Dunn, Huong Ly Tong, Ahmet Baki Kocaballi, Jessica Chen, Rabia Bashir, Didi Surian, Blanca Gallego, Farah Magrabi, Annie Y. S. Lau, and Enrico Coiera. 2018. Conversational agents in healthcare: A systematic review. Journal of the American Medical Informatics Association, 25(… [find](https://scholar.google.com/scholar?q=Liliana+Laranjo%2C+Adam+G%2E+Dunn%2C+Huong+Ly+Tong%2C+Ahmet+Baki+Kocaballi%2C+Jessica+Chen%2C+Rabia+Bashir%2C+Didi+Surian%2C+Blanca+Gallego%2C+Farah+Magrabi%2C+Annie+Y%2E+S%2E+Lau%2C+and+Enrico+Coiera%2E+2018%2E+Conversational+age)
39. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
40. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. [A diversity-promoting objective function for neural conversation models](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag…
41. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
42. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR (arXiv preprint), abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+C)
43. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. [Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, Online. Association for Computation…
44. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020a. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. https://… [find](https://scholar.google.com/scholar?q=Joshua+Maynez%2C+Shashi+Narayan%2C+Bernd+Bohnet%2C+and+Ryan+McDonald%2E+2020a%2E+On+faithfulness+and+factuality+in+abstractive+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Com)
45. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020b. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. https://… [find](https://scholar.google.com/scholar?q=Joshua+Maynez%2C+Shashi+Narayan%2C+Bernd+Bohnet%2C+and+Ryan+McDonald%2E+2020b%2E+On+faithfulness+and+factuality+in+abstractive+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Com)
46. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Ling…
47. George A. Miller. 1994. [WordNet: A lexical database for English](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8–11, 1994. https://doi.org/10.3115/1075812 .1075938 [↗](https://doi.org/10.3115/1075812)
48. Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. 2020. [Syntactic data augmentation increases robustness to inference heuristics](https://md.research.elamur.ai/free/en/papers/min-2020-syntactic-data-augmentation-increases-robustness/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2339–2352, Online. Association for Computationa…
49. Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. 2021. [Improving factual consistency of abstractive summarization via question answering](https://md.research.elamur.ai/free/en/papers/nan-2021-improving-factual-consistency-abstractive-summarization/). In Proceedings of the 59th Annual Meeting of the As…
50. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. [Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics](https://md.research.elamur.ai/free/en/papers/pagnoni-2021-understanding-factuality-abstractive-summarization-frank/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag…
51. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [BLEU: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Lin…
52. Shrimai Prabhumoye, Kazuma Hashimoto, Yingbo Zhou, Alan W. Black, and Ruslan Salakhutdinov. 2021. Focused attention improves document-grounded generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… [find](https://scholar.google.com/scholar?q=Shrimai+Prabhumoye%2C+Kazuma+Hashimoto%2C+Yingbo+Zhou%2C+Alan+W%2E+Black%2C+and+Ruslan+Salakhutdinov%2E+2021%2E+Focused+attention+improves+document%2Dgrounded+generation%2E+In+Proceedings+of+the+2021+Conference+of+the+)
53. Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. [Learning compact metrics for MT](https://md.research.elamur.ai/free/en/papers/pu-2021-learning-compact-metrics-mt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 751–762, Online and Punta Cana, Dominican Republic. Association for Computational Li…
54. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2E)
55. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
56. Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021a. [Measuring attribution in natural language generation models](https://md.research.elamur.ai/free/en/papers/rashkin-2023-measuring-attribution-natural-language-generation/). CoRR (arXiv preprint), abs/2112.12870.
57. Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021b. [Increasing faithfulness in knowledge-grounded dialogue with controllable features](https://md.research.elamur.ai/free/en/papers/rashkin-2021-increasing-faithfulness-knowledge-grounded-dialogue/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Nat…
58. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. [CoQA: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266. https://doi.org/10.1162/tacl_a_00266 [↗](https://doi.org/10.1162/tacl_a_00266)
59. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. [Recipes for building an open-domain chatbot](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational…
60. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. [What makes a good conversation? How controllable attributes affect human judgments](https://md.research.elamur.ai/free/en/papers/see-2019-what-makes-good-conversation-how/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol…
61. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1 /202… [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
62. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. [Retrieval augmentation reduces hallucination in conversation](https://md.research.elamur.ai/free/en/papers/shuster-2021-retrieval-augmentation-reduces-hallucination-conversation/). In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784–3803, Punta Cana, Dominican Republic. Association for Computational Linguisti…
63. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+A+simple+way+to+prevent+neural+networks+from+overfitting%2E+Journal+of+Machine+Learning+Rese)
64. William B. Stiles. 1992. Describing Talk: A Taxonomy of Verbal Response Modes. Sage Publications. [find](https://scholar.google.com/scholar?q=William+B%2E+Stiles%2E+1992%2E+Describing+Talk%3A+A+Taxonomy+of+Verbal+Response+Modes%2E+Sage+Publications%2E)
65. Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P. Parikh. 2019. Sticking to the facts: Confident decoding for faithful data-to-text generation. CoRR (arXiv preprint), abs/1910.08684. — [arXiv:1910.08684](https://arxiv.org/abs/1910.08684) [find](https://scholar.google.com/scholar?q=Ran+Tian%2C+Shashi+Narayan%2C+Thibault+Sellam%2C+and+Ankur+P%2E+Parikh%2E+2019%2E+Sticking+to+the+facts%3A+Confident+decoding+for+faithful+data%2Dto%2Dtext+generation%2E+CoRR+%28arXiv+preprint%29%2C+abs%2F1910%2E08684%2E+%E2%80%94+%5BarXiv%3A19)
66. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
67. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. https://doi… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comput)
68. Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019a. [Dialogue natural language inference](https://md.research.elamur.ai/free/en/papers/welleck-2019-dialogue-natural-language-inference/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731–3741, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653 /…
69. Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019b. [Dialogue natural language inference](https://md.research.elamur.ai/free/en/papers/welleck-2019-dialogue-natural-language-inference/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731–3741, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653 /…
70. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
71. [Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
72. Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. CoRR (arXiv preprint), abs/1901.08149. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Victor+Sanh%2C+Julien+Chaumond%2C+and+Clement+Delangue%2E+2019%2E+Transfertransfo%3A+A+transfer+learning+approach+for+neural+network+based+conversational+agents%2E+CoRR+%28arXiv+preprint%29%2C+abs%2F1901%2E081)
73. Shanshan Yang and Chris Evans. 2019. Opportunities and challenges in using AI chatbots in higher education. In Proceedings of the 2019 3rd International Conference on Education and E-Learning, ICEEL 2019, pages 79–83, New York, NY, USA. Association for Computing Machinery. https://doi.org/10 .1145/3… [find](https://scholar.google.com/scholar?q=Shanshan+Yang+and+Chris+Evans%2E+2019%2E+Opportunities+and+challenges+in+using+AI+chatbots+in+higher+education%2E+In+Proceedings+of+the+2019+3rd+International+Conference+on+Education+and+E%2DLearning%2C+ICEEL+2)
74. Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. [A comprehensive assessment of dialog evaluation metrics](https://md.research.elamur.ai/free/en/papers/yeh-2021-comprehensive-assessment-dialog-evaluation-metrics/). In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15–33, Online. Association for Computational Linguistics.
75. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). In Advances in Neural Information Processing Systems, volume 34, pages 27263–27277. Curran Associates, Inc.
76. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. [Personalizing dialogue agents: I have a dog, do you have pets too?](https://md.research.elamur.ai/free/en/papers/zhang-2018-personalizing-dialogue-agents-i-have/) In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Melbou…
77. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [https://openreview.net/forum?id=SkeHuCVFDr](https://openreview.net/forum?id=SkeHuCVFDr) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brocke… [↗](https://openreview.net/forum?id=SkeHuCVFDr)
78. Kangyan Zhou, Shrimai Prabhumoye, and Alan W. Black. 2018. [A dataset for document grounded conversations](https://md.research.elamur.ai/free/en/papers/zhou-2018-dataset-document-grounded-conversations/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708–713, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10 .18653/…

*Updated 24 October 2026*