---
title: "Q²: Evaluating Factual Consistency in — Honovich 2021"
description: "Knowledge-grounded dialogue systems often produce fluent replies that contradict or invent facts beyond the knowledge they were given."
authors: ["Or Honovich", "Leshem Choshen", "Roee Aharoni", "Ella Neeman", "Idan Szpektor", "Omri Abend"]
year: 2021
venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2021.emnlp-main.619/"
doi: "10.18653/v1/2021.emnlp-main.619"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/honovich-2021-q-2-evaluating-factual-consistency/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/honovich-2021-q-2-evaluating-factual-consistency/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Q²: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering

# Q²: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering

**Authors:** [Or Honovich](https://md.research.elamur.ai/free/en/authors/or-honovich/), [Leshem Choshen](https://md.research.elamur.ai/free/en/authors/leshem-choshen/), [Roee Aharoni](https://md.research.elamur.ai/free/en/authors/roee-aharoni/), Ella Neeman, [Idan Szpektor](https://md.research.elamur.ai/free/en/authors/idan-szpektor/), [Omri Abend](https://md.research.elamur.ai/free/en/authors/omri-abend/) · **Year:** 2021 · **Venue:** Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.619/) · [PDF](https://aclanthology.org/2021.emnlp-main.619.pdf) · [DOI 10.18653/v1/2021.emnlp-main.619](https://doi.org/10.18653/v1/2021.emnlp-main.619) · [OpenAlex](https://openalex.org/W3153947101)

**Citations:** 112 citations (openalex-snapshot · 2026-10-14) · [83](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

Knowledge-grounded dialogue systems often produce fluent replies that contradict or invent facts beyond the knowledge they were given. Standard metrics rarely catch this, and many need reference responses, which suit open-ended dialogue poorly. The authors propose Q², a reference-free metric that checks factual consistency in three steps:

1. It extracts informative spans (named entities and noun phrases) from a system response.
2. It generates questions whose answers are those spans.
3. It answers the same questions against the grounding knowledge and compares the two answers.

Earlier QG/QA metrics for summarization compared answers by token overlap. Q² instead uses an NLI model, so paraphrases such as "LA" and "Los Angeles" still count as matches. It also filters out questions about personal opinions, which matters because dialogue mixes facts with chit-chat.

To test the metric, the authors built a new human-annotated dataset of consistent and inconsistent outputs from two systems on Wizard of Wikipedia. Across this dataset, Topical-Chat and DNLI, Q² correlates better with human judgements than competing reference-free metrics. This makes it a practical tool for assessing hallucination in grounded dialogue.

## Key points

- **Pipeline:**
  - spaCy selects the answer spans.
  - T5-base fine-tuned on SQuAD1.1 generates the questions (beam search, top-5 candidates).
  - Albert-Xlarge fine-tuned on SQuAD2.0 answers them and can return "no answer."
  - RoBERTa fine-tuned on SNLI (AllenNLP) compares the answers.
- **Scoring:**
  - Entailment counts as 1, while contradiction or a missing answer counts as 0.
  - Neutral cases fall back to token-level F1.
  - If no valid question survives filtering, an end-to-end NLI prediction decides the score.
- **New dataset:** responses from dodecaDialogue and MemNet on Wizard of Wikipedia, labeled consistent or inconsistent, in splits of 150 responses each. It is released on GitHub (orhonovich/q-squared).
- **Score separation:** on dodecaDialogue, Q² gives about 0.24 to inconsistent responses versus about 0.70 to consistent ones. On MemNet, the figures are about 0.14 versus 0.76.
- **Sanity check with random knowledge:** when the knowledge is swapped for a random passage, over 91% of questions find no answer. The rate rises to 99.6% when the passage comes from a different dialogue.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

83 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 72 not yet.

- [Multilingual Sequence-to-Sequence Models for Hebrew NLP](https://md.research.elamur.ai/free/en/papers/findings-2023-eyal-multilingual-sequence-sequence-models-hebrew/)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://md.research.elamur.ai/free/en/papers/gao-2023-rarr-researching-revising-what-language/)
- [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/)
- [TRUE: Re-evaluating Factual Consistency Evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/)
- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [大语言模型的幻觉问题研究综述](https://md.research.elamur.ai/free/en/papers/liu-2024-hallucinations-large-language/)
- [Measuring Attribution in Natural Language Generation Models](https://md.research.elamur.ai/free/en/papers/rashkin-2023-measuring-attribution-natural-language-generation/)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://md.research.elamur.ai/free/en/papers/stelmakh-2022-asqa-factoid-questions-meet-long/)
- [LaMDA: Language Models for Dialog Applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
- [AlignScore: Evaluating Factual Consistency with A Unified Alignment Function](https://md.research.elamur.ai/free/en/papers/zha-2023-alignscore-evaluating-factual-consistency-unified/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{honovich2021evaluating,
  title = {{Q²: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering}},
  author = {Honovich, Or and Choshen, Leshem and Aharoni, Roee and Neeman, Ella and Szpektor, Idan and Abend, Omri},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.619},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.619}
}
```

- [← previous QuestEval: Summarization Asks for Fact-based Evaluation](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/)
- [next → Realistic Evaluation Principles for Cross-document Coreference Resolution](https://md.research.elamur.ai/free/en/papers/cattan-2021-realistic-evaluation-principles-cross-document/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

57 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Automatic Evaluation of Dialogue Systems. Automatically evaluating natural language generation is a notoriously difficult problem, especially when considering open-ended tasks such as dialogue. Standard token-matching metrics, such as BLEU (Papineni et al., 2002) or METEOR (Banerjee and Lavie, 2005)… [find](https://scholar.google.com/scholar?q=Automatic+Evaluation+of+Dialogue+Systems%2E+Automatically+evaluating+natural+language+generation+is+a+notoriously+difficult+problem%2C+especially+when+considering+open%2Dended+tasks+such+as+dialogue%2E+Standa)
2. Recently, Mehri and Eskenazi (2020) and Pang et al. (2020) suggested to use large pretrained language models (Liu et al., 2019; Radford et al., 2019) to develop reference-response-free metrics for dialogue evaluation. Such LMs are also the backbone of the QG, QA and NLI models employed in Q 2 . [find](https://scholar.google.com/scholar?q=Recently%2C+Mehri+and+Eskenazi+%282020%29+and+Pang+et+al%2E+%282020%29+suggested+to+use+large+pretrained+language+models+%28Liu+et+al%2E%2C+2019%3B+Radford+et+al%2E%2C+2019%29+to+develop+reference%2Dresponse%2Dfree+metrics+for+dia)
3. Qualitative Analysis. To get a better impression of Q 2 ’s operation, we give examples of how it operates in its various stages. Figure 2 presents an example for an inconsistent response, together with a generated question and the answer Q 2 obtained based on the knowledge. In this example, the ques… [find](https://scholar.google.com/scholar?q=Qualitative+Analysis%2E+To+get+a+better+impression+of+Q+2+%E2%80%99s+operation%2C+we+give+examples+of+how+it+operates+in+its+various+stages%2E+Figure+2+presents+an+example+for+an+inconsistent+response%2C+together+wit)
4. Conclusion and Future Work We presented Q 2 , an automatic evaluation method for factual consistency in knowledge grounded dialogue. Q 2 employs question generation, question answering and NLI models, and does not require reference responses. To test our approach, we compiled a dataset of dialogue r… [find](https://scholar.google.com/scholar?q=Conclusion+and+Future+Work+We+presented+Q+2+%2C+an+automatic+evaluation+method+for+factual+consistency+in+knowledge+grounded+dialogue%2E+Q+2+employs+question+generation%2C+question+answering+and+NLI+models%2C)
5. Evaluation via Question Answering and Question Generation. QA-based evaluation metrics have been proposed as a means for measuring content coverage in text generation tasks. For example, Eyal et al. (2019) used QA models for abstractive summarization both as an evaluation metric and as an optimizati… [find](https://scholar.google.com/scholar?q=Evaluation+via+Question+Answering+and+Question+Generation%2E+QA%2Dbased+evaluation+metrics+have+been+proposed+as+a+means+for+measuring+content+coverage+in+text+generation+tasks%2E+For+example%2C+Eyal+et+al%2E+%28)
6. Durmus et al. (2020) and Wang et al. (2020) suggested using QG and QA to identify factual inconsistencies in abstractive summaries, which is more Precision-oriented. Their approach is based on the intuition Acknowledgements that if a summary is consistent with its source, This work was carried out a… [find](https://scholar.google.com/scholar?q=Durmus+et+al%2E+%282020%29+and+Wang+et+al%2E+%282020%29+suggested+using+QG+and+QA+to+identify+factual+inconsistencies+in+abstractive+summaries%2C+which+is+more+Precision%2Doriented%2E+Their+approach+is+based+on+the+int) [our card](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/)
7. Antoine Bosselut, Esin Durmus, Varun Prashant Gangal, Sebastian Gehrmann, Yacine Jernite, Laura Perez-Beltrachini, Samira Shaikh, and Wei Xu, editors. 2021. Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021). Association for Computational Linguistics,… [find](https://scholar.google.com/scholar?q=Antoine+Bosselut%2C+Esin+Durmus%2C+Varun+Prashant+Gangal%2C+Sebastian+Gehrmann%2C+Yacine+Jernite%2C+Laura+Perez%2DBeltrachini%2C+Samira+Shaikh%2C+and+Wei+Xu%2C+editors%2E+2021%2E+Proceedings+of+the+1st+Workshop+on+Natural+)
8. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
9. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+2018%2E+Hierarchical+neural+story+generation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29)
10. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform. [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Joel+Grus%2C+Mark+Neumann%2C+Oyvind+Tafjord%2C+Pradeep+Dasigi%2C+Nelson+F%2E+Liu%2C+Matthew+Peters%2C+Michael+Schmitz%2C+and+Luke+S%2E+Zettlemoyer%2E+2017%2E+Allennlp%3A+A+deep+semantic+natural+language+process) [our card](https://md.research.elamur.ai/free/en/papers/gardner-2018-allennlp-deep-semantic-natural-language/)
11. Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In KDD ’19, KDD ’19, page 166–175, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Ben+Goodrich%2C+Vinay+Rao%2C+Peter+J%2E+Liu%2C+and+Mohammad+Saleh%2E+2019%2E+Assessing+the+factual+accuracy+of+generated+text%2E+In+KDD+%E2%80%9919%2C+KDD+%E2%80%9919%2C+page+166%E2%80%93175%2C+New+York%2C+NY%2C+USA%2E+Association+for+Computing+Machi)
12. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. [find](https://scholar.google.com/scholar?q=Asli+Celikyilmaz%2C+Elizabeth+Clark%2C+and+Jianfeng+Gao%2E+2020%2E+Evaluation+of+text+generation%3A+A+survey%2E)
13. Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019. [find](https://scholar.google.com/scholar?q=Karthik+Gopalakrishnan%2C+Behnam+Hedayatnia%2C+Qinlang+Chen%2C+Anna+Gottardi%2C+Sanjeev+Kwatra%2C+Anu+Venkatesh%2C+Raefer+Gabriel%2C+and+Dilek+Hakkani%2DT%C3%BCr%2E+2019%2E)
14. Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In Proc. Interspeech 2019, pages 1891–1895. [find](https://scholar.google.com/scholar?q=Topical%2DChat%3A+Towards+Knowledge%2DGrounded+Open%2DDomain+Conversations%2E+In+Proc%2E+Interspeech+2019%2C+pages+1891%E2%80%931895%2E)
15. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177–190, Berlin, Heidelberg. Springer Berlin Heidelber… [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges%2E+Evaluating+Predictive+Uncertainty%2C+Visual+Object+Classificati)
16. Yvette Graham and Qun Liu. 2016. Achieving accurate conclusions in evaluation of automatic machine translation metrics. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1–10, San Diego, California… [find](https://scholar.google.com/scholar?q=Yvette+Graham+and+Qun+Liu%2E+2016%2E+Achieving+accurate+conclusions+in+evaluation+of+automatic+machine+translation+metrics%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Assoc)
17. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Emily+Dinan%2C+Stephen+Roller%2C+Kurt+Shuster%2C+Angela+Fan%2C+Michael+Auli%2C+and+Jason+Weston%2E+2019%2E+Wizard+of+Wikipedia%3A+Knowledge%2Dpowered+conversational+agents%2E+In+Proceedings+of+the+International+Conferenc)
18. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2020%2E+The+curious+case+of+neural+text+degeneration%2E+In+International+Conference+on+Learning+Representations%2E)
19. Ziqiang Cao, Furu Wei, W. Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In AAAI. [find](https://scholar.google.com/scholar?q=Ziqiang+Cao%2C+Furu+Wei%2C+W%2E+Li%2C+and+Sujian+Li%2E+2018%2E+Faithful+to+the+original%3A+Fact+aware+neural+abstractive+summarization%2E+In+AAAI%2E)
20. Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055– 5070, Online. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Esin+Durmus%2C+He+He%2C+and+Mona+Diab%2E+2020%2E+FEQA%3A+A+question+answering+evaluation+framework+for+faithfulness+assessment+in+abstractive+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Asso) [our card](https://md.research.elamur.ai/free/en/papers/durmus-2020-feqa-question-answering-evaluation-framework/)
21. Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2021. Evaluating groundedness in dialogue systems: The begin benchmark. [find](https://scholar.google.com/scholar?q=Nouha+Dziri%2C+Hannah+Rashkin%2C+Tal+Linzen%2C+and+David+Reitter%2E+2021%2E+Evaluating+groundedness+in+dialogue+systems%3A+The+begin+benchmark%2E)
22. B. Efron. 1987. Better bootstrap confidence intervals. Journal of the American Statistical Association, 82:171–185. [find](https://scholar.google.com/scholar?q=B%2E+Efron%2E+1987%2E+Better+bootstrap+confidence+intervals%2E+Journal+of+the+American+Statistical+Association%2C+82%3A171%E2%80%93185%2E)
23. Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long an… [find](https://scholar.google.com/scholar?q=Matan+Eyal%2C+Tal+Baumel%2C+and+Michael+Elhadad%2E+2019%2E+Question+answering+as+an+automatic+evaluation+metric+for+news+article+summarization%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chap)
24. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Rebecca+Knowles%2E+2017%2E+Six+challenges+for+neural+machine+translation%2E+In+Proceedings+of+the+First+Workshop+on+Neural+Machine+Translation%2C+pages+28%E2%80%9339%2C+Vancouver%2E+Association+for+Comp)
25. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2019%2E+Evaluating+the+factual+consistency+of+abstractive+text+summarization%2E+arXiv+preprint+arXiv%3A1910%2E12840%2E)
26. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+International+Confe)
27. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2019. Hallucinations in neural machine translation. [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Orhan+Firat%2C+Ashish+Agarwal%2C+Clara+Fannjiang%2C+and+David+Sussillo%2E+2019%2E+Hallucinations+in+neural+machine+translation%2E)
28. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E+Association+for+Computational+Linguistics%2E)
29. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural L… [find](https://scholar.google.com/scholar?q=Chia%2DWei+Liu%2C+Ryan+Lowe%2C+Iulian+Serban%2C+Mike+Noseworthy%2C+Laurent+Charlin%2C+and+Joelle+Pineau%2E+2016%2E+How+NOT+to+evaluate+your+dialogue+system%3A+An+empirical+study+of+unsupervised+evaluation+metrics+for+d)
30. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
31. Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… [find](https://scholar.google.com/scholar?q=Ryan+Lowe%2C+Michael+Noseworthy%2C+Iulian+Vlad+Serban%2C+Nicolas+Angelard%2DGontier%2C+Yoshua+Bengio%2C+and+Joelle+Pineau%2E+2017%2E+Towards+an+automatic+Turing+test%3A+Learning+to+evaluate+dialogue+responses%2E+In+Proce) [our card](https://md.research.elamur.ai/free/en/papers/lowe-2017-towards-automatic-turing-test-learning/)
32. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Joshua+Maynez%2C+Shashi+Narayan%2C+Bernd+Bohnet%2C+and+Ryan+McDonald%2E+2020%2E+On+faithfulness+and+factuality+in+abstractive+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comp)
33. Shikib Mehri and Maxine Eskenazi. 2020. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shikib+Mehri+and+Maxine+Eskenazi%2E+2020%2E+USR%3A+An+unsupervised+and+reference+free+evaluation+metric+for+dialog+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+) [our card](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/)
34. A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. 2017. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476. [find](https://scholar.google.com/scholar?q=A%2E+H%2E+Miller%2C+W%2E+Feng%2C+A%2E+Fisch%2C+J%2E+Lu%2C+D%2E+Batra%2C+A%2E+Bordes%2C+D%2E+Parikh%2C+and+J%2E+Weston%2E+2017%2E+Parlai%3A+A+dialog+research+software+platform%2E+arXiv+preprint+arXiv%3A1705%2E06476%2E) [our card](https://md.research.elamur.ai/free/en/papers/miller-2017-parlai-dialog-research-software-platform/)
35. Mathias Müller, Annette Rios, and Rico Sennrich. 2020. Domain robustness in neural machine translation. In Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 151–164, Virtual. Association for Machine Translation in the Amer… [find](https://scholar.google.com/scholar?q=Mathias+M%C3%BCller%2C+Annette+Rios%2C+and+Rico+Sennrich%2E+2020%2E+Domain+robustness+in+neural+machine+translation%2E+In+Proceedings+of+the+14th+Conference+of+the+Association+for+Machine+Translation+in+the+Americas) [our card](https://md.research.elamur.ai/free/en/papers/muller-2020-domain-robustness-neural-machine-translation/)
36. Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. 2021. Improving factual consistency of abstractive summarization via question answering. In Proceedings of the 59th Annual Meeting of the As… [find](https://scholar.google.com/scholar?q=Feng+Nan%2C+Cicero+Nogueira+dos+Santos%2C+Henghui+Zhu%2C+Patrick+Ng%2C+Kathleen+McKeown%2C+Ramesh+Nallapati%2C+Dejiao+Zhang%2C+Zhiguo+Wang%2C+Andrew+O%2E+Arnold%2C+and+Bing+Xiang%2E+2021%2E+Improving+factual+consistency+of+a)
37. Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. 2020. Towards holistic and automatic evaluation of open-domain dialogue generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3619–3629, Online. Association for Computati… [find](https://scholar.google.com/scholar?q=Bo+Pang%2C+Erik+Nijkamp%2C+Wenjuan+Han%2C+Linqi+Zhou%2C+Yixian+Liu%2C+and+Kewei+Tu%2E+2020%2E+Towards+holistic+and+automatic+evaluation+of+open%2Ddomain+dialogue+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+)
38. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
39. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical Report. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+Report%2E)
40. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
41. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789, Melbourne, Australia. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+SQuAD%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
42. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
43. Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021. Increasing faithfulness in knowledge-grounded dialogue with controllable features. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu… [find](https://scholar.google.com/scholar?q=Hannah+Rashkin%2C+David+Reitter%2C+Gaurav+Singh+Tomar%2C+and+Dipanjan+Das%2E+2021%2E+Increasing+faithfulness+in+knowledge%2Dgrounded+dialogue+with+controllable+features%2E+In+Proceedings+of+the+59th+Annual+Meeting+)
44. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston. 2020. Recipes for building an open-domain chatbot. [find](https://scholar.google.com/scholar?q=Stephen+Roller%2C+Emily+Dinan%2C+Naman+Goyal%2C+Da+Ju%2C+Mary+Williamson%2C+Yinhan+Liu%2C+Jing+Xu%2C+Myle+Ott%2C+Kurt+Shuster%2C+Eric+M%2E+Smith%2C+Y%2DLan+Boureau%2C+and+Jason+Weston%2E+2020%2E+Recipes+for+building+an+open%2Ddomain) [our card](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/)
45. Thomas Scialom, Paul-Alexis Dray, Gallinari Patrick, Lamprier Sylvain, Piwowarski Benjamin, Staiano Jacopo, and Wang Alex. 2021. Questeval: Summarization asks for fact-based evaluation. arXiv preprint arXiv:2103.12693. [find](https://scholar.google.com/scholar?q=Thomas+Scialom%2C+Paul%2DAlexis+Dray%2C+Gallinari+Patrick%2C+Lamprier+Sylvain%2C+Piwowarski+Benjamin%2C+Staiano+Jacopo%2C+and+Wang+Alex%2E+2021%2E+Questeval%3A+Summarization+asks+for+fact%2Dbased+evaluation%2E+arXiv+preprint) [our card](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/)
46. Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La… [find](https://scholar.google.com/scholar?q=Thomas+Scialom%2C+Sylvain+Lamprier%2C+Benjamin+Piwowarski%2C+and+Jacopo+Staiano%2E+2019%2E+Answers+unite%21+unsupervised+metrics+for+reinforced+summarization+models%2E+In+Proceedings+of+the+2019+Conference+on+Empir) [our card](https://md.research.elamur.ai/free/en/papers/scialom-2019-answers-unite-unsupervised-metrics-reinforced/)
47. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
48. Linguistics (Volume 1: Long Papers), pages 2204– 2213, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pages+2204%E2%80%93+2213%2C+Melbourne%2C+Australia%2E+Association+for+Computational+Linguistics%2E)
49. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
50. Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020. Designing precise and robust dialogue response evaluators. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 26–33, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tianyu+Zhao%2C+Divesh+Lala%2C+and+Tatsuya+Kawahara%2E+2020%2E+Designing+precise+and+robust+dialogue+response+evaluators%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguist)
51. Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, and Jason Weston. 2020. The dialogue dodecathlon: Open-domain knowledge and image grounded conversational agents. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2453–2470, Online. Asso… [find](https://scholar.google.com/scholar?q=Kurt+Shuster%2C+Da+Ju%2C+Stephen+Roller%2C+Emily+Dinan%2C+Y%2DLan+Boureau%2C+and+Jason+Weston%2E+2020%2E+The+dialogue+dodecathlon%3A+Open%2Ddomain+knowledge+and+image+grounded+conversational+agents%2E+In+Proceedings+of+the)
52. James Thorne and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3346–3359, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=James+Thorne+and+Andreas+Vlachos%2E+2018%2E+Automated+fact+checking%3A+Task+formulations%2C+methods+and+future+directions%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+pag) [our card](https://md.research.elamur.ai/free/en/papers/thorne-2018-automated-fact-checking-task-formulations/)
53. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comput)
54. Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544–3552, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chaojun+Wang+and+Rico+Sennrich%2E+2020%2E+On+exposure+bias%2C+hallucination+and+domain+shift+in+neural+machine+translation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Lin) [our card](https://md.research.elamur.ai/free/en/papers/wang-2020-exposure-bias-hallucination-domain-shift/)
55. Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731–3741, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sean+Welleck%2C+Jason+Weston%2C+Arthur+Szlam%2C+and+Kyunghyun+Cho%2E+2019%2E+Dialogue+natural+language+inference%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+page) [our card](https://md.research.elamur.ai/free/en/papers/welleck-2019-dialogue-natural-language-inference/)
56. Xinnuo Xu, Ondřej Dušek, Jingyi Li, Verena Rieser, and Ioannis Konstas. 2020. Fact-based content weighting for evaluating abstractive summarisation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5071–5081, Online. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Xinnuo+Xu%2C+Ondr%CC%8Cej+Du%C5%A1ek%2C+Jingyi+Li%2C+Verena+Rieser%2C+and+Ioannis+Konstas%2E+2020%2E+Fact%2Dbased+content+weighting+for+evaluating+abstractive+summarisation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+A)
57. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational [find](https://scholar.google.com/scholar?q=Saizheng+Zhang%2C+Emily+Dinan%2C+Jack+Urbanek%2C+Arthur+Szlam%2C+Douwe+Kiela%2C+and+Jason+Weston%2E+2018%2E+Personalizing+dialogue+agents%3A+I+have+a+dog%2C+do+you+have+pets+too%3F+In+Proceedings+of+the+56th+Annual+Meeti) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2018-personalizing-dialogue-agents-i-have/)