---
title: "Tomayto, Tomahto. Beyond Token-level Answer Equivalence for"
description: "The authors define Answer Equivalence (AE), an asymmetric relation that depends on the question… 50 references, cited by 56 papers here."
authors: ["Jannis Bulian", "Christian Buck", "Wojciech Gajewski", "Benjamin Börschinger", "Tal Schuster"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.20/"
doi: "10.18653/v1/2022.emnlp-main.20"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bulian-2022-tomayto-tomahto-beyond-token-level/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bulian-2022-tomayto-tomahto-beyond-token-level/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation

# Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation

**Authors:** [Jannis Bulian](https://md.research.elamur.ai/free/en/authors/jannis-bulian/), [Christian Buck](https://md.research.elamur.ai/free/en/authors/christian-buck/), Wojciech Gajewski, Benjamin Börschinger and [Tal Schuster](https://md.research.elamur.ai/free/en/authors/tal-schuster/) · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.20/) · [PDF](https://aclanthology.org/2022.emnlp-main.20.pdf) · [DOI 10.18653/v1/2022.emnlp-main.20](https://doi.org/10.18653/v1/2022.emnlp-main.20) · [OpenAlex](https://openalex.org/W4385572714)

**Citations:** 44 citations (OpenAlex, 2026-10-14) · [56](https://md.research.elamur.ai/free/en/papers/bulian-2022-tomayto-tomahto-beyond-token-level/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Question answering systems are usually scored by comparing their output to a small set of human-written reference answers using exact match (EM) or token-level F1. These metrics miss correct answers that are worded differently and can reward wrong answers that happen to share tokens with the reference. The authors define Answer Equivalence (AE), an asymmetric relation that depends on the question and the context. Under AE, a candidate is acceptable if it keeps all relevant content of the reference and adds nothing misleading or superfluous, so a candidate that improves on the reference also gets full credit.

They gathered more than 23k human AE judgments on SQuAD for predictions from several QA systems. This data shows that EM and F1 underestimate system quality, depend strongly on how many references are available, and imply a graded scale that human ratings do not support.

They then trained BEM, a BERT classifier that predicts equivalence. BEM agrees with human judgments much better than F1 and generalizes to unseen models and questions. Because it is simple to use, BEM can stand in for F1 in downstream applications that rely on QA outputs.

## Key points
- **Released resources:** the AE annotations (over 23k ratings on SQuAD candidates from multiple QA systems) are on GitHub (google-research-datasets/answer-equivalence-dataset) and in TFDS. The trained BEM model is available on TF Hub.
- **Documented F1 failures:**
  - Tokenization mismatches, such as "Napoleon's" vs "Napoleon".
  - Blindness to the question, as in fused-head cases like "secondary school" vs "secondary school teachers".
  - Blindness to the context, as when "Queen Bees" is used as a metaphor for "women".
  - Unit insensitivity: "50–140 cm" vs "0.5–1.4 m" scores 0.
  - High overlap for wrong answers: "P is not equal to NP" scores 0.84.
  - No graduality: candidates with any non-zero F1 have a similar proportion of equivalent to non-equivalent answers.
- **BEM evaluation:** experiments with three QA systems show BEM tracks human AE judgments more closely than F1 and gives more accurate system rankings. The final BEM does not use the passage context.
- **Prediction sets:** when used in Fisch et al.'s (2021) conformal prediction approach with expanded admission, AE and BEM cut the number of answers needed in accurate prediction sets by up to ×2.6.
- **Limitations:**
  - Covers only English data and short, extractive answers.
  - Gives only preliminary results on NQ-Open.
  - Tends to accept generative answers that contain unsupported, more specific details.
  - Does not handle answers whose correctness depends on time or place.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 242 works

## Cite

Cite as: Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Börschinger and Tal Schuster. 2022. Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bulian2022tomayto,
  title = {{Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation}},
  author = {Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Börschinger and Tal Schuster},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.20},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.20}
}
```

- [← previous On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations](https://md.research.elamur.ai/free/en/papers/cao-2022-intrinsic-extrinsic-fairness-evaluation-metrics/)
- [next → UniTE: Unified Translation Evaluation](https://md.research.elamur.ai/free/en/papers/wan-2022-unite-unified-translation-evaluation/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

50 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anastasios N. Angelopoulos and Stephen Bates. 2021. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. [find](https://scholar.google.com/scholar?q=Anastasios+N%2E+Angelopoulos+and+Stephen+Bates%2E+2021%2E+A+gentle+introduction+to+conformal+prediction+and+distribution%2Dfree+uncertainty+quantification%2E)
2. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass…
3. Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. 2021. [find](https://scholar.google.com/scholar?q=Stephen+Bates%2C+Anastasios+Angelopoulos%2C+Lihua+Lei%2C+Jitendra+Malik%2C+and+Michael+Jordan%2E+2021%2E)
4. Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM), 68(6):1–34. [find](https://scholar.google.com/scholar?q=Distribution%2Dfree%2C+risk%2Dcontrolling+prediction+sets%2E+Journal+of+the+ACM+%28JACM%29%2C+68%286%29%3A1%E2%80%9334%2E)
5. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, page 1247–1250, New York, NY, USA.… [find](https://scholar.google.com/scholar?q=Kurt+Bollacker%2C+Colin+Evans%2C+Praveen+Paritosh%2C+Tim+Sturge%2C+and+Jamie+Taylor%2E+2008%2E+Freebase%3A+A+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+Proceedings+of+the+2008+ACM+SI)
6. Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. [Question answering as an automatic evaluation metric for news article summarization](https://md.research.elamur.ai/free/en/papers/eyal-2019-question-answering-as-automatic-evaluation/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long an…
7. Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2021. [Qafacteval: Improved qa-based factual consistency evaluation for summarization](https://md.research.elamur.ai/free/en/papers/fabbri-2022-qafacteval-improved-qa-based-factual/). CoRR, abs/2112.08542.
8. Eric Breck, John Burger, Lisa Ferro, David House, Marc Light, and Inderjeet Mani. 1999. A sys called qanda. In Proceedings of The Eighth Text REtrieval Conference. [find](https://scholar.google.com/scholar?q=Eric+Breck%2C+John+Burger%2C+Lisa+Ferro%2C+David+House%2C+Marc+Light%2C+and+Inderjeet+Mani%2E+1999%2E+A+sys+called+qanda%2E+In+Proceedings+of+The+Eighth+Text+REtrieval+Conference%2E)
9. Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Evaluating question answering evaluation. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 119–124, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anthony+Chen%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2019%2E+Evaluating+question+answering+evaluation%2E+In+Proceedings+of+the+2nd+Workshop+on+Machine+Reading+for+Question+Answering%2C+pages+119)
10. Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2020. MOCHA: A dataset for training and evaluating generative reading comprehension metrics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 652… [find](https://scholar.google.com/scholar?q=Anthony+Chen%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2020%2E+MOCHA%3A+A+dataset+for+training+and+evaluating+generative+reading+comprehension+metrics%2E+In+Proceedings+of+the+2020+Conference+on+E)
11. Jifan Chen, Eunsol Choi, and Greg Durrett. 2021. Can NLI models verify QA systems’ predictions? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3841–3854, Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jifan+Chen%2C+Eunsol+Choi%2C+and+Greg+Durrett%2E+2021%2E+Can+NLI+models+verify+QA+systems%E2%80%99+predictions%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2021%2C+pages+3841%E2%80%933854%2C+Punta+Cana%2C+D)
12. Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019. [Sentence mover’s similarity: Automatic evaluation for multi-sentence texts](https://md.research.elamur.ai/free/en/papers/clark-2019-sentence-mover-s-similarity-automatic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2748–2760, Florence, Italy. Association for Computational Linguist…
13. William W. Cohen, Pradeep Ravikumar, and Stephen E. Fienberg. 2003. A comparison of string distance metrics for name-matching tasks. In IIWeb. [find](https://scholar.google.com/scholar?q=William+W%2E+Cohen%2C+Pradeep+Ravikumar%2C+and+Stephen+E%2E+Fienberg%2E+2003%2E+A+comparison+of+string+distance+metrics+for+name%2Dmatching+tasks%2E+In+IIWeb%2E)
14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
15. Yanai Elazar and Yoav Goldberg. 2019. Where’s My Head? Definition, Data Set, and Models for Numeric Fused-Head Identification and Resolution. Transactions of the Association for Computational Linguistics, 7:519–535. [find](https://scholar.google.com/scholar?q=Yanai+Elazar+and+Yoav+Goldberg%2E+2019%2E+Where%E2%80%99s+My+Head%3F+Definition%2C+Data+Set%2C+and+Models+for+Numeric+Fused%2DHead+Identification+and+Resolution%2E+Transactions+of+the+Association+for+Computational+Linguist)
16. Adam Fisch, Robin Jia, and Tal Schuster. 2022. Uncertainty estimation for natural language processing. In COLING. [find](https://scholar.google.com/scholar?q=Adam+Fisch%2C+Robin+Jia%2C+and+Tal+Schuster%2E+2022%2E+Uncertainty+estimation+for+natural+language+processing%2E+In+COLING%2E)
17. Adam Fisch, Tal Schuster, Tommi S. Jaakkola, and Regina Barzilay. 2021. Efficient conformal prediction via cascaded inference with expanded admission. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Adam+Fisch%2C+Tal+Schuster%2C+Tommi+S%2E+Jaakkola%2C+and+Regina+Barzilay%2E+2021%2E+Efficient+conformal+prediction+via+cascaded+inference+with+expanded+admission%2E+In+International+Conference+on+Learning+Represent)
18. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yu… [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Tosin+Adewumi%2C+Karmanya+Aggarwal%2C+Pawan+Sasanka+Ammanamanchi%2C+Aremu+Anuoluwapo%2C+Antoine+Bosselut%2C+Khyathi+Raghavi+Chandu%2C+Miruna+Clinciu%2C+Dipanjan+Das%2C+Kaustubh+D%2E+Dhole%2C+Wanyu+Du%2C)
19. Uri Gneezy and Aldo Rustichini. 2000. Pay enough or don’t pay at all. The Quarterly Journal of Economics, 115(3):791–810. [find](https://scholar.google.com/scholar?q=Uri+Gneezy+and+Aldo+Rustichini%2E+2000%2E+Pay+enough+or+don%E2%80%99t+pay+at+all%2E+The+Quarterly+Journal+of+Economics%2C+115%283%29%3A791%E2%80%93810%2E)
20. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. [TRUE: Re-evaluating factual consistency evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/). In Proceedings of the 2022 Conference of the North American Chapter of the Associa…
21. Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. [q 2 : Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering](https://md.research.elamur.ai/free/en/papers/honovich-2021-q-2-evaluating-factual-consistency/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,…
22. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. [find](https://scholar.google.com/scholar?q=Tom+Kocmi%2C+Christian+Federmann%2C+Roman+Grundkiewicz%2C+Marcin+Junczys%2DDowmunt%2C+Hitokazu+Matsushita%2C+and+Arul+Menezes%2E+2021%2E+To+ship+or+not+to+ship%3A+An+extensive+evaluation+of+automatic+metrics+for+machin)
23. Klaus Krippendorff. 2004. Content Analysis: An Introduction to Its Methodology (second edition). Sage Publications. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2004%2E+Content+Analysis%3A+An+Introduction+to+Its+Methodology+%28second+edition%29%2E+Sage+Publications%2E)
24. Matt J Kusner, Yu Sun, Nicholas I Kolkin, and Kilian Q Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, pages 957–966. JMLR.org. [find](https://scholar.google.com/scholar?q=Matt+J+Kusner%2C+Yu+Sun%2C+Nicholas+I+Kolkin%2C+and+Kilian+Q+Weinberger%2E+2015%2E+From+word+embeddings+to+document+distances%2E+In+Proceedings+of+the+32nd+International+Conference+on+International+Conference+on+)
25. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. CoRR, abs/1909.11942. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2019%2E+ALBERT%3A+A+lite+BERT+for+self%2Dsupervised+learning+of+language+representations%2E+CoRR%2C+abs%2F1909%2E11942%2E)
26. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. [Latent retrieval for weakly supervised open domain question answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
27. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
28. Sewon Min, Jordan L. Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick S. H. Lewis, Yuxiang Wu, Heinrich Küttler, Linqing Liu, Pasquale Minervini, Pontus Stenet… [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Jordan+L%2E+Boyd%2DGraber%2C+Chris+Alberti%2C+Danqi+Chen%2C+Eunsol+Choi%2C+Michael+Collins%2C+Kelvin+Guu%2C+Hannaneh+Hajishirzi%2C+Kenton+Lee%2C+Jennimaria+Palomaki%2C+Colin+Raffel%2C+Adam+Roberts%2C+Tom+Kwiatkowski)
29. Neurips 2020 efficientqa competition: Systems, analyses and lessons learned. In NeurIPS 2020 Competition and Demonstration Track, 6-12 December 2020, Virtual Event / Vancouver, BC, Canada, volume 133 of Proceedings of Machine Learning Research, pages 86–111. PMLR. [find](https://scholar.google.com/scholar?q=Neurips+2020+efficientqa+competition%3A+Systems%2C+analyses+and+lessons+learned%2E+In+NeurIPS+2020+Competition+and+Demonstration+Track%2C+6%2D12+December+2020%2C+Virtual+Event+%2F+Vancouver%2C+BC%2C+Canada%2C+volume+133+)
30. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020b. Ambigqa: Answering ambiguous open-domain questions. CoRR, abs/2004.10645. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2020b%2E+Ambigqa%3A+Answering+ambiguous+open%2Ddomain+questions%2E+CoRR%2C+abs%2F2004%2E10645%2E)
31. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics.
32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
33. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
34. Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. 2021. Semantic answer similarity for evaluating question answering models. [find](https://scholar.google.com/scholar?q=Julian+Risch%2C+Timo+Mo%CC%88ller%2C+Julian+Gutsch%2C+and+Malte+Pietsch%2E+2021%2E+Semantic+answer+similarity+for+evaluating+question+answering+models%2E)
35. Tal Schuster, Adam Fisch, and Regina Barzilay. 2021a. [Get your vitamin C! robust fact verification with contrastive evidence](https://md.research.elamur.ai/free/en/papers/schuster-2021-get-your-vitamin-c-robust/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624–643, Online. Asso…
36. Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Quang Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Tal+Schuster%2C+Adam+Fisch%2C+Jai+Gupta%2C+Mostafa+Dehghani%2C+Dara+Bahri%2C+Vinh+Quang+Tran%2C+Yi+Tay%2C+and+Donald+Metzler%2E+2022%2E+Confident+adaptive+language+modeling%2E+In+Advances+in+Neural+Information+Processing)
37. Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. 2021b. Consistent accelerated inference via confident adaptive transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4962– 4979, Online and Punta Cana, Dominican Republic. Associat… [find](https://scholar.google.com/scholar?q=Tal+Schuster%2C+Adam+Fisch%2C+Tommi+Jaakkola%2C+and+Regina+Barzilay%2E+2021b%2E+Consistent+accelerated+inference+via+confident+adaptive+transformers%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+i)
38. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020a. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020a%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C)
39. Thibault Sellam, Amy Pu, Hyung Won Chung, Sebastian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur Parikh. 2020b. [Learning to evaluate translation beyond English: BLEURT submissions to the WMT metrics 2020 shared task](https://md.research.elamur.ai/free/en/papers/sellam-2020-learning-evaluate-translation-beyond-english/). In Proceedings of the Fifth Conference on Machine Translation, page…
40. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. [Bidirectional attention flow for machine comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/). CoRR, abs/1611.01603.
41. Glenn Shafer and Vladimir Vovk. 2007. A tutorial on conformal prediction. CoRR, abs/0706.3188. — [arXiv:0706.3188](https://arxiv.org/abs/0706.3188) [find](https://scholar.google.com/scholar?q=Glenn+Shafer+and+Vladimir+Vovk%2E+2007%2E+A+tutorial+on+conformal+prediction%2E+CoRR%2C+abs%2F0706%2E3188%2E+%E2%80%94+%5BarXiv%3A0706%2E3188%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F0706%2E3188%29)
42. Chenglei Si, Chen Zhao, and Jordan Boyd-Graber. 2021. What’s in a name? answer equivalence for [open-domain question answering](https://md.research.elamur.ai/free/en/papers/chen-2020-open-domain-question-answering/). In Empirical Methods in Natural Language Processing.
43. Ellen M. Voorhees and Dawn M. Tice. 2000. Building a question answering test collection. In SIGIR 2000: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 24-28, 2000, Athens, Greece, pages 200–207. ACM. [find](https://scholar.google.com/scholar?q=Ellen+M%2E+Voorhees+and+Dawn+M%2E+Tice%2E+2000%2E+Building+a+question+answering+test+collection%2E+In+SIGIR+2000%3A+Proceedings+of+the+23rd+Annual+International+ACM+SIGIR+Conference+on+Research+and+Development+in)
44. Vladimir Vovk, Alex Gammerman, and Glenn Shafer. 2005. Algorithmic Learning in a Random World. Springer-Verlag, Berlin, Heidelberg. [find](https://scholar.google.com/scholar?q=Vladimir+Vovk%2C+Alex+Gammerman%2C+and+Glenn+Shafer%2E+2005%2E+Algorithmic+Learning+in+a+Random+World%2E+Springer%2DVerlag%2C+Berlin%2C+Heidelberg%2E)
45. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. [Towards ai-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). CoRR.
46. Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020,… [find](https://scholar.google.com/scholar?q=Ikuya+Yamada%2C+Akari+Asai%2C+Hiroyuki+Shindo%2C+Hideaki+Takeda%2C+and+Yuji+Matsumoto%2E+2020%2E+LUKE%3A+deep+contextualized+entity+representations+with+entity%2Daware+self%2Dattention%2E+In+Proceedings+of+the+2020+Confe)
47. An Yang, Kai Liu, Jing Liu, Yajuan Lyu, and Sujian Li. 2018. Adaptations of ROUGE and BLEU to better evaluate machine reading comprehension task. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 98–104, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Kai+Liu%2C+Jing+Liu%2C+Yajuan+Lyu%2C+and+Sujian+Li%2E+2018%2E+Adaptations+of+ROUGE+and+BLEU+to+better+evaluate+machine+reading+comprehension+task%2E+In+Proceedings+of+the+Workshop+on+Machine+Reading+for+)
48. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
49. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+BERT%2E+In+8th+International+Conference+on+Learning+Representations%2C+ICLR+2)
50. Metric Threshold F 1 LERC 0.2 2.52 85.06 81.97 82.47 80.74 BEM 0.56 89.99 89.80 XLNet BiDAF Luke Total Non-exact matches AE-examples Ratings 1,205 2,448 7,932 3,000 5,655 7,522 1,082 2,240 4,590 4,369 8,565 14,170 *Table 10: Data statistics for human rated (question, context, reference, candidate) 4… [find](https://scholar.google.com/scholar?q=Metric+Threshold+F+1+LERC+0%2E2+2%2E52+85%2E06+81%2E97+82%2E47+80%2E74+BEM+0%2E56+89%2E99+89%2E80+XLNet+BiDAF+Luke+Total+Non%2Dexact+matches+AE%2Dexamples+Ratings+1%2C205+2%2C448+7%2C932+3%2C000+5%2C655+7%2C522+1%2C082+2%2C240+4%2C590+4%2C369)

*Updated 24 October 2026*