---
title: "Generating Benchmarks for Factuality Evaluation of Language Models — paper by Muhlgay et al. (2024)"
description: "The authors propose FACTOR (Factual Assessment via Corpus TransfORmation). 44 references, cited by 57 papers here."
authors: ["Dor Muhlgay", "Ori Ram", "Inbal Magar", "Yoav Levine", "Nir Ratner", "Yonatan Belinkov", "Omri Abend", "Kevin Leyton-Brown", "Amnon Shashua", "Yoav Shoham"]
year: 2024
venue: "Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.eacl-long.4/"
doi: "10.18653/v1/2024.eacl-long.4"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/muhlgay-2024-generating-benchmarks-factuality-evaluation-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/muhlgay-2024-generating-benchmarks-factuality-evaluation-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Generating Benchmarks for Factuality Evaluation of Language Models

# Generating Benchmarks for Factuality Evaluation of Language Models

**Authors:** Dor Muhlgay, [Ori Ram](https://md.research.elamur.ai/free/en/authors/ori-ram/), [Inbal Magar](https://md.research.elamur.ai/free/en/authors/inbal-magar/), [Yoav Levine](https://md.research.elamur.ai/free/en/authors/yoav-levine/), [Nir Ratner](https://md.research.elamur.ai/free/en/authors/nir-ratner/), [Yonatan Belinkov](https://md.research.elamur.ai/free/en/authors/yonatan-belinkov/), [Omri Abend](https://md.research.elamur.ai/free/en/authors/omri-abend/), [Kevin Leyton-Brown](https://md.research.elamur.ai/free/en/authors/kevin-leyton-brown/), [Amnon Shashua](https://md.research.elamur.ai/free/en/authors/amnon-shashua/) and [Yoav Shoham](https://md.research.elamur.ai/free/en/authors/yoav-shoham/) · **Year:** 2024 · **Venue:** [Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2024](https://md.research.elamur.ai/free/en/venues/eacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.eacl-long.4/) · [PDF](https://aclanthology.org/2024.eacl-long.4.pdf) · [DOI 10.18653/v1/2024.eacl-long.4](https://doi.org/10.18653/v1/2024.eacl-long.4) · [OpenAlex](https://openalex.org/W4411630296)

**Citations:** 22 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/muhlgay-2024-generating-benchmarks-factuality-evaluation-language/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Measuring how often a language model produces false statements in a given domain is hard. Methods that fact-check a model's own samples cannot choose which facts get tested, and they favour common knowledge over rare or domain-specific facts. Perplexity is often used as a stand-in, but it reflects many linguistic factors besides truthfulness.

The authors propose FACTOR (Factual Assessment via Corpus TransfORmation). It automatically converts any factual corpus into a multiple-choice benchmark. Each item pairs a prefix with its true next sentence and three minimally edited false versions written by InstructGPT. A model passes an item if it gives the true sentence the highest length-normalised log-likelihood.

They built Wiki-FACTOR, News-FACTOR and Expert-FACTOR and tested GPT-2, GPT-Neo and OPT models. Scores rise with model size and with retrieval augmentation, yet stay modest overall. FACTOR sometimes orders models differently from perplexity. Human judgments of open-ended generations side with FACTOR in those cases, which makes it a useful, controllable complement to sampling-based factuality checks.

## Key points
- **Benchmark construction:** False completions follow five error types adapted from the FRANK typology: predicate, entity, circumstance, coreference and link. They are produced with an edit-planning prompt that keeps changes small and fluent.
- **Models and scores:** The evaluated models range from 110M to 66B parameters. The best scores reached only about 58% on Wiki-FACTOR, 68% on News-FACTOR and 55% on Expert-FACTOR.
- **Retrieval:** Adding the simple retrieval component of Ram et al. (2023) consistently improved FACTOR accuracy.
- **FACTOR vs. perplexity:** OPT-66B has higher Wikipedia perplexity than GPT-J-6B but scores better on Wiki-FACTOR. In manual annotation of 1,200 completions, OPT-66B produced true claims 38.8% of the time versus 24.8% for GPT-J 6B, matching the FACTOR ranking.
- **Availability:** Data and code are released at github.com/AI21Labs/factor.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 221 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua and Yoav Shoham. 2024. Generating Benchmarks for Factuality Evaluation of Language Models. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{muhlgay2024generating,
  title = {{Generating Benchmarks for Factuality Evaluation of Language Models}},
  author = {Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua and Yoav Shoham},
  booktitle = {Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.eacl-long.4},
  url = {https://doi.org/10.18653/v1/2024.eacl-long.4}
}
```

- [← previous You Only Look at Screens: Multimodal Chain-of-Action Agents](https://md.research.elamur.ai/free/en/papers/zhang-2024-you-only-look-screens-multimodal/)
- [next → MART: Improving LLM Safety with Multi-round Automatic Red-Teaming](https://md.research.elamur.ai/free/en/papers/ge-2024-mart-improving-llm-safety-multi/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

44 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved QA-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa… [our card](https://md.research.elamur.ai/free/en/papers/fabbri-2022-qafacteval-improved-qa-based-factual/)
2. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Leo+Gao%2C+Phil+Wang%2C+Connor+Leahy%2C+and+Stella+Biderman%2E+2021%2E)
3. GPT-Neo: Large Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The Pile: An 800gb dataset of diverse text for language modeling. [find](https://scholar.google.com/scholar?q=GPT%2DNeo%3A+Large+Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2021%2E+The+Pile%3A)
4. Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Divyansh+Kaushik%2C+Eduard+Hovy%2C+and+Zachary+Lipton%2E+2020%2E+Learning+the+difference+that+makes+a+difference+with+counterfactually%2Daugmented+data%2E+In+International+Conference+on+Learning+Representations%2E)
5. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655, Melbourne, Australia. Association for… [our card](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/)
6. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+Omer+Levy%2C+Dan+Jurafsky%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2020%2E+Generalization+through+memorization%3A+Nearest+neighbor+language+models%2E+In+International+Conference+on+Learning+Repr)
7. Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Factuality enhanced language models for open-ended text generation. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Nayeon+Lee%2C+Wei+Ping%2C+Peng+Xu%2C+Mostofa+Patwary%2C+Pascale+Fung%2C+Mohammad+Shoeybi%2C+and+Bryan+Catanzaro%2E+2022%2E+Factuality+enhanced+language+models+for+open%2Dended+text+generation%2E+In+Advances+in+Neural+Inf)
8. Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. DialFact: A benchmark for fact-checking in dialogue. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3785–3801, Dublin, Ireland. Association for Computational… [our card](https://md.research.elamur.ai/free/en/papers/gupta-2022-dialfact-benchmark-fact-checking-dialogue/)
9. Shaobo Li, Xiaoguang Li, Lifeng Shang, Zhenhua Dong, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang, and Qun Liu. 2022. How pre-trained language models capture factual knowledge? a causal-inspired analysis. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1720–1732,… [find](https://scholar.google.com/scholar?q=Shaobo+Li%2C+Xiaoguang+Li%2C+Lifeng+Shang%2C+Zhenhua+Dong%2C+Chengjie+Sun%2C+Bingquan+Liu%2C+Zhenzhou+Ji%2C+Xin+Jiang%2C+and+Qun+Liu%2E+2022%2E+How+pre%2Dtrained+language+models+capture+factual+knowledge%3F+a+causal%2Dinspired)
10. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with disentangled attention. In International Conference on Learning Representations. [our card](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/)
11. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Co… [our card](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/)
12. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/)
13. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics… [our card](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/)
14. Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020. Challenges in building intelligent open-domain dialog systems. ACM Trans. Inf. Syst., 38(3). [find](https://scholar.google.com/scholar?q=Minlie+Huang%2C+Xiaoyan+Zhu%2C+and+Jianfeng+Gao%2E+2020%2E+Challenges+in+building+intelligent+open%2Ddomain+dialog+systems%2E+ACM+Trans%2E+Inf%2E+Syst%2E%2C+38%283%29%2E)
15. Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2022. A token-level reference-free hallucination detection benchmark for free-form text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa… [our card](https://md.research.elamur.ai/free/en/papers/liu-2022-token-level-reference-free-hallucination/)
16. Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021. The factual inconsistency problem in abstractive text summarization: A survey. arXiv preprint arXiv:2104.14839. — [arXiv:2104.14839](https://arxiv.org/abs/2104.14839) [find](https://scholar.google.com/scholar?q=Yichong+Huang%2C+Xiachong+Feng%2C+Xiaocheng+Feng%2C+and+Bing+Qin%2E+2021%2E+The+factual+inconsistency+problem+in+abstractive+text+summarization%3A+A+survey%2E+arXiv+preprint+arXiv%3A2104%2E14839%2E+%E2%80%94+%5BarXiv%3A2104%2E14839%5D%28h)
17. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Atlas: Few-shot learning with retrieval augmented language models. [find](https://scholar.google.com/scholar?q=Gautier+Izacard%2C+Patrick+Lewis%2C+Maria+Lomeli%2C+Lucas+Hosseini%2C+Fabio+Petroni%2C+Timo+Schick%2C+Jane+Dwivedi%2DYu%2C+Armand+Joulin%2C+Sebastian+Riedel%2C+and+Edouard+Grave%2E+2022%2E+Atlas%3A+Few%2Dshot+learning+with+retri)
18. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2023. Expertqa: Expert-curated questions and attributed answers. In arXiv. [our card](https://md.research.elamur.ai/free/en/papers/malaviya-2024-expertqa-expert-curated-questions-attributed/)
19. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438. [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/)
20. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… [find](https://scholar.google.com/scholar?q=Alex+Mallen%2C+Akari+Asai%2C+Victor+Zhong%2C+Rajarshi+Das%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023%2E+When+not+to+trust+language+models%3A+Investigating+effectiveness+of+parametric+and+non%2Dparametric+mem)
21. Jushi Kai, Tianhang Zhang, Hai Hu, and Zhouhan Lin. 2024. SH2: Self-highlighted hesitation helps you decode more truthfully. [find](https://scholar.google.com/scholar?q=Jushi+Kai%2C+Tianhang+Zhang%2C+Hai+Hu%2C+and+Zhouhan+Lin%2E+2024%2E+SH2%3A+Self%2Dhighlighted+hesitation+helps+you+decode+more+truthfully%2E)
22. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli… [our card](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)
23. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Joshua+Maynez%2C+Shashi+Narayan%2C+Bernd+Bohnet%2C+and+Ryan+McDonald%2E+2020%2E+On+faithfulness+and+factuality+in+abstractive+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comp)
24. Language Technologies, pages 2523–2544, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Language+Technologies%2C+pages+2523%E2%80%932544%2C+Online%2E+Association+for+Computational+Linguistics%2E)
25. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. [our card](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/)
26. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
27. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActscore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. arXiv preprint arXiv:2305.14251. [our card](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/)
28. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… [our card](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
30. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ori+Ram%2C+Yoav+Levine%2C+Itay+Dalmedigos%2C+Dor+Muhlgay%2C+Amnon+Shashua%2C+Kevin+Leyton%2DBrown%2C+and+Yoav+Shoham%2E+2023%2E+In%2Dcontext+retrieval%2Daugmented+language+models%2E+Transactions+of+the+Association+for+Comput)
31. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Langu)
32. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag… [our card](https://md.research.elamur.ai/free/en/papers/pagnoni-2021-understanding-factuality-abstractive-summarization-frank/)
33. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. [find](https://scholar.google.com/scholar?q=Stephen+Robertson+and+Hugo+Zaragoza%2E+2009%2E+The+probabilistic+relevance+framework%3A+BM25+and+beyond%2E+Found%2E+Trends+Inf%2E+Retr%2E%2C+3%284%29%3A333%E2%80%93389%2E)
34. Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624–643, Online. Assoc… [our card](https://md.research.elamur.ai/free/en/papers/schuster-2021-get-your-vitamin-c-robust/)
35. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023%2E+The+RefinedWeb+dataset+for+F)
36. Rico Sennrich. 2017. How grammatical is character-level neural machine translation? assessing MT quality with contrastive translation pairs. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 376–382, Valencia… [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2017-how-grammatical-is-character-level/)
37. Hao Peng, Xiaozhi Wang, Shengding Hu, Hailong Jin, Lei Hou, Juanzi Li, Zhiyuan Liu, and Qun Liu. 2022. COPEN: Probing conceptual knowledge in pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5015–5035, Abu Dhabi, United Ara… [find](https://scholar.google.com/scholar?q=Hao+Peng%2C+Xiaozhi+Wang%2C+Shengding+Hu%2C+Hailong+Jin%2C+Lei+Hou%2C+Juanzi+Li%2C+Zhiyuan+Liu%2C+and+Qun+Liu%2E+2022%2E+COPEN%3A+Probing+conceptual+knowledge+in+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2022+Co)
38. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652. [our card](https://md.research.elamur.ai/free/en/papers/shi-2023-replug-retrieval-augmented-black-box/)
39. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. 2023. Discovering language model behaviors with model-written evaluations. In Findings of ACL. [our card](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/)
40. Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. 2023. Evaluating the factual consistency of large language models through summarization. In Findings of ACL. [find](https://scholar.google.com/scholar?q=Derek+Tam%2C+Anisha+Mascarenhas%2C+Shiyue+Zhang%2C+Sarah+Kwan%2C+Mohit+Bansal%2C+and+Colin+Raffel%2E+2023%2E+Evaluating+the+factual+consistency+of+large+language+models+through+summarization%2E+In+Findings+of+ACL%2E)
41. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of th… [our card](https://md.research.elamur.ai/free/en/papers/petroni-2020-kilt-benchmark-knowledge-intensive-language/)
42. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [find](https://scholar.google.com/scholar?q=Ben+Wang+and+Aran+Komatsuzaki%2E+2021%2E+GPT%2DJ%2D6B%3A+A+6+Billion+Parameter+Autoregressive+Language+Model%2E)
43. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems. [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
44. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)

*Updated 24 October 2026*