---
title: "Are LLMs Better than Reported? Detecting Label Errors and"
description: "They study four binary factual-consistency datasets drawn from the TRUE benchmark, covering tasks… 92 references, cited by 11 papers here."
authors: ["Omer Nahum", "Nitay Calderon", "Orgad Keller", "Idan Szpektor", "Roi Reichart"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.1360/"
doi: "10.18653/v1/2025.emnlp-main.1360"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nahum-2025-are-llms-better-than-reported/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nahum-2025-are-llms-better-than-reported/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance

# Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance

**Authors:** Omer Nahum, [Nitay Calderon](https://md.research.elamur.ai/free/en/authors/nitay-calderon/), [Orgad Keller](https://md.research.elamur.ai/free/en/authors/orgad-keller/), [Idan Szpektor](https://md.research.elamur.ai/free/en/authors/idan-szpektor/) and [Roi Reichart](https://md.research.elamur.ai/free/en/authors/roi-reichart/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.1360/) · [PDF](https://aclanthology.org/2025.emnlp-main.1360.pdf) · [DOI 10.18653/v1/2025.emnlp-main.1360](https://doi.org/10.18653/v1/2025.emnlp-main.1360) · [OpenAlex](https://openalex.org/W4416035743)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/nahum-2025-are-llms-better-than-reported/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Benchmark labels are often treated as ground truth, yet both expert and crowd annotation introduce mistakes. Nahum et al. (Technion and Google Research) ask how common such errors are, whether LLMs can find them, and how they distort training and evaluation. They study four binary factual-consistency datasets drawn from the TRUE benchmark, covering tasks such as summarization, dialogue, paraphrasing and fact verification, plus SummEval with its 1–5 ratings across four quality dimensions. Their detector is a zero-shot LLM-as-a-judge ensemble: several LLMs, each run with varied prompts, produce class probabilities from label-token logits, which are then averaged. Examples where the ensemble confidently picks a label different from the original are flagged. Flagged test items are rechecked by experts, while flagged training items are either dropped or relabeled. Between 6% and 21% of labels turn out to be wrong. Once these are corrected, LLM scores rise substantially. This suggests that part of what looks like model failure is actually annotation noise and that current LLMs may be stronger than benchmarks indicate.

## Key points
- Detection needs no training. Averaged probabilities from multiple LLMs and prompts are compared against the original labels, and only high-confidence disagreements are flagged.
- The flagging precision grows with ensemble confidence. Above 95% confidence, more than two-thirds of flagged cases are genuine human labeling errors.
- After expert correction of the test sets, measured LLM performance increases by up to 15%.
- Filtering or flipping flagged training examples improves fine-tuned models by up to 4%.
- When compared with expert and crowd-sourced annotation, LLM labels matched or beat crowd workers on accuracy, consistency and cost. Crowd labels were evaluated after filtering out low-quality submissions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations
- [Technion – Israel Institute of Technology](https://md.research.elamur.ai/free/en/universities/technion-israel-institute-of-technology/) — University, Haifa, Israel · 49 works · 5,306 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Omer Nahum, Nitay Calderon, Orgad Keller, Idan Szpektor and Roi Reichart. 2025. Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nahum2025are,
  title = {{Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance}},
  author = {Omer Nahum, Nitay Calderon, Orgad Keller, Idan Szpektor and Roi Reichart},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1360},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1360}
}
```

- [← previous AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents](https://md.research.elamur.ai/free/en/papers/xu-2025-androidlab-training-systematic-benchmarking-android/)
- [next → CaLMQA: Exploring culturally specific long-form question answering across 23 languages](https://md.research.elamur.ai/free/en/papers/arora-2025-calmqa-exploring-culturally-specific-long/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

92 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
2. Mohammad Allahbakhsh, Boualem Benatallah, Aleksandar Ignjatovic, Hamid Reza Motahari-Nezhad, Elisa Bertino, and Schahram Dustdar. 2013. Quality control in crowdsourcing systems: Issues and directions. IEEE Internet Computing, 17(2):76–81. [find](https://scholar.google.com/scholar?q=Mohammad+Allahbakhsh%2C+Boualem+Benatallah%2C+Aleksandar+Ignjatovic%2C+Hamid+Reza+Motahari%2DNezhad%2C+Elisa+Bertino%2C+and+Schahram+Dustdar%2E+2013%2E+Quality+control+in+crowdsourcing+systems%3A+Issues+and+directions%2E)
3. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… — [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
4. Anaëlle Baledent, Yann Mathet, Antoine Widlöcher, Christophe Couronne, and Jean-Luc Manguin. 2022. Validity, agreement, consensuality and annotated data quality. In International Conference on Language Resources and Evaluation. [find](https://scholar.google.com/scholar?q=Ana%C3%ABlle+Baledent%2C+Yann+Mathet%2C+Antoine+Widl%C3%B6cher%2C+Christophe+Couronne%2C+and+Jean%2DLuc+Manguin%2E+2022%2E+Validity%2C+agreement%2C+consensuality+and+annotated+data+quality%2E+In+International+Conference+on+Languag)
5. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Sur… — [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://md.research.elamur.ai/free/en/papers/bavaresco-2024-llms-instead-human-judges-large/)
6. Savita Bhat and Vasudeva Varma. 2023. Large language models as annotators: A preliminary evaluation for annotating low-resource language content. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pages 100–107, Bali, Indonesia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Savita+Bhat+and+Vasudeva+Varma%2E+2023%2E+Large+language+models+as+annotators%3A+A+preliminary+evaluation+for+annotating+low%2Dresource+language+content%2E+In+Proceedings+of+the+4th+Workshop+on+Evaluation+and+C)
7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
8. Nitay Calderon and Roi Reichart. 2024. On behalf of the stakeholders: Trends in NLP model interpretability in the era of llms. CoRR, abs/2407.19200. [find](https://scholar.google.com/scholar?q=Nitay+Calderon+and+Roi+Reichart%2E+2024%2E+On+behalf+of+the+stakeholders%3A+Trends+in+NLP+model+interpretability+in+the+era+of+llms%2E+CoRR%2C+abs%2F2407%2E19200%2E)
9. Hung Chau, Saeid Balaneshin, Kai Liu, and Ondrej Linda. 2020. Understanding the tradeoff between cost and quality of expert annotations for keyphrase extraction. In Law. [find](https://scholar.google.com/scholar?q=Hung+Chau%2C+Saeid+Balaneshin%2C+Kai+Liu%2C+and+Ondrej+Linda%2E+2020%2E+Understanding+the+tradeoff+between+cost+and+quality+of+expert+annotations+for+keyphrase+extraction%2E+In+Law%2E)
10. Honghua Chen and Nai Ding. 2023. Probing the “creativity” of large language models: Can models produce divergent semantic association? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12881–12888, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Honghua+Chen+and+Nai+Ding%2E+2023%2E+Probing+the+%E2%80%9Ccreativity%E2%80%9D+of+large+language+models%3A+Can+models+produce+divergent+semantic+association%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMN)
11. Ruirui Chen, Chengwei Qin, Weifeng Jiang, and Dongkyu Choi. 2024. Is a large language model a good annotator for event extraction? In AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Ruirui+Chen%2C+Chengwei+Qin%2C+Weifeng+Jiang%2C+and+Dongkyu+Choi%2E+2024%2E+Is+a+large+language+model+a+good+annotator+for+event+extraction%3F+In+AAAI+Conference+on+Artificial+Intelligence%2E)
12. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+2023%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vo)
13. Michael Chmielewski and Sarah C. Kucker. 2019. An MTurk crisis? shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11:464 – 473. [find](https://scholar.google.com/scholar?q=Michael+Chmielewski+and+Sarah+C%2E+Kucker%2E+2019%2E+An+MTurk+crisis%3F+shifts+in+data+quality+and+the+impact+on+study+results%2E+Social+Psychological+and+Personality+Science%2C+11%3A464+%E2%80%93+473%2E)
14. Derek Chong, Jenny Hong, and Christopher D. Manning. 2022. Detecting label errors by using pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 9074–9091. Associ… [find](https://scholar.google.com/scholar?q=Derek+Chong%2C+Jenny+Hong%2C+and+Christopher+D%2E+Manning%2E+2022%2E+Detecting+label+errors+by+using+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+P)
15. C. J. Clopper and E. S. Pearson. 1934. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413. [find](https://scholar.google.com/scholar?q=C%2E+J%2E+Clopper+and+E%2E+S%2E+Pearson%2E+1934%2E+The+use+of+confidence+or+fiducial+limits+illustrated+in+the+case+of+the+binomial%2E+Biometrika%2C+26%284%29%3A404%E2%80%93413%2E)
16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
17. Thomas G. Dietterich. 2007. Ensemble methods in machine learning. [find](https://scholar.google.com/scholar?q=Thomas+G%2E+Dietterich%2E+2007%2E+Ensemble+methods+in+machine+learning%2E)
18. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. [Wizard of wikipedia: Knowledge-powered conversational agents](https://md.research.elamur.ai/free/en/papers/dinan-2018-wizard-wikipedia-knowledge-powered-conversational/). In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
19. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. [Improving factuality and reasoning in language models through multiagent debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.
20. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). CoRR, abs/2407.21783.
21. Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022. [Evaluating attribution in dialogue systems: The BEGIN benchmark](https://md.research.elamur.ai/free/en/papers/dziri-2022-evaluating-attribution-dialogue-systems-begin/). Transactions of the Association for Computational Linguistics, 10:1066– 1083.
22. Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2021. Summeval: Re-evaluating summarization evaluation. Trans. Assoc. Comput. Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R%2E+Fabbri%2C+Wojciech+Kryscinski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+R%2E+Radev%2E+2021%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+Trans%2E+Assoc%2E+Comput%2E+Linguistics%2C)
23. Virginia K. Felkner, Jennifer A. Thompson, and Jonathan May. 2024. Gpt is not an annotator: The necessity of human annotation in fairness benchmark construction. ArXiv, abs/2405.15760. [find](https://scholar.google.com/scholar?q=Virginia+K%2E+Felkner%2C+Jennifer+A%2E+Thompson%2C+and+Jonathan+May%2E+2024%2E+Gpt+is+not+an+annotator%3A+The+necessity+of+human+annotation+in+fairness+benchmark+construction%2E+ArXiv%2C+abs%2F2405%2E15760%2E)
24. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–382. [find](https://scholar.google.com/scholar?q=Joseph+L%2E+Fleiss%2E+1971%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+Bulletin%2C+76%3A378%E2%80%93382%2E)
25. Benoît Frénay and Michel Verleysen. 2014. Classification in the presence of label noise: A survey. IEEE Transactions on Neural Networks and Learning Systems, 25:845–869. [find](https://scholar.google.com/scholar?q=Beno%C3%AEt+Fr%C3%A9nay+and+Michel+Verleysen%2E+2014%2E+Classification+in+the+presence+of+label+noise%3A+A+survey%2E+IEEE+Transactions+on+Neural+Networks+and+Learning+Systems%2C+25%3A845%E2%80%93869%2E)
26. Yoav Freund and Robert E Schapire. 1997. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119–139. [find](https://scholar.google.com/scholar?q=Yoav+Freund+and+Robert+E+Schapire%2E+1997%2E+A+decision%2Dtheoretic+generalization+of+on%2Dline+learning+and+an+application+to+boosting%2E+Journal+of+Computer+and+System+Sciences%2C+55%281%29%3A119%E2%80%93139%2E)
27. Yair Ori Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, and Roi Reichart. 2024. Faithful explanations of black-box NLP models using LLM-generated counterfactuals. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenR… [find](https://scholar.google.com/scholar?q=Yair+Ori+Gat%2C+Nitay+Calderon%2C+Amir+Feder%2C+Alexander+Chapanin%2C+Amit+Sharma%2C+and+Roi+Reichart%2E+2024%2E+Faithful+explanations+of+black%2Dbox+NLP+models+using+LLM%2Dgenerated+counterfactuals%2E+In+The+Twelfth+Int)
28. Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023. [TrueTeacher: Learning factual consistency evaluation with large language models](https://md.research.elamur.ai/free/en/papers/gekhman-2023-trueteacher-learning-factual-consistency-evaluation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2053–2070, Singapore. Association for…
29. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. [ChatGPT outperforms crowd workers for text-annotation tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/). Proceedings of the National Academy of Sciences of the United States of America, 120.
30. Degan Hao, Lei Zhang, Jules H. Sumkin, Aly A. Mohamed, and Shandong Wu. 2020. Inaccurate labels in weakly-supervised deep learning: Automatic identification and correction and their impact on classification performance. IEEE Journal of Biomedical and Health Informatics, 24:2701–2710. [find](https://scholar.google.com/scholar?q=Degan+Hao%2C+Lei+Zhang%2C+Jules+H%2E+Sumkin%2C+Aly+A%2E+Mohamed%2C+and+Shandong+Wu%2E+2020%2E+Inaccurate+labels+in+weakly%2Dsupervised+deep+learning%3A+Automatic+identification+and+correction+and+their+impact+on+classifi)
31. David N. Hauser, Aaron J. Moss, Cheskie Rosenzweig, Shalom N. Jaffe, Jonathan Robinson, and Leib Litman. 2021. Evaluating CloudResearch’s approved group as a solution for problematic data quality on MTurk. Behavior Research Methods, 55:3953 – 3964. [find](https://scholar.google.com/scholar?q=David+N%2E+Hauser%2C+Aaron+J%2E+Moss%2C+Cheskie+Rosenzweig%2C+Shalom+N%2E+Jaffe%2C+Jonathan+Robinson%2C+and+Leib+Litman%2E+2021%2E+Evaluating+CloudResearch%E2%80%99s+approved+group+as+a+solution+for+problematic+data+quality+on+M)
32. Xingwei He, Zheng-Wen Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2023. [AnnoLLM: Making large language models to be better crowdsourced annotators](https://md.research.elamur.ai/free/en/papers/he-2023-annollm-making-large-language-models/). In North American Chapter of the Association for Computational Linguistics.
33. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
34. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansky, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. [TRUE: re-evaluating factual consistency evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/). In Proceedings of the 2022 Conference of the North American Chapter of the Associ…
35. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+Las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
36. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [find](https://scholar.google.com/scholar?q=Jared+Kaplan%2C+Sam+McCandlish%2C+Tom+Henighan%2C+Tom+B%2E+Brown%2C+Benjamin+Chess%2C+Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+Jeffrey+Wu%2C+and+Dario+Amodei%2E+2020%2E)
37. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). CoRR, abs/2001.08361.
38. Gabriella Kazai, Jaap Kamps, and Natasa Milic-Frayling. 2013. An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Inf. Retr., 16(2):138–178. [find](https://scholar.google.com/scholar?q=Gabriella+Kazai%2C+Jaap+Kamps%2C+and+Natasa+Milic%2DFrayling%2E+2013%2E+An+analysis+of+human+factors+and+label+accuracy+in+crowdsourcing+relevance+judgments%2E+Inf%2E+Retr%2E%2C+16%282%29%3A138%E2%80%93178%2E)
39. Ryan Kennedy, Scott Clifford, Tyler Burleigh, Philip D Waggoner, Ryan Jewell, and Nicholas JG Winter. 2020. The shape of and solutions to the MTurk quality crisis. Political Science Research and Methods, 8(4):614–629. [find](https://scholar.google.com/scholar?q=Ryan+Kennedy%2C+Scott+Clifford%2C+Tyler+Burleigh%2C+Philip+D+Waggoner%2C+Ryan+Jewell%2C+and+Nicholas+JG+Winter%2E+2020%2E+The+shape+of+and+solutions+to+the+MTurk+quality+crisis%2E+Political+Science+Research+and+Metho)
40. Nataliia Kholodna, Sahib Julka, Mohammad Khodadadi, Muhammed Nurullah Gumus, and Michael Granitzer. 2024. Llms in the loop: Leveraging large language model annotations for active learning in low-resource languages. ArXiv, abs/2404.02261. — [arXiv:2404.02261](https://arxiv.org/abs/2404.02261) [find](https://scholar.google.com/scholar?q=Nataliia+Kholodna%2C+Sahib+Julka%2C+Mohammad+Khodadadi%2C+Muhammed+Nurullah+Gumus%2C+and+Michael+Granitzer%2E+2024%2E+Llms+in+the+loop%3A+Leveraging+large+language+model+annotations+for+active+learning+in+low%2Dresou)
41. Han Jun Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang. 2024. Meganno+: A human-llm collaborative annotation system. In Conference of the European Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Han+Jun+Kim%2C+Kushan+Mitra%2C+Rafael+Li+Chen%2C+Sajjadur+Rahman%2C+and+Dan+Zhang%2E+2024%2E+Meganno%2B%3A+A+human%2Dllm+collaborative+annotation+system%2E+In+Conference+of+the+European+Chapter+of+the+Association+for+Com)
42. Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023. [Annotation error detection: Analyzing the past and present for a more coherent future](https://md.research.elamur.ai/free/en/papers/klie-2023-annotation-error-detection-analyzing-past/). Computational Linguistics, 49(1):157–198.
43. Klaus Krippendorff. 1970. Estimating the reliability, systematic error, and random error of interval data. Educational and Psychological Measurement, 30(1):61–70. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+1970%2E+Estimating+the+reliability%2C+systematic+error%2C+and+random+error+of+interval+data%2E+Educational+and+Psychological+Measurement%2C+30%281%29%3A61%E2%80%9370%2E)
44. Chi-Hoon Lee. 2010. Learning to combine discriminative classifiers: confidence based. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, July 25-28, 2010, pages 743–752. ACM. [find](https://scholar.google.com/scholar?q=Chi%2DHoon+Lee%2E+2010%2E+Learning+to+combine+discriminative+classifiers%3A+confidence+based%2E+In+Proceedings+of+the+16th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+Washington%2C)
45. Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy F. Chen, Zhengyuan Liu, and Diyi Yang. 2023. [Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation](https://md.research.elamur.ai/free/en/papers/li-2023-coannotating-uncertainty-guided-work-allocation/). ArXiv, abs/2310.15638.
46. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. [Encouraging divergent thinking in large language models through multi-agent debate](https://md.research.elamur.ai/free/en/papers/liang-2023-encouraging-divergent-thinking-large-language/). CoRR, abs/2305.19118.
47. Shir Lissak, Nitay Calderon, Geva Shenkman, Yaakov Ophir, Eyal Fruchter, Anat Brunstein Klomek, and Roi Reichart. 2024. The colorful future of llms: Evaluating and improving llms as emotional supporters for queer youth. In Proceedings of the 2024 Conference of the North American Chapter of the Assoc… [find](https://scholar.google.com/scholar?q=Shir+Lissak%2C+Nitay+Calderon%2C+Geva+Shenkman%2C+Yaakov+Ophir%2C+Eyal+Fruchter%2C+Anat+Brunstein+Klomek%2C+and+Roi+Reichart%2E+2024%2E+The+colorful+future+of+llms%3A+Evaluating+and+improving+llms+as+emotional+supporte)
48. Jian Lu, Wei Li, Qingren Wang, and Yiwen Zhang. 2020. Research on data quality control of crowdsourcing annotation: A survey. In 2020 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf… [find](https://scholar.google.com/scholar?q=Jian+Lu%2C+Wei+Li%2C+Qingren+Wang%2C+and+Yiwen+Zhang%2E+2020%2E+Research+on+data+quality+control+of+crowdsourcing+annotation%3A+A+survey%2E+In+2020+IEEE+Intl+Conf+on+Dependable%2C+Autonomic+and+Secure+Computing%2C+Intl)
49. Zhihe Lu, Jiawang Bai, Xin Li, Zeyu Xiao, and Xinchao Wang. 2024. Beyond sole strength: Customized ensembles for generalized vision-language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. [find](https://scholar.google.com/scholar?q=Zhihe+Lu%2C+Jiawang+Bai%2C+Xin+Li%2C+Zeyu+Xiao%2C+and+Xinchao+Wang%2E+2024%2E+Beyond+sole+strength%3A+Customized+ensembles+for+generalized+vision%2Dlanguage+models%2E+In+Forty%2Dfirst+International+Conference+on+Machine+)
50. Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceedings of the Eight International Conference on Computational Semantics, pages 140–156, Tilburg, The Netherlands. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bill+MacCartney+and+Christopher+D%2E+Manning%2E+2009%2E+An+extended+model+of+natural+logic%2E+In+Proceedings+of+the+Eight+International+Conference+on+Computational+Semantics%2C+pages+140%E2%80%93156%2C+Tilburg%2C+The+Nethe)
51. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her)
52. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. 2020. [On faithfulness and factuality in abstractive summarization](https://md.research.elamur.ai/free/en/papers/maynez-2020-faithfulness-factuality-abstractive-summarization/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1906–1919. Association for Comput…
53. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. As…
54. Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi-Phuong-Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. 2019. Self: Learning to filter noisy labels with self-ensembling. ArXiv, abs/1910.01842. — [arXiv:1910.01842](https://arxiv.org/abs/1910.01842) [find](https://scholar.google.com/scholar?q=Duc+Tam+Nguyen%2C+Chaithanya+Kumar+Mummadi%2C+Thi%2DPhuong%2DNhung+Ngo%2C+Thi+Hoai+Phuong+Nguyen%2C+Laura+Beggel%2C+and+Thomas+Brox%2E+2019%2E+Self%3A+Learning+to+filter+noisy+labels+with+self%2Densembling%2E+ArXiv%2C+abs%2F1910)
55. Curtis G. Northcutt, Anish Athalye, and Jonas W. Mueller. 2021. Pervasive label errors in test sets destabilize machine learning benchmarks. ArXiv, abs/2103.14749. [find](https://scholar.google.com/scholar?q=Curtis+G%2E+Northcutt%2C+Anish+Athalye%2C+and+Jonas+W%2E+Mueller%2E+2021%2E+Pervasive+label+errors+in+test+sets+destabilize+machine+learning+benchmarks%2E+ArXiv%2C+abs%2F2103%2E14749%2E)
56. Curtis G. Northcutt, Lu Jiang, and Isaac L. Chuang. 2019. Confident learning: Estimating uncertainty in dataset labels. J. Artif. Intell. Res., 70:1373–1411. [find](https://scholar.google.com/scholar?q=Curtis+G%2E+Northcutt%2C+Lu+Jiang%2C+and+Isaac+L%2E+Chuang%2E+2019%2E+Confident+learning%3A+Estimating+uncertainty+in+dataset+labels%2E+J%2E+Artif%2E+Intell%2E+Res%2E%2C+70%3A1373%E2%80%931411%2E)
57. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
58. CoRR, Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe.… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
59. Geoff Pleiss, Tianyi Zhang, Ethan R. Elenberg, and Kilian Q. Weinberger. 2020. Identifying mislabeled data using the area under the margin ranking. ArXiv, abs/2001.10528. — [arXiv:2001.10528](https://arxiv.org/abs/2001.10528) [find](https://scholar.google.com/scholar?q=Geoff+Pleiss%2C+Tianyi+Zhang%2C+Ethan+R%2E+Elenberg%2C+and+Kilian+Q%2E+Weinberger%2E+2020%2E+Identifying+mislabeled+data+using+the+area+under+the+margin+ranking%2E+ArXiv%2C+abs%2F2001%2E10528%2E+%E2%80%94+%5BarXiv%3A2001%2E10528%5D%28https%3A%2F%2F)
60. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
61. Frederick Reiss, Hong Xu, Bryan Cutler, Karthik Muthuraman, and Zachary Eichenberger. 2020. Identifying incorrect labels in the conll-2003 corpus. In Proceedings of the 24th Conference on Computational Natural Language Learning, CoNLL 2020, Online, November 19-20, 2020, pages 215–226. Association fo… [find](https://scholar.google.com/scholar?q=Frederick+Reiss%2C+Hong+Xu%2C+Bryan+Cutler%2C+Karthik+Muthuraman%2C+and+Zachary+Eichenberger%2E+2020%2E+Identifying+incorrect+labels+in+the+conll%2D2003+corpus%2E+In+Proceedings+of+the+24th+Conference+on+Computationa)
62. Simon Rogers, Derek H. Sleeman, and John Kinsella. 2013. Investigating the disagreement between clinicians’ ratings of patients in icus. IEEE J. Biomed. Health Informatics, 17(4):843–852. [find](https://scholar.google.com/scholar?q=Simon+Rogers%2C+Derek+H%2E+Sleeman%2C+and+John+Kinsella%2E+2013%2E+Investigating+the+disagreement+between+clinicians%E2%80%99+ratings+of+patients+in+icus%2E+IEEE+J%2E+Biomed%2E+Health+Informatics%2C+17%284%29%3A843%E2%80%93852%2E)
63. Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. [Get your vitamin C! robust fact verification with contrastive evidence](https://md.research.elamur.ai/free/en/papers/schuster-2021-get-your-vitamin-c-robust/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624–643, Online. Assoc…
64. Katharine Sherratt, Hugo Gruson, Rok Grah, Helen Johnson, Rene Niehus, Bastian Prasse, and et al. 2023. Predictive performance of multi-model ensemble forecasts of covid-19 across european nations. eLife, 12:e81916. [find](https://scholar.google.com/scholar?q=Katharine+Sherratt%2C+Hugo+Gruson%2C+Rok+Grah%2C+Helen+Johnson%2C+Rene+Niehus%2C+Bastian+Prasse%2C+and+et+al%2E+2023%2E+Predictive+performance+of+multi%2Dmodel+ensemble+forecasts+of+covid%2D19+across+european+nations%2E+eL)
65. Rion Snow, Brendan T. O’Connor, Dan Jurafsky, and A. Ng. 2008. [Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/). In Conference on Empirical Methods in Natural Language Processing.
66. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Al… — [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
67. Trans. Mach. Learn. Res., 2023. [find](https://scholar.google.com/scholar?q=Trans%2E+Mach%2E+Learn%2E+Res%2E%2C+2023%2E)
68. Julius Steen, Juri Opitz, Anette Frank, and Katja Markert. 2023. With a little push, NLI models can robustly and efficiently predict faithfulness. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 914–924, Toronto, Canada. Asso… [find](https://scholar.google.com/scholar?q=Julius+Steen%2C+Juri+Opitz%2C+Anette+Frank%2C+and+Katja+Markert%2E+2023%2E+With+a+little+push%2C+NLI+models+can+robustly+and+efficiently+predict+faithfulness%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Asso)
69. Aneeta Sylolypavan, Derek H. Sleeman, Honghan Wu, and Malcolm Sim. 2023. The impact of inconsistent human annotations on AI driven clinical decision making. npj Digit. Medicine, 6. [find](https://scholar.google.com/scholar?q=Aneeta+Sylolypavan%2C+Derek+H%2E+Sleeman%2C+Honghan+Wu%2C+and+Malcolm+Sim%2E+2023%2E+The+impact+of+inconsistent+human+annotations+on+AI+driven+clinical+decision+making%2E+npj+Digit%2E+Medicine%2C+6%2E)
70. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2818–2826. IEEE Compute… [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Vincent+Vanhoucke%2C+Sergey+Ioffe%2C+Jonathon+Shlens%2C+and+Zbigniew+Wojna%2E+2016%2E+Rethinking+the+inception+architecture+for+computer+vision%2E+In+2016+IEEE+Conference+on+Computer+Vision+and)
71. Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. 2023. Evaluating the factual consistency of large language models through news summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5220–5255, Toronto, Canada. Association… [find](https://scholar.google.com/scholar?q=Derek+Tam%2C+Anisha+Mascarenhas%2C+Shiyue+Zhang%2C+Sarah+Kwan%2C+Mohit+Bansal%2C+and+Colin+Raffel%2E+2023%2E+Evaluating+the+factual+consistency+of+large+language+models+through+news+summarization%2E+In+Findings+of+th)
72. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [FEVER: a large-scale dataset for fact extraction and VERification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
73. Petter Törnberg. 2023. [ChatGPT-4 outperforms experts and crowd workers in annotating political Twitter messages with zero-shot learning](https://md.research.elamur.ai/free/en/papers/tornberg-2023-chatgpt-4-outperforms-experts-crowd/). ArXiv, abs/2304.06588.
74. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. J. Artif. Intell. Res., 72:1385–1470. [find](https://scholar.google.com/scholar?q=Alexandra+Uma%2C+Tommaso+Fornaciari%2C+Dirk+Hovy%2C+Silviu+Paun%2C+Barbara+Plank%2C+and+Massimo+Poesio%2E+2021%2E+Learning+from+disagreement%3A+A+survey%2E+J%2E+Artif%2E+Intell%2E+Res%2E%2C+72%3A1385%E2%80%931470%2E)
75. Mor Ventura, Eyal Ben-David, Anna Korhonen, and Roi Reichart. 2023. [Navigating cultural chasms: Exploring and unlocking the cultural POV of text-to-image models](https://md.research.elamur.ai/free/en/papers/ventura-2025-navigating-cultural-chasms-exploring-unlocking/). CoRR, abs/2310.01929.
76. Veniamin Veselovsky, Manoel Horta Ribeiro, Philip Cozzolino, Andrew Gordon, David Rothschild, and Robert West. 2023a. Prevalence and prevention of large language model use in crowd work. CoRR, abs/2310.15683. — [arXiv:2310.15683](https://arxiv.org/abs/2310.15683) [find](https://scholar.google.com/scholar?q=Veniamin+Veselovsky%2C+Manoel+Horta+Ribeiro%2C+Philip+Cozzolino%2C+Andrew+Gordon%2C+David+Rothschild%2C+and+Robert+West%2E+2023a%2E+Prevalence+and+prevention+of+large+language+model+use+in+crowd+work%2E+CoRR%2C+abs%2F231)
77. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023b. [Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks](https://md.research.elamur.ai/free/en/papers/veselovsky-2023-artificial-artificial-artificial-intelligence-crowd/). CoRR, abs/2306.07899.
78. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRevie…
79. Tong Wang, Ninad Kulkarni, and Yanjun Qi. 2024. Less is more for improving automatic evaluation of factual consistency. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pages… [find](https://scholar.google.com/scholar?q=Tong+Wang%2C+Ninad+Kulkarni%2C+and+Yanjun+Qi%2E+2024%2E+Less+is+more+for+improving+automatic+evaluation+of+factual+consistency%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapter+of+the+Assoc)
80. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Atharva+Naik%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Anjana+Arunkumar%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karaman)
81. Leon Weber and Barbara Plank. 2023. ActiveAED: A human in the loop improves annotation error detection. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8834–8845, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Leon+Weber+and+Barbara+Plank%2E+2023%2E+ActiveAED%3A+A+human+in+the+loop+improves+annotation+error+detection%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pages+8834%E2%80%938845%2C+Toronto)
82. Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. [VariErr NLI: Separating annotation error from human label variation](https://md.research.elamur.ai/free/en/papers/genzel-2024-varierr-nli-separating-annotation-error/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2256–2269, Bangkok, Thai…
83. Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. [find](https://scholar.google.com/scholar?q=Annual+Conference+on+Neural+Information+Processing+Systems+2022%2C+NeurIPS+2022%2C+New+Orleans%2C+LA%2C+USA%2C+November+28+%2D+December+9%2C+2022%2E)
84. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
85. Jianguo Xia, David I. Broadhurst, Michael Wilson, and David Scott Wishart. 2012. Translational biomarker discovery in clinical metabolomics: an introductory tutorial. Metabolomics, 9:280 – 299. [find](https://scholar.google.com/scholar?q=Jianguo+Xia%2C+David+I%2E+Broadhurst%2C+Michael+Wilson%2C+and+David+Scott+Wishart%2E+2012%2E+Translational+biomarker+discovery+in+clinical+metabolomics%3A+an+introductory+tutorial%2E+Metabolomics%2C+9%3A280+%E2%80%93+299%2E)
86. Xiao Yang, Ahmed K. Mohamed, Shashank Jain, Stanislav Peshterliev, Debojeet Chatterjee, Hanwen Zha, Nikita Bhalla, Gagan Aneja, and Pranab Mohanty. 2023. Improving opinion-based question answering systems through label error detection and overwrite. Preprint, arXiv:2306.07499. — [arXiv:2306.07499](https://arxiv.org/abs/2306.07499) [find](https://scholar.google.com/scholar?q=Xiao+Yang%2C+Ahmed+K%2E+Mohamed%2C+Shashank+Jain%2C+Stanislav+Peshterliev%2C+Debojeet+Chatterjee%2C+Hanwen+Zha%2C+Nikita+Bhalla%2C+Gagan+Aneja%2C+and+Pranab+Mohanty%2E+2023%2E+Improving+opinion%2Dbased+question+answering+sys)
87. Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. [Alignscore: Evaluating factual consistency with a unified alignment function](https://md.research.elamur.ai/free/en/papers/zha-2023-alignscore-evaluating-factual-consistency-unified/). In Annual Meeting of the Association for Computational Linguistics.
88. Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. 2018. [mixup: Beyond empirical risk minimization](https://md.research.elamur.ai/free/en/papers/zhang-2017-mixup-empirical-risk-minimization/). In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
89. Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023. LLMaAA: Making large language models as active annotators. ArXiv, abs/2310.19596. [find](https://scholar.google.com/scholar?q=Ruoyu+Zhang%2C+Yanzeng+Li%2C+Yongliang+Ma%2C+Ming+Zhou%2C+and+Lei+Zou%2E+2023%2E+LLMaAA%3A+Making+large+language+models+as+active+annotators%2E+ArXiv%2C+abs%2F2310%2E19596%2E)
90. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. [PAWS: Paraphrase adversaries from word scrambling](https://md.research.elamur.ai/free/en/papers/zhang-2019-paws-paraphrase-adversaries-word-scrambling/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308, M…
91. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Advances in Neural Information Processing Systems 36: A…
92. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems 35:

*Updated 30 October 2026*