---
title: "Can Unconfident LLM Annotations Be Used for Confident"
description: "The authors propose CONFIDENCE-DRIVEN INFERENCE, which asks an LLM for both a label and a… 63 references, cited by 10 papers here."
authors: ["Kristina Gligorić", "Tijana Zrnic", "Cinoo Lee", "Emmanuel Candès", "Dan Jurafsky"]
year: 2025
venue: "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.naacl-long.179/"
doi: "10.18653/v1/2025.naacl-long.179"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gligoric-2025-can-unconfident-llm-annotations-be/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gligoric-2025-can-unconfident-llm-annotations-be/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / Can Unconfident LLM Annotations Be Used for Confident Conclusions?

# Can Unconfident LLM Annotations Be Used for Confident Conclusions?

**Authors:** [Kristina Gligorić](https://md.research.elamur.ai/free/en/authors/kristina-gligoric/), Tijana Zrnic, [Cinoo Lee](https://md.research.elamur.ai/free/en/authors/cinoo-lee/), [Emmanuel Candès](https://md.research.elamur.ai/free/en/authors/emmanuel-j-candes/) and [Dan Jurafsky](https://md.research.elamur.ai/free/en/authors/dan-jurafsky/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2025](https://md.research.elamur.ai/free/en/venues/naacl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.naacl-long.179/) · [PDF](https://aclanthology.org/2025.naacl-long.179.pdf) · [DOI 10.18653/v1/2025.naacl-long.179](https://doi.org/10.18653/v1/2025.naacl-long.179) · [OpenAlex](https://openalex.org/W4411119592)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/gligoric-2025-can-unconfident-llm-annotations-be/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Computational social scientists increasingly add LLM annotations to costly human labels. However, LLM errors and biases can quietly distort downstream statistics such as prevalences or regression coefficients. The authors propose CONFIDENCE-DRIVEN INFERENCE, which asks an LLM for both a label and a verbalized confidence score on every text. Items where the model is less confident are more likely to be sent for human annotation. Human and LLM labels are then combined with an estimator built on active inference (Zrnic and Candès, 2024). Power tuning ensures that adding LLM labels never makes the estimate worse than using human labels alone. The output is an unbiased estimate with a valid confidence interval, however good or bad the LLM is. Across five estimation tasks covering politeness, stance, and media bias, the method keeps target coverage and beats baselines in effective sample size. It cuts the human labels needed by over 25%. In contrast, treating LLM outputs as if they were human data can produce badly wrong conclusions. The approach needs only API access, making reliable, cost-efficient annotation practical across disciplines.

## Key points
- Two stages: LLM confidence guides active selection of a small human-annotated subset, and then statistical inference merges both label sources. Sampling probabilities approximate the optimal rule, which is proportional to the expected LLM error.
- The method works for any M-estimator, including means, prevalences, quantiles, and linear or logistic regression coefficients. It is model-free, unlike earlier Bayesian prevalence approaches.
- In every tested setting, higher verbalized confidence corresponded to higher agreement with human labels, which makes the adaptive sampling effective.
- Using only LLM annotations led to wrong conclusions in some cases, such as an odds-ratio estimate in the stance setting, even though LLM–human agreement was moderate.
- Limitations: few LLMs tested, English-only CSS datasets, human labels treated as gold, and no causal experiments. Code and data are at github.com/kristinagligoric/confidence-driven-inference.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Kristina Gligorić, Tijana Zrnic, Cinoo Lee, Emmanuel Candès and Dan Jurafsky. 2025. Can Unconfident LLM Annotations Be Used for Confident Conclusions?. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gligoric2025can,
  title = {{Can Unconfident LLM Annotations Be Used for Confident Conclusions?}},
  author = {Kristina Gligorić, Tijana Zrnic, Cinoo Lee, Emmanuel Candès and Dan Jurafsky},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.179},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.179}
}
```

- [← previous Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs](https://md.research.elamur.ai/free/en/papers/xia-2025-beyond-chain-thought-survey-chain/)
- [next → Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs](https://md.research.elamur.ai/free/en/papers/yang-2025-code-think-think-code-survey/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/)

## References

63 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. James E Allen, Curry I Guinn, and Eric Horvitz. 1999. Mixed-initiative interaction. IEEE Intelligent Systems and their Applications, 14(5):14–23. [find](https://scholar.google.com/scholar?q=James+E+Allen%2C+Curry+I+Guinn%2C+and+Eric+Horvitz%2E+1999%2E+Mixed%2Dinitiative+interaction%2E+IEEE+Intelligent+Systems+and+their+Applications%2C+14%285%29%3A14%E2%80%9323%2E)
3. Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I Jordan, and Tijana Zrnic. 2023a. Prediction-powered inference. Science, 382(6671):669–674. [find](https://scholar.google.com/scholar?q=Anastasios+N+Angelopoulos%2C+Stephen+Bates%2C+Clara+Fannjiang%2C+Michael+I+Jordan%2C+and+Tijana+Zrnic%2E+2023a%2E+Prediction%2Dpowered+inference%2E+Science%2C+382%286671%29%3A669%E2%80%93674%2E)
4. Anastasios N Angelopoulos, John C Duchi, and Tijana Zrnic. 2023b. PPI++: Efficient prediction-powered inference. arXiv preprint arXiv:2311.01453. — [arXiv:2311.01453](https://arxiv.org/abs/2311.01453) [find](https://scholar.google.com/scholar?q=Anastasios+N+Angelopoulos%2C+John+C+Duchi%2C+and+Tijana+Zrnic%2E+2023b%2E+PPI%2B%2B%3A+Efficient+prediction%2Dpowered+inference%2E+arXiv+preprint+arXiv%3A2311%2E01453%2E+%E2%80%94+%5BarXiv%3A2311%2E01453%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E01453%29)
5. Shubham Atreja, Joshua Ashkinaze, Lingyao Li, Julia Mendelsohn, and Libby Hemphill. 2024. Prompt design matters for computational social science tasks but in unpredictable ways. arXiv preprint arXiv:2406.11980. — [arXiv:2406.11980](https://arxiv.org/abs/2406.11980) [find](https://scholar.google.com/scholar?q=Shubham+Atreja%2C+Joshua+Ashkinaze%2C+Lingyao+Li%2C+Julia+Mendelsohn%2C+and+Libby+Hemphill%2E+2024%2E+Prompt+design+matters+for+computational+social+science+tasks+but+in+unpredictable+ways%2E+arXiv+preprint+arXiv%3A2)
6. Christopher A Bail. 2024. Can generative AI improve social science? Proceedings of the National Academy of Sciences, 121(21):e2314021121. [find](https://scholar.google.com/scholar?q=Christopher+A+Bail%2E+2024%2E+Can+generative+AI+improve+social+science%3F+Proceedings+of+the+National+Academy+of+Sciences%2C+121%2821%29%3Ae2314021121%2E)
7. Neil Band, Xuechen Li, Tengyu Ma, and Tatsunori Hashimoto. 2024. Linguistic calibration of long-form generations. In Proceedings of the Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Neil+Band%2C+Xuechen+Li%2C+Tengyu+Ma%2C+and+Tatsunori+Hashimoto%2E+2024%2E+Linguistic+calibration+of+long%2Dform+generations%2E+In+Proceedings+of+the+Forty%2Dfirst+International+Conference+on+Machine+Learning%2E)
8. Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan. 2024. Autoeval done right: Using synthetic data for model evaluation. arXiv preprint arXiv:2403.07008. — [arXiv:2403.07008](https://arxiv.org/abs/2403.07008) [find](https://scholar.google.com/scholar?q=Pierre+Boyeau%2C+Anastasios+N+Angelopoulos%2C+Nir+Yosef%2C+Jitendra+Malik%2C+and+Michael+I+Jordan%2E+2024%2E+Autoeval+done+right%3A+Using+synthetic+data+for+model+evaluation%2E+arXiv+preprint+arXiv%3A2403%2E07008%2E+%E2%80%94+%5BarX)
9. Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263–9274. [find](https://scholar.google.com/scholar?q=Dallas+Card%2C+Peter+Henderson%2C+Urvashi+Khandelwal%2C+Robin+Jia%2C+Kyle+Mahowald%2C+and+Dan+Jurafsky%2E+2020%2E+With+little+power+comes+great+responsibility%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Met)
10. Mauro Cazzaniga, Ms Florence Jaumotte, Longji Li, Mr Giovanni Melina, Augustus J Panton, Carlo Pizzinelli, Emma J Rockall, and Ms Marina Mendes Tavares. 2024. Gen-AI: Artificial intelligence and the future of work. International Monetary Fund. [find](https://scholar.google.com/scholar?q=Mauro+Cazzaniga%2C+Ms+Florence+Jaumotte%2C+Longji+Li%2C+Mr+Giovanni+Melina%2C+Augustus+J+Panton%2C+Carlo+Pizzinelli%2C+Emma+J+Rockall%2C+and+Ms+Marina+Mendes+Tavares%2E+2024%2E+Gen%2DAI%3A+Artificial+intelligence+and+the+f)
11. Ivi Chatzi, Eleni Straitouri, Suhas Thejaswi, and Manuel Gomez Rodriguez. 2024. Prediction-powered ranking of large language models. arXiv preprint arXiv:2402.17826. — [arXiv:2402.17826](https://arxiv.org/abs/2402.17826) [find](https://scholar.google.com/scholar?q=Ivi+Chatzi%2C+Eleni+Straitouri%2C+Suhas+Thejaswi%2C+and+Manuel+Gomez+Rodriguez%2E+2024%2E+Prediction%2Dpowered+ranking+of+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E17826%2E+%E2%80%94+%5BarXiv%3A2402%2E17826%5D%28https%3A%2F%2Farxiv)
12. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. [find](https://scholar.google.com/scholar?q=Tianqi+Chen+and+Carlos+Guestrin%2E+2016%2E+XGBoost%3A+A+scalable+tree+boosting+system%2E+In+Proceedings+of+the+22nd+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C+pages+785%E2%80%93794%2E)
13. Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023. [Marked personas: Using natural language prompts to measure stereotypes in language models](https://md.research.elamur.ai/free/en/papers/cheng-2023-marked-personas-natural-language-prompts/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1504–1532.
14. Cheng-Han Chiang and Hung-Yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2DYi+Lee%2E+2023%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vo)
15. Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. [A computational approach to politeness with application to social factors](https://md.research.elamur.ai/free/en/papers/mizil-2013-computational-approach-politeness-application-social/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa…
16. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proc… [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Suchin+Gururangan%2C+Dallas+Card%2C+Roy+Schwartz%2C+and+Noah+A+Smith%2E+2019%2E+Show+your+work%3A+Improved+reporting+of+experimental+results%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Method)
17. Naoki Egami, Musashi Hinck, Brandon Stewart, and Hanying Wei. 2024. Using imperfect surrogates for downstream inference: Design-based supervised learning for social science applications of large language models. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Naoki+Egami%2C+Musashi+Hinck%2C+Brandon+Stewart%2C+and+Hanying+Wei%2E+2024%2E+Using+imperfect+surrogates+for+downstream+inference%3A+Design%2Dbased+supervised+learning+for+social+science+applications+of+large+langu)
18. Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. 2022. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. Transactions of the Asso… [find](https://scholar.google.com/scholar?q=Amir+Feder%2C+Katherine+A+Keith%2C+Emaad+Manzoor%2C+Reid+Pryzant%2C+Dhanya+Sridhar%2C+Zach+Wood%2DDoughty%2C+Jacob+Eisenstein%2C+Justin+Grimmer%2C+Roi+Reichart%2C+Margaret+E+Roberts%2C+et+al%2E+2022%2E+Causal+inference+in+natu)
19. Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagreement for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6715–6726. [find](https://scholar.google.com/scholar?q=Eve+Fleisig%2C+Rediet+Abebe%2C+and+Dan+Klein%2E+2023%2E+When+the+majority+is+wrong%3A+Modeling+annotator+disagreement+for+subjective+tasks%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+)
20. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… [find](https://scholar.google.com/scholar?q=Jiahui+Geng%2C+Fengyu+Cai%2C+Yuxia+Wang%2C+Heinz+Koeppl%2C+Preslav+Nakov%2C+and+Iryna+Gurevych%2E+2024%2E+A+survey+of+confidence+estimation+and+calibration+in+large+language+models%2E+In+Proceedings+of+the+2024+Confe)
21. Jing Huang and Diyi Yang. 2023. [Culturally aware natural language inference](https://md.research.elamur.ai/free/en/papers/huang-2023-culturally-aware-natural-language-inference/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7591–7609, Singapore. Association for Computational Linguistics.
22. Katherine Keith and Brendan O’Connor. 2018. Uncertainty-aware generative models for inferring document class prevalence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4575–4585, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Katherine+Keith+and+Brendan+O%E2%80%99Connor%2E+2018%2E+Uncertainty%2Daware+generative+models+for+inferring+document+class+prevalence%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+)
23. Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang. 2024. MEGAnno+: A human-LLM collaborative annotation system. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 168–176, St. Julians, Ma… [find](https://scholar.google.com/scholar?q=Hannah+Kim%2C+Kushan+Mitra%2C+Rafael+Li+Chen%2C+Sajjadur+Rahman%2C+and+Dan+Zhang%2E+2024%2E+MEGAnno%2B%3A+A+human%2DLLM+collaborative+annotation+system%2E+In+Proceedings+of+the+18th+Conference+of+the+European+Chapter+of+)
24. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). In The Twelfth International Conference on Learning Representations.
25. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In Advances in Neural Information Processing Systems.
26. Michelle S Lam, Janice Teoh, James A Landay, Jeffrey Heer, and Michael S Bernstein. 2024. Concept induction: Analyzing unstructured text with high-level concepts using lloom. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–28. [find](https://scholar.google.com/scholar?q=Michelle+S+Lam%2C+Janice+Teoh%2C+James+A+Landay%2C+Jeffrey+Heer%2C+and+Michael+S+Bernstein%2E+2024%2E+Concept+induction%3A+Analyzing+unstructured+text+with+high%2Dlevel+concepts+using+lloom%2E+In+Proceedings+of+the+CHI)
27. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. [ChatGPT outperforms crowd workers for text-annotation tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/). Proceedings of the National Academy of Sciences, 120(30):e2305016120.
28. Richard N Landers and Tara S Behrend. 2023. Auditing the AI auditors: A framework for evaluating fairness and bias in high stakes AI predictive models. American Psychologist, 78(1):36. [find](https://scholar.google.com/scholar?q=Richard+N+Landers+and+Tara+S+Behrend%2E+2023%2E+Auditing+the+AI+auditors%3A+A+framework+for+evaluating+fairness+and+bias+in+high+stakes+AI+predictive+models%2E+American+Psychologist%2C+78%281%29%3A36%2E)
29. Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135–18143. [find](https://scholar.google.com/scholar?q=Anisha+Gunjal%2C+Jihan+Yin%2C+and+Erhan+Bas%2E+2024%2E+Detecting+and+preventing+hallucinations+in+large+vision+language+models%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+38%2C+pag)
30. Hanlin Li, Nicholas Vincent, Stevie Chancellor, and Brent Hecht. 2023a. The dimensions of data labor: A road map for researchers, activists, and policymakers to empower data producers. In Proceedings of the 2023 ACM conference on fairness, accountability, and transparency, pages 1151–1161. [find](https://scholar.google.com/scholar?q=Hanlin+Li%2C+Nicholas+Vincent%2C+Stevie+Chancellor%2C+and+Brent+Hecht%2E+2023a%2E+The+dimensions+of+data+labor%3A+A+road+map+for+researchers%2C+activists%2C+and+policymakers+to+empower+data+producers%2E+In+Proceedings+)
31. Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. 2024. [Llm-rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts](https://md.research.elamur.ai/free/en/papers/hashemi-2024-llm-rubric-multidimensional-calibrated-approach/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap…
32. Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023b. [HaluEval: A large-scale hallucination evaluation benchmark for large language models](https://md.research.elamur.ai/free/en/papers/li-2023-halueval-large-scale-hallucination-evaluation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464, Singapore. Association for Comput…
33. Jay D Hmielowski, Lauren Feldman, Teresa A Myers, Anthony Leiserowitz, and Edward Maibach. 2014. An attack on science? media use, trust in scientists, and perceptions of global warming. Public Understanding of Science, 23(7):866–883. [find](https://scholar.google.com/scholar?q=Jay+D+Hmielowski%2C+Lauren+Feldman%2C+Teresa+A+Myers%2C+Anthony+Leiserowitz%2C+and+Edward+Maibach%2E+2014%2E+An+attack+on+science%3F+media+use%2C+trust+in+scientists%2C+and+perceptions+of+global+warming%2E+Public+Underst)
34. Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy Chen, Zhengyuan Liu, and Diyi Yang. 2023c. [Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation](https://md.research.elamur.ai/free/en/papers/li-2023-coannotating-uncertainty-guided-work-allocation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p…
35. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-Eval: NLG evaluation using Gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522.
36. Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020. Detecting stance in media on global warming. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3296–3315. [find](https://scholar.google.com/scholar?q=Yiwei+Luo%2C+Dallas+Card%2C+and+Dan+Jurafsky%2E+2020%2E+Detecting+stance+in+media+on+global+warming%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+3296%E2%80%933315%2E)
37. Katerina Margatina, Giorgos Vernikos, Loïc Barrault, and Nikolaos Aletras. 2021. Active learning by acquiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 650–663, Online and Punta Cana, Dominican Republic. Association for Com… [find](https://scholar.google.com/scholar?q=Katerina+Margatina%2C+Giorgos+Vernikos%2C+Lo%C3%AFc+Barrault%2C+and+Nikolaos+Aletras%2E+2021%2E+Active+learning+by+acquiring+contrastive+examples%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natura)
38. Matthew L Newman, Carla J Groom, Lori D Handelman, and James W Pennebaker. 2008. Gender differences in language use: An analysis of 14,000 text samples. Discourse processes, 45(3):211–236. [find](https://scholar.google.com/scholar?q=Matthew+L+Newman%2C+Carla+J+Groom%2C+Lori+D+Handelman%2C+and+James+W+Pennebaker%2E+2008%2E+Gender+differences+in+language+use%3A+An+analysis+of+14%2C000+text+samples%2E+Discourse+processes%2C+45%283%29%3A211%E2%80%93236%2E)
39. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22. [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Joseph+O%E2%80%99Brien%2C+Carrie+Jun+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S+Bernstein%2E+2023%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+In+Proceedings+of+the+36)
40. Chau Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, and Mohit Iyyer. 2024. [TopicGPT: A prompt-based topic modeling framework](https://md.research.elamur.ai/free/en/papers/pham-2024-topicgpt-prompt-based-topic-modeling/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers),…
41. Filipe Ribeiro, Lucas Henrique, Fabricio Benevenuto, Abhijnan Chakraborty, Juhi Kulshrestha, Mah-moudreza Babaei, and Krishna Gummadi. 2018. Media bias monitor: Quantifying biases of social media news outlets at large-scale. In Proceedings of the International AAAI Conference on Web and Social Media… [find](https://scholar.google.com/scholar?q=Filipe+Ribeiro%2C+Lucas+Henrique%2C+Fabricio+Benevenuto%2C+Abhijnan+Chakraborty%2C+Juhi+Kulshrestha%2C+Mah%2Dmoudreza+Babaei%2C+and+Krishna+Gummadi%2E+2018%2E+Media+bias+monitor%3A+Quantifying+biases+of+social+media+news)
42. Ronald E Robertson, Shan Jiang, Kenneth Joseph, Lisa Friedland, David Lazer, and Christo Wilson. 2018. Auditing partisan audience bias within Google search. Proceedings of the ACM on human-computer interaction, 2(CSCW):1–22. [find](https://scholar.google.com/scholar?q=Ronald+E+Robertson%2C+Shan+Jiang%2C+Kenneth+Joseph%2C+Lisa+Friedland%2C+David+Lazer%2C+and+Christo+Wilson%2E+2018%2E+Auditing+partisan+audience+bias+within+Google+search%2E+Proceedings+of+the+ACM+on+human%2Dcomputer+in)
43. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. ARES: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… [find](https://scholar.google.com/scholar?q=Jon+Saad%2DFalcon%2C+Omar+Khattab%2C+Christopher+Potts%2C+and+Matei+Zaharia%2E+2024%2E+ARES%3A+An+automated+evaluation+framework+for+retrieval%2Daugmented+generation+systems%2E+In+Proceedings+of+the+2024+Conference+of+)
44. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. [Whose opinions do language models reflect?](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/) In International Conference on Machine Learning, pages 29971–30004. PMLR.
45. Sebastin Santy, Jenny Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. [NLPositionality: Characterizing design biases of datasets and models](https://md.research.elamur.ai/free/en/papers/santy-2023-nlpositionality-characterizing-design-biases-datasets/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9080–9102, Toronto, C…
46. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. [Annotators with attitudes: How annotator beliefs and identities bias toxic language detection](https://md.research.elamur.ai/free/en/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Ling…
47. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Melanie+Sclar%2C+Yejin+Choi%2C+Yulia+Tsvetkov%2C+and+Alane+Suhr%2E+2023%2E+Quantifying+language+models%E2%80%99+sensitivity+to+spurious+features+in+prompt+design+or%3A+How+I+learned+to+start+worrying+about+prompt+formatt)
48. Yanchuan Sim, Brice D. L. Acree, Justin H. Gross, and Noah A. Smith. 2013. Measuring ideological proportions in political speeches. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 91–101, Seattle, Washington, USA. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Yanchuan+Sim%2C+Brice+D%2E+L%2E+Acree%2C+Justin+H%2E+Gross%2C+and+Noah+A%2E+Smith%2E+2013%2E+Measuring+ideological+proportions+in+political+speeches%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natura)
49. Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. From humans to machines: Can ChatGPT-like LLMs effectively replace human annotators in NLP tasks. In Workshop Proceedings of the 17th International AAAI Conference on Web and Social Media. [find](https://scholar.google.com/scholar?q=Surendrabikram+Thapa%2C+Usman+Naseem%2C+and+Mehwish+Nasim%2E+2023%2E+From+humans+to+machines%3A+Can+ChatGPT%2Dlike+LLMs+effectively+replace+human+annotators+in+NLP+tasks%2E+In+Workshop+Proceedings+of+the+17th+Inter)
50. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Confer… [find](https://scholar.google.com/scholar?q=Katherine+Tian%2C+Eric+Mitchell%2C+Allan+Zhou%2C+Archit+Sharma%2C+Rafael+Rafailov%2C+Huaxiu+Yao%2C+Chelsea+Finn%2C+and+Christopher+D+Manning%2E+2023%2E+Just+ask+for+calibration%3A+Strategies+for+eliciting+calibrated+conf)
51. Aad W Van der Vaart. 2000. Asymptotic statistics, volume 3. Cambridge university press. [find](https://scholar.google.com/scholar?q=Aad+W+Van+der+Vaart%2E+2000%2E+Asymptotic+statistics%2C+volume+3%2E+Cambridge+university+press%2E)
52. Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. 2024. [Ghostbuster: Detecting text ghostwritten by large language models](https://md.research.elamur.ai/free/en/papers/verma-2024-ghostbuster-detecting-text-ghostwritten-large/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)…
53. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. [Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks](https://md.research.elamur.ai/free/en/papers/veselovsky-2023-artificial-artificial-artificial-intelligence-crowd/). arXiv preprint arXiv:2306.07899.
54. Rob Voigt, Nicholas P Camp, Vinodkumar Prabhakaran, William L Hamilton, Rebecca C Hetey, Camilla M Griffiths, David Jurgens, Dan Jurafsky, and Jennifer L Eberhardt. 2017. Language from police body camera footage shows racial disparities in officer respect. Proceedings of the National Academy of Scie… [find](https://scholar.google.com/scholar?q=Rob+Voigt%2C+Nicholas+P+Camp%2C+Vinodkumar+Prabhakaran%2C+William+L+Hamilton%2C+Rebecca+C+Hetey%2C+Camilla+M+Griffiths%2C+David+Jurgens%2C+Dan+Jurafsky%2C+and+Jennifer+L+Eberhardt%2E+2017%2E+Language+from+police+body+cam)
55. Tharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, and Christopher Homan. 2023. Disagreement matters: Preserving label diversity by jointly modeling item and annotator label distributions with DisCo. In Findings of the Association for Computational Linguistics: ACL… [find](https://scholar.google.com/scholar?q=Tharindu+Cyril+Weerasooriya%2C+Alexander+Ororbia%2C+Raj+Bhensadadia%2C+Ashiqur+KhudaBukhsh%2C+and+Christopher+Homan%2E+2023%2E+Disagreement+matters%3A+Preserving+label+diversity+by+jointly+modeling+item+and+annotat)
56. Johnny Tian-Zheng Wei, Frederike Zufall, and Robin Jia. 2024. Operationalizing content moderation “accuracy”’ in the Digital Services Act. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 1527–1538. [find](https://scholar.google.com/scholar?q=Johnny+Tian%2DZheng+Wei%2C+Frederike+Zufall%2C+and+Robin+Jia%2E+2024%2E+Operationalizing+content+moderation+%E2%80%9Caccuracy%E2%80%9D%E2%80%99+in+the+Digital+Services+Act%2E+In+Proceedings+of+the+AAAI%2FACM+Conference+on+AI%2C+Ethics%2C+and+)
57. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, page… [find](https://scholar.google.com/scholar?q=Laura+Weidinger%2C+Jonathan+Uesato%2C+Maribeth+Rauh%2C+Conor+Griffin%2C+Po%2DSen+Huang%2C+John+Mellor%2C+Amelia+Glaese%2C+Myra+Cheng%2C+Borja+Balle%2C+Atoosa+Kasirzadeh%2C+et+al%2E+2022%2E+Taxonomy+of+risks+posed+by+language+m)
58. Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023. LLMaAA: Making large language models as active annotators. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13088–13103, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ruoyu+Zhang%2C+Yanzeng+Li%2C+Yongliang+Ma%2C+Ming+Zhou%2C+and+Lei+Zou%2E+2023%2E+LLMaAA%3A+Making+large+language+models+as+active+annotators%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2023)
59. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-Bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36:46595–4…
60. Kaitlyn Zhou, Jena D Hwang, Xiang Ren, and Maarten Sap. 2024. Relying on the unreliable: The impact of language models’ reluctance to express uncertainty. arXiv preprint arXiv:2401.06730. [find](https://scholar.google.com/scholar?q=Kaitlyn+Zhou%2C+Jena+D+Hwang%2C+Xiang+Ren%2C+and+Maarten+Sap%2E+2024%2E+Relying+on+the+unreliable%3A+The+impact+of+language+models%E2%80%99+reluctance+to+express+uncertainty%2E+arXiv+preprint+arXiv%3A2401%2E06730%2E)
61. Caleb Ziems, Jane Dwivedi-Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2023. [Normbank: A knowledge bank of situational social norms](https://md.research.elamur.ai/free/en/papers/ziems-2023-normbank-knowledge-bank-situational-social/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7756–7776.
62. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. [Can large language models transform computational social science?](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/) Computational Linguistics, 50(1):237–291.
63. Tijana Zrnic and Emmanuel J Candès. 2024. Active statistical inference. arXiv preprint arXiv:2403.03208. — [arXiv:2403.03208](https://arxiv.org/abs/2403.03208) [find](https://scholar.google.com/scholar?q=Tijana+Zrnic+and+Emmanuel+J+Cand%C3%A8s%2E+2024%2E+Active+statistical+inference%2E+arXiv+preprint+arXiv%3A2403%2E03208%2E+%E2%80%94+%5BarXiv%3A2403%2E03208%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E03208%29)

*Updated 31 October 2026*