---
title: "Thesis Proposal — paper by Pandita and Homan (2026)"
description: "The authors build on the Variance Estimation Toolkit (VET). 87 references. Among them: 0."
authors: ["Deepak Pandita", "Christopher M. Homan"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)"
source: "https://aclanthology.org/2026.acl-srw.74/"
doi: "10.18653/v1/2026.acl-srw.74"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pandita-2026-thesis-proposal-human-centered-perspective/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pandita-2026-thesis-proposal-human-centered-perspective/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/53/) / Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment

# Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment

**Authors:** Deepak Pandita and Christopher M. Homan · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-srw.74/) · [PDF](https://aclanthology.org/2026.acl-srw.74.pdf) · [DOI 10.18653/v1/2026.acl-srw.74](https://doi.org/10.18653/v1/2026.acl-srw.74) · [OpenAlex](https://openalex.org/W7166799601)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
This thesis proposal argues that ML evaluation and LLM alignment are less reliable because human disagreement is usually treated as noise. Evaluators typically collapse a few labels per item into one by majority vote, which hides minority views and makes results harder to reproduce. The authors build on the Variance Estimation Toolkit (VET). Their method fits a two-stage probabilistic response model to real annotated data, then simulates comparisons between an ideal model and a perturbed one to estimate p-values and, as a new addition, statistical power. Experiments on seven multi-annotator datasets show that, with a fixed annotation budget, collecting more responses per item usually makes comparisons more reliable than adding more items. The best balance depends on the metric used. On alignment, they study how cohesive demographic groups are when rating offense, and they introduce ProRefine, an inference-time prompt refinement method driven by textual feedback, which improves multi-step reasoning. Planned work targets pluralistic alignment through perspective-specific feedback models and bandit-based prompt selection. The overall goal is budget-aware, perspective-sensitive evaluation that practitioners can trust.

## Key points
- The proposal is organized around six research questions. RQ1–RQ3 cover reproducible evaluation: disaggregated labels, budget allocation, and rater diversity. RQ4–RQ6 cover alignment: rater cohesion, textual feedback, and pluralistic adaptation.
- The seven datasets are MultiDomain Agreement, Stanford Toxicity, Amazon reviews, HS-Brexit, ConvAbuse, ArMIS, and Measuring Hate Speech (MHS).
- The comparison metrics are mean absolute error difference (MAE), item-wise wins (Wins), and mean EMD difference (MEMD).
- In the simulations, the Toxicity dataset was run with a perturbation of ϵ = 0.05, MultiDomain with ϵ = 0.1, and each data point drew on 10,000 samples. p-values fell as K (responses per item) grew, and the same trend held across five metrics at N×K = 2500.
- The authors acknowledge several limitations:
  - responses are assumed independent across items;
  - political views are reduced to three groups (Democrats, Republicans, Independents);
  - ProRefine adds latency and cost and has no convergence guarantee.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Rochester Institute of Technology](https://md.research.elamur.ai/free/en/universities/rochester-institute-of-technology/) — University, Rochester, United States · 15 works · 5,233 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works

## Cite

Cite as: Deepak Pandita and Christopher M. Homan. 2026. Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pandita2026thesis,
  title = {{Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment}},
  author = {Deepak Pandita and Christopher M. Homan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.74},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.74}
}
```

- [← previous Thesis Proposal: Self-Adaptive and Epistemic Uncertainty-Guided ASR of Dense Intra-Sentential Code-Switched Speech for African Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/umar-2026-thesis-proposal-self-adaptive-epistemic/)
- [next → Thesis Proposal: When Does an Agent Know It Is Lost? Confidence Trajectory Analysis for Tool-Using LLMs](https://md.research.elamur.ai/free/en/papers/mao-2026-thesis-proposal-when-does-agent/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/53/)

## References

87 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection. arXiv preprint arXiv:2106.15896. — [arXiv:2106.15896](https://arxiv.org/abs/2106.15896) [find](https://scholar.google.com/scholar?q=Sohail+Akhtar%2C+Valerio+Basile%2C+and+Viviana+Patti%2E+2021%2E+Whose+opinions+matter%3F+perspective%2Daware+models+to+identify+opinions+of+hate+speech+victims+in+abusive+language+detection%2E+arXiv+preprint+arXiv%3A)
2. Afra Feyza Akyurek, Ekin Akyurek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, and Niket Tandon. 2023. [RL4F: Generating natural language feedback with reinforcement learning for repairing model outputs](https://md.research.elamur.ai/free/en/papers/akyurek-2023-rl4f-generating-natural-language-feedback/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vo…
3. Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020. [Identifying and measuring annotator bias based on annotators’ demographic characteristics](https://md.research.elamur.ai/free/en/papers/kuwatly-2020-identifying-measuring-annotator-bias-based/). In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 184–190, Online. Association for Computational Linguistics.
4. Dina Almanea and Massimo Poesio. 2022. [ArMIS -the Arabic misogyny and sexism corpus with annotator subjective disagreements](https://md.research.elamur.ai/free/en/papers/almanea-2022-armis-arabic-misogyny-sexism-corpus/). In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2282–2291, Marseille, France. European Language Resources Association.
5. Lora Aroyo, Alex Taylor, Mark Díaz, Christopher Homan, Alicia Parrish, Gregory Serapio-García, Vinodkumar Prabhakaran, and Ding Wang. 2023. Dices dataset: Diversity in conversational ai evaluation for safety. In Advances in Neural Information Processing Systems, volume 36, pages 53330–53342, New Orl… [find](https://scholar.google.com/scholar?q=Lora+Aroyo%2C+Alex+Taylor%2C+Mark+D%C3%ADaz%2C+Christopher+Homan%2C+Alicia+Parrish%2C+Gregory+Serapio%2DGarc%C3%ADa%2C+Vinodkumar+Prabhakaran%2C+and+Ding+Wang%2E+2023%2E+Dices+dataset%3A+Diversity+in+conversational+ai+evaluation+for)
6. Mohammad Arvan, Luís Pina, and Natalie Parde. 2022. Reproducibility in computational linguistics: Is source code enough? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2350–2361, Abu Dhabi, United Arab Emirates. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Mohammad+Arvan%2C+Lu%C3%ADs+Pina%2C+and+Natalie+Parde%2E+2022%2E+Reproducibility+in+computational+linguistics%3A+Is+source+code+enough%3F+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+)
7. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, and 32 others. 20… — [Constitutional AI: Harmlessness from AI Feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
8. Monya Baker. 2016. 1,500 scientists lift the lid on reproducibility. Nature, 533(7604):452–454. [find](https://scholar.google.com/scholar?q=Monya+Baker%2E+2016%2E+1%2C500+scientists+lift+the+lid+on+reproducibility%2E+Nature%2C+533%287604%29%3A452%E2%80%93454%2E)
9. Francesco Barile, Shabnam Najafian, Tim Draws, Oana Inel, Alisa Rieger, Rishav Hada, and Nava Tintarev. 2021. Toward benchmarking group explanations: Evaluating the effect of aggregation strategies versus explanation. In Perspectives on the Evaluation of Recommender Systems Workshop 2021: co-located… [find](https://scholar.google.com/scholar?q=Francesco+Barile%2C+Shabnam+Najafian%2C+Tim+Draws%2C+Oana+Inel%2C+Alisa+Rieger%2C+Rishav+Hada%2C+and+Nava+Tintarev%2E+2021%2E+Toward+benchmarking+group+explanations%3A+Evaluating+the+effect+of+aggregation+strategies+ve)
10. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. [We need to consider disagreement in evaluation](https://md.research.elamur.ai/free/en/papers/basile-2021-we-need-consider-disagreement-evaluation/). In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computa…
11. Anya Belz, Craig Thomson, Ehud Reiter, and Simon Mille. 2023. Non-Repeatable Experiments and Non-Reproducible Results: The Reproducibility Crisis in Human Evaluation in NLP. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3676–3687, Toronto, Canada. Association for Comp… [find](https://scholar.google.com/scholar?q=Anya+Belz%2C+Craig+Thomson%2C+Ehud+Reiter%2C+and+Simon+Mille%2E+2023%2E+Non%2DRepeatable+Experiments+and+Non%2DReproducible+Results%3A+The+Reproducibility+Crisis+in+Human+Evaluation+in+NLP%2E+In+Findings+of+the+Associa)
12. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. [Toward a perspectivist turn in ground truthing for predictive computing](https://md.research.elamur.ai/free/en/papers/basile-2021-toward-perspectivist-turn-ground-truthing/). Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):6860–6868.
13. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar… [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Xander+Davies%2C+Claudia+Shi%2C+Thomas+Krendl+Gilbert%2C+J%C3%A9r%C3%A9my+Scheurer%2C+Javier+Rando%2C+Rachel+Freedman%2C+Tomasz+Korbak%2C+David+Lindner%2C+Pedro+Freire%2C+Tony+Wang%2C+Samuel+Marks%2C+Charbel%2DRapha%C3%ABl+)
14. Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser. 2021. ConvAbuse: Data, analysis, and benchmarks for nuanced abuse detection in conversational AI. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7388–7403, Online and Punta Cana, Dominican Re… [find](https://scholar.google.com/scholar?q=Amanda+Cercas+Curry%2C+Gavin+Abercrombie%2C+and+Verena+Rieser%2E+2021%2E+ConvAbuse%3A+Data%2C+analysis%2C+and+benchmarks+for+nuanced+abuse+detection+in+conversational+AI%2E+In+Proceedings+of+the+2021+Conference+on+Em)
15. Tanise Ceron, Neele Falk, Ana Barić, Dmitry Nikolaev, and Sebastian Padó. 2024. [Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs](https://md.research.elamur.ai/free/en/papers/ceron-2024-beyond-prompt-brittleness-evaluating-reliability/). Transactions of the Association for Computational Linguistics, 12:1378– 1400.
16. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+Reinforcement+Learning+from+Human+Preferences%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+vo)
17. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
18. Aida Mostafazadeh Davani, Mark Díaz, Dylan Baker, and Vinodkumar Prabhakaran. 2024. D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation. arXiv preprint. ArXiv:2404.10857 [cs]. [find](https://scholar.google.com/scholar?q=Aida+Mostafazadeh+Davani%2C+Mark+D%C3%ADaz%2C+Dylan+Baker%2C+and+Vinodkumar+Prabhakaran%2E+2024%2E+D3CODE%3A+Disentangling+Disagreements+in+Data+across+Cultures+on+Offensiveness+Detection+and+Evaluation%2E+arXiv+preprin)
19. Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. 2022. [RLPrompt: Optimizing discrete text prompts with reinforcement learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/).
20. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3369–3391, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+3369%E2%80%933391%2C+Abu+Dhabi%2C+United+Arab+Emirates%2E+Association+for+Computational+Linguistics%2E)
21. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, A… [find](https://scholar.google.com/scholar?q=Rotem+Dror%2C+Gili+Baumer%2C+Segev+Shlomov%2C+and+Roi+Reichart%2E+2018%2E+The+hitchhiker%E2%80%99s+guide+to+testing+statistical+significance+in+natural+language+processing%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+)
22. Anca Dumitrache, Oana Inel, Lora Aroyo, Benjamin Timmermans, and Chris Welty. 2018. Crowdtruth 2.0: Quality metrics for crowdsourcing with disagreement. In 1st Workshop on Subjectivity, Ambiguity and Disagreement (SAD) in Crowdsourcing 2018, and CrowdBias’18: Disentangling the Relation Between Crowd… [find](https://scholar.google.com/scholar?q=Anca+Dumitrache%2C+Oana+Inel%2C+Lora+Aroyo%2C+Benjamin+Timmermans%2C+and+Chris+Welty%2E+2018%2E+Crowdtruth+2%2E0%3A+Quality+metrics+for+crowdsourcing+with+disagreement%2E+In+1st+Workshop+on+Subjectivity%2C+Ambiguity+and+)
23. Suyash Fulay, William Brannon, Shrestha Mohanty, Cassandra Overney, Elinor Poole-Dayan, Deb Roy, and Jad Kabbara. 2024. On the relationship between truth and political bias in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9004– 9018… [find](https://scholar.google.com/scholar?q=Suyash+Fulay%2C+William+Brannon%2C+Shrestha+Mohanty%2C+Cassandra+Overney%2C+Elinor+Poole%2DDayan%2C+Deb+Roy%2C+and+Jad+Kabbara%2E+2024%2E+On+the+relationship+between+truth+and+political+bias+in+language+models%2E+In+Proc)
24. Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–19. [find](https://scholar.google.com/scholar?q=Mitchell+L+Gordon%2C+Michelle+S+Lam%2C+Joon+Sung+Park%2C+Kayur+Patel%2C+Jeff+Hancock%2C+Tatsunori+Hashimoto%2C+and+Michael+S+Bernstein%2E+2022%2E+Jury+learning%3A+Integrating+dissenting+voices+into+machine+learning+mod)
25. Nitesh Goyal, Ian D Kivlichan, Rachel Rosen, and Lucy Vasserman. 2022. Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):1–28. [find](https://scholar.google.com/scholar?q=Nitesh+Goyal%2C+Ian+D+Kivlichan%2C+Rachel+Rosen%2C+and+Lucy+Vasserman%2E+2022%2E+Is+your+toxicity+my+toxicity%3F+exploring+the+impact+of+rater+identity+on+toxicity+annotation%2E+Proceedings+of+the+ACM+on+Human%2DComp)
26. Odd Erik Gundersen. 2020. The reproducibility crisis is real. AI Magazine, 41(3):103–106. [find](https://scholar.google.com/scholar?q=Odd+Erik+Gundersen%2E+2020%2E+The+reproducibility+crisis+is+real%2E+AI+Magazine%2C+41%283%29%3A103%E2%80%93106%2E)
27. Odd Erik Gundersen and Sigbjørn Kjensmo. 2018. State of the art: Reproducibility in artificial intelligence. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1):1644–1651. [find](https://scholar.google.com/scholar?q=Odd+Erik+Gundersen+and+Sigbj%C3%B8rn+Kjensmo%2E+2018%2E+State+of+the+art%3A+Reproducibility+in+artificial+intelligence%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+32%281%29%3A1644%E2%80%931651%2E)
28. Qianyue Hao, Sibo Li, Jian Yuan, and Yong Li. 2025. Rl of thoughts: Navigating llm reasoning with inference-time reinforcement learning. Preprint, arXiv:2505.14140. — [arXiv:2505.14140](https://arxiv.org/abs/2505.14140) [find](https://scholar.google.com/scholar?q=Qianyue+Hao%2C+Sibo+Li%2C+Jian+Yuan%2C+and+Yong+Li%2E+2025%2E+Rl+of+thoughts%3A+Navigating+llm+reasoning+with+inference%2Dtime+reinforcement+learning%2E+Preprint%2C+arXiv%3A2505%2E14140%2E+%E2%80%94+%5BarXiv%3A2505%2E14140%5D%28https%3A%2F%2Farxiv%2E)
29. Christopher Homan, Gregory Serapio-Garcia, Lora Aroyo, Mark Diaz, Alicia Parrish, Vinodkumar Prabhakaran, Alex Taylor, and Ding Wang. 2024. Intersectionality in AI safety: Using multilevel models to understand diverse perceptions of safety in conversational AI. In Proceedings of the 3rd Workshop on… [find](https://scholar.google.com/scholar?q=Christopher+Homan%2C+Gregory+Serapio%2DGarcia%2C+Lora+Aroyo%2C+Mark+Diaz%2C+Alicia+Parrish%2C+Vinodkumar+Prabhakaran%2C+Alex+Taylor%2C+and+Ding+Wang%2E+2024%2E+Intersectionality+in+AI+safety%3A+Using+multilevel+models+to+u)
30. Christopher M Homan, Flip Korn, Deepak Pandita, and Chris Welty. 2026. How many ratings per item are necessary for reliable significance testing? In Findings of the Association for Computational Linguistics: EACL 2026, pages 4258–4273, Rabat, Morocco. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Christopher+M+Homan%2C+Flip+Korn%2C+Deepak+Pandita%2C+and+Chris+Welty%2E+2026%2E+How+many+ratings+per+item+are+necessary+for+reliable+significance+testing%3F+In+Findings+of+the+Association+for+Computational+Lingu)
31. Christopher M Homan, Shira Wein, Lora M Aroyo, and Chris Welty. 2023. How many raters do you need? power analysis for foundation models. In Proceedings of I Can’t Believe It’s Not Better (ICBINB): Failure Modes in the Age of Foundation Models. [find](https://scholar.google.com/scholar?q=Christopher+M+Homan%2C+Shira+Wein%2C+Lora+M+Aroyo%2C+and+Chris+Welty%2E+2023%2E+How+many+raters+do+you+need%3F+power+analysis+for+foundation+models%2E+In+Proceedings+of+I+Can%E2%80%99t+Believe+It%E2%80%99s+Not+Better+%28ICBINB%29%3A+Fai)
32. Matthew Hutson. 2018. Artificial intelligence faces reproducibility crisis. Science, 359(6377):725–726. [find](https://scholar.google.com/scholar?q=Matthew+Hutson%2E+2018%2E+Artificial+intelligence+faces+reproducibility+crisis%2E+Science%2C+359%286377%29%3A725%E2%80%93726%2E)
33. Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for a diversity of perspectives. In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021), pages 299–318. [find](https://scholar.google.com/scholar?q=Deepak+Kumar%2C+Patrick+Gage+Kelley%2C+Sunny+Consolvo%2C+Joshua+Mason%2C+Elie+Bursztein%2C+Zakir+Durumeric%2C+Kurt+Thomas%2C+and+Michael+Bailey%2E+2021%2E+Designing+toxic+content+classification+for+a+diversity+of+persp)
34. Savannah Larimore, Ian Kennedy, Breon Haskett, and Alina Arseniev-Koehler. 2021. Reconsidering annotator disagreement about racist language: Noise or signal? In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media, pages 81–90, Online. Association for Compu… [find](https://scholar.google.com/scholar?q=Savannah+Larimore%2C+Ian+Kennedy%2C+Breon+Haskett%2C+and+Alina+Arseniev%2DKoehler%2E+2021%2E+Reconsidering+annotator+disagreement+about+racist+language%3A+Noise+or+signal%3F+In+Proceedings+of+the+Ninth+International+)
35. Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. [Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement](https://md.research.elamur.ai/free/en/papers/leonardelli-2021-agreeing-disagree-annotating-offensive-language/). arXiv preprint arXiv:2109.13563.
36. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167, Vancouver… [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+In+Proceedings+of+the+55th+Annual+Meetin)
37. Tong Liu, Christopher Homan, Cecilia Ovesdotter Alm, Megan Lytle, Ann Marie White, and Henry Kautz. 2016. Understanding discourse on work and job-related well-being in public social media. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… [find](https://scholar.google.com/scholar?q=Tong+Liu%2C+Christopher+Homan%2C+Cecilia+Ovesdotter+Alm%2C+Megan+Lytle%2C+Ann+Marie+White%2C+and+Henry+Kautz%2E+2016%2E+Understanding+discourse+on+work+and+job%2Drelated+well%2Dbeing+in+public+social+media%2E+In+Proceedi)
38. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her)
39. Maitrey Mehta, Valentina Pyatkin, and Vivek Srikumar. 2024. Promptly predicting structures: The return of inference. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 112– 1… [find](https://scholar.google.com/scholar?q=Maitrey+Mehta%2C+Valentina+Pyatkin%2C+and+Vivek+Srikumar%2E+2024%2E+Promptly+predicting+structures%3A+The+return+of+inference%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapter+of+the+Associat)
40. Meta. 2024. llama-models/models/llama3_2/MODEL_card.md at main · meta-llama/llama-models. [find](https://scholar.google.com/scholar?q=Meta%2E+2024%2E+llama%2Dmodels%2Fmodels%2Fllama3%5F2%2FMODEL%5Fcard%2Emd+at+main+%C2%B7+meta%2Dllama%2Fllama%2Dmodels%2E)
41. Margot Mieskes, Karën Fort, Aurélie Névéol, Cyril Grouin, and Kevin Cohen. 2019. Community perspective on replicability in natural language processing. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 768–775, Varna, Bulgaria. INCOM… [find](https://scholar.google.com/scholar?q=Margot+Mieskes%2C+Kar%C3%ABn+Fort%2C+Aur%C3%A9lie+N%C3%A9v%C3%A9ol%2C+Cyril+Grouin%2C+and+Kevin+Cohen%2E+2019%2E+Community+perspective+on+replicability+in+natural+language+processing%2E+In+Proceedings+of+the+International+Conference+o)
42. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. [Dealing with disagreements: Looking beyond the majority vote in subjective annotations](https://md.research.elamur.ai/free/en/papers/davani-2022-dealing-disagreements-looking-beyond-majority/). Transactions of the Association for Computational Linguistics, 10:92–110.
43. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/). arXiv preprint. ArXiv:2501.19393 [cs].
44. Rithesh Murthy, Ming Zhu, Liangwei Yang, Jielin Qiu, Juntao Tan, Shelby Heinecke, Caiming Xiong, Silvio Savarese, and Huan Wang. 2025. Promptomatix: An automatic prompt optimization framework for large language models. Preprint, arXiv:2507.14241. — [arXiv:2507.14241](https://arxiv.org/abs/2507.14241) [find](https://scholar.google.com/scholar?q=Rithesh+Murthy%2C+Ming+Zhu%2C+Liangwei+Yang%2C+Jielin+Qiu%2C+Juntao+Tan%2C+Shelby+Heinecke%2C+Caiming+Xiong%2C+Silvio+Savarese%2C+and+Huan+Wang%2E+2025%2E+Promptomatix%3A+An+automatic+prompt+optimization+framework+for+larg)
45. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Tr… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
46. Deepak Pandita, Flip Korn, Chris Welty, and Christopher M Homan. 2026a. Forest vs tree: The (n, k) trade-off in reproducible ml evaluation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(29):24736–24744. [find](https://scholar.google.com/scholar?q=Deepak+Pandita%2C+Flip+Korn%2C+Chris+Welty%2C+and+Christopher+M+Homan%2E+2026a%2E+Forest+vs+tree%3A+The+%28n%2C+k%29+trade%2Doff+in+reproducible+ml+evaluation%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligenc)
47. Deepak Pandita, Tharindu Cyril Weerasooriya, Sujan Dutta, Sarah K. Luger, Tharindu Ranasinghe, Ashiqur R. KhudaBukhsh, Marcos Zampieri, and Christopher M. Homan. 2024. Rater cohesion and quality from a vicarious perspective. In Findings of the Association for Computational Linguistics: EMNLP 2024, p… [find](https://scholar.google.com/scholar?q=Deepak+Pandita%2C+Tharindu+Cyril+Weerasooriya%2C+Sujan+Dutta%2C+Sarah+K%2E+Luger%2C+Tharindu+Ranasinghe%2C+Ashiqur+R%2E+KhudaBukhsh%2C+Marcos+Zampieri%2C+and+Christopher+M%2E+Homan%2E+2024%2E+Rater+cohesion+and+quality+from+)
48. Deepak Pandita, Tharindu Cyril Weerasooriya, Ankit Shah, Isabelle Diana May-Xin Ng, Christopher M. Homan, and Wei Wei. 2026b. Prorefine: Inference-time prompt refinement with textual feedback (student abstract). Proceedings of the AAAI Conference on Artificial Intelligence, 40(48):41343–41345. [find](https://scholar.google.com/scholar?q=Deepak+Pandita%2C+Tharindu+Cyril+Weerasooriya%2C+Ankit+Shah%2C+Isabelle+Diana+May%2DXin+Ng%2C+Christopher+M%2E+Homan%2C+and+Wei+Wei%2E+2026b%2E+Prorefine%3A+Inference%2Dtime+prompt+refinement+with+textual+feedback+%28student)
49. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are NLP models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Associat…
50. Jiaxin Pei and David Jurgens. 2023. When do annotator demographics matter? measuring the influence of annotator demographics with the POPQUORN dataset. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 252–265, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiaxin+Pei+and+David+Jurgens%2E+2023%2E+When+do+annotator+demographics+matter%3F+measuring+the+influence+of+annotator+demographics+with+the+POPQUORN+dataset%2E+In+Proceedings+of+the+17th+Linguistic+Annotation)
51. Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. 2020. Problems and Opportunities in Training Deep Learning Software Systems: An Analysis of Variance. In 2020 35th IEEE/ACM International Conference on Automated Softwar… [find](https://scholar.google.com/scholar?q=Hung+Viet+Pham%2C+Shangshu+Qian%2C+Jiannan+Wang%2C+Thibaud+Lutellier%2C+Jonathan+Rosenthal%2C+Lin+Tan%2C+Yaoliang+Yu%2C+and+Nachiappan+Nagappan%2E+2020%2E+Problems+and+Opportunities+in+Training+Deep+Learning+Software+S)
52. Barbara Plank. 2022. [The “problem” of human label variation: On ground truth in data, modeling and evaluation](https://md.research.elamur.ai/free/en/papers/plank-2022-problem-human-label-variation-ground/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
53. Vinodkumar Prabhakaran, Christopher Homan, Lora Aroyo, Aida Mostafazadeh Davani, Alicia Parrish, Alex Taylor, Mark Diaz, Ding Wang, and Gregory Serapio-García. 2024. GRASP: A disagreement analysis framework to assess group associations in perspectives. In Proceedings of the 2024 Conference of the No… [find](https://scholar.google.com/scholar?q=Vinodkumar+Prabhakaran%2C+Christopher+Homan%2C+Lora+Aroyo%2C+Aida+Mostafazadeh+Davani%2C+Alicia+Parrish%2C+Alex+Taylor%2C+Mark+Diaz%2C+Ding+Wang%2C+and+Gregory+Serapio%2DGarc%C3%ADa%2E+2024%2E+GRASP%3A+A+disagreement+analysis+fra)
54. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. [On releasing annotator-level labels and information in datasets](https://md.research.elamur.ai/free/en/papers/prabhakaran-2021-releasing-annotator-level-labels-information/). In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133–138, Punta Cana, Dominican…
55. Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. [Automatic prompt optimization with “gradient descent” and beam search](https://md.research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7957–7968, Singapore. Association for Computationa…
56. Edward Raff. 2019. A step toward quantifying independently reproducible machine learning research. In Advances in Neural Information Processing Systems, volume 32, Vancouver, Canada. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Edward+Raff%2E+2019%2E+A+step+toward+quantifying+independently+reproducible+machine+learning+research%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+32%2C+Vancouver%2C+Canada%2E+Curran+Associates)
57. Pratik Sachdeva, Renata Barreto, Geoff Bacon, Alexander Sahn, Claudia von Vacano, and Chris Kennedy. 2022. The measuring hate speech corpus: Leveraging rasch measurement theory for data perspectivism. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, pages 83–94, Marse… [find](https://scholar.google.com/scholar?q=Pratik+Sachdeva%2C+Renata+Barreto%2C+Geoff+Bacon%2C+Alexander+Sahn%2C+Claudia+von+Vacano%2C+and+Chris+Kennedy%2E+2022%2E+The+measuring+hate+speech+corpus%3A+Leveraging+rasch+measurement+theory+for+data+perspectivism%2E)
58. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. [Whose Opinions Do Language Models Reflect?](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/) In Proceedings of the 40th International Conference on Machine Learning, pages 29971–30004. PMLR.
59. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. [The risk of racial bias in hate speech detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics.
60. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. [Annotators with attitudes: How annotator beliefs and identities bias toxic language detection](https://md.research.elamur.ai/free/en/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Ling…
61. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. [Self-critiquing models for assisting human evaluators](https://md.research.elamur.ai/free/en/papers/saunders-2022-self-critiquing-models-assisting-human/). arXiv preprint. ArXiv:2206.05802 [cs].
62. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin…
63. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/). arXiv preprint. ArXiv:2408.03314 [cs].
64. Rion Snow, Brendan O’connor, Dan Jurafsky, and Andrew Y Ng. 2008. [Cheap and fast–but is it good?](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/) evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing, pages 254–263.
65. Anders Søgaard, Anders Johannsen, Barbara Plank, Dirk Hovy, and Hector Martínez Alonso. 2014. What’s in a p-value in NLP? In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 1–10, Ann Arbor, Michigan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard%2C+Anders+Johannsen%2C+Barbara+Plank%2C+Dirk+Hovy%2C+and+Hector+Mart%C3%ADnez+Alonso%2E+2014%2E+What%E2%80%99s+in+a+p%2Dvalue+in+NLP%3F+In+Proceedings+of+the+Eighteenth+Conference+on+Computational+Natural+Language+)
66. Kefan Song, Amir Moeini, Peng Wang, Lei Gong, Rohan Chandra, Yanjun Qi, and Shangtong Zhang. 2025. Reward is enough: Llms are in-context reinforcement learners. Preprint, arXiv:2506.06303. — [arXiv:2506.06303](https://arxiv.org/abs/2506.06303) [find](https://scholar.google.com/scholar?q=Kefan+Song%2C+Amir+Moeini%2C+Peng+Wang%2C+Lei+Gong%2C+Rohan+Chandra%2C+Yanjun+Qi%2C+and+Shangtong+Zhang%2E+2025%2E+Reward+is+enough%3A+Llms+are+in%2Dcontext+reinforcement+learners%2E+Preprint%2C+arXiv%3A2506%2E06303%2E+%E2%80%94+%5BarXiv%3A25)
67. Taylor Sorensen, Liwei Jiang, Jena D. Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, Maarten Sap, John Tasioulas, and Yejin Choi. 2024a. [Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties](https://md.research.elamur.ai/free/en/papers/sorensen-2023-value-kaleidoscope-engaging-ai-pluralistic/). Proceedings of the…
68. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. 2024b. [A Roadmap to Pluralistic Alignment](https://md.research.elamur.ai/free/en/papers/sorensen-2024-roadmap-pluralistic-alignment/). arXiv preprint. ArXiv:2402.05070 [cs].
69. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, an… — [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
70. Tharindu Weerasooriya, Sujan Dutta, Tharindu Ranasinghe, Marcos Zampieri, Christopher Homan, and Ashiqur KhudaBukhsh. 2023. Vicarious offense and noise audit of offensive speech classifiers: Unifying human and machine disagreement on what is offensive. In Proceedings of the 2023 Conference on Empiri… [find](https://scholar.google.com/scholar?q=Tharindu+Weerasooriya%2C+Sujan+Dutta%2C+Tharindu+Ranasinghe%2C+Marcos+Zampieri%2C+Christopher+Homan%2C+and+Ashiqur+KhudaBukhsh%2E+2023%2E+Vicarious+offense+and+noise+audit+of+offensive+speech+classifiers%3A+Unifying+)
71. Tharindu Cyril Weerasooriya, Alexander Ororbia, and Christopher Homan. 2022. Improving label quality by jointly modeling items and annotators. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, pages 95–99, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Tharindu+Cyril+Weerasooriya%2C+Alexander+Ororbia%2C+and+Christopher+Homan%2E+2022%2E+Improving+label+quality+by+jointly+modeling+items+and+annotators%2E+In+Proceedings+of+the+1st+Workshop+on+Perspectivist+Appro)
72. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
73. Dominik Stammbach, Philine Widmer, Eunjung Cho, Caglar Gulcehre, and Elliott Ash. 2024. Aligning large language models with diverse political viewpoints. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7257–7267, Miami, Florida, USA. Association for C… [find](https://scholar.google.com/scholar?q=Dominik+Stammbach%2C+Philine+Widmer%2C+Eunjung+Cho%2C+Caglar+Gulcehre%2C+and+Elliott+Ash%2E+2024%2E+Aligning+large+language+models+with+diverse+political+viewpoints%2E+In+Proceedings+of+the+2024+Conference+on+Empir)
74. Lisa Torrey and Matthew Taylor. 2013. Teaching on a budget: Agents advising agents in reinforcement learning. In Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, pages 1053–1060. [find](https://scholar.google.com/scholar?q=Lisa+Torrey+and+Matthew+Taylor%2E+2013%2E+Teaching+on+a+budget%3A+Agents+advising+agents+in+reinforcement+learning%2E+In+Proceedings+of+the+2013+international+conference+on+Autonomous+agents+and+multi%2Dagent+s)
75. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. [find](https://scholar.google.com/scholar?q=Alexandra+N+Uma%2C+Tommaso+Fornaciari%2C+Dirk+Hovy%2C+Silviu+Paun%2C+Barbara+Plank%2C+and+Massimo+Poesio%2E+2021%2E+Learning+from+disagreement%3A+A+survey%2E+Journal+of+Artificial+Intelligence+Research%2C+72%3A1385%E2%80%931470%2E)
76. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pages 355–368, Tokyo, Japan. Association for Comp… [find](https://scholar.google.com/scholar?q=Chris+van+der+Lee%2C+Albert+Gatt%2C+Emiel+van+Miltenburg%2C+Sander+Wubben%2C+and+Emiel+Krahmer%2E+2019%2E+Best+practices+for+the+human+evaluation+of+automatically+generated+text%2E+In+Proceedings+of+the+12th+Intern)
77. Yiming Wang, Pei Zhang, Siyuan Huang, Baosong Yang, Zhuosheng Zhang, Fei Huang, and Rui Wang. 2025a. Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decoding. arXiv preprint. ArXiv:2503.01422 [cs]. [find](https://scholar.google.com/scholar?q=Yiming+Wang%2C+Pei+Zhang%2C+Siyuan+Huang%2C+Baosong+Yang%2C+Zhuosheng+Zhang%2C+Fei+Huang%2C+and+Rui+Wang%2E+2025a%2E+Sampling%2DEfficient+Test%2DTime+Scaling%3A+Self%2DEstimating+the+Best%2Dof%2DN+Sampling+in+Early+Decoding%2E+arX)
78. Yingxu Wang, Siwei Liu, Jinyuan Fang, and Zaiqiao Meng. 2025b. Evoagentx: An automated framework for evolving agentic workflows. Preprint, arXiv:2507.03616. [find](https://scholar.google.com/scholar?q=Yingxu+Wang%2C+Siwei+Liu%2C+Jinyuan+Fang%2C+and+Zaiqiao+Meng%2E+2025b%2E+Evoagentx%3A+An+automated+framework+for+evolving+agentic+workflows%2E+Preprint%2C+arXiv%3A2507%2E03616%2E)
79. Shira Wein, Christopher Homan, Lora Aroyo, and Chris Welty. 2023. Follow the leader(board) with confidence: Estimating p-values from a single test set with item and response variance. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3138–3161, Toronto, Canada. Associatio… [find](https://scholar.google.com/scholar?q=Shira+Wein%2C+Christopher+Homan%2C+Lora+Aroyo%2C+and+Chris+Welty%2E+2023%2E+Follow+the+leader%28board%29+with+confidence%3A+Estimating+p%2Dvalues+from+a+single+test+set+with+item+and+response+variance%2E+In+Findings+of+t)
80. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. [Large language models as optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/). Preprint, arXiv:2309.03409.
81. Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4393–4479, Abu Dhabi, United Arab Emirates. Association for Computatio… [find](https://scholar.google.com/scholar?q=Kevin+Yang%2C+Yuandong+Tian%2C+Nanyun+Peng%2C+and+Dan+Klein%2E+2022%2E+Re3%3A+Generating+longer+stories+with+recursive+reprompting+and+revision%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natur)
82. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. [Textgrad: Automatic" differentiation" via text](https://md.research.elamur.ai/free/en/papers/yuksekgonul-2024-textgrad-automatic-differentiation-text/). arXiv preprint arXiv:2406.07496.
83. Haoke Zhang, Xiaobo Liang, Cunxiang Wang, Juntao Li, and Min Zhang. 2025. Unlocking recursive thinking of llms: Alignment via refinement. Preprint, arXiv:2506.06009. [find](https://scholar.google.com/scholar?q=Haoke+Zhang%2C+Xiaobo+Liang%2C+Cunxiang+Wang%2C+Juntao+Li%2C+and+Min+Zhang%2E+2025%2E+Unlocking+recursive+thinking+of+llms%3A+Alignment+via+refinement%2E+Preprint%2C+arXiv%3A2506%2E06009%2E)
84. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). Advances in neural information processing systems, 28.
85. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Advances in Neural Information Processing Systems, volume 3…
86. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. [Large language models are human-level prompt engineers](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/). arXiv preprint arXiv:2211.01910. i.e., the data is inconsistent with the null hypothesis at least 95% of the time. While the power of…
87. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-Tuning Language Models from Human Preferences. [find](https://scholar.google.com/scholar?q=Daniel+M%2E+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B%2E+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+2019%2E+Fine%2DTuning+Language+Models+from+Human+Preferences%2E)

*Updated 1 November 2026*