---
title: "Aligning with Human Judgement — paper by Liu et al. (2024)"
description: "They propose Pairwise-preference Search (PAIRS), which asks an LLM to compare selected pairs of… 60 references, cited by 66 papers here."
authors: ["Yinhong Liu", "Han Zhou", "Zhijiang Guo", "Ehsan Shareghi", "Ivan Vulic", "Anna Korhonen", "Nigel Collier"]
year: 2024
venue: "arXiv:2403.16950"
source: "https://arxiv.org/abs/2403.16950"
doi: "10.48550/arxiv.2403.16950"
arxiv: "2403.16950"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2024-aligning-human-judgement-role-pairwise/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2024-aligning-human-judgement-role-pairwise/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

# Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

**Authors:** Yinhong Liu, Han Zhou, [Zhijiang Guo](https://md.research.elamur.ai/free/en/authors/zhijiang-guo/), [Ehsan Shareghi](https://md.research.elamur.ai/free/en/authors/ehsan-shareghi/), [Ivan Vulic](https://md.research.elamur.ai/free/en/authors/ivan-vulic/), [Anna Korhonen](https://md.research.elamur.ai/free/en/authors/anna-korhonen/) and [Nigel Collier](https://md.research.elamur.ai/free/en/authors/nigel-collier/) · **Year:** 2024 · **Venue:** arXiv:2403.16950 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.16950) · [PDF](https://arxiv.org/pdf/2403.16950) · [DOI 10.48550/arxiv.2403.16950](https://doi.org/10.48550/arxiv.2403.16950) · [arXiv 2403.16950](https://arxiv.org/abs/2403.16950) · [OpenAlex](https://openalex.org/W4393247764)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [66](https://md.research.elamur.ai/free/en/papers/liu-2024-aligning-human-judgement-role-pairwise/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
LLMs are increasingly used as cheap, reference-free judges of generated text, but their ratings often diverge from human assessments. The authors first test whether score-calibration can fix this. On HANNA and SummEval, they rescale model predictions so the model's score prior matches the human one. The gap remains, which suggests the problem lies in the models' underlying evaluation standards (the likelihood term), not in skewed score priors. Drawing on how RLHF learns from preference data, they recast evaluation as ranking. They propose Pairwise-preference Search (PAIRS), which asks an LLM to compare selected pairs of candidates and uses uncertainty to guide a search toward a maximum-likelihood (Kemeny-optimal) global ranking. This avoids exhaustive quadratic comparisons and the NP-hard exact solution. Across summarization and open-ended generation benchmarks, PAIRS correlates better with human judgements than direct scoring and needs far fewer comparisons than standard rank aggregation. The study also shows that pairwise preferences can measure how transitive an LLM evaluator is. This makes PAIRS a practical, scalable route to more human-aligned automatic evaluation.

## Key points
- Prior calibration does not close the human–LLM gap. MAE fell only from 1.62 to 1.16 on HANNA and actually rose from 0.78 to 0.86 on SummEval.
- PAIRS comes in greedy and beam variants and was tested with Mistral-7B, Llama-2 7B and GPT-3.5-turbo on NewsRoom, SummEval and HANNA. Spearman correlation on the coherence aspect served as the metric.
- Compared with Win-loss rate and ELO rating aggregation, PAIRS-greedy reaches ELO-level correlation using roughly 30% of the comparisons. Calibrated PAIRS generally beats baselines that use all N(N−1) pairs.
- Transitivity was estimated from variance over 10 seeded runs, using 200 sampled HANNA stories. GPT-3.5-turbo was the most self-consistent model. HANNA proved hardest, likely because creative-writing judgements are subjective.
- Debiasing pairwise comparisons by averaging both candidate orderings helps Mistral-7B and Llama-2 7B (e.g., SummEval 25.6 → 32.9 for Mistral). It brings little gain for the over-confident GPT-3.5-turbo. Code: github.com/cambridgeltl/PairS.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 155 works · 115,562 citations
- [Monash University](https://md.research.elamur.ai/free/en/universities/monash-university/) — University, Melbourne, Australia · 31 works · 54,201 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulic, Anna Korhonen and Nigel Collier. 2024. Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators. arXiv:2403.16950.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2024aligning,
  title = {{Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators}},
  author = {Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulic, Anna Korhonen and Nigel Collier},
  year = {2024},
  doi = {10.48550/arxiv.2403.16950},
  eprint = {2403.16950},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.16950}
}
```

- [← previous Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps](https://md.research.elamur.ai/free/en/papers/chuang-2024-lookback-lens-detecting-mitigating-contextual/)
- [next → PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities](https://md.research.elamur.ai/free/en/papers/sravanthi-2024-pub-pragmatics-understanding-benchmark-assessing/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

60 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap,…](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/)
2. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023b.
3. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. [Exploring the use of large language models for reference-free text quality evaluation: An empirical study](https://md.research.elamur.ai/free/en/papers/chen-2023-exploring-use-large-language-models/). In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Findings of the As…
5. Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chloé Clavel. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli,… [find](https://scholar.google.com/scholar?q=Cyril+Chhun%2C+Pierre+Colombo%2C+Fabian+M%2E+Suchanek%2C+and+Chloe%CC%81+Clavel%2E+Of+human+criteria+and+automatic+metrics%3A+A+benchmark+of+the+evaluation+of+story+generation%2E+In+Nicoletta+Calzolari%2C+Chu%2DRen+Huang%2C+H)
6. Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607–15631, Toronto… [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+In+Anna+Rogers%2C+Jordan+Boyd%2DGraber%2C+and+Naoaki+Okazaki+%28eds%2E%29%2C+Proceedings+of+the+61st+Annual+Meetin)
7. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Informat… [find](https://scholar.google.com/scholar?q=Paul+F%2E+Christiano%2C+Jan+Leike%2C+Tom+B%2E+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+Isabelle+Guyon%2C+Ulrike+von+Luxburg%2C+Samy+Bengio%2C+Hanna+)
8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. [Scaling instruction-finetuned language models](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/). arXiv preprint arXiv:2210.11416, 2022.
9. William G. Cochran. Sampling Techniques, 3rd Edition. John Wiley, 1977. ISBN 0-471-16240-X. [find](https://scholar.google.com/scholar?q=William+G%2E+Cochran%2E+Sampling+Techniques%2C+3rd+Edition%2E+John+Wiley%2C+1977%2E+ISBN+0%2D471%2D16240%2DX%2E)
10. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. [Safe RLHF: Safe reinforcement learning from human feedback](https://md.research.elamur.ai/free/en/papers/dai-2023-safe-rlhf-safe-reinforcement-learning/). In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=TyFrPOKYXw](https://openreview.net/forum?id=TyFrPOKYXw). [↗](https://openreview.net/forum?id=TyFrPOKYXw)
11. Cynthia Dwork, Ravi Kumar, Moni Naor, and D Sivakumar. Rank aggregation methods for the web. In Proceedings of the 10th International Conference on World Wide Web, pp. 613–622. ACM, 2001. [find](https://scholar.google.com/scholar?q=Cynthia+Dwork%2C+Ravi+Kumar%2C+Moni+Naor%2C+and+D+Sivakumar%2E+Rank+aggregation+methods+for+the+web%2E+In+Proceedings+of+the+10th+International+Conference+on+World+Wide+Web%2C+pp%2E+613%E2%80%93622%2E+ACM%2C+2001%2E)
12. Cynthia Dwork, Ravi Kumar, Moni Naor, and D. Sivakumar. Rank aggregation revisited. 2002. URL [https://api.semanticscholar.org/CorpusID:13976791](https://api.semanticscholar.org/CorpusID:13976791). [↗](https://api.semanticscholar.org/CorpusID:13976791)
13. Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409, 2021. doi: 10.1162/tacl a 00373. URL https://aclanthology.org/2021.… [↗](https://aclanthology.org/2021.…)
14. Peter C Fishburn. Binary choice probabilities: on the varieties of stochastic transitivity. Journal of Mathematical psychology, 10(4):327–352, 1973. [find](https://scholar.google.com/scholar?q=Peter+C+Fishburn%2E+Binary+choice+probabilities%3A+on+the+varieties+of+stochastic+transitivity%2E+Journal+of+Mathematical+psychology%2C+10%284%29%3A327%E2%80%93352%2C+1973%2E)
15. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+Gptscore%3A+Evaluate+as+you+desire%2E+arXiv+preprint+arXiv%3A2302%2E04166%2C+2023%2E)
16. Max Grusky, Mor Naaman, and Yoav Artzi. [Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/). In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu…
17. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroi… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+Le%CC%81lio+Renard+La)
18. Zhongtao Jiang, Yuanzhe Zhang, Cao Liu, Jun Zhao, and Kang Liu. Generative calibration for in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2312–2333, Singapore, December 2023b. Association for Comput… [find](https://scholar.google.com/scholar?q=Zhongtao+Jiang%2C+Yuanzhe+Zhang%2C+Cao+Liu%2C+Jun+Zhao%2C+and+Kang+Liu%2E+Generative+calibration+for+in%2Dcontext+learning%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kalika+Bali+%28eds%2E%29%2C+Findings+of+the+Association+for+Com)
19. John G Kemeny. Mathematics without numbers. Daedalus, 88(4):577–591, 1959. [find](https://scholar.google.com/scholar?q=John+G+Kemeny%2E+Mathematics+without+numbers%2E+Daedalus%2C+88%284%29%3A577%E2%80%93591%2C+1959%2E)
20. Tom Kocmi and Christian Federmann. Large language models are state-of-the-art evaluators of translation quality. In Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara N… [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+In+Mary+Nurminen%2C+Judith+Brenner%2C+Maarit+Koponen%2C+Sirkku+Latomaa%2C+Mikhail+Mikhailov%2C+Fr)
21. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural In…
22. Robert B Latta. Composition rules for probabilities from paired comparisons. The Annals of Statistics, pp. 349–371, 1979. [find](https://scholar.google.com/scholar?q=Robert+B+Latta%2E+Composition+rules+for+probabilities+from+paired+comparisons%2E+The+Annals+of+Statistics%2C+pp%2E+349%E2%80%93371%2C+1979%2E)
23. Chengzu Li, Han Zhou, Goran Glavaš, Anna Korhonen, and Ivan Vulić. On task performance and model calibration with supervised and self-ensembled in-context learning. arXiv preprint arXiv:2312.13772, 2023a. — [arXiv:2312.13772](https://arxiv.org/abs/2312.13772) [find](https://scholar.google.com/scholar?q=Chengzu+Li%2C+Han+Zhou%2C+Goran+Glavas%CC%8C%2C+Anna+Korhonen%2C+and+Ivan+Vulic%CC%81%2E+On+task+performance+and+model+calibration+with+supervised+and+self%2Densembled+in%2Dcontext+learning%2E+arXiv+preprint+arXiv%3A2312%2E13772%2C+)
24. Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, and Chongyang Tao. Leveraging large language models for nlg evaluation: A survey. arXiv preprint arXiv:2401.07103, 2024. — [arXiv:2401.07103](https://arxiv.org/abs/2401.07103) [find](https://scholar.google.com/scholar?q=Zhen+Li%2C+Xiaohan+Xu%2C+Tao+Shen%2C+Can+Xu%2C+Jia%2DChen+Gu%2C+and+Chongyang+Tao%2E+Leveraging+large+language+models+for+nlg+evaluation%3A+A+survey%2E+arXiv+preprint+arXiv%3A2401%2E07103%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E07103%5D%28https%3A)
25. Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in large language model based evaluators. arXiv preprint arXiv:2310.01432, 2023b. — [arXiv:2310.01432](https://arxiv.org/abs/2310.01432) [find](https://scholar.google.com/scholar?q=Zongjie+Li%2C+Chaozheng+Wang%2C+Pingchuan+Ma%2C+Daoyuan+Wu%2C+Shuai+Wang%2C+Cuiyun+Gao%2C+and+Yang+Liu%2E+Split+and+merge%3A+Aligning+position+biases+in+large+language+model+based+evaluators%2E+arXiv+preprint+arXiv%3A231)
26. Rensis Likert. A technique for the measurement of attitudes. Archives of psychology, 1932. [find](https://scholar.google.com/scholar?q=Rensis+Likert%2E+A+technique+for+the+measurement+of+attitudes%2E+Archives+of+psychology%2C+1932%2E)
27. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Geval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511–2522, Sin… [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+Geval%3A+NLG+evaluation+using+gpt%2D4+with+better+human+alignment%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kalika+Bali+%28eds%2E%29%2C+Procee)
28. Yinhong Liu, Yimai Fang, David Vandyke, and Nigel Collier. Toad: Task-oriented automatic dialogs with diverse response styles. arXiv preprint arXiv:2402.10137, 2024a. [find](https://scholar.google.com/scholar?q=Yinhong+Liu%2C+Yimai+Fang%2C+David+Vandyke%2C+and+Nigel+Collier%2E+Toad%3A+Task%2Doriented+automatic+dialogs+with+diverse+response+styles%2E+arXiv+preprint+arXiv%3A2402%2E10137%2C+2024a%2E)
29. Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vulić, and Nigel Collier. Aligning with logic: Measuring, evaluating and improving logical consistency in large language models. arXiv preprint arXiv:2410.02205, 2024b. — [arXiv:2410.02205](https://arxiv.org/abs/2410.02205) [find](https://scholar.google.com/scholar?q=Yinhong+Liu%2C+Zhijiang+Guo%2C+Tianya+Liang%2C+Ehsan+Shareghi%2C+Ivan+Vulic%CC%81%2C+and+Nigel+Collier%2E+Aligning+with+logic%3A+Measuring%2C+evaluating+and+improving+logical+consistency+in+large+language+models%2E+arXiv+pr)
30. Yinhong Liu, Yixuan Su, Ehsan Shareghi, and Nigel Collier. Unlocking structure measuring: Introducing PDD, an automatic metric for positional discourse coherence. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Associatio… [find](https://scholar.google.com/scholar?q=Yinhong+Liu%2C+Yixuan+Su%2C+Ehsan+Shareghi%2C+and+Nigel+Collier%2E+Unlocking+structure+measuring%3A+Introducing+PDD%2C+an+automatic+metric+for+positional+discourse+coherence%2E+In+Kevin+Duh%2C+Helena+Gomez%2C+and+Steve)
31. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. [Calibrating llm-based evaluator](https://md.research.elamur.ai/free/en/papers/liu-2024-calibrating-llm-based-evaluator/). arXiv preprint arXiv:2309.13308, 2023b.
32. Adian Liusie, Potsawee Manakul, and Mark Gales. [LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models](https://md.research.elamur.ai/free/en/papers/liusie-2024-llm-comparative-assessment-zero-shot/). In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational L…
33. IFD Oliveira, S Zehavi, and O Davidov. Stochastic transitivity: Axioms and models. Journal of Mathematical Psychology, 85:25–35, 2018. [find](https://scholar.google.com/scholar?q=IFD+Oliveira%2C+S+Zehavi%2C+and+O+Davidov%2E+Stochastic+transitivity%3A+Axioms+and+models%2E+Journal+of+Mathematical+Psychology%2C+85%3A25%E2%80%9335%2C+2018%2E)
34. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
35. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
36. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318, Philadelphia,…
37. Pouya Pezeshkpour and Estevam Hruschka. [Large language models sensitivity to the order of options in multiple-choice questions](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-large-language-models-sensitivity-order/). arXiv preprint arXiv:2308.11483, 2023.
38. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, et al. [Large language models are effective text rankers with pairwise ranking prompting](https://md.research.elamur.ai/free/en/papers/qin-2024-large-language-models-are-effective/). arXiv preprint arXiv:2306.17563, 2023.
39. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076, 2023. — [arXiv:2310.10076](https://arxiv.org/abs/2310.10076) [find](https://scholar.google.com/scholar?q=Keita+Saito%2C+Akifumi+Wachi%2C+Koki+Wataoka%2C+and+Youhei+Akimoto%2E+Verbosity+bias+in+preference+labeling+by+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E10076%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E10076%5D%28https%3A%2F%2Farxiv%2E)
40. Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. [Large language models are not yet human-level evaluators for abstractive summarization](https://md.research.elamur.ai/free/en/papers/shen-2023-large-language-models-are-not/). In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 4215–4233, S…
41. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Infor)
42. Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and Linqi Song. Proxyqa: An alternative framework for evaluating long-form text generation with large language models. arXiv preprint arXiv:2401.15042, 2024. [find](https://scholar.google.com/scholar?q=Haochen+Tan%2C+Zhijiang+Guo%2C+Zhan+Shi%2C+Lu+Xu%2C+Zhili+Liu%2C+Xiaoguang+Li%2C+Yasheng+Wang%2C+Lifeng+Shang%2C+Qun+Liu%2C+and+Linqi+Song%2E+Proxyqa%3A+An+alternative+framework+for+evaluating+long%2Dform+text+generation+wit)
43. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
44. Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Arik, and Tomas Pfister. Better zero-shot reasoning with self-adaptive prompting. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 3493–3514, Toronto, Canada, July 2023a… [find](https://scholar.google.com/scholar?q=Xingchen+Wan%2C+Ruoxi+Sun%2C+Hanjun+Dai%2C+Sercan+Arik%2C+and+Tomas+Pfister%2E+Better+zero%2Dshot+reasoning+with+self%2Dadaptive+prompting%2E+In+Anna+Rogers%2C+Jordan+Boyd%2DGraber%2C+and+Naoaki+Okazaki+%28eds%2E%29%2C+Findings+of)
45. Xingchen Wan, Ruoxi Sun, Hootan Nakhost, Hanjun Dai, Julian Eisenschlos, Sercan Arik, and Tomas Pfister. Universal self-adaptive prompting. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7437–7462, Sing… [find](https://scholar.google.com/scholar?q=Xingchen+Wan%2C+Ruoxi+Sun%2C+Hootan+Nakhost%2C+Hanjun+Dai%2C+Julian+Eisenschlos%2C+Sercan+Arik%2C+and+Tomas+Pfister%2E+Universal+self%2Dadaptive+prompting%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kalika+Bali+%28eds%2E%29%2C+Proceed)
46. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. [Is ChatGPT a good NLG evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). In Yue Dong, Wen Xiao, Lu Wang, Fei Liu, and Giuseppe Carenini (eds.), Proceedings of the 4th New Frontiers in Summarization Worksh…
47. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023b. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2C+2023b%2E)
48. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=gEZrGCozdqR](https://openreview.net/forum?id=gEZrGCozdqR). [↗](https://openreview.net/forum?id=gEZrGCozdqR)
49. Weizhe Yuan, Graham Neubig, and Pengfei Liu. [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). Advances in Neural Information Processing Systems, 34:27263–27277, 2021.
50. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) tr0KidwPLc. [↗](https://openreview.net/forum?id=)
51. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=SkeHuCVFDr](https://openreview.net/forum?id=SkeHuCVFDr). [↗](https://openreview.net/forum?id=SkeHuCVFDr)
52. Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. [Wider and deeper llm networks are fairer llm evaluators](https://md.research.elamur.ai/free/en/papers/zhang-2023-wider-deeper-llm-networks-fairer/). arXiv preprint arXiv:2308.01862, 2023.
53. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pp. 12697–12706, 2021.
54. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+Large+language+models+are+not+robust+multiple+choice+selectors%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2)
55. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+Advance)
56. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. [Towards a unified multi-dimensional evaluator for text generation](https://md.research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/). In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural…
57. Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. Survival of the most influential prompts: Efficient black-box prompt search via clustering and pruning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13064–13077,… [find](https://scholar.google.com/scholar?q=Han+Zhou%2C+Xingchen+Wan%2C+Ivan+Vulic%CC%81%2C+and+Anna+Korhonen%2E+Survival+of+the+most+influential+prompts%3A+Efficient+black%2Dbox+prompt+search+via+clustering+and+pruning%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kalika+)
58. Han Zhou, Xingchen Wan, Yinhong Liu, Nigel Collier, Ivan Vulić, and Anna Korhonen. Fairer preferences elicit improved human-aligned large language model judgments. arXiv preprint arXiv:2406.11370, 2024a. [find](https://scholar.google.com/scholar?q=Han+Zhou%2C+Xingchen+Wan%2C+Yinhong+Liu%2C+Nigel+Collier%2C+Ivan+Vulic%CC%81%2C+and+Anna+Korhonen%2E+Fairer+preferences+elicit+improved+human%2Daligned+large+language+model+judgments%2E+arXiv+preprint+arXiv%3A2406%2E11370%2C+20)
59. Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine A Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In The Twelfth International Conference on Learning Representations, 2024b. URL [https://openreview.net/forum?…](https://openreview.net/forum?…) [↗](https://openreview.net/forum?…)
60. Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. Autopeft: Automatic configuration search for parameter-efficient fine-tuning. Transactions of the Association for Computational Linguistics, 12:525–542, 2024c. [find](https://scholar.google.com/scholar?q=Han+Zhou%2C+Xingchen+Wan%2C+Ivan+Vulic%CC%81%2C+and+Anna+Korhonen%2E+Autopeft%3A+Automatic+configuration+search+for+parameter%2Defficient+fine%2Dtuning%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+12%3A)

*Updated 24 October 2026*