---
title: "Label Effects — paper by Sun et al. (2026) — summary and ideas"
description: "The paper asks whether LLMs used as automatic evaluators (LLM-as-a-Judge) are swayed by… 54 references, cited by 1 paper here."
authors: ["Xin Sun", "Di Wu", "Sijing Qin", "Isao Echizen", "Abdallah El Ali", "Saku Sugawara"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1495/"
doi: "10.18653/v1/2026.acl-long.1495"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sun-2026-label-effects-shared-heuristic-reliance/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sun-2026-label-effects-shared-heuristic-reliance/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/) / Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

# Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

**Authors:** [Xin Sun](https://md.research.elamur.ai/free/en/authors/xin-sun/), [Di Wu](https://md.research.elamur.ai/free/en/authors/di-wu/), Sijing Qin, Isao Echizen, Abdallah El Ali and [Saku Sugawara](https://md.research.elamur.ai/free/en/authors/saku-sugawara/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1495/) · [PDF](https://aclanthology.org/2026.acl-long.1495.pdf) · [DOI 10.18653/v1/2026.acl-long.1495](https://doi.org/10.18653/v1/2026.acl-long.1495) · [OpenAlex](https://openalex.org/W7166846628)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/sun-2026-label-effects-shared-heuristic-reliance/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether LLMs used as automatic evaluators (LLM-as-a-Judge) are swayed by disclosed authorship, and whether this resembles how people judge health information. The authors took 150 health question-answer pairs from Yadav et al. (2022). They paired expert-written answers with GPT-4o answers matched in length and format, and crossed true source with a displayed label ("Human Authored" vs. "AI-Generated") in a 2 × 2 counterfactual design. Both lab participants, tracked with an eye tracker, and LLM judges rated trust on a 5-point scale adapted from a validated online-health-trust questionnaire. Both groups trusted identical text more when it was labelled human-written. On the model side, attention concentrated more on the label than on the content, especially under the Human label. This mirrored human gaze. Logit-based uncertainty was higher under AI labels. The authors conclude that provenance labels act as a shared heuristic shortcut. This undermines LLM-as-a-Judge validity and suggests preference alignment may pass human biases to models. They recommend blind or identity-controlled judging.

## Key points
- Data: 150 health QA pairs. Expert answers come from Yadav et al. (2022), and comparable answers were generated with GPT-4o and checked by domain experts.
- Design: source (human vs. LLM) × label (Human vs. AI), with label swaps that hold content constant. A placebo label condition tests whether the effect comes from label meaning.
- Human study: eye tracking on predefined AoIs (Label, Content, Rating). Participants looked at the label area more under Human labels and gave Human-labelled text higher trust.
- LLM internals: models attend more densely to the label region than to the content region, more so under Human labels. Entropy from logits shows greater decision uncertainty under AI labels.
- Limitations include health-only scope, only two label types, and attention/logit signals that are correlational rather than causal. Materials are released at github.com/XIN-von-SUN/Label-Effects.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,649 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [💭 Social Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/social-psychology/) · 330 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 206 works · [🎯 General Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/general-decision-sciences/) · 4 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Xin Sun, Di Wu, Sijing Qin, Isao Echizen, Abdallah El Ali and Saku Sugawara. 2026. Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sun2026label,
  title = {{Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge}},
  author = {Xin Sun, Di Wu, Sijing Qin, Isao Echizen, Abdallah {El Ali} and Saku Sugawara},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1495},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1495}
}
```

- [← previous KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs](https://md.research.elamur.ai/free/en/papers/tang-2026-kv-embedding-training-free-text/)
- [next → Label-Consistent Data Generation for Aspect-Based Sentiment Analysis Using LLM Agents](https://md.research.elamur.ai/free/en/papers/monfared-2026-label-consistent-data-generation-aspect/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/)

## References

54 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tobii AB. 2024. Tobii pro lab. Computer software. [find](https://scholar.google.com/scholar?q=Tobii+AB%2E+2024%2E+Tobii+pro+lab%2E+Computer+software%2E)
2. Benjamin R Bates, Sharon Romina, Rukhsana Ahmed, and Danielle Hopson. 2006. The effect of source credibility on consumers’ perceptions of the quality of health information on the internet. Medical informatics and the Internet in medicine, 31(1):45–52. [find](https://scholar.google.com/scholar?q=Benjamin+R+Bates%2C+Sharon+Romina%2C+Rukhsana+Ahmed%2C+and+Danielle+Hopson%2E+2006%2E+The+effect+of+source+credibility+on+consumers%E2%80%99+perceptions+of+the+quality+of+health+information+on+the+internet%2E+Medical+inf)
3. Oliver Brady, Paul Nulty, Lili Zhang, Tomás E Ward, and David P McGovern. 2025. Dual-process theory and decision-making in large language models. Nat. Rev. Psychol., 4(12):777–792. [find](https://scholar.google.com/scholar?q=Oliver+Brady%2C+Paul+Nulty%2C+Lili+Zhang%2C+Tom%C3%A1s+E+Ward%2C+and+David+P+McGovern%2E+2025%2E+Dual%2Dprocess+theory+and+decision%2Dmaking+in+large+language+models%2E+Nat%2E+Rev%2E+Psychol%2E%2C+4%2812%29%3A777%E2%80%93792%2E)
4. John T. Cacioppo, Louis G. Tassinary, and Gary G. Berntson. 2016. Strong Inference in Psychophysi-ological Science, page 3–15. Cambridge Handbooks in Psychology. Cambridge University Press. [find](https://scholar.google.com/scholar?q=John+T%2E+Cacioppo%2C+Louis+G%2E+Tassinary%2C+and+Gary+G%2E+Berntson%2E+2016%2E+Strong+Inference+in+Psychophysi%2Dological+Science%2C+page+3%E2%80%9315%2E+Cambridge+Handbooks+in+Psychology%2E+Cambridge+University+Press%2E)
5. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. [Humans or LLMs as the judge? a study on judgement bias](https://md.research.elamur.ai/free/en/papers/chen-2024-humans-or-llms-as-judge/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327, Miami, Florida, USA. Association for Computational Lingu…
6. Vanessa Cheung, Maximilian Maier, and Falk Lieder. 2025. Large language models show amplified cognitive biases in moral decision-making. Proceedings of the National Academy of Sciences, 122(25):e2412015122. [find](https://scholar.google.com/scholar?q=Vanessa+Cheung%2C+Maximilian+Maier%2C+and+Falk+Lieder%2E+2025%2E+Large+language+models+show+amplified+cognitive+biases+in+moral+decision%2Dmaking%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+122%2825%29%3Ae24120)
7. Sunhao Dai, Yuqi Zhou, Liang Pang, Weihao Liu, Xiaolin Hu, Yong Liu, Xiao Zhang, Gang Wang, and Jun Xu. 2024. Neural retrievers are biased towards llm-generated content. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page 526–537, New York, NY, USA.… [find](https://scholar.google.com/scholar?q=Sunhao+Dai%2C+Yuqi+Zhou%2C+Liang+Pang%2C+Weihao+Liu%2C+Xiaolin+Hu%2C+Yong+Liu%2C+Xiao+Zhang%2C+Gang+Wang%2C+and+Jun+Xu%2E+2024%2E+Neural+retrievers+are+biased+towards+llm%2Dgenerated+content%2E+In+Proceedings+of+the+30th+ACM)
8. Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner. 2024. Questioning the survey responses of large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. [find](https://scholar.google.com/scholar?q=Ricardo+Dominguez%2DOlmedo%2C+Moritz+Hardt%2C+and+Celestine+Mendler%2DD%C3%BCnner%2E+2024%2E+Questioning+the+survey+responses+of+large+language+models%2E+In+Proceedings+of+the+38th+International+Conference+on+Neural+Inf)
9. Jessica Maria Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He. 2024. Cognitive bias in decision-making with LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12640–12653, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jessica+Maria+Echterhoff%2C+Yao+Liu%2C+Abeer+Alessa%2C+Julian+McAuley%2C+and+Zexue+He%2E+2024%2E+Cognitive+bias+in+decision%2Dmaking+with+LLMs%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+20)
10. Abdallah El Ali, Karthikeya Puttur Venkatraj, Sophie Morosoli, Laurens Naudts, Natali Helberger, and Pablo Cesar. 2024. Transparent ai disclosure obligations: Who, what, when, where, why, how. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA ’24, New York, NY… [find](https://scholar.google.com/scholar?q=Abdallah+El+Ali%2C+Karthikeya+Puttur+Venkatraj%2C+Sophie+Morosoli%2C+Laurens+Naudts%2C+Natali+Helberger%2C+and+Pablo+Cesar%2E+2024%2E+Transparent+ai+disclosure+obligations%3A+Who%2C+what%2C+when%2C+where%2C+why%2C+how%2E+In+Exte)
11. Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. 2007. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods, 39(2):175–191. [find](https://scholar.google.com/scholar?q=Franz+Faul%2C+Edgar+Erdfelder%2C+Albert%2DGeorg+Lang%2C+and+Axel+Buchner%2E+2007%2E+G%2APower+3%3A+a+flexible+statistical+power+analysis+program+for+the+social%2C+behavioral%2C+and+biomedical+sciences%2E+Behav%2E+Res%2E+Method)
12. Bertram Gawronski, Dillon M Luke, and Laura A Creighton. 2024. Dual-process theories. In The Oxford Handbook of Social Cognition, Second Edition, pages 319–353. Oxford University Press. [find](https://scholar.google.com/scholar?q=Bertram+Gawronski%2C+Dillon+M+Luke%2C+and+Laura+A+Creighton%2E+2024%2E+Dual%2Dprocess+theories%2E+In+The+Oxford+Handbook+of+Social+Cognition%2C+Second+Edition%2C+pages+319%E2%80%93353%2E+Oxford+University+Press%2E)
13. Ellen R Girden. 1992. ANOVA: Repeated measures. 84. Sage. [find](https://scholar.google.com/scholar?q=Ellen+R+Girden%2E+1992%2E+ANOVA%3A+Repeated+measures%2E+84%2E+Sage%2E)
14. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. [find](https://scholar.google.com/scholar?q=Jiawei+Gu%2C+Xuhui+Jiang%2C+Zhichao+Shi%2C+Hexiang+Tan%2C+Xuehao+Zhai%2C+Chengjin+Xu%2C+Wei+Li%2C+Yinghan+Shen%2C+Shengjie+Ma%2C+Honghao+Liu%2C+Saizhuo+Wang%2C+Kun+Zhang%2C+Yuanzhuo+Wang%2C+Wen+Gao%2C+Lionel+Ni%2C+and+Jian+Guo%2E+20)
15. Rajarshi Haldar and Julia Hockenmaier. 2025. Rating roulette: Self-inconsistency in LLM-as-a-judge frameworks. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 24986– 25004, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rajarshi+Haldar+and+Julia+Hockenmaier%2E+2025%2E+Rating+roulette%3A+Self%2Dinconsistency+in+LLM%2Das%2Da%2Djudge+frameworks%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2025%2C+pages+24986%E2%80%93+25)
16. James W Hardin and Joseph M Hilbe. 2012. Generalized estimating equations, second edition, 2 edition. Chapman & Hall/CRC, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=James+W+Hardin+and+Joseph+M+Hilbe%2E+2012%2E+Generalized+estimating+equations%2C+second+edition%2C+2+edition%2E+Chapman+%26+Hall%2FCRC%2C+Philadelphia%2C+PA%2E)
17. Winston Haynes. 2013. Benjamini–Hochberg Method, pages 78–78. Springer New York, New York, NY. [find](https://scholar.google.com/scholar?q=Winston+Haynes%2E+2013%2E+Benjamini%E2%80%93Hochberg+Method%2C+pages+78%E2%80%9378%2E+Springer+New+York%2C+New+York%2C+NY%2E)
18. Maurice Jakesch, Megan French, Xiao Ma, Jeffrey T. Hancock, and Mor Naaman. 2019. Ai-mediated communication: How the perception that profile text was written by ai affects trustworthiness. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, page 1–13, New York,… [find](https://scholar.google.com/scholar?q=Maurice+Jakesch%2C+Megan+French%2C+Xiao+Ma%2C+Jeffrey+T%2E+Hancock%2C+and+Mor+Naaman%2E+2019%2E+Ai%2Dmediated+communication%3A+How+the+perception+that+profile+text+was+written+by+ai+affects+trustworthiness%2E+In+Proceedi)
19. Ruili Jiang, Kehai Chen, Xuefeng Bai, Zhixuan He, Juntao Li, Muyun Yang, Tiejun Zhao, Liqiang Nie, and Min Zhang. 2024. A survey on human preference learning for large language models. arXiv preprint arXiv:2406.11191. — [arXiv:2406.11191](https://arxiv.org/abs/2406.11191) [find](https://scholar.google.com/scholar?q=Ruili+Jiang%2C+Kehai+Chen%2C+Xuefeng+Bai%2C+Zhixuan+He%2C+Juntao+Li%2C+Muyun+Yang%2C+Tiejun+Zhao%2C+Liqiang+Nie%2C+and+Min+Zhang%2E+2024%2E+A+survey+on+human+preference+learning+for+large+language+models%2E+arXiv+preprint+)
20. Frances C. Johnson, Jennifer E. Rowley, and Laura Sbaffi. 2015. Modelling trust formation in health information contexts. Journal of Information Science, 41:415 – 429. [find](https://scholar.google.com/scholar?q=Frances+C%2E+Johnson%2C+Jennifer+E%2E+Rowley%2C+and+Laura+Sbaffi%2E+2015%2E+Modelling+trust+formation+in+health+information+contexts%2E+Journal+of+Information+Science%2C+41%3A415+%E2%80%93+429%2E)
21. Marcel A Just and Patricia A Carpenter. 1980. A theory of reading: From eye fixations to comprehension. Psychol. Rev., 87(4):329–354. [find](https://scholar.google.com/scholar?q=Marcel+A+Just+and+Patricia+A+Carpenter%2E+1980%2E+A+theory+of+reading%3A+From+eye+fixations+to+comprehension%2E+Psychol%2E+Rev%2E%2C+87%284%29%3A329%E2%80%93354%2E)
22. Tatsuki Kuribayashi, Yohei Oseki, Souhaib Ben Taieb, Kentaro Inui, and Timothy Baldwin. 2025. [Large language models are human-like internally](https://md.research.elamur.ai/free/en/papers/kuribayashi-2025-large-language-models-are-human/). Transactions of the Association for Computational Linguistics, 13:1743–1766.
23. Walter Laurito, Benjamin Davis, Peli Grietzer, Tomáš Gavenčiak, Ada Böhm, and Jan Kulveit. 2025. Ai–ai bias: Large language models favor communications generated by large language models. Proceedings of the National Academy of Sciences, 122(31):e2415697122. [find](https://scholar.google.com/scholar?q=Walter+Laurito%2C+Benjamin+Davis%2C+Peli+Grietzer%2C+Tom%C3%A1%C5%A1+Gavenc%CC%8Ciak%2C+Ada+B%C3%B6hm%2C+and+Jan+Kulveit%2E+2025%2E+Ai%E2%80%93ai+bias%3A+Large+language+models+favor+communications+generated+by+large+language+models%2E+Proceedings)
24. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2025a. From generation to judgment: Opportunities and challenges of LLM-as-a-judge. In Proceedings of the 2025 Conference… [find](https://scholar.google.com/scholar?q=Dawei+Li%2C+Bohan+Jiang%2C+Liangjie+Huang%2C+Alimohammad+Beigi%2C+Chengshuai+Zhao%2C+Zhen+Tan%2C+Amrita+Bhattacharjee%2C+Yuxuan+Jiang%2C+Canyu+Chen%2C+Tianhao+Wu%2C+Kai+Shu%2C+Lu+Cheng%2C+and+Huan+Liu%2E+2025a%2E+From+generation)
25. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. [Llms-as-judges: a comprehensive survey on llm-based evaluation methods](https://md.research.elamur.ai/free/en/papers/li-2024-llms-judges-comprehensive-survey-llm/). arXiv preprint arXiv:2412.05579.
26. Qingquan Li, Shaoyu Dou, Kailai Shao, Chao Chen, and Haixiang Hu. 2025b. Evaluating scoring bias in llm-as-a-judge. Preprint, arXiv:2506.22316. — [arXiv:2506.22316](https://arxiv.org/abs/2506.22316) [find](https://scholar.google.com/scholar?q=Qingquan+Li%2C+Shaoyu+Dou%2C+Kailai+Shao%2C+Chao+Chen%2C+and+Haixiang+Hu%2E+2025b%2E+Evaluating+scoring+bias+in+llm%2Das%2Da%2Djudge%2E+Preprint%2C+arXiv%3A2506%2E22316%2E+%E2%80%94+%5BarXiv%3A2506%2E22316%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2506%2E22316%29)
27. Songze Li, Chuokun Xu, Jiaying Wang, Xueluan Gong, Chen Chen, Jirui Zhang, Jun Wang, Kwok-Yan Lam, and Shouling Ji. 2025c. Llms cannot reliably judge (yet?): A comprehensive assessment on the robustness of llm-as-a-judge. Preprint, arXiv:2506.09443. — [arXiv:2506.09443](https://arxiv.org/abs/2506.09443) [find](https://scholar.google.com/scholar?q=Songze+Li%2C+Chuokun+Xu%2C+Jiaying+Wang%2C+Xueluan+Gong%2C+Chen+Chen%2C+Jirui+Zhang%2C+Jun+Wang%2C+Kwok%2DYan+Lam%2C+and+Shouling+Ji%2E+2025c%2E+Llms+cannot+reliably+judge+%28yet%3F%29%3A+A+comprehensive+assessment+on+the+robustne)
28. Q.Vera Liao and S. Shyam Sundar. 2022. Designing for responsible trust in ai systems: A communication perspective. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 1257–1268, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Q%2EVera+Liao+and+S%2E+Shyam+Sundar%2E+2022%2E+Designing+for+responsible+trust+in+ai+systems%3A+A+communication+perspective%2E+In+Proceedings+of+the+2022+ACM+Conference+on+Fairness%2C+Accountability%2C+and+Transparen)
29. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-eval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). Preprint, arXiv:2303.16634.
30. Joao Marecos, Duarte Tude Graça, Francisco Goiana-da Silva, Hutan Ashrafian, and Ara Darzi. 2024. Source credibility labels and other nudging interventions in the context of online health misinformation: A systematic literature review. Journalism and Media, 5(2):702–717. [find](https://scholar.google.com/scholar?q=Joao+Marecos%2C+Duarte+Tude+Gra%C3%A7a%2C+Francisco+Goiana%2Dda+Silva%2C+Hutan+Ashrafian%2C+and+Ara+Darzi%2E+2024%2E+Source+credibility+labels+and+other+nudging+interventions+in+the+context+of+online+health+misinformati)
31. Arash Marioriyad, Mohammad Hossein Rohban, and Mahdieh Soleymani Baghshah. 2025. The silent judge: Unacknowledged shortcut bias in llm-as-a-judge. Preprint, arXiv:2509.26072. — [arXiv:2509.26072](https://arxiv.org/abs/2509.26072) [find](https://scholar.google.com/scholar?q=Arash+Marioriyad%2C+Mohammad+Hossein+Rohban%2C+and+Mahdieh+Soleymani+Baghshah%2E+2025%2E+The+silent+judge%3A+Unacknowledged+shortcut+bias+in+llm%2Das%2Da%2Djudge%2E+Preprint%2C+arXiv%3A2509%2E26072%2E+%E2%80%94+%5BarXiv%3A2509%2E26072%5D%28http)
32. OpenAI. 2024. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774.
33. Preprint, Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:2773…
34. Jonathan Peirce, Jeremy R Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hiroyuki Sogo, Erik Kastman, and Jonas Kristoffer Lindeløv. 2019. PsychoPy2: Experiments in behavior made easy. Behavior Research Methods, 51(1):195–203. [find](https://scholar.google.com/scholar?q=Jonathan+Peirce%2C+Jeremy+R+Gray%2C+Sol+Simpson%2C+Michael+MacAskill%2C+Richard+H%C3%B6chenberger%2C+Hiroyuki+Sogo%2C+Erik+Kastman%2C+and+Jonas+Kristoffer+Lindel%C3%B8v%2E+2019%2E+PsychoPy2%3A+Experiments+in+behavior+made+easy%2E+Be)
35. Moritz Reis, Florian Reis, and Wilfried Kunde. 2024. Influence of believed AI involvement on the perception of digital medical advice. Nature Medicine. [find](https://scholar.google.com/scholar?q=Moritz+Reis%2C+Florian+Reis%2C+and+Wilfried+Kunde%2E+2024%2E+Influence+of+believed+AI+involvement+on+the+perception+of+digital+medical+advice%2E+Nature+Medicine%2E)
36. Bernard Rosner, Robert J Glynn, and Mei-Ling T Lee. 2006. The wilcoxon signed rank test for paired comparisons of clustered data. Biometrics, 62(1):185– 192. [find](https://scholar.google.com/scholar?q=Bernard+Rosner%2C+Robert+J+Glynn%2C+and+Mei%2DLing+T+Lee%2E+2006%2E+The+wilcoxon+signed+rank+test+for+paired+comparisons+of+clustered+data%2E+Biometrics%2C+62%281%29%3A185%E2%80%93+192%2E)
37. Jennifer E. Rowley, Frances C. Johnson, and Laura Sbaffi. 2015. Students’ trust judgements in online health information seeking. Health Informatics Journal, 21:316 – 327. [find](https://scholar.google.com/scholar?q=Jennifer+E%2E+Rowley%2C+Frances+C%2E+Johnson%2C+and+Laura+Sbaffi%2E+2015%2E+Students%E2%80%99+trust+judgements+in+online+health+information+seeking%2E+Health+Informatics+Journal%2C+21%3A316+%E2%80%93+327%2E)
38. Nicolas Scharowski, Michaela Benk, Swen J. KÃŒhne, Léane Wettstein, and Florian BrÃŒhlmann. 2023. Certification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study. In 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 248–260, Chicago IL USA. ACM. [find](https://scholar.google.com/scholar?q=Nicolas+Scharowski%2C+Michaela+Benk%2C+Swen+J%2E+K%C3%83%C5%92hne%2C+L%C3%83%C2%A9ane+Wettstein%2C+and+Florian+Br%C3%83%C5%92hlmann%2E+2023%2E+Certification+Labels+for+Trustworthy+AI%3A+Insights+From+an+Empirical+Mixed%2DMethod+Study%2E+In+2023+ACM+C)
39. Kayla Schroeder and Zach Wood-Doughty. 2025. Can you trust llm judgments? reliability of llm-as-a-judge. Preprint, arXiv:2412.12509. — [arXiv:2412.12509](https://arxiv.org/abs/2412.12509) [find](https://scholar.google.com/scholar?q=Kayla+Schroeder+and+Zach+Wood%2DDoughty%2E+2025%2E+Can+you+trust+llm+judgments%3F+reliability+of+llm%2Das%2Da%2Djudge%2E+Preprint%2C+arXiv%3A2412%2E12509%2E+%E2%80%94+%5BarXiv%3A2412%2E12509%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2412%2E12509%29)
40. Huanxin Sheng, Xinyi Liu, Hangfeng He, Jieyu Zhao, and Jian Kang. 2025. Analyzing uncertainty of LLM-as-a-judge: Interval evaluations with conformal prediction. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 11297–11339, Suzhou, China. Association fo… [find](https://scholar.google.com/scholar?q=Huanxin+Sheng%2C+Xinyi+Liu%2C+Hangfeng+He%2C+Jieyu+Zhao%2C+and+Jian+Kang%2E+2025%2E+Analyzing+uncertainty+of+LLM%2Das%2Da%2Djudge%3A+Interval+evaluations+with+conformal+prediction%2E+In+Proceedings+of+the+2025+Conference+o)
41. Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. 2025. [Judging the judges: A systematic study of position bias in llm-as-a-judge](https://md.research.elamur.ai/free/en/papers/shi-2025-judging-judges-systematic-study-position/). Preprint, arXiv:2406.07791.
42. Evangelia Spiliopoulou, Riccardo Fogliato, Hanna Burnsky, Tamer Soliman, Jie Ma, Graham Horwood, and Miguel Ballesteros. 2025. Play favorites: A statistical method to measure self-bias in llm-as-a-judge. Preprint, arXiv:2508.06709. — [arXiv:2508.06709](https://arxiv.org/abs/2508.06709) [find](https://scholar.google.com/scholar?q=Evangelia+Spiliopoulou%2C+Riccardo+Fogliato%2C+Hanna+Burnsky%2C+Tamer+Soliman%2C+Jie+Ma%2C+Graham+Horwood%2C+and+Miguel+Ballesteros%2E+2025%2E+Play+favorites%3A+A+statistical+method+to+measure+self%2Dbias+in+llm%2Das%2Da%2Djud)
43. Xin Sun, Rongjun Ma, Shu Wei, Pablo Cesar, Jos A. Bosch, and Abdallah El Ali. 2025. Understanding trust toward human versus ai-generated health information through behavioral and physiological sensing. International Journal of Human-Computer Studies, page 103714. [find](https://scholar.google.com/scholar?q=Xin+Sun%2C+Rongjun+Ma%2C+Shu+Wei%2C+Pablo+Cesar%2C+Jos+A%2E+Bosch%2C+and+Abdallah+El+Ali%2E+2025%2E+Understanding+trust+toward+human+versus+ai%2Dgenerated+health+information+through+behavioral+and+physiological+sensing)
44. Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. [Do LLMs exhibit human-like response biases? a case study in survey design](https://md.research.elamur.ai/free/en/papers/tjuatja-2024-do-llms-exhibit-human-like/). Transactions of the Association for Computational Linguistics, 12:1011–1026.
45. Qian Wang, Zhanzhi Lou, Zhenheng Tang, Nuo Chen, Xuandong Zhao, Wenxuan Zhang, Dawn Song, and Bingsheng He. 2025a. Assessing judging bias in large reasoning models: An empirical study. Preprint, arXiv:2504.09946. — [arXiv:2504.09946](https://arxiv.org/abs/2504.09946) [find](https://scholar.google.com/scholar?q=Qian+Wang%2C+Zhanzhi+Lou%2C+Zhenheng+Tang%2C+Nuo+Chen%2C+Xuandong+Zhao%2C+Wenxuan+Zhang%2C+Dawn+Song%2C+and+Bingsheng+He%2E+2025a%2E+Assessing+judging+bias+in+large+reasoning+models%3A+An+empirical+study%2E+Preprint%2C+arXiv)
46. Yidong Wang, Yunze Song, Tingyuan Zhu, Xuanwang Zhang, Zhuohao Yu, Hao Chen, Chiyu Song, Qiufeng Wang, Cunxiang Wang, Zhen Wu, Xinyu Dai, Yue Zhang, Wei Ye, and Shikun Zhang. 2025b. Trustjudge: Inconsistencies of llm-as-a-judge and how to alleviate them. Preprint, arXiv:2509.21117. — [arXiv:2509.21117](https://arxiv.org/abs/2509.21117) [find](https://scholar.google.com/scholar?q=Yidong+Wang%2C+Yunze+Song%2C+Tingyuan+Zhu%2C+Xuanwang+Zhang%2C+Zhuohao+Yu%2C+Hao+Chen%2C+Chiyu+Song%2C+Qiufeng+Wang%2C+Cunxiang+Wang%2C+Zhen+Wu%2C+Xinyu+Dai%2C+Yue+Zhang%2C+Wei+Ye%2C+and+Shikun+Zhang%2E+2025b%2E+Trustjudge%3A+Incons)
47. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2025. [Self-preference bias in llm-as-a-judge](https://md.research.elamur.ai/free/en/papers/wataoka-2024-self-preference-bias-llm-judge/). Preprint, arXiv:2410.21819.
48. Sarah Wiegreffe and Yuval Pinter. 2019. [Attention is not not explanation](https://md.research.elamur.ai/free/en/papers/wiegreffe-2019-attention-is-not-not-explanation/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20, Hong Kong, China. Association for…
49. Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Shiyang Lai, Kai Shu, Jindong Gu, Adel Bibi, Ziniu Hu, David Jurgens, James Evans, Philip H.S. Torr, Bernard Ghanem, and Guohao Li. 2024. Can large language model agents simulate human trust behavior? In Proceedings of the 38th International Conference… [find](https://scholar.google.com/scholar?q=Chengxing+Xie%2C+Canyu+Chen%2C+Feiran+Jia%2C+Ziyu+Ye%2C+Shiyang+Lai%2C+Kai+Shu%2C+Jindong+Gu%2C+Adel+Bibi%2C+Ziniu+Hu%2C+David+Jurgens%2C+James+Evans%2C+Philip+H%2ES%2E+Torr%2C+Bernard+Ghanem%2C+and+Guohao+Li%2E+2024%2E+Can+large+lang)
50. Shweta Yadav, Deepak Gupta, and Dina Demner-Fushman. 2022. Chq-summ: A dataset for consumer healthcare question summarization. Preprint, arXiv:2206.06581. — [arXiv:2206.06581](https://arxiv.org/abs/2206.06581) [find](https://scholar.google.com/scholar?q=Shweta+Yadav%2C+Deepak+Gupta%2C+and+Dina+Demner%2DFushman%2E+2022%2E+Chq%2Dsumm%3A+A+dataset+for+consumer+healthcare+question+summarization%2E+Preprint%2C+arXiv%3A2206%2E06581%2E+%E2%80%94+%5BarXiv%3A2206%2E06581%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F22)
51. Haoyan Yang, Runxue Bao, Cao Xiao, Jun Ma, Parminder Bhatia, Shangqian Gao, and Taha Kass-Hout. 2025. Any large language model can be a reliable judge: Debiasing with a reasoning-based bias detector. Preprint, arXiv:2505.17100. — [arXiv:2505.17100](https://arxiv.org/abs/2505.17100) [find](https://scholar.google.com/scholar?q=Haoyan+Yang%2C+Runxue+Bao%2C+Cao+Xiao%2C+Jun+Ma%2C+Parminder+Bhatia%2C+Shangqian+Gao%2C+and+Taha+Kass%2DHout%2E+2025%2E+Any+large+language+model+can+be+a+reliable+judge%3A+Debiasing+with+a+reasoning%2Dbased+bias+detector%2E+)
52. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2024. [Justice or prejudice? quantifying biases in llm-as-a-judge](https://md.research.elamur.ai/free/en/papers/ye-2024-justice-prejudice-quantifying-biases-llm/). Preprint, arXiv:2410.02736.
53. Yidan Yin, Nan Jia, and Cheryl J. Wakslak. 2024. Ai can help people feel heard, but an ai label diminishes this impact. Proceedings of the National Academy of Sciences, 121(14):e2319112121. [find](https://scholar.google.com/scholar?q=Yidan+Yin%2C+Nan+Jia%2C+and+Cheryl+J%2E+Wakslak%2E+2024%2E+Ai+can+help+people+feel+heard%2C+but+an+ai+label+diminishes+this+impact%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+121%2814%29%3Ae2319112121%2E)
54. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Preprint, arXiv:2306.05685.

*Updated 31 October 2026*