---
title: "UNH @ Rag4Reports: A Broad Exploration of LLM-Judges for RAG"
description: "This paper describes how a University of New Hampshire team approached Rag4Reports Task A. The task asks for automatic… 12 references."
authors: ["Minna Tran", "Ryan McCarthy", "Aiden Parsons", "Jaren Unzen", "Laura Dietz"]
year: 2026
venue: "Proceedings of the 1st Workshop on Multilingual Report Generation via Retrieval Augmented Generation (RAG4Reports 2026)"
source: "https://aclanthology.org/2026.rag4reports-1.9/"
doi: "10.18653/v1/2026.rag4reports-1.9"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tran-2026-unh-rag4reports-broad-exploration-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tran-2026-unh-rag4reports-broad-exploration-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/) / UNH @ Rag4Reports: A Broad Exploration of LLM-Judges for RAG

# UNH @ Rag4Reports: A Broad Exploration of LLM-Judges for RAG

**Authors:** Minna Tran, Ryan McCarthy, Aiden Parsons, Jaren Unzen and Laura Dietz · **Year:** 2026 · **Venue:** [Proceedings of the 1st Workshop on Multilingual Report Generation via Retrieval Augmented Generation (RAG4Reports 2026)](https://md.research.elamur.ai/free/en/venues/rag4reports/) ([Workshop on Multilingual Report Generation via Retrieval Augmented Generation 2026](https://md.research.elamur.ai/free/en/venues/rag4reports/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.rag4reports-1.9/) · [PDF](https://aclanthology.org/2026.rag4reports-1.9.pdf) · [DOI 10.18653/v1/2026.rag4reports-1.9](https://doi.org/10.18653/v1/2026.rag4reports-1.9) · [OpenAlex](https://openalex.org/W7166809268)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
This paper describes how a University of New Hampshire team approached Rag4Reports Task A. The task asks for automatic judges ("AutoJudges", or LLM-as-a-Judge) that score RAG report-generation systems and rank them. The ranking is then checked against a human reference ranking using Kendall's τ. Students in a Spring 2026 course each built one judge independently. The RAG-TIME portion of the TREC AutoJudge Pilot Dataset v0.2 served for development, while testing relied on the official Rag4Reports data, which is based on TREC 2025 RAGTIME.

The judges cover a wide range of signals:
- NLI-based citation checking
- claim-level attribution
- nugget coverage of the retrieved documents
- LLM-generated checklists
- query negation
- graded relevance
- naive baselines

The NLI-based Citation Accuracy judge ranked first among all submitted systems and came in just below the organizers' AUTO-ARGUE reference. The authors draw two conclusions. Citation faithfulness is the most important signal. Content quality is best assessed by testing whether cited documents cover nuggets that the LLM generates from its own knowledge. These results offer practical guidance for building cheaper, more auditable evaluation of RAG systems.

## Key points
- **Citation Accuracy** pairs each cited fragment with its first resolvable cited document. It scores the pair with `cross-encoder/nli-deberta-v3-base` and counts the fragment as supported only when entailment is the top class. On the test set it reached Kendall τ = 0.559, versus 0.636 for the organizers' `autoargue-f1`.
- **Retrieval Quality** uses gpt-4.1-nano to break each topic into sub-question nuggets. It then grades the retrieved documents on a 0/1/2 scale for each nugget and scored τ = 0.469. **Final** takes the larger of the citation and attribution scores, averages it with Retrieval Quality, and scored τ = 0.430. Final outperformed Citation Accuracy on the development set.
- The other judges scored lower on the test set:
  - **Attribution** (HHEMv2 with gpt-4.1-nano claim extraction): τ = 0.291
  - **Negation Judge** (gpt-4o-mini): τ = 0.284
  - **Checklist** (five items, gpt-4o-mini): τ = 0.300
  - The naive length and random baselines did worst.
- **Graded Relevance** (DeepSeek v3.2) agreed best with development-set nugget coverage, but it was not finished in time to be submitted.
- The authors list several limitations:
  - Most methods rely on non-deterministic commercial APIs.
  - Each method was built by a different author, so method choices are confounded with other design differences, and no ablations were run.
  - The test set comes from a single set of human assessments.
  - The development reference is noisy because it is derived from AUTO-ARGUE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of New Hampshire](https://md.research.elamur.ai/free/en/universities/university-of-new-hampshire/) — University, Durham, United States · 2 works

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 182 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Minna Tran, Ryan McCarthy, Aiden Parsons, Jaren Unzen and Laura Dietz. 2026. UNH @ Rag4Reports: A Broad Exploration of LLM-Judges for RAG. Proceedings of the 1st Workshop on Multilingual Report Generation via Retrieval Augmented Generation (RAG4Reports 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tran2026unh,
  title = {{UNH @ Rag4Reports: A Broad Exploration of LLM-Judges for RAG}},
  author = {Minna Tran, Ryan McCarthy, Aiden Parsons, Jaren Unzen and Laura Dietz},
  booktitle = {Proceedings of the 1st Workshop on Multilingual Report Generation via Retrieval Augmented Generation (RAG4Reports 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.rag4reports-1.9},
  url = {https://doi.org/10.18653/v1/2026.rag4reports-1.9}
}
```

- [← previous UNED at SemEval-2026 Task 9: Sentiment-Aware Transformer Models with Back-Translation Augmentation for Online polarisation Detection](https://md.research.elamur.ai/free/en/papers/sanabria-2026-uned-semeval-2026-task-9/)
- [next → UniCite: A Dataset and Unified Hierarchical Taxonomy for Multi-Dimensional Citation Analysis](https://md.research.elamur.ai/free/en/papers/mourky-2026-unicite-dataset-unified-hierarchical-taxonomy/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/)

## References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. [find](https://scholar.google.com/scholar?q=Antonia+Creswell%2C+Murray+Shanahan%2C+and+Irina+Higgins%2E+2022%2E+Selection%2Dinference%3A+Exploiting+large+language+models+for+interpretable+logical+reasoning%2E)
2. Kaustubh D. Dhole and Eugene Agichtein. 2026. Rubricrag: Towards interpretable and reliable llm evaluation via domain knowledge retrieval for rubric generation. [find](https://scholar.google.com/scholar?q=Kaustubh+D%2E+Dhole+and+Eugene+Agichtein%2E+2026%2E+Rubricrag%3A+Towards+interpretable+and+reliable+llm+evaluation+via+domain+knowledge+retrieval+for+rubric+generation%2E)
3. Laura Dietz. 2024. A workbench for autograding retrieve/generate systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1963–1972. [find](https://scholar.google.com/scholar?q=Laura+Dietz%2E+2024%2E+A+workbench+for+autograding+retrieve%2Fgenerate+systems%2E+In+Proceedings+of+the+47th+International+ACM+SIGIR+Conference+on+Research+and+Development+in+Information+Retrieval%2C+pages+1963)
4. Laura Dietz, Ian Soboroff, and Coordinators of various TREC tracks. 2025. Trec autojudge pilot data. Licensed under CC BY-SA 3.0. Based on content from TREC DRAGUN, TREC RAG, TREC RAGTIME, their license applies. [find](https://scholar.google.com/scholar?q=Laura+Dietz%2C+Ian+Soboroff%2C+and+Coordinators+of+various+TREC+tracks%2E+2025%2E+Trec+autojudge+pilot+data%2E+Licensed+under+CC+BY%2DSA+3%2E0%2E+Based+on+content+from+TREC+DRAGUN%2C+TREC+RAG%2C+TREC+RAGTIME%2C+their+licen)
5. Nelson F. Liu, Tianyi Zhang, and Percy Liang. 2023. [Evaluating verifiability in generative search engines](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-verifiability-generative-search-engines/). In Findings of the Association for Computational Linguistics: EMNLP 2023.
6. Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145–152, Boston, Massachusetts… [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Rebecca+Passonneau%2E+2004%2E+Evaluating+content+selection+in+summarization%3A+The+pyramid+method%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+North+American+Chapter+of)
7. Dani Roytburg, Matthew Bozoukov, Matthew Nguyen, Jou Barzdukas, Mackenzie Puig-Hall, and Narmeen Oozeer. 2026. Are llm evaluators really narcissists? sanity checking self-preference evaluations. [find](https://scholar.google.com/scholar?q=Dani+Roytburg%2C+Matthew+Bozoukov%2C+Matthew+Nguyen%2C+Jou+Barzdukas%2C+Mackenzie+Puig%2DHall%2C+and+Narmeen+Oozeer%2E+2026%2E+Are+llm+evaluators+really+narcissists%3F+sanity+checking+self%2Dpreference+evaluations%2E)
8. Minzhu Tu, Shiyu Ni, and Keping Bi. 2026. [How long reasoning chains influence llms’ judgment of answer factuality](https://md.research.elamur.ai/free/en/papers/tu-2026-how-long-reasoning-chains-influence/).
9. Ellen M. Voorhees. 2004. Overview of the TREC 2003 question answering track. In Proceedings of the Twelfth Text REtrieval Conference (TREC 2003), pages 54–68, Gaithersburg, MD. NIST. [find](https://scholar.google.com/scholar?q=Ellen+M%2E+Voorhees%2E+2004%2E+Overview+of+the+TREC+2003+question+answering+track%2E+In+Proceedings+of+the+Twelfth+Text+REtrieval+Conference+%28TREC+2003%29%2C+pages+54%E2%80%9368%2C+Gaithersburg%2C+MD%2E+NIST%2E)
10. William Walden, Marc Mason, Orion Weller, Laura Dietz, John Conroy, Neil Molino, Hannah Recknor, Bryan Li, Gabrielle Kaili-May Liu, Yu Hou, Dawn Lawrie, James Mayfield, and Eugene Yang. 2026. Auto-argue: Llm-based report generation evaluation. [find](https://scholar.google.com/scholar?q=William+Walden%2C+Marc+Mason%2C+Orion+Weller%2C+Laura+Dietz%2C+John+Conroy%2C+Neil+Molino%2C+Hannah+Recknor%2C+Bryan+Li%2C+Gabrielle+Kaili%2DMay+Liu%2C+Yu+Hou%2C+Dawn+Lawrie%2C+James+Mayfield%2C+and+Eugene+Yang%2E+2026%2E+Auto%2Darg)
11. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems 35 (NeurIPS 2022).
12. Chuting Yu, Hang Li, Guido Zuccon, Joel Mackenzie, and Teerapong Leelanupab. 2026. When llm judges inflate scores: Exploring overrating in relevance assessment. [find](https://scholar.google.com/scholar?q=Chuting+Yu%2C+Hang+Li%2C+Guido+Zuccon%2C+Joel+Mackenzie%2C+and+Teerapong+Leelanupab%2E+2026%2E+When+llm+judges+inflate+scores%3A+Exploring+overrating+in+relevance+assessment%2E)

*Updated 1 November 2026*