--- title: "Argument-Based Comparative Question Answering Evaluation" description: "Large language models can produce fluent answers to comparative questions such as \"Simpsons or Family Guy?\". 1 reference." authors: ["Irina Nikishina", "Saba Anwar", "Nikolay Dolgov", "Maria Manina", "Daria Ignatenko", "Viktor Moskvoretskii", "Artem Shelmanov", "Tim Baldwin", "Chris Biemann"] year: 2026 venue: "Proceedings of the 13th Workshop on Argument Mining and Reasoning" source: "https://aclanthology.org/2026.argmining-1.6/" doi: "10.18653/v1/2026.argmining-1.6" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/nikishina-2026-argument-based-comparative-question-answering/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nikishina-2026-argument-based-comparative-question-answering/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/22/) / Argument-Based Comparative Question Answering Evaluation Benchmark # Argument-Based Comparative Question Answering Evaluation Benchmark **Authors:** [Irina Nikishina](https://md.research.elamur.ai/free/en/authors/irina-nikishina/), Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, [Viktor Moskvoretskii](https://md.research.elamur.ai/free/en/authors/viktor-moskvoretskii/), [Artem Shelmanov](https://md.research.elamur.ai/free/en/authors/artem-shelmanov/), Tim Baldwin and [Chris Biemann](https://md.research.elamur.ai/free/en/authors/chris-biemann/) · **Year:** 2026 · **Venue:** [Proceedings of the 13th Workshop on Argument Mining and Reasoning](https://md.research.elamur.ai/free/en/venues/argmining/) ([Workshop on Argument Mining 2026](https://md.research.elamur.ai/free/en/venues/argmining/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.argmining-1.6/) · [PDF](https://aclanthology.org/2026.argmining-1.6.pdf) · [DOI 10.18653/v1/2026.argmining-1.6](https://doi.org/10.18653/v1/2026.argmining-1.6) · [OpenAlex](https://openalex.org/W4407806885) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Large language models can produce fluent answers to comparative questions such as "Simpsons or Family Guy?". Judging those answers is still difficult, because there are no agreed standards for comparative question answering (CQA) and manual annotation is expensive. The authors address this with CompQA, an LLM-as-a-Judge benchmark. It scores comparative summaries on 15 criteria grouped into structure, relevance and quality, giving a total of 0 to 19 points. They built evaluation sets of object pairs with optional aspects and supporting arguments, drawing on CAM 2.0, Yahoo! Answers and earlier CQA data. Several LLMs then generated answers under four prompting scenarios, and both human annotators and LLM judges rated the results. Automatic and human scores largely agreed, although the LLM judge was somewhat more generous. Proprietary systems led the ranking. However, a small open model fine-tuned on top-rated answers reached comparable quality. The benchmark matters because it gives CQA research a reproducible, low-cost evaluation protocol. All data, code and evaluation results are publicly released. ## Key points - **Scoring scheme.** CompQA uses 15 criteria worth up to 19 points: structure accounts for 7, relevance for 5, and the rest covers quality aspects such as conciseness, factuality, informativeness and coherence. The judge LLM returns per-criterion scores as JSON and does not compute a total. - **Datasets.** There is a main set of 50 comparative questions and an extended set of 432. In the main set, 20 pairs have a defined aspect, and each pair has 3–20 arguments (10 on average). - **Models evaluated.** - Main experiments: GPT-3.5, GPT-4/GPT-4o, Llama-3-8B-Instruct, Llama-3-70B-Instruct, Perplexity and Mixtral. - Additional runs: Gemma-3-27B, Qwen3-8B, DeepSeek-R1 and Gemini-2.5-Flash. - GPT-4o and Perplexity scored highest. - **Judge reliability.** Llama-3-70B used as a judge broadly matched human ratings but tended to overrate answers that humans scored low. - **Fine-tuning result.** Llama-3-8B-Instruct was fine-tuned with LoRA on 1,244 answers that had scored ≥18. It reached 18.40 ± 1.25, roughly on par with GPT-4o. The resources are available on GitHub at `lilaspourpre/cqa-evaluation-framework`. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works ## Cite Cite as: Irina Nikishina, Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, Viktor Moskvoretskii, Artem Shelmanov, Tim Baldwin and Chris Biemann. 2026. Argument-Based Comparative Question Answering Evaluation Benchmark. Proceedings of the 13th Workshop on Argument Mining and Reasoning. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{nikishina2026argument, title = {{Argument-Based Comparative Question Answering Evaluation Benchmark}}, author = {Irina Nikishina, Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, Viktor Moskvoretskii, Artem Shelmanov, Tim Baldwin and Chris Biemann}, booktitle = {Proceedings of the 13th Workshop on Argument Mining and Reasoning}, year = {2026}, doi = {10.18653/v1/2026.argmining-1.6}, url = {https://doi.org/10.18653/v1/2026.argmining-1.6} } ``` - [← previous ArGuard Shared Task: Harmful Content Detection in Arabic Memes and LLM Prompts](https://md.research.elamur.ai/free/en/papers/alam-2026-arguard-shared-task-harmful-content/) - [next → Arguments that Alter Minds: LLM Rationales Sway Human (and LLM) Notions of Plausibility](https://md.research.elamur.ai/free/en/papers/palta-2026-arguments-that-alter-minds-llm/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/22/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya… — [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://md.research.elamur.ai/free/en/papers/bavaresco-2024-llms-instead-human-judges-large/) *Updated 1 November 2026*