---
title: "Automatic Evaluation of Open-Domain Real Conversations:"
description: "This work builds a reference-free evaluator that predicts the 1–5 scores real Alexa users assigned to a socialbot while it… 30 references."
authors: ["Cristina Conforto-López", "Marcos Estecha-Garitagoitia", "Mario Rodriguez-Cantelar", "Ricardo de Córdoba", "Luis Fernando D'Haro"]
year: 2026
venue: "Proceedings of the 16th International Workshop on Spoken Dialogue System Technology"
source: "https://aclanthology.org/2026.iwsds-1.5/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lopez-2026-automatic-evaluation-open-domain-real/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lopez-2026-automatic-evaluation-open-domain-real/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/) / Automatic Evaluation of Open-Domain Real Conversations: Combining Encoder-Based, Dialogue-Based Features and Large Language Models Ratings

# Automatic Evaluation of Open-Domain Real Conversations: Combining Encoder-Based, Dialogue-Based Features and Large Language Models Ratings

**Authors:** Cristina Conforto-López, [Marcos Estecha-Garitagoitia](https://md.research.elamur.ai/free/en/authors/marcos-estecha-garitagoitia/), Mario Rodriguez-Cantelar, Ricardo de Córdoba and [Luis Fernando D'Haro](https://md.research.elamur.ai/free/en/authors/luis-fernando-d-haro/) · **Year:** 2026 · **Venue:** [Proceedings of the 16th International Workshop on Spoken Dialogue System Technology](https://md.research.elamur.ai/free/en/venues/iwsds/) ([International Workshop on Spoken Dialogue System Technology 2026](https://md.research.elamur.ai/free/en/venues/iwsds/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.iwsds-1.5/) · [PDF](https://aclanthology.org/2026.iwsds-1.5.pdf)

## Summary
Judging the quality of open-domain chatbot conversations automatically is difficult because user satisfaction is subjective. Most learned evaluators are also benchmarked on expert-annotated static datasets rather than on ratings from real users. This work builds a reference-free evaluator that predicts the 1–5 scores real Alexa users assigned to a socialbot while it competed in the fifth edition of the Alexa Prize SocialBot Grand Challenge. The authors adapt the encoder-based Panel of Experts (PoE) to dialogue-level ratings by scoring chatbot responses against sliding windows of preceding context. They also test a locally deployable LLM, both zero-shot with prompts and after fine-tuning, and design hand-crafted dialogue flags and length features. Finally, an XGBoost regressor fuses all of these signals. PoE is the strongest single component. The full fusion gives the best agreement with user ratings, a Pearson correlation of about 0.40 on the held-out set. The authors present this as competitive with earlier work that used roughly ten times more dialogues, though the setups are not directly comparable. The study shows that combining complementary cheap and neural signals can help with noisy, real-world evaluation data.

## Key points
- **Data:** About 16,000 rated conversations (290k turns) were filtered to 13,000 dialogues with at least five turns (260k turns). Of these, 10% (~1,300) form the Eval set, and the rest is used for 5-fold cross-validation.
- **PoE adaptation:** PoE was fine-tuned with input contexts of 3, 4 and 5 turns, giving 183k restructured turns in total. The 5-turn variant (PoE-ft-5-turns) worked best, reaching Pearson 0.309 on the test folds and 0.325 on the Eval set.
- **LLM results:** LLMs on their own trailed PoE. The zero-shot LLM scored 0.212 on test and 0.225 on Eval, and the fine-tuned LLM scored 0.257 on test and 0.264 on Eval.
- **Best model:** The best model is an XGBoost regressor combining flags, conversation length, PoE-ft-5-turns predictions and LLM-ft predictions. It reaches 0.346 on test and 0.404 on Eval. Its gain over the PoE+features regressor (0.339 / 0.387) is not statistically significant.
- **Limitations:** All data come from a single chatbot, which leaves generalization to other systems unclear. Only small, non-proprietary LLMs were used.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 624 works

## Cite

Cite as: Cristina Conforto-López, Marcos Estecha-Garitagoitia, Mario Rodriguez-Cantelar, Ricardo de Córdoba and Luis Fernando D'Haro. 2026. Automatic Evaluation of Open-Domain Real Conversations: Combining Encoder-Based, Dialogue-Based Features and Large Language Models Ratings. Proceedings of the 16th International Workshop on Spoken Dialogue System Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{confortolopez2026automatic,
  title = {{Automatic Evaluation of Open-Domain Real Conversations: Combining Encoder-Based, Dialogue-Based Features and Large Language Models Ratings}},
  author = {Cristina Conforto-López, Marcos Estecha-Garitagoitia, Mario Rodriguez-Cantelar, Ricardo {de Córdoba} and Luis Fernando D'Haro},
  booktitle = {Proceedings of the 16th International Workshop on Spoken Dialogue System Technology},
  year = {2026},
  url = {https://aclanthology.org/2026.iwsds-1.5/}
}
```

- [← previous Automatic Classification of Arabic Literature into Historical Eras](https://md.research.elamur.ai/free/en/papers/alhathloul-2026-automatic-classification-arabic-literature-historical/)
- [next → Automatic Extraction of Textual and Phonemic Complexity for French Cued Speech](https://md.research.elamur.ai/free/en/papers/norre-2026-automatic-extraction-textual-phonemic-complexity/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/)

## References

30 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michael Johnston, Cris Flagg, Anna Gottardi, Sattvik Sahai, Yao Lu, Samyuth Sagi, Luke Dai, Prasoon Goyal, Behnam Hedayatnia, Lucy Hu, and 1 others. 2023. Advancing open domain dialog: The fifth alexa prize socialbot grand challenge. [find](https://scholar.google.com/scholar?q=Michael+Johnston%2C+Cris+Flagg%2C+Anna+Gottardi%2C+Sattvik+Sahai%2C+Yao+Lu%2C+Samyuth+Sagi%2C+Luke+Dai%2C+Prasoon+Goyal%2C+Behnam+Hedayatnia%2C+Lucy+Hu%2C+and+1+others%2E+2023%2E+Advancing+open+domain+dialog%3A+The+fifth+alexa)
2. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785– 794. [find](https://scholar.google.com/scholar?q=Tianqi+Chen+and+Carlos+Guestrin%2E+2016%2E+Xgboost%3A+A+scalable+tree+boosting+system%2E+In+Proceedings+of+the+22nd+acm+sigkdd+international+conference+on+knowledge+discovery+and+data+mining%2C+pages+785%E2%80%93+794%2E)
3. Cat P Le, Luke Dai, Michael Johnston, Yang Liu, Marilyn Walker, and Reza Ghanadan. 2023. Improving open-domain dialogue evaluation with a causal inference model. arXiv preprint arXiv:2301.13372. — [arXiv:2301.13372](https://arxiv.org/abs/2301.13372) [find](https://scholar.google.com/scholar?q=Cat+P+Le%2C+Luke+Dai%2C+Michael+Johnston%2C+Yang+Liu%2C+Marilyn+Walker%2C+and+Reza+Ghanadan%2E+2023%2E+Improving+open%2Ddomain+dialogue+evaluation+with+a+causal+inference+model%2E+arXiv+preprint+arXiv%3A2301%2E13372%2E+%E2%80%94+%5Bar)
4. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, and 1 others. 2024. [Chatbot arena: An open platform for evaluating llms by human preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/). In Forty-first International Conference on Machin…
5. Dong-Ho Lee, Adyasha Maharana, Jay Pujara, Xiang Ren, and Francesco Barbieri. 2025. Realtalk: A 21-day real-world dataset for long-term conversation. arXiv preprint arXiv:2502.13270. — [arXiv:2502.13270](https://arxiv.org/abs/2502.13270) [find](https://scholar.google.com/scholar?q=Dong%2DHo+Lee%2C+Adyasha+Maharana%2C+Jay+Pujara%2C+Xiang+Ren%2C+and+Francesco+Barbieri%2E+2025%2E+Realtalk%3A+A+21%2Dday+real%2Dworld+dataset+for+long%2Dterm+conversation%2E+arXiv+preprint+arXiv%3A2502%2E13270%2E+%E2%80%94+%5BarXiv%3A2502%2E132)
6. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024a. [Llms-as-judges: A comprehensive survey on llm-based evaluation methods](https://md.research.elamur.ai/free/en/papers/li-2024-llms-judges-comprehensive-survey-llm/). Preprint, arXiv:2412.05579.
7. Corinna Cortes and Vladimir Vapnik. 1995. [Support-vector networks](https://md.research.elamur.ai/free/en/papers/cortes-1995-support-vector-networks/). Machine learning, 20:273–297.
8. Kyle Dai, Maurice Burger, Roman Engeler, Max Bartolo, Clémentine Fourrier, Toby Drane, Mathias Leys, and Jackson Golden. 2024. Judge arena: Benchmarking llms as evaluators. HuggingFace Blogs. [find](https://scholar.google.com/scholar?q=Kyle+Dai%2C+Maurice+Burger%2C+Roman+Engeler%2C+Max+Bartolo%2C+Cl%C3%A9mentine+Fourrier%2C+Toby+Drane%2C+Mathias+Leys%2C+and+Jackson+Golden%2E+2024%2E+Judge+arena%3A+Benchmarking+llms+as+evaluators%2E+HuggingFace+Blogs%2E)
9. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024b. [Llms-as-judges: a comprehensive survey on llm-based evaluation methods](https://md.research.elamur.ai/free/en/papers/li-2024-llms-judges-comprehensive-survey-llm/). arXiv preprint arXiv:2412.05579.
10. Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. 2020. Dialogue response ranking training with large-scale human feedback data. Preprint, arXiv:2009.06978. [find](https://scholar.google.com/scholar?q=Xiang+Gao%2C+Yizhe+Zhang%2C+Michel+Galley%2C+Chris+Brockett%2C+and+Bill+Dolan%2E+2020%2E+Dialogue+response+ranking+training+with+large%2Dscale+human+feedback+data%2E+Preprint%2C+arXiv%3A2009%2E06978%2E)
11. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. [Dailydialog: A manually labelled multi-turn dialogue dataset](https://md.research.elamur.ai/free/en/papers/li-2017-dailydialog-manually-labelled-multi-turn/). arXiv preprint arXiv:1710.03957.
12. Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur. 2023. Topical-chat: Towards knowledge-grounded open-domain conversations. arXiv preprint arXiv:2308.11995. — [arXiv:2308.11995](https://arxiv.org/abs/2308.11995) [find](https://scholar.google.com/scholar?q=Karthik+Gopalakrishnan%2C+Behnam+Hedayatnia%2C+Qinlang+Chen%2C+Anna+Gottardi%2C+Sanjeev+Kwatra%2C+Anu+Venkatesh%2C+Raefer+Gabriel%2C+and+Dilek+Hakkani%2DTur%2E+2023%2E+Topical%2Dchat%3A+Towards+knowledge%2Dgrounded+open%2Ddomain)
13. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-eval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv preprint arXiv:2303.16634.
14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
15. Varvara Logacheva, Valentin Malykh, Aleksey Litinsky, and Mikhail Burtsev. 2019. Convai2 dataset of non-goal-oriented human-to-bot dialogues. In The NeurIPS’18 Competition: From Machine Learning to Intelligent Conversations, pages 277–294. Springer. [find](https://scholar.google.com/scholar?q=Varvara+Logacheva%2C+Valentin+Malykh%2C+Aleksey+Litinsky%2C+and+Mikhail+Burtsev%2E+2019%2E+Convai2+dataset+of+non%2Dgoal%2Doriented+human%2Dto%2Dbot+dialogues%2E+In+The+NeurIPS%E2%80%9918+Competition%3A+From+Machine+Learning+to+In)
16. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. [find](https://scholar.google.com/scholar?q=Jiawei+Gu%2C+Xuhui+Jiang%2C+Zhichao+Shi%2C+Hexiang+Tan%2C+Xuehao+Zhai%2C+Chengjin+Xu%2C+Wei+Li%2C+Yinghan+Shen%2C+Shengjie+Ma%2C+Honghao+Liu%2C+and+1+others%2E+2024%2E+A+survey+on+llm%2Das%2Da%2Djudge%2E+arXiv+preprint+arXiv%3A2411%2E15)
17. Ryan Lowe, Michael Noseworthy, Iulian V Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017a. [Towards an automatic turing test: Learning to evaluate dialogue responses](https://md.research.elamur.ai/free/en/papers/lowe-2017-towards-automatic-turing-test-learning/). arXiv preprint arXiv:1708.07149.
18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). ICLR, 1(2):3.
19. Ryan Lowe, Michael Noseworthy, Iulian V Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017b. [Towards an automatic turing test: Learning to evaluate dialogue responses](https://md.research.elamur.ai/free/en/papers/lowe-2017-towards-automatic-turing-test-learning/). arXiv preprint arXiv:1708.07149.
20. Daniil Huryn, William M. Hutsell, and Jinho D. Choi. 2022. Automatic generation of large-scale multi-turn dialogues from Reddit. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3360–3373, Gyeongju, Republic of Korea. International Committee on Computational Li… [find](https://scholar.google.com/scholar?q=Daniil+Huryn%2C+William+M%2E+Hutsell%2C+and+Jinho+D%2E+Choi%2E+2022%2E+Automatic+generation+of+large%2Dscale+multi%2Dturn+dialogues+from+Reddit%2E+In+Proceedings+of+the+29th+International+Conference+on+Computational+Li)
21. Shikib Mehri, Jinho Choi, Luis Fernando D’Haro, Jan Deriu, Maxine Eskenazi, Milica Gasic, Kallirroi Georgila, Dilek Hakkani-Tur, Zekang Li, Verena Rieser, and 1 others. 2022. Report from the nsf future directions workshop on automatic evaluation of dialog: Research directions and challenges. arXiv p… — [arXiv:2203.10012](https://arxiv.org/abs/2203.10012) [find](https://scholar.google.com/scholar?q=Shikib+Mehri%2C+Jinho+Choi%2C+Luis+Fernando+D%E2%80%99Haro%2C+Jan+Deriu%2C+Maxine+Eskenazi%2C+Milica+Gasic%2C+Kallirroi+Georgila%2C+Dilek+Hakkani%2DTur%2C+Zekang+Li%2C+Verena+Rieser%2C+and+1+others%2E+2022%2E+Report+from+the+nsf+futur)
22. Takumi Ito, Kees van Deemter, and Jun Suzuki. 2025. Reference-free evaluation metrics for text generation: A survey. arXiv preprint arXiv:2501.12011. — [arXiv:2501.12011](https://arxiv.org/abs/2501.12011) [find](https://scholar.google.com/scholar?q=Takumi+Ito%2C+Kees+van+Deemter%2C+and+Jun+Suzuki%2E+2025%2E+Reference%2Dfree+evaluation+metrics+for+text+generation%3A+A+survey%2E+arXiv+preprint+arXiv%3A2501%2E12011%2E+%E2%80%94+%5BarXiv%3A2501%2E12011%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E12)
23. Shikib Mehri and Maxine Eskenazi. 2020. [Usr: An unsupervised and reference free evaluation metric for dialog generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/). arXiv preprint arXiv:2005.00456.
24. John Mendonça, Alon Lavie, and Isabel Trancoso. 2024. On the benchmarking of llms for open-domain dialogue evaluation. arXiv preprint arXiv:2407.03841. [find](https://scholar.google.com/scholar?q=John+Mendon%C3%A7a%2C+Alon+Lavie%2C+and+Isabel+Trancoso%2E+2024%2E+On+the+benchmarking+of+llms+for+open%2Ddomain+dialogue+evaluation%2E+arXiv+preprint+arXiv%3A2407%2E03841%2E)
25. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36:46595–46623.
26. John Mendonça, Patrícia Pereira, Helena Moniz, Joao Paulo Carvalho, Alon Lavie, and Isabel Trancoso. 2023. Simple llm prompting is state-of-the-art for robust and multilingual dialogue evaluation. arXiv preprint arXiv:2308.16797. [find](https://scholar.google.com/scholar?q=John+Mendon%C3%A7a%2C+Patr%C3%ADcia+Pereira%2C+Helena+Moniz%2C+Joao+Paulo+Carvalho%2C+Alon+Lavie%2C+and+Isabel+Trancoso%2E+2023%2E+Simple+llm+prompting+is+state%2Dof%2Dthe%2Dart+for+robust+and+multilingual+dialogue+evaluation%2E+arX)
27. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. [Llamafactory: Unified efficient fine-tuning of 100+ language models](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/). arXiv preprint arXiv:2403.13372.
28. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207. [find](https://scholar.google.com/scholar?q=Hannah+Rashkin%2C+Eric+Michael+Smith%2C+Margaret+Li%2C+and+Y%2DLan+Boureau%2E+2018%2E+Towards+empathetic+open%2Ddomain+conversation+models%3A+A+new+benchmark+and+dataset%2E+arXiv+preprint+arXiv%3A1811%2E00207%2E)
29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
30. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+and+1+others%2E+2024%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115)

*Updated 1 November 2026*