---
title: "Text Analytics Evaluation Framework — paper by Shi et al. (2026)"
description: "LLMs are increasingly used for data analysis, but it is unclear how reliably they can draw quantitative conclusions from large… 32 references."
authors: ["Yuefeng Shi", "Nedjma Ousidhoum", "Jose Camacho-Collados"]
year: 2026
venue: "Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)"
source: "https://aclanthology.org/2026.gem-main.69/"
doi: "10.18653/v1/2026.gem-main.69"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shi-2026-text-analytics-evaluation-framework-case/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shi-2026-text-analytics-evaluation-framework-case/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/44/) / Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media

# Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media

**Authors:** [Yuefeng Shi](https://md.research.elamur.ai/free/en/authors/yuefeng-shi/), [Nedjma Ousidhoum](https://md.research.elamur.ai/free/en/authors/nedjma-ousidhoum/) and [Jose Camacho-Collados](https://md.research.elamur.ai/free/en/authors/jose-camacho-collados/) · **Year:** 2026 · **Venue:** [Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)](https://md.research.elamur.ai/free/en/venues/gem/) ([GEM 2026](https://md.research.elamur.ai/free/en/venues/gem/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.gem-main.69/) · [PDF](https://aclanthology.org/2026.gem-main.69.pdf) · [DOI 10.18653/v1/2026.gem-main.69](https://doi.org/10.18653/v1/2026.gem-main.69) · [OpenAlex](https://openalex.org/W7166802833)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LLMs are increasingly used for data analysis, but it is unclear how reliably they can draw quantitative conclusions from large sets of short, unstructured texts. This paper addresses that gap with a question-answering benchmark built on social media posts. The authors repurpose eight English Twitter datasets from TweetEval and SuperTweetEval. They group these by annotation complexity into single-label, multi-label and targeted (stance or target-dependent sentiment) classification. Over sampled subsets of increasing size, they pose 470 manually curated questions with human-verified answers. The questions span four reasoning levels: existence, comparison, counting and calculation.

Several open-weights and closed-source LLMs are tested through prompting alone, without external tools. Closed-source models do better on average. Yet all models weaken as questions shift from detecting whether a label occurs to comparing, counting and computing proportions. They also weaken as sources become multi-label or target-dependent. Roughly 500 input posts marks a breaking point: open-weights models deteriorate broadly, while closed-source ones falter on numeric aggregation. This warns against treating LLMs as standalone analysts of large text collections.

## Key points
- The benchmark has 470 questions over eight datasets: TE-SA, TE-HS, TE-OL, STE-HS (SLC), STE-ER, STE-TC (MLC), TE-SD and STE-SA (Targeted). The code is at github.com/yuefeng-shi/text_data_analysis.
- There are four question types (Existence, Count, Comparison, Calculation) with yes/no, numeric or percentage answers. Answers are scored with F1 and NRMSE-style error metrics.
- Existence questions are easiest, with F1 up to 0.887 on SLC and 0.726 on Targeted. On Comparison, F1 falls to about 0.55 for closed-source models and 0.49–0.53 for open-weights ones.
- Counting errors stay high: in SLC, NRMSE is 0.255 for closed-source models and 0.457 for open-weights models. Calculation is hardest, with open-weights error near 0.49.
- Limitations include English-only data, a modest dataset size and a possible contamination risk. The study also omits top-tier proprietary models and tool-using agents, so the authors present results as a lower bound.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Cardiff University](https://md.research.elamur.ai/free/en/universities/cardiff-university/) — University, Cardiff, United Kingdom · 22 works · 2,444 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 227 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Yuefeng Shi, Nedjma Ousidhoum and Jose Camacho-Collados. 2026. Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shi2026text,
  title = {{Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media}},
  author = {Yuefeng Shi, Nedjma Ousidhoum and Jose Camacho-Collados},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.69},
  url = {https://doi.org/10.18653/v1/2026.gem-main.69}
}
```

- [← previous Test-Time Training for Zero-Resource Dense Retrieval Reranking](https://md.research.elamur.ai/free/en/papers/liu-2026-test-time-training-zero-resource/)
- [next → Text-to-SQL Task-oriented Dialogue Ontology Construction](https://md.research.elamur.ai/free/en/papers/vukovic-2026-text-sql-task-oriented-dialogue/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/44/)

## References

32 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Alibaba. 2026. Introducing qwen 3.5 official blog. [find](https://scholar.google.com/scholar?q=Alibaba%2E+2026%2E+Introducing+qwen+3%2E5+official+blog%2E)
3. Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, Leonardo Neves, Kiamehr Rezaee, Luis Espinosa Anke, Jiaxin Pei, and Jose Camacho-Collados. 2023. Supertweeteval: A challenging, unified and heterogeneous benchmark for social media nlp research. In Findings of the Association for Computational Li… [find](https://scholar.google.com/scholar?q=Dimosthenis+Antypas%2C+Asahi+Ushio%2C+Francesco+Barbieri%2C+Leonardo+Neves%2C+Kiamehr+Rezaee%2C+Luis+Espinosa+Anke%2C+Jiaxin+Pei%2C+and+Jose+Camacho%2DCollados%2E+2023%2E+Supertweeteval%3A+A+challenging%2C+unified+and+hetero)
4. Dimosthenis Antypas, Asahi Ushio, Jose Camacho-Collados, Vitor Silva, Leonardo Neves, and Francesco Barbieri. 2022. Twitter topic classification. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3386– 3400, Gyeongju, Republic of Korea. International Committee o… [find](https://scholar.google.com/scholar?q=Dimosthenis+Antypas%2C+Asahi+Ushio%2C+Jose+Camacho%2DCollados%2C+Vitor+Silva%2C+Leonardo+Neves%2C+and+Francesco+Barbieri%2E+2022%2E+Twitter+topic+classification%2E+In+Proceedings+of+the+29th+International+Conference+on)
5. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. [Tweeteval: Unified benchmark and comparative evaluation for tweet classification](https://md.research.elamur.ai/free/en/papers/barbieri-2020-tweeteval-unified-benchmark-comparative-evaluation/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650.
6. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. [SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter](https://md.research.elamur.ai/free/en/papers/basile-2019-semeval-2019-task-5-multilingual/). In Proceedings of the 13th International Wor…
7. Amanda Bertsch, Adithya Pratapa, Teruko Mitamura, Graham Neubig, and Matthew R. Gormley. 2025. Oolong: Evaluating long context reasoning and aggregation capabilities. arXiv preprint arXiv:2511.02817. — [arXiv:2511.02817](https://arxiv.org/abs/2511.02817) [find](https://scholar.google.com/scholar?q=Amanda+Bertsch%2C+Adithya+Pratapa%2C+Teruko+Mitamura%2C+Graham+Neubig%2C+and+Matthew+R%2E+Gormley%2E+2025%2E+Oolong%3A+Evaluating+long+context+reasoning+and+aggregation+capabilities%2E+arXiv+preprint+arXiv%3A2511%2E02817%2E+)
8. Jingxuan Chen, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2026. Understanding LLM performance degradation in multi-instance processing: The roles of instance count and context length. arXiv preprint arXiv:2603.22608. [find](https://scholar.google.com/scholar?q=Jingxuan+Chen%2C+Mohammad+Taher+Pilehvar%2C+and+Jose+Camacho%2DCollados%2E+2026%2E+Understanding+LLM+performance+degradation+in+multi%2Dinstance+processing%3A+The+roles+of+instance+count+and+context+length%2E+arXiv+p)
9. Qiyan Deng, Jianhui Li, Chengliang Chai, Jinqi Liu, Junzhi She, Kaisen Jin, Zhaoze Sun, Yuhao Deng, Jia Yuan, Ye Yuan, and 1 others. 2025. Unstructured data analysis using llms: A comprehensive benchmark. arXiv preprint arXiv:2510.27119. — [arXiv:2510.27119](https://arxiv.org/abs/2510.27119) [find](https://scholar.google.com/scholar?q=Qiyan+Deng%2C+Jianhui+Li%2C+Chengliang+Chai%2C+Jinqi+Liu%2C+Junzhi+She%2C+Kaisen+Jin%2C+Zhaoze+Sun%2C+Yuhao+Deng%2C+Jia+Yuan%2C+Ye+Yuan%2C+and+1+others%2E+2025%2E+Unstructured+data+analysis+using+llms%3A+A+comprehensive+benchm)
10. Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, and Thomas Wolf. 2025. Dabstep: Data agent benchmark for multi-step reasoning. arXiv preprint arXiv:2506.23719. — [arXiv:2506.23719](https://arxiv.org/abs/2506.23719) [find](https://scholar.google.com/scholar?q=Alex+Egg%2C+Martin+Iglesias+Goyanes%2C+Friso+Kingma%2C+Andreu+Mora%2C+Leandro+von+Werra%2C+and+Thomas+Wolf%2E+2025%2E+Dabstep%3A+Data+agent+benchmark+for+multi%2Dstep+reasoning%2E+arXiv+preprint+arXiv%3A2506%2E23719%2E+%E2%80%94+%5BarXi)
11. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. [Eli5: Long form question answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567.
12. Google. 2026. Gemini-3-1-flash-lite official blog. [find](https://scholar.google.com/scholar?q=Google%2E+2026%2E+Gemini%2D3%2D1%2Dflash%2Dlite+official+blog%2E)
13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
14. Xinyi He, Mengyu Zhou, Xinrun Xu, Xiaojun Ma, Rui Ding, Lun Du, Yan Gao, Ran Jia, Xu Chen, Shi Han, and 1 others. 2024. Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38,… [find](https://scholar.google.com/scholar?q=Xinyi+He%2C+Mengyu+Zhou%2C+Xinrun+Xu%2C+Xiaojun+Ma%2C+Rui+Ding%2C+Lun+Du%2C+Yan+Gao%2C+Ran+Jia%2C+Xu+Chen%2C+Shi+Han%2C+and+1+others%2E+2024%2E+Text2analysis%3A+A+benchmark+of+table+question+answering+with+advanced+data+analys)
15. Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, and 1 others. 2024. Infiagent-dabench: Evaluating agents on data analysis tasks. In International Conference on Machine Learning, pages 19544–19572. PMLR. [find](https://scholar.google.com/scholar?q=Xueyu+Hu%2C+Ziyu+Zhao%2C+Shuang+Wei%2C+Ziwei+Chai%2C+Qianli+Ma%2C+Guoyin+Wang%2C+Xuwu+Wang%2C+Jing+Su%2C+Jingjing+Xu%2C+Ming+Zhu%2C+and+1+others%2E+2024%2E+Infiagent%2Ddabench%3A+Evaluating+agents+on+data+analysis+tasks%2E+In+Inte)
16. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Yih, Daniel Fried, Si yi Wang, and Tao Yu. 2022. Ds-1000: A natural and reliable benchmark for data science code generation. arxiv abs/2211.11501 (2022). [find](https://scholar.google.com/scholar?q=Yuhang+Lai%2C+Chengxi+Li%2C+Yiming+Wang%2C+Tianyi+Zhang%2C+Ruiqi+Zhong%2C+Luke+Zettlemoyer%2C+Scott+Yih%2C+Daniel+Fried%2C+Si+yi+Wang%2C+and+Tao+Yu%2E+2022%2E+Ds%2D1000%3A+A+natural+and+reliable+benchmark+for+data+science+code)
17. Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu, and Yue Zhang. 2025. Logical reasoning in [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). arXiv preprint arXiv:2502.09100.
18. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. [SemEval-2018 task 1: Affect in tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2018-semeval-2018-task-1-affect/). In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana. Association for Computational Linguistics.
19. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. [Semeval-2016 task 6: Detecting stance in tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2016-semeval-2016-task-6-detecting/). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41.
20. OpenAI. 2026. Introducing gpt-5.4 mini and nano official blog. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2026%2E+Introducing+gpt%2D5%2E4+mini+and+nano+official+blog%2E)
21. Zafaryab Rasool, Stefanus Kurniawan, Sherwin Balugo, Scott Barnett, Rajesh Vasa, Courtney Chesser, Benjamin M Hampstead, Sylvie Belleville, Kon Mouzakis, and Alex Bahar-Fuchs. 2024. Evaluating llms on document-based qa: Exact answer selection and numerical extraction using cogtale dataset. Natural L… [find](https://scholar.google.com/scholar?q=Zafaryab+Rasool%2C+Stefanus+Kurniawan%2C+Sherwin+Balugo%2C+Scott+Barnett%2C+Rajesh+Vasa%2C+Courtney+Chesser%2C+Benjamin+M+Hampstead%2C+Sylvie+Belleville%2C+Kon+Mouzakis%2C+and+Alex+Bahar%2DFuchs%2E+2024%2E+Evaluating+llms+on)
22. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017a. [Semeval-2017 task 4: Sentiment analysis in twitter](https://md.research.elamur.ai/free/en/papers/rosenthal-2017-semeval-2017-task-4-sentiment/). In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pages 502– 518.
23. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017b. [SemEval-2017 task 4: Sentiment analysis in Twitter](https://md.research.elamur.ai/free/en/papers/rosenthal-2017-semeval-2017-task-4-sentiment/). In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502– 518, Vancouver, Canada. Association for Computational Linguistics.
24. Pratik Sachdeva, Renata Barreto, Geoff Bacon, Alexander Sahn, Claudia von Vacano, and Chris Kennedy. 2022. The measuring hate speech corpus: Leveraging rasch measurement theory for data perspectivism. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, pages 83–94, Marse… [find](https://scholar.google.com/scholar?q=Pratik+Sachdeva%2C+Renata+Barreto%2C+Geoff+Bacon%2C+Alexander+Sahn%2C+Claudia+von+Vacano%2C+and+Chris+Kennedy%2E+2022%2E+The+measuring+hate+speech+corpus%3A+Leveraging+rasch+measurement+theory+for+data+perspectivism%2E)
25. Eliyahu Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, and Assaf Arbelle. 2024. Numerologic: Number encoding for enhanced llms’ numerical reasoning. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Eliyahu+Schwartz%2C+Leshem+Choshen%2C+Joseph+Shtok%2C+Sivan+Doveh%2C+Leonid+Karlinsky%2C+and+Assaf+Arbelle%2E+2024%2E+Numerologic%3A+Number+encoding+for+enhanced+llms%E2%80%99+numerical+reasoning%2E+In+Conference+on+Empirical+)
26. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and 1 others. 2022. [Scrolls: Standardized comparison over long language sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pag…
27. Xinyi Song, Lina Lee, Kexin Xie, Xueying Liu, Xinwei Deng, and Yili Hong. 2025. Statllm: A dataset for evaluating the performance of large language models in statistical analysis. arXiv preprint arXiv:2502.17657. — [arXiv:2502.17657](https://arxiv.org/abs/2502.17657) [find](https://scholar.google.com/scholar?q=Xinyi+Song%2C+Lina+Lee%2C+Kexin+Xie%2C+Xueying+Liu%2C+Xinwei+Deng%2C+and+Yili+Hong%2E+2025%2E+Statllm%3A+A+dataset+for+evaluating+the+performance+of+large+language+models+in+statistical+analysis%2E+arXiv+preprint+arXiv)
28. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
29. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971. xAI. 2025. Grok 4.1 blog.
30. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
31. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. [Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval)](https://md.research.elamur.ai/free/en/papers/zampieri-2019-semeval-2019-task-6-identifying/). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86.
32. Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lekang Yang, Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, and Yisong Yue. 2025. [Datascibench: An llm agent benchmark for data science](https://md.research.elamur.ai/free/en/papers/zhang-2026-datascibench-llm-agent-benchmark-data/). arXiv preprint arXiv:2502.13897.

*Updated 1 November 2026*