--- title: "From TextBlob to LLM Agents — paper by Vidigal (2026)" description: "The paper addresses predicting customer dissatisfaction from ticket text in B2B database support. 96.4% accuracy. 10 references." authors: ["Pedro Vidigal"] year: 2026 venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)" source: "https://aclanthology.org/2026.acl-industry.121/" doi: "10.18653/v1/2026.acl-industry.121" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/vidigal-2026-textblob-llm-agents-sentiment-model/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vidigal-2026-textblob-llm-agents-sentiment-model/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/) / From TextBlob to LLM Agents: Sentiment Model Selection for B2B Technical Support with CSAT Ground Truth # From TextBlob to LLM Agents: Sentiment Model Selection for B2B Technical Support with CSAT Ground Truth **Authors:** Pedro Vidigal · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.acl-industry.121/) · [PDF](https://aclanthology.org/2026.acl-industry.121.pdf) · [DOI 10.18653/v1/2026.acl-industry.121](https://doi.org/10.18653/v1/2026.acl-industry.121) · [OpenAlex](https://openalex.org/W7166847886) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary The paper addresses predicting customer dissatisfaction from ticket text in B2B database support. In this setting, language is formal and technical, negative ratings are rare, and the label (post-resolution CSAT) reflects the outcome rather than the emotions expressed during the case. The author uses every CSAT-rated ticket from Yugabyte's support operation over five years: 533 tickets, of which only 26 were rated bad. On this data, 17 approaches are compared across five paradigms (lexicon, off-the-shelf transformers, NLI zero-shot, a multi-task LLM agent and dedicated LLM agents), along with fine-tuning attempts. MCC and McNemar tests serve as the main evaluation tools. The study also traces how the deployed system evolved from TextBlob in 2022 to a dedicated agent in 2025. A single-purpose sentiment agent that sees only customer comments performs best. Gemini 3 Flash leads at low cost. Anthropic's flagship Claude Opus 4.6 does worse than a cheaper sibling because it over-predicts Neutral, which the author links to the Alignment Tax. Some unhappy customers send purely administrative messages and cannot be detected from text alone, so non-textual signals are needed. ## Key points - The dataset contains 533 CSAT-rated tickets (507 good, 26 bad; about 2,500 customer comments) from 100+ organizations, mostly in North America. The 19.5:1 imbalance reflects the natural rate of dissatisfaction, not a sampling choice. - Moving sentiment out of a 20-field multi-task agent into a dedicated agent cut Neutral predictions from 69% to 22% and raised MCC from −0.018 to 0.347. The multi-task agent had 96.4% accuracy but caught no bad tickets. - Gemini 3 Flash scored best (MCC 0.347, recall 65.4%, $0.60/1K tickets). Claude Opus 4.6 ($67.50/1K) predicted Neutral 41% of the time and reached only MCC 0.165, below Haiku 4.5 at 0.176 (McNemar p=0.003). - All 12 LLMs missed 5 of the 26 bad tickets, and 10 of the 26 were flagged by fewer than 30% of models. - Eleven fine-tuning experiments on Siebert RoBERTa-large (focal loss, LoRA, DAPT, among others) all ended at MCC≤0. The best off-the-shelf transformer reached MCC 0.086, and DeBERTa v3 NLI zero-shot reached 0.174. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [💼 Business, Management and Accounting](https://md.research.elamur.ai/free/en/shelves/business-management-and-accounting/) · 40 works · [💼 Organizational Behavior and Human Resource Management](https://md.research.elamur.ai/free/en/shelves/business-management-and-accounting/organizational-behavior-and-human-resource-management/) · 15 works 🧭 Topics (1): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 1,035 works ## Cite Cite as: Pedro Vidigal. 2026. From TextBlob to LLM Agents: Sentiment Model Selection for B2B Technical Support with CSAT Ground Truth. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{vidigal2026textblob, title = {{From TextBlob to LLM Agents: Sentiment Model Selection for B2B Technical Support with CSAT Ground Truth}}, author = {Pedro Vidigal}, booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)}, year = {2026}, doi = {10.18653/v1/2026.acl-industry.121}, url = {https://doi.org/10.18653/v1/2026.acl-industry.121} } ``` - [← previous From Tablet to Text: AI-Assisted Reconstruction and Translation of Akkadian Passages in the Epic of Gilgamesh](https://md.research.elamur.ai/free/en/papers/mahmood-2026-tablet-text-ai-assisted-reconstruction/) - [next → From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning](https://md.research.elamur.ai/free/en/papers/purohit-2026-tokens-steps-verification-aware-speculative/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/) ## References 10 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Toyin Aguda, Erik Wilson, Allan Anzagira, Simerjot Kaur, and Charese Smiley. 2025. Conservative bias in large language models: Measuring relation predictions. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18989–18998, Vienna, Austria. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Toyin+Aguda%2C+Erik+Wilson%2C+Allan+Anzagira%2C+Simerjot+Kaur%2C+and+Charese+Smiley%2E+2025%2E+Conservative+bias+in+large+language+models%3A+Measuring+relation+predictions%2E+In+Findings+of+the+Association+for+Comput) 2. Davide Chicco and Giuseppe Jurman. 2020. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21:6. [find](https://scholar.google.com/scholar?q=Davide+Chicco+and+Giuseppe+Jurman%2E+2020%2E+The+advantages+of+the+Matthews+correlation+coefficient+%28MCC%29+over+F1+score+and+accuracy+in+binary+classification+evaluation%2E+BMC+Genomics%2C+21%3A6%2E) 3. Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A. Huerta, and Hao Peng. 2025. Context length alone hurts LLM performance despite perfect retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2… [find](https://scholar.google.com/scholar?q=Yufeng+Du%2C+Minyang+Tian%2C+Srikanth+Ronanki%2C+Subendhu+Rongali%2C+Sravan+Babu+Bodapati%2C+Aram+Galstyan%2C+Azton+Wells%2C+Roy+Schwartz%2C+Eliu+A%2E+Huerta%2C+and+Hao+Peng%2E+2025%2E+Context+length+alone+hurts+LLM+performa) 4. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Zachary Yahn, Yichang Xu, and Ling Liu. 2025. Safety tax: Safety alignment makes your large reasoning models less reasonable. arXiv preprint arXiv:2503.00555. — [arXiv:2503.00555](https://arxiv.org/abs/2503.00555) [find](https://scholar.google.com/scholar?q=Tiansheng+Huang%2C+Sihao+Hu%2C+Fatih+Ilhan%2C+Selim+Furkan+Tekin%2C+Zachary+Yahn%2C+Yichang+Xu%2C+and+Ling+Liu%2E+2025%2E+Safety+tax%3A+Safety+alignment+makes+your+large+reasoning+models+less+reasonable%2E+arXiv+preprint) 5. Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. 2024. [Mitigating the alignment tax of RLHF](https://md.research.elamur.ai/free/en/papers/lin-2024-mitigating-alignment-tax-rlhf/). In Proceedings of the 2024 Conference on Empirical Me… 6. Etienne Manderscheid and Matthias Lee. 2023. Predicting customer satisfaction with soft labels for ordinal classification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 652– 659, Toronto, Canada. Association for Computati… [find](https://scholar.google.com/scholar?q=Etienne+Manderscheid+and+Matthias+Lee%2E+2023%2E+Predicting+customer+satisfaction+with+soft+labels+for+ordinal+classification%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computationa) 7. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. [SemEval-2017 task 4: Sentiment analysis in Twitter](https://md.research.elamur.ai/free/en/papers/rosenthal-2017-semeval-2017-task-4-sentiment/). In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502–518. Association for Computational Linguistics. 8. Jinyan Su, Jennifer Healey, Preslav Nakov, and Claire Cardie. 2025. Between underthinking and overthinking: An empirical study of reasoning length and correctness in LLMs. arXiv preprint arXiv:2505.00127. — [arXiv:2505.00127](https://arxiv.org/abs/2505.00127) [find](https://scholar.google.com/scholar?q=Jinyan+Su%2C+Jennifer+Healey%2C+Preslav+Nakov%2C+and+Claire+Cardie%2E+2025%2E+Between+underthinking+and+overthinking%3A+An+empirical+study+of+reasoning+length+and+correctness+in+LLMs%2E+arXiv+preprint+arXiv%3A2505%2E00) 9. Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2024. [Sentiment analysis in the era of large language models: A reality check](https://md.research.elamur.ai/free/en/papers/zhang-2023-sentiment-analysis-era-large-language/). In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3881–3906, Mexico City, Mexico. Association for Computational Linguisti… 10. Cen Zhao, Tiantian Zhang, Hanchen Su, Yufeng Zhang, Shaowei Su, Mingzhi Xu, Yu Liu, Wei Han, Jeremy Werner, Claire Na Cheng, and Yashar Mehdad. 2025. Agent-in-the-loop: A data flywheel for continuous improvement in LLM-based customer support. In Proceedings of the 2025 Conference on Empirical Method… [find](https://scholar.google.com/scholar?q=Cen+Zhao%2C+Tiantian+Zhang%2C+Hanchen+Su%2C+Yufeng+Zhang%2C+Shaowei+Su%2C+Mingzhi+Xu%2C+Yu+Liu%2C+Wei+Han%2C+Jeremy+Werner%2C+Claire+Na+Cheng%2C+and+Yashar+Mehdad%2E+2025%2E+Agent%2Din%2Dthe%2Dloop%3A+A+data+flywheel+for+continuous+) *Updated 2 November 2026*