---
title: "A Hybrid Supervised-LLM Pipeline for Actionable Suggestion"
description: "The authors propose a two-stage hybrid pipeline. 21 references. Among them: 0."
authors: ["Aakash Trivedi", "Aniket Upadhyay", "Pratik Narang", "Dhruv Kumar", "Praveen Kumar"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)"
source: "https://aclanthology.org/2026.eacl-industry.61/"
doi: "10.18653/v1/2026.eacl-industry.61"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/trivedi-2026-hybrid-supervised-llm-pipeline-actionable/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/trivedi-2026-hybrid-supervised-llm-pipeline-actionable/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/) / A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews

# A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews

**Authors:** Aakash Trivedi, Aniket Upadhyay, Pratik Narang, [Dhruv Kumar](https://md.research.elamur.ai/free/en/authors/dhruv-kumar/) and Praveen Kumar · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-industry.61/) · [PDF](https://aclanthology.org/2026.eacl-industry.61.pdf) · [DOI 10.18653/v1/2026.eacl-industry.61](https://doi.org/10.18653/v1/2026.eacl-industry.61) · [OpenAlex](https://openalex.org/W7140087257)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

Customer reviews sometimes contain concrete requests for businesses, such as adding menu options or speeding up check-in. These requests are rare (13–18% of reviews) and appear alongside praise, complaints, and advice aimed at other customers. Earlier suggestion-mining work mostly flags sentences that contain a suggestion, and opinion summarization produces broad themes. Neither approach yields the precise operational fix a business needs.

The authors propose a two-stage hybrid pipeline. First, a RoBERTa-base classifier decides whether a review contains an actionable suggestion. It is trained with cross-entropy plus a differentiable precision–recall surrogate loss that pushes for high recall, since a missed suggestion cannot be recovered later. Second, a quantized, instruction-tuned Gemma-3 27B model running under Ollama handles the remaining work through few-shot prompts: it extracts and rewrites suggestions, assigns categories, clusters them, and summarizes each cluster.

On restaurant and ice-cream review data, the hybrid system beats prompt-only LLM, rule-based, and classifier-only baselines in extraction, categorization, clustering, and summarization. Industry experts also rated its outputs as clear and faithful. The study shows that classifier gating combined with tightly controlled LLM prompting can be deployed locally at industrial scale.

## Key points

- The classifier was trained on a proprietary set of 1,110 reviews (440 positive, 670 negative), and its learning curve levels off at about 70% of the data. The raw texts are not released, but the prompts, hyperparameters, and hardware specifications are provided.
- On 150 held-out instances, category assignment accuracy was 0.90 for the hybrid pipeline, 0.78 for the prompt-only LLM, and 0.62 for the rule-based system.
- Cluster summaries scored ROUGE-L 0.46 and BERTScore F1 0.91 for the hybrid pipeline, compared with 0.34/0.86 for prompt-only and 0.22/0.75 for rule-based. Three experts rated the outputs on a 1–5 Likert scale with κ = 0.74–0.85, giving clustering a score of 5.0.
- Ablations showed the following effects:
  - Removing the PR loss lowered recall by 3.49%.
  - Removing clustering cut human-rated interpretability by 22%.
  - Dropping category assignment reduced AMI by 0.12.
  - Running without quantization required 2.4× more memory and 47% more latency, with ΔF1 \< 0.01.
- In cross-domain tests (real estate, healthcare, finance, automotive), recall stayed robust but precision fell where specialized terminology was common. The main remaining errors were sarcasm and occasional mis-clustering by the LLM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Birla Institute of Technology and Science, Pilani](https://md.research.elamur.ai/free/en/universities/birla-institute-of-technology-and-science-pilani/) — University, Pilāni, India · 16 works · 41,879 citations
- [VMware](https://md.research.elamur.ai/free/en/companies/vmware/) — Company, Palo Alto, United States · 2 works

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Aakash Trivedi, Aniket Upadhyay, Pratik Narang, Dhruv Kumar and Praveen Kumar. 2026. A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{trivedi2026hybrid,
  title = {{A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews}},
  author = {Aakash Trivedi, Aniket Upadhyay, Pratik Narang, Dhruv Kumar and Praveen Kumar},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.61},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.61}
}
```

- [← previous A Hybrid Confidence-Aware Framework for Arabic Toxicity Detection in Social Media](https://md.research.elamur.ai/free/en/papers/alanazi-2026-hybrid-confidence-aware-framework-arabic/)
- [next → A Joint Detection Framework for Latvian Loanwords and Calques Using Monolingual Data](https://md.research.elamur.ai/free/en/papers/zhang-2026-joint-detection-framework-latvian-loanwords/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/20/)

## References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stefanos Angelidis and Mirella Lapata. 2021. Summarizing opinions with gsum. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Stefanos+Angelidis+and+Mirella+Lapata%2E+2021%2E+Summarizing+opinions+with+gsum%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28NAACL)
2. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. [Latent dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). Journal of Machine Learning Research, 3:993–1022.
3. Algirdas Bražinskas, Róbert Busa-Fekete, and Daniel Preotiuc-Pietro. 2020. Learning to summarize product reviews by exploiting aspect-level ground truth. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Algirdas+Bra%C5%BEinskas%2C+R%C3%B3bert+Busa%2DFekete%2C+and+Daniel+Preotiuc%2DPietro%2E+2020%2E+Learning+to+summarize+product+reviews+by+exploiting+aspect%2Dlevel+ground+truth%2E+In+Proceedings+of+the+2020+Conference+on+Empir)
4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877–…
5. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. [Reading wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL).
6. Adji B. Dieng, Chong Wang, Jianfeng Gao, and John Paisley. 2020. [Topic modeling in embedding spaces](https://md.research.elamur.ai/free/en/papers/dieng-2020-topic-modeling-embedding-spaces/). In Transactions of the Association for Computational Linguistics, volume 8, pages 439–453.
7. Li Dong, Qi Qian, and Lei Jiang. 2017. Attention-based neural networks for suggestion mining. In Proceedings of the 2017 Conference on Natural Language Processing (ACL). [find](https://scholar.google.com/scholar?q=Li+Dong%2C+Qi+Qian%2C+and+Lei+Jiang%2E+2017%2E+Attention%2Dbased+neural+networks+for+suggestion+mining%2E+In+Proceedings+of+the+2017+Conference+on+Natural+Language+Processing+%28ACL%29%2E)
8. Zhenzhong Ji, Richard Lee, Joseph Fries, and Roger Levy. 2023. A survey of hallucination in large language models. ACM Computing Surveys, 56(12):1– 42. [find](https://scholar.google.com/scholar?q=Zhenzhong+Ji%2C+Richard+Lee%2C+Joseph+Fries%2C+and+Roger+Levy%2E+2023%2E+A+survey+of+hallucination+in+large+language+models%2E+ACM+Computing+Surveys%2C+56%2812%29%3A1%E2%80%93+42%2E)
9. Xinyu Jiang, Min Chen, and Zhen Li. 2023. Mistral: Open-weight instruction-tuned language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Xinyu+Jiang%2C+Min+Chen%2C+and+Zhen+Li%2E+2023%2E+Mistral%3A+Open%2Dweight+instruction%2Dtuned+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
10. Mandar Joshi, Danqi Chen, and Yinhan Liu. 2020. [Spanbert: Improving pre-training by representing and predicting spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/). In Transactions of the Association for Computational Linguistics, volume 8, pages 64–77.
11. Ryota Koto, Keisuke Yoshida, and Takuya Tanaka. 2022. Span-level inconsistencies in llm-based extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Ryota+Koto%2C+Keisuke+Yoshida%2C+and+Takuya+Tanaka%2E+2022%2E+Span%2Dlevel+inconsistencies+in+llm%2Dbased+extraction%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMN)
12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [RoBERTa: A robustly optimized BERT pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
13. Sai Mukku and Praneeth Mukku. 2024. Insightnet: A neural network for semantic theme extraction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Sai+Mukku+and+Praneeth+Mukku%2E+2024%2E+Insightnet%3A+A+neural+network+for+semantic+theme+extraction%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
14. Mir Tafseer Nayeem and Davood Rafiei. 2024. LFO-Sum: Large-scale summarization of fine-grained opinions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). identified the top-performing models and further filtered them to select the best candidate for… [find](https://scholar.google.com/scholar?q=Mir+Tafseer+Nayeem+and+Davood+Rafiei%2E+2024%2E+LFO%2DSum%3A+Large%2Dscale+summarization+of+fine%2Dgrained+opinions%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNL)
15. Model GPT-2 Small GPT-2 Medium ROBERTa-Large DeBERTa-Large BERT-Large XLNet-Large BART-Large ROBERTa-Base Sapna Negi and Paul Buitelaar. 2015. Towards the extraction of customer-to-customer suggestions from reviews. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Proce… [find](https://scholar.google.com/scholar?q=Model+GPT%2D2+Small+GPT%2D2+Medium+ROBERTa%2DLarge+DeBERTa%2DLarge+BERT%2DLarge+XLNet%2DLarge+BART%2DLarge+ROBERTa%2DBase+Sapna+Negi+and+Paul+Buitelaar%2E+2015%2E+Towards+the+extraction+of+customer%2Dto%2Dcustomer+suggestion)
16. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Journal of Machine Learning Research, 23:1–60.
17. Muhammad Riaz, Ayesha Raza, and Hira Javed. 2024. TransLSTM: Transformer-enhanced LSTM for suggestion mining. Natural Language Engineering, 30(2):123–145. [find](https://scholar.google.com/scholar?q=Muhammad+Riaz%2C+Ayesha+Raza%2C+and+Hira+Javed%2E+2024%2E+TransLSTM%3A+Transformer%2Denhanced+LSTM+for+suggestion+mining%2E+Natural+Language+Engineering%2C+30%282%29%3A123%E2%80%93145%2E)
18. Gemma Team. 2024. Gemma: Instruction-tuned large language model for structured extraction. [https://gemma-model.org](https://gemma-model.org). [↗](https://gemma-model.org)
19. Budi Wicaksono and Sung-Hyon Myaeng. 2013. Mining product improvement suggestions from customer reviews. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Budi+Wicaksono+and+Sung%2DHyon+Myaeng%2E+2013%2E+Mining+product+improvement+suggestions+from+customer+reviews%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL)
20. Yuxin Wu, Lili Wang, and Ming Chen. 2023. A survey of hybrid approaches for nlp tasks: Classifier-language model pipelines. ACM Computing Surveys, 56(7):1–36. [find](https://scholar.google.com/scholar?q=Yuxin+Wu%2C+Lili+Wang%2C+and+Ming+Chen%2E+2023%2E+A+survey+of+hybrid+approaches+for+nlp+tasks%3A+Classifier%2Dlanguage+model+pipelines%2E+ACM+Computing+Surveys%2C+56%287%29%3A1%E2%80%9336%2E)
21. Peng Zhou and Wei Xu. 2018. [Joint extraction of entities and relations based on a novel tagging scheme](https://md.research.elamur.ai/free/en/papers/zheng-2017-joint-extraction-entities-relations-based/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).

*Updated 1 November 2026*