---
title: "Position — paper by Perez et al. (2026) — summary and ideas"
description: "They propose the Inductive Conceptual Rating (ICR) metric. 21 references. Among them: 0."
authors: ["Natalie Perez", "Sreyoshi Bhaduri", "Aman Chadha"]
year: 2026
venue: "Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)"
source: "https://aclanthology.org/2026.gem-main.81/"
doi: "10.18653/v1/2026.gem-main.81"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/perez-2026-position-semiotic-hermeneutic-approach-evaluating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/perez-2026-position-semiotic-hermeneutic-approach-evaluating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/) / Position: A Semiotic-Hermeneutic Approach to Evaluating Meaning in LLM Summaries via the Inductive Conceptual Rating Metric

# Position: A Semiotic-Hermeneutic Approach to Evaluating Meaning in LLM Summaries via the Inductive Conceptual Rating Metric

**Authors:** Natalie Perez, Sreyoshi Bhaduri and [Aman Chadha](https://md.research.elamur.ai/free/en/authors/aman-chadha/) · **Year:** 2026 · **Venue:** [Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)](https://md.research.elamur.ai/free/en/venues/gem/) ([GEM 2026](https://md.research.elamur.ai/free/en/venues/gem/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.gem-main.81/) · [PDF](https://aclanthology.org/2026.gem-main.81.pdf) · [DOI 10.18653/v1/2026.gem-main.81](https://doi.org/10.18653/v1/2026.gem-main.81) · [OpenAlex](https://openalex.org/W7166829864)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper questions whether standard automatic metrics, which treat words as stable units, can tell us if an LLM summary actually preserves the meaning of its source. Drawing on Saussure's semiotics and Gadamer's hermeneutics, the authors argue that meaning depends on context and interpretation. They propose the Inductive Conceptual Rating (ICR) metric. First, human coders run a blind Reflective Thematic Analysis (RTA, Braun and Clarke's six phases) on the reference data to build an interpretive baseline of themes. Then Inductive Content Analysis (ICA, Zhang and Wildemuth's eight steps) checks which of those concepts LLM outputs retain, weaken, or omit, and the result is turned into a coverage score. The case study compares thematic summaries of open-text survey answers to a single question, written by humans and by two LLMs, across five datasets. The models score well on lexical similarity metrics but cover far fewer of the human-identified themes. The authors conclude that surface fluency can conceal semantic misalignment, so interpretive evaluation needs human baselines.

## Key points
- **Data and models:** five datasets of 50, 100, 200, 400 and 800 survey responses. Human analysts are compared with Sonnet 3.5 and Nova Pro using cosine similarity, recall, precision, F1 and ICR.
- **Main gap:** human ICR stays high, at about 0.93 on average (range 0.86–1.00). LLM ICR ranges from 0.35 to 0.76, even where the models' cosine similarity and F1 match or beat the human scores.
- **Effect of dataset size:** the best LLM result appears at N = 800, where Nova Pro reaches ICR 0.76. Larger inputs help, but they do not bring models to human-level interpretation. Newer or more sophisticated models were not reliably better.
- **Scope:** ICR is intended for abstractive summarization, thematic synthesis, and reference-based open generation. It is not meant for factual QA or creative generation.
- **Limitations:**
  - The study covers a single survey question and only two LLMs.
  - The metric relies on presence or absence of concepts and does not capture discourse-level meaning.
  - Its validity depends on the coders' expertise and positionality.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Hawaiʻi at Mānoa](https://md.research.elamur.ai/free/en/universities/university-of-hawai-i-at-manoa/) — University, Honolulu, United States · 4 works · 20 citations

Also on the byline: [University of Hawaii System](https://ror.org/03tzaeb71).

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 254 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Natalie Perez, Sreyoshi Bhaduri and Aman Chadha. 2026. Position: A Semiotic-Hermeneutic Approach to Evaluating Meaning in LLM Summaries via the Inductive Conceptual Rating Metric. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{perez2026position,
  title = {{Position: A Semiotic-Hermeneutic Approach to Evaluating Meaning in LLM Summaries via the Inductive Conceptual Rating Metric}},
  author = {Natalie Perez, Sreyoshi Bhaduri and Aman Chadha},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.81},
  url = {https://doi.org/10.18653/v1/2026.gem-main.81}
}
```

- [← previous Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers](https://md.research.elamur.ai/free/en/papers/shimizu-2026-position-encoding-random-float-sampling/)
- [next → Position: Biomedical NLP Demands Specialization, Not Generalization](https://md.research.elamur.ai/free/en/papers/wasi-2026-position-biomedical-nlp-demands-specialization/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/)

## References

21 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Alizadeh and A. Seilsepour. 2025. A novel self-supervised sentiment classification approach using semantic labeling based on contextual embeddings. Multimedia Tools and Applications, 84(12):10195– 10220. [find](https://scholar.google.com/scholar?q=M%2E+Alizadeh+and+A%2E+Seilsepour%2E+2025%2E+A+novel+self%2Dsupervised+sentiment+classification+approach+using+semantic+labeling+based+on+contextual+embeddings%2E+Multimedia+Tools+and+Applications%2C+84%2812%29%3A10195%E2%80%93+)
2. M. Apidianaki. 2023. [From word types to tokens and back: A survey of approaches to word meaning representation and interpretation](https://md.research.elamur.ai/free/en/papers/apidianaki-2023-word-types-tokens-back-survey/). Computational Linguistics, 49(2):465–523.
3. A. Arseniev-Koehler. 2024. Theoretical foundations and limits of word embeddings: what types of meaning can they capture? Sociological Methods & Research, 53(4):1753–1793. [find](https://scholar.google.com/scholar?q=A%2E+Arseniev%2DKoehler%2E+2024%2E+Theoretical+foundations+and+limits+of+word+embeddings%3A+what+types+of+meaning+can+they+capture%3F+Sociological+Methods+%26+Research%2C+53%284%29%3A1753%E2%80%931793%2E)
4. R. Bedemariam, N. Perez, S. Bhaduri, S. Kapoor, A. Gil, E. Conjar, and N. Nayyar. 2025. Potential and perils of large language models as judges of unstructured textual data. arXiv preprint arXiv:2501.08167. — [arXiv:2501.08167](https://arxiv.org/abs/2501.08167) [find](https://scholar.google.com/scholar?q=R%2E+Bedemariam%2C+N%2E+Perez%2C+S%2E+Bhaduri%2C+S%2E+Kapoor%2C+A%2E+Gil%2C+E%2E+Conjar%2C+and+N%2E+Nayyar%2E+2025%2E+Potential+and+perils+of+large+language+models+as+judges+of+unstructured+textual+data%2E+arXiv+preprint+arXiv%3A2501%2E)
5. S. Bhaduri, S. Kapoor, A. Gil, A. Mittal, and R. Mulkar. 2024. Reconciling methodological paradigms: Employing large language models as novice qualitative research assistants in talent management research. arXiv preprint, abs/2408.11043. — [arXiv:2408.11043](https://arxiv.org/abs/2408.11043) [find](https://scholar.google.com/scholar?q=S%2E+Bhaduri%2C+S%2E+Kapoor%2C+A%2E+Gil%2C+A%2E+Mittal%2C+and+R%2E+Mulkar%2E+2024%2E+Reconciling+methodological+paradigms%3A+Employing+large+language+models+as+novice+qualitative+research+assistants+in+talent+management+rese)
6. A. D. Bhargavi. 2025. Comparative study of static and contextual text vectorization for sentiment analysis. International Journal for Research in Applied Science & Engineering Technology. [find](https://scholar.google.com/scholar?q=A%2E+D%2E+Bhargavi%2E+2025%2E+Comparative+study+of+static+and+contextual+text+vectorization+for+sentiment+analysis%2E+International+Journal+for+Research+in+Applied+Science+%26+Engineering+Technology%2E)
7. M. Borgstede and M. Scholz. 2021. Quantitative and qualitative approaches to generalization and replication – a representational perspective. Frontiers in Psychology, 12:605823. [find](https://scholar.google.com/scholar?q=M%2E+Borgstede+and+M%2E+Scholz%2E+2021%2E+Quantitative+and+qualitative+approaches+to+generalization+and+replication+%E2%80%93+a+representational+perspective%2E+Frontiers+in+Psychology%2C+12%3A605823%2E)
8. V. Braun and V. Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2):77–101. [find](https://scholar.google.com/scholar?q=V%2E+Braun+and+V%2E+Clarke%2E+2006%2E+Using+thematic+analysis+in+psychology%2E+Qualitative+Research+in+Psychology%2C+3%282%29%3A77%E2%80%93101%2E)
9. E. N. Purba and B. A. R. Damanik. 2025. Polysemy and homonymy in semantic interpretation. Young Journal of Social Sciences and Humanities, 1(3):101–109. [find](https://scholar.google.com/scholar?q=E%2E+N%2E+Purba+and+B%2E+A%2E+R%2E+Damanik%2E+2025%2E+Polysemy+and+homonymy+in+semantic+interpretation%2E+Young+Journal+of+Social+Sciences+and+Humanities%2C+1%283%29%3A101%E2%80%93109%2E)
10. V. Braun and V. Clarke. 2022. Conceptual and design thinking for thematic analysis. Qualitative Psychology, 9(1):3. [find](https://scholar.google.com/scholar?q=V%2E+Braun+and+V%2E+Clarke%2E+2022%2E+Conceptual+and+design+thinking+for+thematic+analysis%2E+Qualitative+Psychology%2C+9%281%29%3A3%2E)
11. F. De Saussure. 1916. Nature of the linguistic sign. Bedford/St. Martin’s Press. [find](https://scholar.google.com/scholar?q=F%2E+De+Saussure%2E+1916%2E+Nature+of+the+linguistic+sign%2E+Bedford%2FSt%2E+Martin%E2%80%99s+Press%2E)
12. J. W. Creswell and J. D. Creswell. 2017. Research design: Qualitative, quantitative, and mixed methods approaches. Sage Publications. [find](https://scholar.google.com/scholar?q=J%2E+W%2E+Creswell+and+J%2E+D%2E+Creswell%2E+2017%2E+Research+design%3A+Qualitative%2C+quantitative%2C+and+mixed+methods+approaches%2E+Sage+Publications%2E)
13. S. Shan. 2025. Towards reflexive ai: A comprehensive exploration of enhancing social science research through nlp. In Future of Information and Communication Conference, pages 765–792. Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=S%2E+Shan%2E+2025%2E+Towards+reflexive+ai%3A+A+comprehensive+exploration+of+enhancing+social+science+research+through+nlp%2E+In+Future+of+Information+and+Communication+Conference%2C+pages+765%E2%80%93792%2E+Springer+Nature)
14. J. W. Creswell and C. N. Poth. 2016. Qualitative inquiry and research design: Choosing among five approaches. Sage Publications. [find](https://scholar.google.com/scholar?q=J%2E+W%2E+Creswell+and+C%2E+N%2E+Poth%2E+2016%2E+Qualitative+inquiry+and+research+design%3A+Choosing+among+five+approaches%2E+Sage+Publications%2E)
15. E. Fitkov-Norris, N. Kocheva, and N. Bhimireddy. 2023. Are we there yet? thematic analysis, nlp and machine learning for academic research: a comparative review. In FBSS Research Conference 2023. [find](https://scholar.google.com/scholar?q=E%2E+Fitkov%2DNorris%2C+N%2E+Kocheva%2C+and+N%2E+Bhimireddy%2E+2023%2E+Are+we+there+yet%3F+thematic+analysis%2C+nlp+and+machine+learning+for+academic+research%3A+a+comparative+review%2E+In+FBSS+Research+Conference+2023%2E)
16. H. G. Gadamer. 1975. Hermeneutics and social science. Cultural Hermeneutics, 2(4):307–316. [find](https://scholar.google.com/scholar?q=H%2E+G%2E+Gadamer%2E+1975%2E+Hermeneutics+and+social+science%2E+Cultural+Hermeneutics%2C+2%284%29%3A307%E2%80%93316%2E)
17. R. Gangundi and R. Sridhar. 2025. Rbca-ets: enhancing extractive text summarization with contextual embedding and word-level attention. International Journal of Information Technology, 17(2):1127–1135. [find](https://scholar.google.com/scholar?q=R%2E+Gangundi+and+R%2E+Sridhar%2E+2025%2E+Rbca%2Dets%3A+enhancing+extractive+text+summarization+with+contextual+embedding+and+word%2Dlevel+attention%2E+International+Journal+of+Information+Technology%2C+17%282%29%3A1127%E2%80%931135)
18. J. Grindrod. 2024. Transformers, contextualism, and polysemy. arXiv preprint arXiv:2404.09577. — [arXiv:2404.09577](https://arxiv.org/abs/2404.09577) [find](https://scholar.google.com/scholar?q=J%2E+Grindrod%2E+2024%2E+Transformers%2C+contextualism%2C+and+polysemy%2E+arXiv+preprint+arXiv%3A2404%2E09577%2E+%E2%80%94+%5BarXiv%3A2404%2E09577%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E09577%29)
19. T. C. Guetterman, T. Chang, M. DeJonckheere, T. Basu, E. Scruggs, and V. V. Vydiswaran. 2018. Augmenting qualitative text analysis with natural language processing: methodological study. Journal of Medical Internet Research, 20(6):e231. [find](https://scholar.google.com/scholar?q=T%2E+C%2E+Guetterman%2C+T%2E+Chang%2C+M%2E+DeJonckheere%2C+T%2E+Basu%2C+E%2E+Scruggs%2C+and+V%2E+V%2E+Vydiswaran%2E+2018%2E+Augmenting+qualitative+text+analysis+with+natural+language+processing%3A+methodological+study%2E+Journal+of+Me)
20. F. Viegas, A. Pereira, W. Cunha, C. França, C. Andrade, E. Tuler, and M. A. Gonçalves. 2025. Exploiting contextual embeddings in hierarchical topic modeling and investigating the limits of the current evaluation metrics. Computational Linguistics, pages 1–41. [find](https://scholar.google.com/scholar?q=F%2E+Viegas%2C+A%2E+Pereira%2C+W%2E+Cunha%2C+C%2E+Fran%C3%A7a%2C+C%2E+Andrade%2C+E%2E+Tuler%2C+and+M%2E+A%2E+Gon%C3%A7alves%2E+2025%2E+Exploiting+contextual+embeddings+in+hierarchical+topic+modeling+and+investigating+the+limits+of+the+current)
21. Y. Zhang and B. M. Wildemuth. 2009. Qualitative analysis of content. In Applications of social research methods to questions in information and library science, volume 308(319), pages 1–12. Libraries Unltd Inc. [find](https://scholar.google.com/scholar?q=Y%2E+Zhang+and+B%2E+M%2E+Wildemuth%2E+2009%2E+Qualitative+analysis+of+content%2E+In+Applications+of+social+research+methods+to+questions+in+information+and+library+science%2C+volume+308%28319%29%2C+pages+1%E2%80%9312%2E+Libraries+)

*Updated 1 November 2026*