---
title: "What Matters to an LLM? Behavioral and Computational"
description: "The authors study this from two sides. 55 references, cited by 1 paper here. Among them: 0."
authors: ["Yongxin Zhou", "Changshun Wu", "Philippe Mulhem", "Didier Schwab", "Maxime Peyrard"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.302/"
doi: "10.18653/v1/2026.findings-eacl.302"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2026-what-matters-llm-behavioral-computational/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2026-what-matters-llm-behavioral-computational/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/19/) / What Matters to an LLM? Behavioral and Computational Evidences from Summarization

# What Matters to an LLM? Behavioral and Computational Evidences from Summarization

**Authors:** [Yongxin Zhou](https://md.research.elamur.ai/free/en/authors/yongxin-zhou/), [Changshun Wu](https://md.research.elamur.ai/free/en/authors/changshun-wu/), [Philippe Mulhem](https://md.research.elamur.ai/free/en/authors/philippe-mulhem/), [Didier Schwab](https://md.research.elamur.ai/free/en/authors/didier-schwab/) and [Maxime Peyrard](https://md.research.elamur.ai/free/en/authors/maxime-peyrard/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.302/) · [PDF](https://aclanthology.org/2026.findings-eacl.302.pdf) · [DOI 10.18653/v1/2026.findings-eacl.302](https://doi.org/10.18653/v1/2026.findings-eacl.302) · [OpenAlex](https://openalex.org/W7140135281)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/zhou-2026-what-matters-llm-behavioral-computational/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs now lead in summarization, but it is unclear what internal sense of importance decides which content they keep. The authors study this from two sides. **Behaviorally**, they summarize each document ten times under length budgets from 10 to 100. A token's importance is the fraction of summaries that contain it. They apply this to 7 LLMs on CNN/DailyMail, SAMSum and DECODA-French. **Computationally**, they compare these distributions with aggregated attention weights. They also train low-capacity linear probes on hidden states, with two controls: probes on randomized-weight models (a "dead salmon" check) and a TextRank baseline.

LLMs largely agree with each other on what matters and depart clearly from pre-LLM methods. They group by model family more than by size. Certain attention heads, and especially middle-to-late layers, predict importance well. Predictability is highest for dialogues and lowest for news. The work is a first step toward interpreting, and later controlling, how LLMs select information.

## Key points
- Released resource: 274,330 summaries from 7 LLMs, preprocessed into token-level importance distributions. Code and data are on GitHub (yongxin2020/llm-importance-summarization).
- Importance is defined as how often a unit persists across 10 length-controlled summaries. The idea is related to Pyramid-style frequency counting.
- In article-level probing, probes on trained weights beat randomized-weight probes by about 0.1–0.2 NDCG@10 and consistently surpass TextRank. Performance peaks in middle-to-late layers.
- Importance encoding shows no consistent gain with model scale. Llama models outperform Qwen, and SAMSum is far more predictable than CNN/DailyMail.
- A case study on Qwen2.5-3B-Instruct shows named entities scoring high in news and participant names scoring high in dialogues.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 243 works

## Cite

Cite as: Yongxin Zhou, Changshun Wu, Philippe Mulhem, Didier Schwab and Maxime Peyrard. 2026. What Matters to an LLM? Behavioral and Computational Evidences from Summarization. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhou2026what,
  title = {{What Matters to an LLM? Behavioral and Computational Evidences from Summarization}},
  author = {Yongxin Zhou, Changshun Wu, Philippe Mulhem, Didier Schwab and Maxime Peyrard},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.302},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.302}
}
```

- [← previous What Makes LLMs Effective Sequential Recommenders? A Study on Preference Intensity and Temporal Context](https://md.research.elamur.ai/free/en/papers/ouyang-2026-what-makes-llms-effective-sequential/)
- [next → When AI Writes, Whose Voice Remains? Quantifying Cultural Marker Erasure Across World English Varieties in Large Language Models](https://md.research.elamur.ai/free/en/papers/navneet-2026-ai-writes-whose-voice-remains/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/19/)

## References

55 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI@Meta. 2024. Llama models card. [find](https://scholar.google.com/scholar?q=AI%40Meta%2E+2024%2E+Llama+models+card%2E)
2. Jie Bao, Prithwish Basu, Mike Dean, Craig Partridge, Ananthram Swami, Will Leland, and James A. Hendler. 2011. Towards a theory of semantic communication. 2011 IEEE Network Science Workshop, pages 110–117. [find](https://scholar.google.com/scholar?q=Jie+Bao%2C+Prithwish+Basu%2C+Mike+Dean%2C+Craig+Partridge%2C+Ananthram+Swami%2C+Will+Leland%2C+and+James+A%2E+Hendler%2E+2011%2E+Towards+a+theory+of+semantic+communication%2E+2011+IEEE+Network+Science+Workshop%2C+pages+110)
3. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Collin+Burns%2C+Haotian+Ye%2C+Dan+Klein%2C+and+Jacob+Steinhardt%2E+2023%2E+Discovering+latent+knowledge+in+language+models+without+supervision%2E+In+The+Eleventh+International+Conference+on+Learning+Representatio)
4. Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. 2020. Thread: Circuits. Distill. Https://distill.pub/2020/circuits. [find](https://scholar.google.com/scholar?q=Nick+Cammarata%2C+Shan+Carter%2C+Gabriel+Goh%2C+Chris+Olah%2C+Michael+Petrov%2C+Ludwig+Schubert%2C+Chelsea+Voss%2C+Ben+Egan%2C+and+Swee+Kiat+Lim%2E+2020%2E+Thread%3A+Circuits%2E+Distill%2E+Https%3A%2F%2Fdistill%2Epub%2F2020%2Fcircuits%2E)
5. Rudolf Carnap and Yehoshua Bar-Hillel. 1952. An outline of a theory of semantic information. [find](https://scholar.google.com/scholar?q=Rudolf+Carnap+and+Yehoshua+Bar%2DHillel%2E+1952%2E+An+outline+of+a+theory+of+semantic+information%2E)
6. Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. 2019. Activation atlas. Distill. [find](https://scholar.google.com/scholar?q=Shan+Carter%2C+Zan+Armstrong%2C+Ludwig+Schubert%2C+Ian+Johnson%2C+and+Chris+Olah%2E+2019%2E+Activation+atlas%2E+Distill%2E)
7. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
8. Zhichen Dong, Zhanhui Zhou, Zhixuan Liu, Chao Yang, and Chaochao Lu. 2025. Emergent response planning in LLMs. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Zhichen+Dong%2C+Zhanhui+Zhou%2C+Zhixuan+Liu%2C+Chao+Yang%2C+and+Chaochao+Lu%2E+2025%2E+Emergent+response+planning+in+LLMs%2E+In+Forty%2Dsecond+International+Conference+on+Machine+Learning%2E)
9. Benoit Favre, Evgeny Stepanov, Jérémy Trione, Frédéric Béchet, and Giuseppe Riccardi. 2015. Call centre conversation summarization: A pilot task at multiling 2015. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 232–236, Prague, Czech Republic… [find](https://scholar.google.com/scholar?q=Benoit+Favre%2C+Evgeny+Stepanov%2C+J%C3%A9r%C3%A9my+Trione%2C+Fr%C3%A9d%C3%A9ric+B%C3%A9chet%2C+and+Giuseppe+Riccardi%2E+2015%2E+Call+centre+conversation+summarization%3A+A+pilot+task+at+multiling+2015%2E+In+Proceedings+of+the+16th+Annual+Me)
10. Mohamed Imed Eddine Ghebriout, Gaël Guibon, Ivan Lerner, and Emmanuel Vincent. 2025. QUARTZ: QA-based unsupervised abstractive refinement for task-oriented dialogue summarization. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 14689–14706, Suzhou, China. Association… [find](https://scholar.google.com/scholar?q=Mohamed+Imed+Eddine+Ghebriout%2C+Ga%C3%ABl+Guibon%2C+Ivan+Lerner%2C+and+Emmanuel+Vincent%2E+2025%2E+QUARTZ%3A+QA%2Dbased+unsupervised+abstractive+refinement+for+task%2Doriented+dialogue+summarization%2E+In+Findings+of+the+A)
11. Daniel Gillick, Benoit Favre, and Dilek Hakkani-Tür. 2008. The icsi summarization system at tac 2008. In Tac. [find](https://scholar.google.com/scholar?q=Daniel+Gillick%2C+Benoit+Favre%2C+and+Dilek+Hakkani%2DT%C3%BCr%2E+2008%2E+The+icsi+summarization+system+at+tac+2008%2E+In+Tac%2E)
12. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. [SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization](https://md.research.elamur.ai/free/en/papers/gliwa-2019-samsum-corpus-human-annotated-dialogue/). In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.
13. Tanya Goyal and Greg Durrett. 2021. [Annotating and modeling fine-grained factuality in summarization](https://md.research.elamur.ai/free/en/papers/goyal-2021-annotating-modeling-fine-grained-factuality/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1449–1462, Online. Association for Computatio…
14. Tianjie Ju, Zhenyu Shao, Bowen Wang, Yujia Chen, Zhuosheng Zhang, Hao Fei, Mong-Li Lee, Wynne Hsu, Sufeng Duan, and Gongshen Liu. 2025. Probing then editing response personality of large language models. In Second Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Tianjie+Ju%2C+Zhenyu+Shao%2C+Bowen+Wang%2C+Yujia+Chen%2C+Zhuosheng+Zhang%2C+Hao+Fei%2C+Mong%2DLi+Lee%2C+Wynne+Hsu%2C+Sufeng+Duan%2C+and+Gongshen+Liu%2E+2025%2E+Probing+then+editing+response+personality+of+large+language+mode)
15. Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018. [Content selection in deep learning models of summarization](https://md.research.elamur.ai/free/en/papers/kedzie-2018-content-selection-deep-learning-models/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818–1828, Brussels, Belgium. Association for Computational Linguistics.
16. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. [Attention is not only a weight: Analyzing transformers with vector norms](https://md.research.elamur.ai/free/en/papers/kobayashi-2020-attention-is-not-only-weight/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057–7075, Online. Association for Computational…
17. Abhinav Kumar, Chenhao Tan, and Amit Sharma. 2022. Probing classifiers are unreliable for concept removal and detection. Advances in Neural Information Processing Systems, 35:17994–18008. [find](https://scholar.google.com/scholar?q=Abhinav+Kumar%2C+Chenhao+Tan%2C+and+Amit+Sharma%2E+2022%2E+Probing+classifiers+are+unreliable+for+concept+removal+and+detection%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A17994%E2%80%9318008%2E)
18. Chen Li, Xian Qian, and Yang Liu. 2013. Using supervised bigram-based ILP for extractive summarization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1004–1013, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chen+Li%2C+Xian+Qian%2C+and+Yang+Liu%2E+2013%2E+Using+supervised+bigram%2Dbased+ILP+for+extractive+summarization%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volu)
19. Raymond Li, Wen Xiao, Linzi Xing, Lanjun Wang, Gabriel Murray, and Giuseppe Carenini. 2022. Human guided exploitation of interpretable attention patterns in summarization and topic segmentation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10189–10… [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Wen+Xiao%2C+Linzi+Xing%2C+Lanjun+Wang%2C+Gabriel+Murray%2C+and+Giuseppe+Carenini%2E+2022%2E+Human+guided+exploitation+of+interpretable+attention+patterns+in+summarization+and+topic+segmentation%2E+In+Pr)
20. Wei Li, Lei He, and Hai Zhuge. 2016. Abstractive news summarization based on event semantic link network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 236–246, Osaka, Japan. [find](https://scholar.google.com/scholar?q=Wei+Li%2C+Lei+He%2C+and+Hai+Zhuge%2E+2016%2E+Abstractive+news+summarization+based+on+event+semantic+link+network%2E+In+Proceedings+of+COLING+2016%2C+the+26th+International+Conference+on+Computational+Linguistics%3A)
21. The COLING 2016 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2016+Organizing+Committee%2E)
22. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
23. Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of research and development, 2(2):159–165. [find](https://scholar.google.com/scholar?q=Hans+Peter+Luhn%2E+1958%2E+The+automatic+creation+of+literature+abstracts%2E+IBM+Journal+of+research+and+development%2C+2%282%29%3A159%E2%80%93165%2E)
24. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. [On faithfulness and factuality in abstractive summarization](https://md.research.elamur.ai/free/en/papers/maynez-2020-faithfulness-factuality-abstractive-summarization/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.
25. Ryan McDonald. 2007. A study of global inference algorithms in multi-document summarization. In Proceedings of the 29th European Conference on IR Research, ECIR’07, page 557–564, Berlin, Heidelberg. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Ryan+McDonald%2E+2007%2E+A+study+of+global+inference+algorithms+in+multi%2Ddocument+summarization%2E+In+Proceedings+of+the+29th+European+Conference+on+IR+Research%2C+ECIR%E2%80%9907%2C+page+557%E2%80%93564%2C+Berlin%2C+Heidelberg%2E+S)
26. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36. ArXiv:2202.05262. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+GPT%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2E+ArXiv%3A2202%2E05262%2E)
27. Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404–411, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea+and+Paul+Tarau%2E+2004%2E+TextRank%3A+Bringing+order+into+text%2E+In+Proceedings+of+the+2004+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+404%E2%80%93411%2C+Barcelona%2C+Spain%2E+Asso)
28. Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kiciman, Hamid Palangi, Barun Patra, and Robert West. 2024. A glitch in the matrix? locating and detecting language model grounding with fakepedia. In Proceedings of the 62nd Annual Meeting of the Association for… [find](https://scholar.google.com/scholar?q=Giovanni+Monea%2C+Maxime+Peyrard%2C+Martin+Josifoski%2C+Vishrav+Chaudhary%2C+Jason+Eisner%2C+Emre+Kiciman%2C+Hamid+Palangi%2C+Barun+Patra%2C+and+Robert+West%2E+2024%2E+A+glitch+in+the+matrix%3F+locating+and+detecting+langu)
29. Maxime Méloux, Giada Dirupo, François Portet, and Maxime Peyrard. 2025. The dead salmons of ai interpretability. Preprint, arXiv:2512.18792. — [arXiv:2512.18792](https://arxiv.org/abs/2512.18792) [find](https://scholar.google.com/scholar?q=Maxime+M%C3%A9loux%2C+Giada+Dirupo%2C+Fran%C3%A7ois+Portet%2C+and+Maxime+Peyrard%2E+2025%2E+The+dead+salmons+of+ai+interpretability%2E+Preprint%2C+arXiv%3A2512%2E18792%2E+%E2%80%94+%5BarXiv%3A2512%2E18792%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2512%2E18792%29)
30. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. As…
31. Ani Nenkova and Kathleen McKeown. 2012. A Survey of Text Summarization Techniques, pages 43–76. Springer US, Boston, MA. [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Kathleen+McKeown%2E+2012%2E+A+Survey+of+Text+Summarization+Techniques%2C+pages+43%E2%80%9376%2E+Springer+US%2C+Boston%2C+MA%2E)
32. Ani Nenkova, Rebecca Passonneau, and Kathleen McKeown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Trans. Speech Lang. Process., 4(2):4–es. [find](https://scholar.google.com/scholar?q=Ani+Nenkova%2C+Rebecca+Passonneau%2C+and+Kathleen+McKeown%2E+2007%2E+The+pyramid+method%3A+Incorporating+human+content+selection+variation+in+summarization+evaluation%2E+ACM+Trans%2E+Speech+Lang%2E+Process%2E%2C+4%282%29%3A4%E2%80%93e)
33. Ani Nenkova and Lucy Vanderwende. 2005. The impact of frequency on summarization. Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005, 101. [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Lucy+Vanderwende%2E+2005%2E+The+impact+of+frequency+on+summarization%2E+Microsoft+Research%2C+Redmond%2C+Washington%2C+Tech%2E+Rep%2E+MSR%2DTR%2D2005%2C+101%2E)
34. Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. 2025. [LLMs know more than they show: On the intrinsic representation of LLM hallucinations](https://md.research.elamur.ai/free/en/papers/orgad-2024-llms-know-more-than-they/). In The Thirteenth International Conference on Learning Representations.
35. Maxime Peyrard. 2019. A simple theoretical model of importance for summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1059–1073, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maxime+Peyrard%2E+2019%2E+A+simple+theoretical+model+of+importance+for+summarization%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1059%E2%80%931073%2C+Florence%2C)
36. Maxime Peyrard and Judith Eckle-Kohler. 2016. Optimizing an approximation of ROUGE - a problem-reduction approach to extractive multi-document summarization. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1825–1836, Berlin, G… [find](https://scholar.google.com/scholar?q=Maxime+Peyrard+and+Judith+Eckle%2DKohler%2E+2016%2E+Optimizing+an+approximation+of+ROUGE+%2D+a+problem%2Dreduction+approach+to+extractive+multi%2Ddocument+summarization%2E+In+Proceedings+of+the+54th+Annual+Meeting+)
37. Sanjana Ramprasad, Elisa Ferracane, and Zachary Lipton. 2024. Analyzing LLM behavior in dialogue summarization: Unveiling circumstantial hallucination trends. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12549–12561, Bangko… [find](https://scholar.google.com/scholar?q=Sanjana+Ramprasad%2C+Elisa+Ferracane%2C+and+Zachary+Lipton%2E+2024%2E+Analyzing+LLM+behavior+in+dialogue+summarization%3A+Unveiling+circumstantial+hallucination+trends%2E+In+Proceedings+of+the+62nd+Annual+Meeting)
38. Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. 2021. [Probing the probing paradigm: Does probing accuracy entail task relevance?](https://md.research.elamur.ai/free/en/papers/ravichander-2021-probing-probing-paradigm-does-probing/) In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3363–3377, Online. Association…
39. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Compu… [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computati)
40. Julius Steen and Katja Markert. 2024. Bias in news summarization: Measures, pitfalls and corpora. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5962–5983, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Julius+Steen+and+Katja+Markert%2E+2024%2E+Bias+in+news+summarization%3A+Measures%2C+pitfalls+and+corpora%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2024%2C+pages+5962%E2%80%935983%2C+Bangkok%2C+Thai)
41. Xu Sun, Lionel Delphin-Poulat, Christèle Tarnec, and Anastasia Shimorina. 2025. PoSum-bench: Benchmarking position bias in LLM-based conversational summarization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 7996–8020, Suzhou, China. Association fo… [find](https://scholar.google.com/scholar?q=Xu+Sun%2C+Lionel+Delphin%2DPoulat%2C+Christ%C3%A8le+Tarnec%2C+and+Anastasia+Shimorina%2E+2025%2E+PoSum%2Dbench%3A+Benchmarking+position+bias+in+LLM%2Dbased+conversational+summarization%2E+In+Proceedings+of+the+2025+Conference)
42. Qwen Team. 2024. Qwen2.5: A party of foundation models. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2E)
43. Damien Teney, Maxime Peyrard, and Ehsan Abbasnejad. 2022. Predicting is not understanding: Recognizing and addressing underspecification in machine learning. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIII, page 458–476, Berlin… [find](https://scholar.google.com/scholar?q=Damien+Teney%2C+Maxime+Peyrard%2C+and+Ehsan+Abbasnejad%2E+2022%2E+Predicting+is+not+understanding%3A+Recognizing+and+addressing+underspecification+in+machine+learning%2E+In+Computer+Vision+%E2%80%93+ECCV+2022%3A+17th+Europ)
44. Jan Trienes, Jörg Schlötterer, Junyi Jessy Li, and Christin Seifert. 2025. Behavioral analysis of information salience in large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 23428–23454, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jan+Trienes%2C+J%C3%B6rg+Schl%C3%B6tterer%2C+Junyi+Jessy+Li%2C+and+Christin+Seifert%2E+2025%2E+Behavioral+analysis+of+information+salience+in+large+language+models%2E+In+Findings+of+the+Association+for+Computational+Lingui)
45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
46. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. [Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, I…
47. David Wan, Jesse Vig, Mohit Bansal, and Shafiq Joty. 2025. On positional bias of faithfulness for long-form summarization. In Proceedings of the 2025 [find](https://scholar.google.com/scholar?q=David+Wan%2C+Jesse+Vig%2C+Mohit+Bansal%2C+and+Shafiq+Joty%2E+2025%2E+On+positional+bias+of+faithfulness+for+long%2Dform+summarization%2E+In+Proceedings+of+the+2025)
48. Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8791–8810, Albuquerque, New Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Conference+of+the+Nations+of+the+Americas+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technologies+%28Volume+1%3A+Long+Papers%29%2C+pages+8791%E2%80%938810%2C+Albuquerque%2C+New+Mexico%2E+Assoc)
49. Linzi Xing, Wen Xiao, and Giuseppe Carenini. 2021. Demoting the lead bias in news summarization via alternating adversarial learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing… [find](https://scholar.google.com/scholar?q=Linzi+Xing%2C+Wen+Xiao%2C+and+Giuseppe+Carenini%2E+2021%2E+Demoting+the+lead+bias+in+news+summarization+via+alternating+adversarial+learning%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+C)
50. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024. [Benchmarking large language models for news summarization](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/). Transactions of the Association for Computational Linguistics, 12:39–57.
51. Yixin Zhong. 2017. A theory of semantic information. Proceedings, 1(3). [find](https://scholar.google.com/scholar?q=Yixin+Zhong%2E+2017%2E+A+theory+of+semantic+information%2E+Proceedings%2C+1%283%29%2E)
52. Yongxin Zhou, François Portet, and Fabien Ringeval. 2022. Effectiveness of French language models on abstractive dialogue summarization task. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3571–3581, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Yongxin+Zhou%2C+Fran%C3%A7ois+Portet%2C+and+Fabien+Ringeval%2E+2022%2E+Effectiveness+of+French+language+models+on+abstractive+dialogue+summarization+task%2E+In+Proceedings+of+the+Thirteenth+Language+Resources+and+Ev)
53. Yongxin Zhou, Fabien Ringeval, and François Portet. 2024. PSentScore: Evaluating sentiment polarity in dialogue summarization. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13290–13302, Torino, Ital… [find](https://scholar.google.com/scholar?q=Yongxin+Zhou%2C+Fabien+Ringeval%2C+and+Fran%C3%A7ois+Portet%2E+2024%2E+PSentScore%3A+Evaluating+sentiment+polarity+in+dialogue+summarization%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Computationa)
54. Yongxin Zhou, Fabien Ringeval, and François Portet. 2025. Can GPT models follow human summarization guidelines? a study for targeted communication goals. In Proceedings of the 18th International Natural Language Generation Conference, pages 249– 273, Hanoi, Vietnam. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Yongxin+Zhou%2C+Fabien+Ringeval%2C+and+Fran%C3%A7ois+Portet%2E+2025%2E+Can+GPT+models+follow+human+summarization+guidelines%3F+a+study+for+targeted+communication+goals%2E+In+Proceedings+of+the+18th+International+Natur)
55. Markus Zopf, Eneldo Loza Mencía, and Johannes Fürnkranz. 2016. Beyond centrality and structural features: Learning information importance for text summarization. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 84–94, Berlin, Germany. Association for Com… [find](https://scholar.google.com/scholar?q=Markus+Zopf%2C+Eneldo+Loza+Menc%C3%ADa%2C+and+Johannes+F%C3%BCrnkranz%2E+2016%2E+Beyond+centrality+and+structural+features%3A+Learning+information+importance+for+text+summarization%2E+In+Proceedings+of+the+20th+SIGNLL+Conf)

*Updated 2 November 2026*