---
title: "Beyond Math — paper by Jiang and Ferraro (2026)"
description: "The authors argue that story understanding works differently. 60 references, cited by 7 papers here."
authors: ["Yuxuan Jiang", "Francis Ferraro"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.eacl-long.261/"
doi: "10.18653/v1/2026.eacl-long.261"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jiang-2026-beyond-math-stories-as-testbed/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jiang-2026-beyond-math-stories-as-testbed/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs

# Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs

**Authors:** [Yuxuan Jiang](https://md.research.elamur.ai/free/en/authors/yuxuan-jiang/) and [Francis Ferraro](https://md.research.elamur.ai/free/en/authors/francis-ferraro/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.261/) · [PDF](https://aclanthology.org/2026.eacl-long.261.pdf) · [DOI 10.18653/v1/2026.eacl-long.261](https://doi.org/10.18653/v1/2026.eacl-long.261) · [OpenAlex](https://openalex.org/W7140122471)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/jiang-2026-beyond-math-stories-as-testbed/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Research on memorization in LLMs has mostly focused on math and logic, where recalling previously seen problems is treated as a pure shortcut. The authors argue that story understanding works differently. There, models should use background knowledge, but selectively and together with the narrative at hand. They borrow the cognitive-science split between verbatim and gist memory and build a two-level intervention scheme:

- **Inductive Setting:** the input is left unchanged, and gist-oriented prompts nudge the model toward context-driven recall.
- **Restrictive Setting:** character names are swapped for culturally distinct alternatives, which blocks exact recall.

GPT-4o, LLaMA 3.3-70B and DeepSeek V3 are tested on CharBench, a collection of six character-centric benchmarks. A new dataset, CharScript, checks how familiar the models are with the underlying fiction. Name replacement cuts accuracy by up to roughly 45%, while inductive prompting keeps scores largely stable. The results suggest that current benchmarks overstate narrative reasoning. They also show that prompting can steer models toward bounded, context-relevant use of memory.

## Key points
- **CharBench** combines MovieCoref, TVSHOWGUESS, PERSONET, CSI Corpus, FriendsQA and SummScreen. It keeps their original splits and metrics (exact match, F1, ROUGE) and covers TV shows, movies and novels.
- **CharScript** contains 200 passages averaging about 312 words. They come from 40 fictional sources (11 TV series, 9 movies, 20 novels), with 5 per source, and models must attribute each passage to its source as a test of verbatim memorization.
- **Size of the drop:** the abstract gives a maximum Restrictive-Setting loss of 45.2%, while the introduction says 45.8%. For every setting, the authors repeated each model's evaluation three times and averaged the scores, and the total API cost was about $100.
- **Task type matters:** action-oriented tasks (Character Guessing, CSI role detection) lose more under the combined GIST+NR intervention than motivation-oriented ones (PERSONET, FriendsQA). On some benchmarks, constraining memory even helps by reducing knowledge conflicts.
- **Stated limitations:** sources are mostly Western popular culture, which may exaggerate the effects. Results may be sensitive to prompt wording, and GPT-4o's training data cannot be verified.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Maryland, Baltimore County](https://md.research.elamur.ai/free/en/universities/university-of-maryland-baltimore-county/) — University, Baltimore, United States · 14 works · 915 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Yuxuan Jiang and Francis Ferraro. 2026. Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jiang2026beyond,
  title = {{Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs}},
  author = {Yuxuan Jiang and Francis Ferraro},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.261},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.261}
}
```

- [← previous Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward](https://md.research.elamur.ai/free/en/papers/guo-2026-backdoors-rlvr-jailbreak-backdoors-llms/)
- [next → BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attacks](https://md.research.elamur.ai/free/en/papers/miao-2026-blindguard-safeguarding-llm-based-multi/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/)

## References

60 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Meta AI. 2024. Llama 3.3 model card. [https://github.com/meta-llama/llama-models/](https://github.com/meta-llama/llama-models/) blob/main/models/llama3_3/MODEL_CARD.md. Accessed: 2025-02-09. [↗](https://github.com/meta-llama/llama-models/)
2. Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. [Llm in a flash: Efficient large language model inference with limited memory](https://md.research.elamur.ai/free/en/papers/alizadeh-2024-llm-flash-efficient-large-language/). In Proceedings of the 62nd Annual Meeting of the Association for Computa…
3. David Bamman, Olivia Lewke, and Anya Mansoor. 2020. [An Annotated Dataset of Coreference in English Literature](https://md.research.elamur.ai/free/en/papers/bamman-2020-annotated-dataset-coreference-english-literature/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 44–54, Marseille, France. European Language Resources Association.
4. Sabyasachee Baruah and Shrikanth Narayanan. 2023. [Character coreference resolution in movie screenplays](https://md.research.elamur.ai/free/en/papers/baruah-2023-character-coreference-resolution-movie-screenplays/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 10300–10313.
5. Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu [https://chatgpt.com/](https://chatgpt.com/) [↗](https://chatgpt.com/)
6. Purohit, and Edward Raff. 2024. [Emergent and predictable memorization in large language models](https://md.research.elamur.ai/free/en/papers/biderman-2023-emergent-predictable-memorization-large-language/). Advances in Neural Information Processing Systems, 36.
7. Terra Blevins and Luke Zettlemoyer. 2022. Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models. arXiv preprint. ArXiv:2204.08110. [find](https://scholar.google.com/scholar?q=Terra+Blevins+and+Luke+Zettlemoyer%2E+2022%2E+Language+Contamination+Helps+Explain+the+Cross%2Dlingual+Capabilities+of+English+Pretrained+Models%2E+arXiv+preprint%2E+ArXiv%3A2204%2E08110%2E)
8. Lea Frermann, Shay B. Cohen, and Mirella Lapata. 2018. [Whodunnit? Crime Drama as a Case for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/frermann-2018-whodunnit-crime-drama-as-case/). Transactions of the Association for Computational Linguistics, 6:1–15. Place: Cambridge, MA Publisher: MIT Press.
9. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2023. News Summarization and Evaluation in the Era of GPT-3. arXiv preprint. ArXiv:2209.12356 [cs]. [find](https://scholar.google.com/scholar?q=Tanya+Goyal%2C+Junyi+Jessy+Li%2C+and+Greg+Durrett%2E+2023%2E+News+Summarization+and+Evaluation+in+the+Era+of+GPT%2D3%2E+arXiv+preprint%2E+ArXiv%3A2209%2E12356+%5Bcs%5D%2E)
10. Gordon H Bower and Daniel G Morrow. 1990. Mental models in narrative comprehension. Science, 247(4938):44–48. [find](https://scholar.google.com/scholar?q=Gordon+H+Bower+and+Daniel+G+Morrow%2E+1990%2E+Mental+models+in+narrative+comprehension%2E+Science%2C+247%284938%29%3A44%E2%80%9348%2E)
11. Yuxuan Jiang and Francis Ferraro. 2026. Scribe: Structured mid-level supervision for tool-using language models. arXiv preprint arXiv:2601.03555. — [arXiv:2601.03555](https://arxiv.org/abs/2601.03555) [find](https://scholar.google.com/scholar?q=Yuxuan+Jiang+and+Francis+Ferraro%2E+2026%2E+Scribe%3A+Structured+mid%2Dlevel+supervision+for+tool%2Dusing+language+models%2E+arXiv+preprint+arXiv%3A2601%2E03555%2E+%E2%80%94+%5BarXiv%3A2601%2E03555%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2601%2E03555%29)
12. Faeze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Mrinmaya Sachan, and Snigdha Chaturvedi. 2021. "Let Your Characters Tell Their Story": A Dataset for Character-Centric Narrative Understanding. arXiv preprint. ArXiv:2109.05438 [cs]. [find](https://scholar.google.com/scholar?q=Faeze+Brahman%2C+Meng+Huang%2C+Oyvind+Tafjord%2C+Chao+Zhao%2C+Mrinmaya+Sachan%2C+and+Snigdha+Chaturvedi%2E+2021%2E+%22Let+Your+Characters+Tell+Their+Story%22%3A+A+Dataset+for+Character%2DCentric+Narrative+Understanding%2E+ar)
13. Yuxuan Jiang, Dawei Li, and Frank Ferraro. 2025. [Drp: Distilled reasoning pruning with skill-aware step decomposition for efficient large reasoning models](https://md.research.elamur.ai/free/en/papers/jiang-2025-drp-distilled-reasoning-pruning-skill/). arXiv preprint arXiv:2505.13975.
14. Charles J Brainerd and Valerie F Reyna. 2002. Fuzzy-trace theory: Dual processes in memory, reasoning, and cognitive neuroscience. Advances in child development and behavior, 28:41–100. [find](https://scholar.google.com/scholar?q=Charles+J+Brainerd+and+Valerie+F+Reyna%2E+2002%2E+Fuzzy%2Dtrace+theory%3A+Dual+processes+in+memory%2C+reasoning%2C+and+cognitive+neuroscience%2E+Advances+in+child+development+and+behavior%2C+28%3A41%E2%80%93100%2E)
15. Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, and Yongfeng Zhang. 2024. [Disentangling memory and reasoning ability in large language models](https://md.research.elamur.ai/free/en/papers/jin-2025-disentangling-memory-reasoning-ability-large/). arXiv preprint arXiv:2411.13504.
16. Nicholas Carlini, Daphne Ippolito, Katherine Lee, Florian Tramèr, Matthew Jagielski, and Chiyuan Zhang. 2023. QUANTIFYING MEMORIZATION ACROSS NEURAL LANGUAGE MODELS. arXiv preprint arXiv:2202.07646. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Daphne+Ippolito%2C+Katherine+Lee%2C+Florian+Tram%C3%A8r%2C+Matthew+Jagielski%2C+and+Chiyuan+Zhang%2E+2023%2E+QUANTIFYING+MEMORIZATION+ACROSS+NEURAL+LANGUAGE+MODELS%2E+arXiv+preprint+arXiv%3A2202%2E07646%2E)
17. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through Memorization: Nearest Neighbor Language Models. arXiv preprint. ArXiv:1911.00172. [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+Omer+Levy%2C+Dan+Jurafsky%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2020%2E+Generalization+through+Memorization%3A+Nearest+Neighbor+Language+Models%2E+arXiv+preprint%2E+ArXiv%3A1911%2E00172%2E)
18. Tommaso Caselli and Piek Vossen. 2017. The Event StoryLine Corpus: A New Benchmark for Causal and Temporal Relation Extraction. In Proceedings of the Events and Stories in the News Workshop, pages 77–86, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tommaso+Caselli+and+Piek+Vossen%2E+2017%2E+The+Event+StoryLine+Corpus%3A+A+New+Benchmark+for+Causal+and+Temporal+Relation+Extraction%2E+In+Proceedings+of+the+Events+and+Stories+in+the+News+Workshop%2C+pages+77%E2%80%93)
19. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
20. Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023a. [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://md.research.elamur.ai/free/en/papers/chang-2023-speak-memory-archaeology-books-known/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7312– 7327, Singapore. Association for Computational Linguistics.
21. Kent K Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023b. [Speak, memory: An archaeology of books known to chatgpt/gpt-4](https://md.research.elamur.ai/free/en/papers/chang-2023-speak-memory-archaeology-books-known/). arXiv preprint arXiv:2305.00118.
22. Bowen Chen, Namgi Han, and Yusuke Miyao. 2024. A multi-perspective analysis of memorization in large language models. arXiv preprint arXiv:2405.11577. [find](https://scholar.google.com/scholar?q=Bowen+Chen%2C+Namgi+Han%2C+and+Yusuke+Miyao%2E+2024%2E+A+multi%2Dperspective+analysis+of+memorization+in+large+language+models%2E+arXiv+preprint+arXiv%3A2405%2E11577%2E)
23. Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. [SummScreen: A Dataset for Abstractive Screenplay Summarization](https://md.research.elamur.ai/free/en/papers/chen-2022-summscreen-dataset-abstractive-screenplay-summarization/). arXiv preprint. ArXiv:2104.07091 [cs].
24. Yu-Hsin Chen and Jinho D Choi. 2016. Character identification on multiparty conversation: Identifying mentions of characters in tv shows. In Proceedings of the 17th annual meeting of the special interest group on discourse and dialogue, pages 90–100. [find](https://scholar.google.com/scholar?q=Yu%2DHsin+Chen+and+Jinho+D+Choi%2E+2016%2E+Character+identification+on+multiparty+conversation%3A+Identifying+mentions+of+characters+in+tv+shows%2E+In+Proceedings+of+the+17th+annual+meeting+of+the+special+inter)
25. Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. 2024. Causal estimation of memorisation profiles. arXiv preprint arXiv:2406.04327. [find](https://scholar.google.com/scholar?q=Pietro+Lesci%2C+Clara+Meister%2C+Thomas+Hofmann%2C+Andreas+Vlachos%2C+and+Tiago+Pimentel%2E+2024%2E+Causal+estimation+of+memorisation+profiles%2E+arXiv+preprint+arXiv%3A2406%2E04327%2E)
26. Bo Li, Qinghua Zhao, and Lijie Wen. 2024. Rome: Memorization insights from text, logits and representation. Preprint, arXiv:2403.00510. — [arXiv:2403.00510](https://arxiv.org/abs/2403.00510) [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Qinghua+Zhao%2C+and+Lijie+Wen%2E+2024%2E+Rome%3A+Memorization+insights+from+text%2C+logits+and+representation%2E+Preprint%2C+arXiv%3A2403%2E00510%2E+%E2%80%94+%5BarXiv%3A2403%2E00510%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E00510%29)
27. Dawei Li, Hengyuan Zhang, Yanran Li, and Shiping Yang. 2023. Multi-level contrastive learning for script-based character understanding. arXiv preprint arXiv:2310.13231. [find](https://scholar.google.com/scholar?q=Dawei+Li%2C+Hengyuan+Zhang%2C+Yanran+Li%2C+and+Shiping+Yang%2E+2023%2E+Multi%2Dlevel+contrastive+learning+for+script%2Dbased+character+understanding%2E+arXiv+preprint+arXiv%3A2310%2E13231%2E)
28. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+et+al%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E19437%2E)
29. Robert McKee. 2010. Story: style, structure, substance, and the principles of screenwriting. Harper Collins. [find](https://scholar.google.com/scholar?q=Robert+McKee%2E+2010%2E+Story%3A+style%2C+structure%2C+substance%2C+and+the+principles+of+screenwriting%2E+Harper+Collins%2E)
30. OpenAI. 2023. Chatgpt: Optimizing language models for dialogue. [https://openai.com/research/](https://openai.com/research/) chatgpt. Accessed: 2025-02-09. [↗](https://openai.com/research/)
31. Alison H Paris and Scott G Paris. 2003. Assessing narrative comprehension in young children. Reading Research Quarterly, 38(1):36–76. [find](https://scholar.google.com/scholar?q=Alison+H+Paris+and+Scott+G+Paris%2E+2003%2E+Assessing+narrative+comprehension+in+young+children%2E+Reading+Research+Quarterly%2C+38%281%29%3A36%E2%80%9376%2E)
32. Akshara Prabhakar, Thomas L Griffiths, and R Thomas McCoy. 2024. Deciphering the factors influencing the efficacy of chain-of-thought: Probability, memorization, and noisy reasoning. arXiv preprint arXiv:2407.01687. [find](https://scholar.google.com/scholar?q=Akshara+Prabhakar%2C+Thomas+L+Griffiths%2C+and+R+Thomas+McCoy%2E+2024%2E+Deciphering+the+factors+influencing+the+efficacy+of+chain%2Dof%2Dthought%3A+Probability%2C+memorization%2C+and+noisy+reasoning%2E+arXiv+preprint+ar)
33. USVSN Sai Prashanth, Alvin Deng, Kyle O’Brien, Jyothir SV, Mohammad Aflah Khan, Jaydeep Borkar, Christopher A Choquette-Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, et al. 2024. Recite, reconstruct, recollect: Memorization in lms as a multifaceted phenomenon. arXiv preprint arXiv:2406.17746. — [arXiv:2406.17746](https://arxiv.org/abs/2406.17746) [find](https://scholar.google.com/scholar?q=USVSN+Sai+Prashanth%2C+Alvin+Deng%2C+Kyle+O%E2%80%99Brien%2C+Jyothir+SV%2C+Mohammad+Aflah+Khan%2C+Jaydeep+Borkar%2C+Christopher+A+Choquette%2DChoo%2C+Jacob+Ray+Fuehne%2C+Stella+Biderman%2C+Tracy+Ke%2C+et+al%2E+2024%2E+Recite%2C+reconstr)
34. Valerie F Reyna and Charles J Brainerd. 1998. Fuzzy-trace theory and false memory: New frontiers. Journal of experimental child psychology, 71(2):194–209. [find](https://scholar.google.com/scholar?q=Valerie+F+Reyna+and+Charles+J+Brainerd%2E+1998%2E+Fuzzy%2Dtrace+theory+and+false+memory%3A+New+frontiers%2E+Journal+of+experimental+child+psychology%2C+71%282%29%3A194%E2%80%93209%2E)
35. Yisi Sang, Xiangyang Mou, Jing Li, Jeffrey Stanton, and Mo Yu. 2022a. A survey of machine narrative reading comprehension assessments. arXiv preprint arXiv:2205.00299. — [arXiv:2205.00299](https://arxiv.org/abs/2205.00299) [find](https://scholar.google.com/scholar?q=Yisi+Sang%2C+Xiangyang+Mou%2C+Jing+Li%2C+Jeffrey+Stanton%2C+and+Mo+Yu%2E+2022a%2E+A+survey+of+machine+narrative+reading+comprehension+assessments%2E+arXiv+preprint+arXiv%3A2205%2E00299%2E+%E2%80%94+%5BarXiv%3A2205%2E00299%5D%28https%3A%2F%2Farx)
36. Yisi Sang, Xiangyang Mou, Mo Yu, Shunyu Yao, Jing Li, and Jeffrey Stanton. 2022b. Tvshowguess: Character comprehension in stories as speaker guessing. arXiv preprint arXiv:2204.07721. [find](https://scholar.google.com/scholar?q=Yisi+Sang%2C+Xiangyang+Mou%2C+Mo+Yu%2C+Shunyu+Yao%2C+Jing+Li%2C+and+Jeffrey+Stanton%2E+2022b%2E+Tvshowguess%3A+Character+comprehension+in+stories+as+speaker+guessing%2E+arXiv+preprint+arXiv%3A2204%2E07721%2E)
37. Yisi Sang, Xiangyang Mou, Mo Yu, Shunyu Yao, Jing Li, and Jeffrey Stanton. 2022c. TVShowGuess: Character Comprehension in Stories as Speaker Guessing. arXiv preprint. ArXiv:2204.07721 [cs]. [find](https://scholar.google.com/scholar?q=Yisi+Sang%2C+Xiangyang+Mou%2C+Mo+Yu%2C+Shunyu+Yao%2C+Jing+Li%2C+and+Jeffrey+Stanton%2E+2022c%2E+TVShowGuess%3A+Character+Comprehension+in+Stories+as+Speaker+Guessing%2E+arXiv+preprint%2E+ArXiv%3A2204%2E07721+%5Bcs%5D%2E)
38. Tianhao Shen, Sun Li, Quan Tu, and Deyi Xiong. 2024. RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models. arXiv preprint. ArXiv:2312.16132 [cs]. — [arXiv:2312.16132](https://arxiv.org/abs/2312.16132) [find](https://scholar.google.com/scholar?q=Tianhao+Shen%2C+Sun+Li%2C+Quan+Tu%2C+and+Deyi+Xiong%2E+2024%2E+RoleEval%3A+A+Bilingual+Role+Evaluation+Benchmark+for+Large+Language+Models%2E+arXiv+preprint%2E+ArXiv%3A2312%2E16132+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A2312%2E16132%5D%28https%3A%2F%2Farxi)
39. Dominik Stammbach, Maria Antoniak, and Elliott Ash. 2022. Heroes, Villains, and Victims, and GPT-3: Automated Extraction of Character Roles Without Training Data. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022), pages 47–56, Seattle, United States. Association for Computation… [find](https://scholar.google.com/scholar?q=Dominik+Stammbach%2C+Maria+Antoniak%2C+and+Elliott+Ash%2E+2022%2E+Heroes%2C+Villains%2C+and+Victims%2C+and+GPT%2D3%3A+Automated+Extraction+of+Character+Roles+Without+Training+Data%2E+In+Proceedings+of+the+4th+Workshop+of)
40. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. MovieQA: Understanding Stories in Movies through Question-Answering. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4631–4640, Las Vegas, NV, USA. IEEE. [find](https://scholar.google.com/scholar?q=Makarand+Tapaswi%2C+Yukun+Zhu%2C+Rainer+Stiefelhagen%2C+Antonio+Torralba%2C+Raquel+Urtasun%2C+and+Sanja+Fidler%2E+2016%2E+MovieQA%3A+Understanding+Stories+in+Movies+through+Question%2DAnswering%2E+In+2016+IEEE+Conference)
41. Yongqi Tong, Yifan Wang, Dawei Li, Sizhe Wang, Zi Lin, Simeng Han, and Jingbo Shang. 2023. Eliminating reasoning via inferring with planning: A new framework to guide llms’ non-linear thinking. arXiv preprint arXiv:2310.12342. — [arXiv:2310.12342](https://arxiv.org/abs/2310.12342) [find](https://scholar.google.com/scholar?q=Yongqi+Tong%2C+Yifan+Wang%2C+Dawei+Li%2C+Sizhe+Wang%2C+Zi+Lin%2C+Simeng+Han%2C+and+Jingbo+Shang%2E+2023%2E+Eliminating+reasoning+via+inferring+with+planning%3A+A+new+framework+to+guide+llms%E2%80%99+non%2Dlinear+thinking%2E+arXiv+)
42. Sai Vallurupalli, Katrin Erk, and Francis Ferraro. 2024. Saga: A participant-specific examination of story alternatives and goal applicability for a deeper understanding of complex events. arXiv preprint arXiv:2408.05793. [find](https://scholar.google.com/scholar?q=Sai+Vallurupalli%2C+Katrin+Erk%2C+and+Francis+Ferraro%2E+2024%2E+Saga%3A+A+participant%2Dspecific+examination+of+story+alternatives+and+goal+applicability+for+a+deeper+understanding+of+complex+events%2E+arXiv+prepr)
43. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461.
44. Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2024. [Resolving Knowledge Conflicts in Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2023-resolving-knowledge-conflicts-large-language/). arXiv preprint. ArXiv:2310.00935.
45. Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. 2024. [On memorization of large language models in logical reasoning](https://md.research.elamur.ai/free/en/papers/xie-2025-memorization-large-language-models-logical/). arXiv preprint arXiv:2410.23123.
46. Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. 2024. Large language models can learn temporal reasoning. arXiv preprint arXiv:2401.06853. [find](https://scholar.google.com/scholar?q=Siheng+Xiong%2C+Ali+Payani%2C+Ramana+Kompella%2C+and+Faramarz+Fekri%2E+2024%2E+Large+language+models+can+learn+temporal+reasoning%2E+arXiv+preprint+arXiv%3A2401%2E06853%2E)
47. Ningning Xu, Yuxuan Jiang, Shubhashis Roy Dipta, and Zhang Hengyuan. 2025. Learning how to use tools, not just when: Pattern-aware tool-integrated reasoning. MATH-AI @ NeurIPS 2025. [find](https://scholar.google.com/scholar?q=Ningning+Xu%2C+Yuxuan+Jiang%2C+Shubhashis+Roy+Dipta%2C+and+Zhang+Hengyuan%2E+2025%2E+Learning+how+to+use+tools%2C+not+just+when%3A+Pattern%2Daware+tool%2Dintegrated+reasoning%2E+MATH%2DAI+%40+NeurIPS+2025%2E)
48. Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. 2024. ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM. arXiv preprint. ArXiv:2408.12076. — [arXiv:2408.12076](https://arxiv.org/abs/2408.12076) [find](https://scholar.google.com/scholar?q=Zhaochen+Su%2C+Jun+Zhang%2C+Xiaoye+Qu%2C+Tong+Zhu%2C+Yanshu+Li%2C+Jiashuo+Sun%2C+Juntao+Li%2C+Min+Zhang%2C+and+Yu+Cheng%2E+2024%2E+ConflictBank%3A+A+Benchmark+for+Evaluating+the+Influence+of+Knowledge+Conflicts+in+LLM%2E+arX)
49. Shiping Yang, Renliang Sun, and Xiaojun Wan. 2023a. A new benchmark and reverse validation method for passage-level hallucination detection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3898–3908. [find](https://scholar.google.com/scholar?q=Shiping+Yang%2C+Renliang+Sun%2C+and+Xiaojun+Wan%2E+2023a%2E+A+new+benchmark+and+reverse+validation+method+for+passage%2Dlevel+hallucination+detection%2E+In+Findings+of+the+Association+for+Computational+Linguistic)
50. Renliang Sun, Mengyuan Liu, Shiping Yang, Rui Wang, Junqing He, and Jiaxing Zhang. 2024. Fostering natural conversation in large language models with nico: a natural interactive conversation dataset. arXiv preprint arXiv:2408.09330. — [arXiv:2408.09330](https://arxiv.org/abs/2408.09330) [find](https://scholar.google.com/scholar?q=Renliang+Sun%2C+Mengyuan+Liu%2C+Shiping+Yang%2C+Rui+Wang%2C+Junqing+He%2C+and+Jiaxing+Zhang%2E+2024%2E+Fostering+natural+conversation+in+large+language+models+with+nico%3A+a+natural+interactive+conversation+dataset%2E+)
51. Shiping Yang, Jie Wu, Wenbiao Ding, Ning Wu, Shining Liang, Ming Gong, Hengyuan Zhang, and Dongmei Zhang. 2025. Quantifying the robustness of retrieval-augmented language models against spurious features in grounding data. arXiv preprint arXiv:2503.05587. — [arXiv:2503.05587](https://arxiv.org/abs/2503.05587) [find](https://scholar.google.com/scholar?q=Shiping+Yang%2C+Jie+Wu%2C+Wenbiao+Ding%2C+Ning+Wu%2C+Shining+Liang%2C+Ming+Gong%2C+Hengyuan+Zhang%2C+and+Dongmei+Zhang%2E+2025%2E+Quantifying+the+robustness+of+retrieval%2Daugmented+language+models+against+spurious+featu)
52. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation: A survey. arXiv preprint arXiv:2402.13446. — [arXiv:2402.13446](https://arxiv.org/abs/2402.13446) [find](https://scholar.google.com/scholar?q=Zhen+Tan%2C+Dawei+Li%2C+Song+Wang%2C+Alimohammad+Beigi%2C+Bohan+Jiang%2C+Amrita+Bhattacharjee%2C+Mansooreh+Karami%2C+Jundong+Li%2C+Lu+Cheng%2C+and+Huan+Liu%2E+2024%2E+Large+language+models+for+data+annotation%3A+A+survey%2E+ar)
53. Yuan Yang, Siheng Xiong, Ali Payani, Ehsan Shareghi, and Faramarz Fekri. 2023b. Harnessing the power of large language models for natural language to first-order logic translation. arXiv preprint arXiv:2305.15541. [find](https://scholar.google.com/scholar?q=Yuan+Yang%2C+Siheng+Xiong%2C+Ali+Payani%2C+Ehsan+Shareghi%2C+and+Faramarz+Fekri%2E+2023b%2E+Harnessing+the+power+of+large+language+models+for+natural+language+to+first%2Dorder+logic+translation%2E+arXiv+preprint+arXi)
54. Zhengzhe Yang and Jinho D. Choi. 2019. FriendsQA: Open-Domain Question Answering on TV Show Transcripts. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 188– 197, Stockholm, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhengzhe+Yang+and+Jinho+D%2E+Choi%2E+2019%2E+FriendsQA%3A+Open%2DDomain+Question+Answering+on+TV+Show+Transcripts%2E+In+Proceedings+of+the+20th+Annual+SIGdial+Meeting+on+Discourse+and+Dialogue%2C+pages+188%E2%80%93+197%2C+St)
55. Mo Yu, Jiangnan Li, Shunyu Yao, Wenjie Pang, Xiaochen Zhou, Zhou Xiao, Fandong Meng, and Jie Zhou. 2023a. Personality Understanding of Fictional Characters during Book Reading. arXiv preprint. ArXiv:2305.10156 [cs]. [find](https://scholar.google.com/scholar?q=Mo+Yu%2C+Jiangnan+Li%2C+Shunyu+Yao%2C+Wenjie+Pang%2C+Xiaochen+Zhou%2C+Zhou+Xiao%2C+Fandong+Meng%2C+and+Jie+Zhou%2E+2023a%2E+Personality+Understanding+of+Fictional+Characters+during+Book+Reading%2E+arXiv+preprint%2E+ArXiv%3A2)
56. Mo Yu, Jiangnan Li, Shunyu Yao, Wenjie Pang, Xiaochen Zhou, Zhou Xiao, Fandong Meng, and Jie Zhou. 2023b. Personality understanding of fictional characters during book reading. arXiv preprint arXiv:2305.10156. [find](https://scholar.google.com/scholar?q=Mo+Yu%2C+Jiangnan+Li%2C+Shunyu+Yao%2C+Wenjie+Pang%2C+Xiaochen+Zhou%2C+Zhou+Xiao%2C+Fandong+Meng%2C+and+Jie+Zhou%2E+2023b%2E+Personality+understanding+of+fictional+characters+during+book+reading%2E+arXiv+preprint+arXiv%3A23)
57. Mo Yu, Qiujing Wang, Shunchi Zhang, Yisi Sang, Kangsheng Pu, Zekai Wei, Han Wang, Liyan Xu, Jing Li, Yue Yu, et al. 2022. Few-shot character understanding in movies as an assessment to meta-learning of theory-of-mind. arXiv preprint arXiv:2211.04684. — [arXiv:2211.04684](https://arxiv.org/abs/2211.04684) [find](https://scholar.google.com/scholar?q=Mo+Yu%2C+Qiujing+Wang%2C+Shunchi+Zhang%2C+Yisi+Sang%2C+Kangsheng+Pu%2C+Zekai+Wei%2C+Han+Wang%2C+Liyan+Xu%2C+Jing+Li%2C+Yue+Yu%2C+et+al%2E+2022%2E+Few%2Dshot+character+understanding+in+movies+as+an+assessment+to+meta%2Dlearning+o)
58. Yijiong Yu. 2025. [Do llms really think step-by-step in implicit reasoning?](https://md.research.elamur.ai/free/en/papers/yu-2024-llms-really-think-step-step/) Preprint, arXiv:2411.15862.
59. Hengyuan Zhang, Shiping Yang, Xiao Liang, Chenming Shang, Yuxuan Jiang, Chaofan Tao, Jing Xiong, Hayden Kwok-Hay So, Ruobing Xie, Angel X Chang, et al. 2025. [Find your optimal teacher: Personalized data synthesis via router-guided multi-teacher distillation](https://md.research.elamur.ai/free/en/papers/zhang-2026-find-your-optimal-teacher-personalized/). arXiv preprint arXiv:2510.10925.
60. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. [Don’t Make Your LLM an Evaluation Benchmark Cheater](https://md.research.elamur.ai/free/en/papers/zhou-2023-don-t-make-llm-evaluation/). arXiv preprint. ArXiv:2311.01964.

*Updated 2 November 2026*