---
title: "Internal Chain-of-Thought — paper by Yang et al. (2025)"
description: "They build a benchmark of 15 such tasks in four categories. 53 references, cited by 7 papers here."
authors: ["Zhipeng Yang", "Junzhuo Li", "Siyu Xia", "Xuming Hu"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.1147/"
doi: "10.18653/v1/2025.emnlp-main.1147"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2025-internal-chain-thought-empirical-evidence/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2025-internal-chain-thought-empirical-evidence/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) / Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs

# Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs

**Authors:** Zhipeng Yang, Junzhuo Li, Siyu Xia and [Xuming Hu](https://md.research.elamur.ai/free/en/authors/xuming-hu/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.1147/) · [PDF](https://aclanthology.org/2025.emnlp-main.1147.pdf) · [DOI 10.18653/v1/2025.emnlp-main.1147](https://doi.org/10.18653/v1/2025.emnlp-main.1147) · [OpenAlex](https://openalex.org/W4416035765)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/yang-2025-internal-chain-thought-empirical-evidence/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This paper asks whether LLMs perform multi-step reasoning silently, across their layers, without writing out intermediate steps. The authors call this an internal chain-of-thought (ICoT). They study two-step composite tasks, such as producing an antonym and then capitalizing it. The tasks are posed through in-context demonstrations that show only final answers. They build a benchmark of 15 such tasks in four categories. Within the Task Vector framework, they test two hypotheses: each subtask is acquired at a different depth, and the subtasks are carried out one after another. Support for the first comes from layer-from context-masking and a new cross-task patching method. This method transplants residual activations from composite prompts into zero-shot single-subtask queries. Support for the second comes from LogitLens decoding of hidden states. The rule for the first subtask is learned earlier than the second. The intermediate answer surfaces in middle layers before the final answer overtakes it. A replication on the TRACE instruction-following benchmark shows similar dynamics. The findings improve transparency and suggest layer-targeted, instruction-level steering, for example for safety.

## Key points
- The benchmark has 15 two-step composite tasks in four categories: Knowledge–Algorithmic, Extractive–Knowledge, Extractive–Algorithmic and Knowledge–Translation. Example: "Cenepa River → Pérou", which means retrieving the country and then translating it into French.
- Four mid-sized open models (3B–8B) were tested: Llama-3.2-3B, Llama-3.1-8B, Mistral-7B and Qwen2.5-7B. In context-masking, all of them show a characteristic "X-shape" in accuracy curves across layers.
- In Llama-3.1-8B, cross-task patching yields subtask vectors that transfer to their corresponding single subtask about 66% of the time on average.
- LogitLens tracking uses mean reciprocal rank. It shows the intermediate answer peaking in mid-layers, with the final answer reaching its own peak several layers deeper.
- The authors list several limitations:
  - The TRACE results rely on LLM-based scoring, so they are treated as exploratory.
  - Closely related constraints produce more entangled patterns.
  - Generalization to frontier-scale models is untested.
- Code is available at github.com/yzp11/Internal-Chain-of-Thought.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 170 works · 26,341 citations
- [The Hong Kong University of Science and Technology (Guangzhou)](https://md.research.elamur.ai/free/en/universities/the-hong-kong-university-of-science-and-technology-guangzhou/) — University, Guangzhou, China · 67 works · 1,765 citations
- [Southeast University](https://md.research.elamur.ai/free/en/universities/southeast-university-cn/) — University, Nanjing, China · 32 works · 453 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Zhipeng Yang, Junzhuo Li, Siyu Xia and Xuming Hu. 2025. Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2025internal,
  title = {{Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs}},
  author = {Zhipeng Yang, Junzhuo Li, Siyu Xia and Xuming Hu},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1147},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1147}
}
```

- [← previous ICLEval: Evaluating In-Context Learning Ability of Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-icleval-evaluating-context-learning-ability/)
- [next → Language Adaptation of Large Language Models: An Empirical Study on LLaMA2](https://md.research.elamur.ai/free/en/papers/wang-2025-language-adaptation-large-language-models/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/)

## References

53 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2024. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. In Transformer Circuits Thread. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Scaling+monosemanticity%3A+Extracting+interpretable+features+from+claude+3+sonnet%2E+In+Transformer+Circuits+Thread%2E)
2. Anthropic. 2025. On the biology of a large language model. In Transformer Circuits Thread. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025%2E+On+the+biology+of+a+large+language+model%2E+In+Transformer+Circuits+Thread%2E)
3. Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. 2024. [Hopping too late: Exploring the limitations of large language models on multi-hop queries](https://md.research.elamur.ai/free/en/papers/biran-2024-hopping-too-late-exploring-limitations/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14113–14130.
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
5. Xinghao Chen, Anhao Zhao, Heming Xia, Xuan Lu, Hanlin Wang, Yanjun Chen, Wei Zhang, Jian Wang, Wenjie Li, and Xiaoyu Shen. 2025. Reasoning beyond language: A comprehensive survey on latent chain-of-thought reasoning. arXiv preprint arXiv:2505.16782. — [arXiv:2505.16782](https://arxiv.org/abs/2505.16782) [find](https://scholar.google.com/scholar?q=Xinghao+Chen%2C+Anhao+Zhao%2C+Heming+Xia%2C+Xuan+Lu%2C+Hanlin+Wang%2C+Yanjun+Chen%2C+Wei+Zhang%2C+Jian+Wang%2C+Wenjie+Li%2C+and+Xiaoyu+Shen%2E+2025%2E+Reasoning+beyond+language%3A+A+comprehensive+survey+on+latent+chain%2Dof%2Dth)
6. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Guillaume+Lample%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Ludovic+Denoyer%2C+and+Herv%C3%A9+J%C3%A9gou%2E+2017%2E+Word+translation+without+parallel+data%2E+arXiv+preprint+arXiv%3A1710%2E04087%2E)
7. Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. [Analyzing transformers in embedding space](https://md.research.elamur.ai/free/en/papers/dar-2023-analyzing-transformers-embedding-space/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16124–16170.
8. Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. [Separating tongue from thought: Activation patching reveals language-agnostic concept representations in transformers](https://md.research.elamur.ai/free/en/papers/dumas-2025-separating-tongue-thought-activation-patching/). arXiv preprint arXiv:2411.08745.
9. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, and 1 others. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652. [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Nicholas+Schiefer%2C+Tom+Henighan%2C+Shauna+Kravec%2C+Zac+Hatfield%2DDodds%2C+Robert+Lasenby%2C+Dawn+Drain%2C+Carol+Chen%2C+and+1+others%2E+2022%2E+Toy+models+of+superpositi)
10. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, and 1 others. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12. [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+and+1+others%2E+2021%2E+A+mathematical+framework+for+transformer+ci)
11. Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. 2024. Do i know this entity? knowledge awareness and hallucinations in language models. arXiv preprint arXiv:2411.14257. — [arXiv:2411.14257](https://arxiv.org/abs/2411.14257) [find](https://scholar.google.com/scholar?q=Javier+Ferrando%2C+Oscar+Obeso%2C+Senthooran+Rajamanoharan%2C+and+Neel+Nanda%2E+2024%2E+Do+i+know+this+entity%3F+knowledge+awareness+and+hallucinations+in+language+models%2E+arXiv+preprint+arXiv%3A2411%2E14257%2E+%E2%80%94+%5BarXi)
12. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Tom+Dupr%C3%A9+la+Tour%2C+Henk+Tillman%2C+Gabriel+Goh%2C+Rajan+Troll%2C+Alec+Radford%2C+Ilya+Sutskever%2C+Jan+Leike%2C+and+Jeffrey+Wu%2E+2024%2E+Scaling+and+evaluating+sparse+autoencoders%2E+arXiv+preprint+arXiv%3A2406)
13. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216–12235. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Jasmijn+Bastings%2C+Katja+Filippova%2C+and+Amir+Globerson%2E+2023%2E+Dissecting+recall+of+factual+associations+in+auto%2Dregressive+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+)
14. Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. 2024. Successor heads: Recurring, interpretable attention heads in the wild. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Rhys+Gould%2C+Euan+Ong%2C+George+Ogden%2C+and+Arthur+Conmy%2E+2024%2E+Successor+heads%3A+Recurring%2C+interpretable+attention+heads+in+the+wild%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
15. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
16. Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. 2024. Overthinking the truth: Understanding how language models process false demonstrations. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Danny+Halawi%2C+Jean%2DStanislas+Denain%2C+and+Jacob+Steinhardt%2E+2024%2E+Overthinking+the+truth%3A+Understanding+how+language+models+process+false+demonstrations%2E+In+The+Twelfth+International+Conference+on+Lear)
17. Roee Hendel, Mor Geva, and Amir Globerson. 2023. [In-context learning creates task vectors](https://md.research.elamur.ai/free/en/papers/hendel-2023-context-learning-creates-task-vectors/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318–9333.
18. Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2024. [Linearity of relation decoding in transformer language models](https://md.research.elamur.ai/free/en/papers/hernandez-2023-linearity-relation-decoding-transformer-language/). In The Twelfth International Conference on Learning Representations.
19. AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, and 1 others. 2024. Mistral 7b. arxiv 2023. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=AQ+Jiang%2C+A+Sablayrolles%2C+A+Mensch%2C+C+Bamford%2C+DS+Chaplot%2C+Ddl+Casas%2C+F+Bressand%2C+G+Lengyel%2C+G+Lample%2C+L+Saulnier%2C+and+1+others%2E+2024%2E+Mistral+7b%2E+arxiv+2023%2E+arXiv+preprint+arXiv%3A2310%2E06825%2E)
20. Tianjie Ju, Yijin Chen, Xinwei Yuan, Zhuosheng Zhang, Wei Du, Yubin Zheng, and Gongshen Liu. 2024. Investigating multi-hop factual shortcuts in knowledge editing of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper… [find](https://scholar.google.com/scholar?q=Tianjie+Ju%2C+Yijin+Chen%2C+Xinwei+Yuan%2C+Zhuosheng+Zhang%2C+Wei+Du%2C+Yubin+Zheng%2C+and+Gongshen+Liu%2E+2024%2E+Investigating+multi%2Dhop+factual+shortcuts+in+knowledge+editing+of+large+language+models%2E+In+Proceedin)
21. Jindong Li, Yali Fu, Li Fan, Jiahong Liu, Yao Shu, Chengwei Qin, Menglin Yang, Irwin King, and Rex Ying. 2025a. [Implicit reasoning in large language models: A comprehensive survey](https://md.research.elamur.ai/free/en/papers/li-2025-implicit-reasoning-large-language-models/). arXiv preprint arXiv:2509.02350.
22. Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. 2024. Understanding and patching compositional reasoning in llms. In Findings of the Association for Computational Linguistics ACL 2024, pages 9668–9688. vector arithmetic. In Proceedings of the 2024 Conference of the North Ame… [find](https://scholar.google.com/scholar?q=Zhaoyi+Li%2C+Gangwei+Jiang%2C+Hong+Xie%2C+Linqi+Song%2C+Defu+Lian%2C+and+Ying+Wei%2E+2024%2E+Understanding+and+patching+compositional+reasoning+in+llms%2E+In+Findings+of+the+Association+for+Computational+Linguistics+)
23. AI Meta. 2024. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. [find](https://scholar.google.com/scholar?q=AI+Meta%2E+2024%2E+Llama+3%2E2%3A+Revolutionizing+edge+ai+and+vision+with+open%2C+customizable+models%2E)
24. Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017. Distinguishing antonyms and synonyms in a pattern-based neural network. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 76–85. [find](https://scholar.google.com/scholar?q=Kim+Anh+Nguyen%2C+Sabine+Schulte+im+Walde%2C+and+Ngoc+Thang+Vu%2E+2017%2E+Distinguishing+antonyms+and+synonyms+in+a+pattern%2Dbased+neural+network%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+)
25. Nostalgebraist. 2020. Interpreting gpt: the logit lens. [https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/) interpreting-gpt-the-logit-lens. [↗](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/)
26. Zhuowei Li, Zihao Xu, Ligong Han, Yunhe Gao, Song Wen, Di Liu, Hao Wang, and Dimitris N Metaxas. 2025b. Implicit in-context learning. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhuowei+Li%2C+Zihao+Xu%2C+Ligong+Han%2C+Yunhe+Gao%2C+Song+Wen%2C+Di+Liu%2C+Hao+Wang%2C+and+Dimitris+N+Metaxas%2E+2025b%2E+Implicit+in%2Dcontext+learning%2E+In+The+Thirteenth+International+Conference+on+Learning+Representat)
27. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, and 1 others. 2022. [In-context learning and induction heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/). arXiv preprint arXiv:2209.11895.
28. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+and+1+others%2E+2024a%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A241)
29. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Muru+Zhang%2C+Sewon+Min%2C+Ludwig+Schmidt%2C+Noah+A+Smith%2C+and+Mike+Lewis%2E+2023%2E+Measuring+and+narrowing+the+compositionality+gap+in+language+models%2E+In+Findings+of+the+Association+for+Computati)
30. Sheng Liu, Haotian Ye, Lei Xing, and James Zou. 2024b. [In-context vectors: making in context learning more effective and controllable through latent space steering](https://md.research.elamur.ai/free/en/papers/liu-2023-context-vectors-making-context-learning/). In Proceedings of the 41st International Conference on Machine Learning, pages 32287–32307.
31. Siddarth Mamidanna, Daking Rai, Ziyu Yao, and Yilun Zhou. 2025. [All for one: Llms solve mental math at the last token with information transferred from other tokens](https://md.research.elamur.ai/free/en/papers/mamidanna-2025-all-one-llms-solve-mental/). arXiv preprint arXiv:2509.09650.
32. Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2024. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647. [find](https://scholar.google.com/scholar?q=Samuel+Marks%2C+Can+Rager%2C+Eric+J+Michaud%2C+Yonatan+Belinkov%2C+David+Bau%2C+and+Aaron+Mueller%2E+2024%2E+Sparse+feature+circuits%3A+Discovering+and+editing+interpretable+causal+graphs+in+language+models%2E+arXiv+pr)
33. Tianyi Men, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. 2024. Unlocking the future: Exploring look-ahead planning mechanistic interpretability in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7713–7724. [find](https://scholar.google.com/scholar?q=Tianyi+Men%2C+Pengfei+Cao%2C+Zhuoran+Jin%2C+Yubo+Chen%2C+Kang+Liu%2C+and+Jun+Zhao%2E+2024%2E+Unlocking+the+future%3A+Exploring+look%2Dahead+planning+mechanistic+interpretability+in+large+language+models%2E+In+Proceedings)
34. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+gpt%2E+Advances+in+neural+information+processing+systems%2C+35%3A17359%E2%80%9317372%2E)
35. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2024. Language models implement simple word2vec-style Baturay Saglam, Zhuoran Yang, Dionysis Kalogerias, and Amin Karbasi. 2025. Learning task representations from in-context learning. arXiv preprint arXiv:2502.05390. [find](https://scholar.google.com/scholar?q=Jack+Merullo%2C+Carsten+Eickhoff%2C+and+Ellie+Pavlick%2E+2024%2E+Language+models+implement+simple+word2vec%2Dstyle+Baturay+Saglam%2C+Zhuoran+Yang%2C+Dionysis+Kalogerias%2C+and+Amin+Karbasi%2E+2025%2E+Learning+task+repres)
36. Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. 2022. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210.01892. — [arXiv:2210.01892](https://arxiv.org/abs/2210.01892) [find](https://scholar.google.com/scholar?q=Adam+Scherlis%2C+Kshitij+Sachan%2C+Adam+S+Jermyn%2C+Joe+Benton%2C+and+Buck+Shlegeris%2E+2022%2E+Polysemanticity+and+capacity+in+neural+networks%2E+arXiv+preprint+arXiv%3A2210%2E01892%2E+%E2%80%94+%5BarXiv%3A2210%2E01892%5D%28https%3A%2F%2Farxiv)
37. Suzanna Sia, David Mueller, and Kevin Duh. 2024. Where does in-context learning happen in large language models? Advances in Neural Information Processing Systems, 37:32761–32786. [find](https://scholar.google.com/scholar?q=Suzanna+Sia%2C+David+Mueller%2C+and+Kevin+Duh%2E+2024%2E+Where+does+in%2Dcontext+learning+happen+in+large+language+models%3F+Advances+in+Neural+Information+Processing+Systems%2C+37%3A32761%E2%80%9332786%2E)
38. Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2024. [Function vectors in large language models](https://md.research.elamur.ai/free/en/papers/todd-2023-function-vectors-large-language-models/). In The Twelfth International Conference on Learning Representations.
39. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388– 12401. [find](https://scholar.google.com/scholar?q=Jesse+Vig%2C+Sebastian+Gehrmann%2C+Yonatan+Belinkov%2C+Sharon+Qian%2C+Daniel+Nevo%2C+Yaron+Singer%2C+and+Stuart+Shieber%2E+2020%2E+Investigating+gender+bias+in+language+models+using+causal+mediation+analysis%2E+Advance)
40. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kevin+Ro+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+2023%2E+Interpretability+in+the+wild%3A+a+circuit+for+indirect+object+identification+in+gpt%2D2+small%2E+In+The+Elevent)
41. Ruoyu Wang, Zhipeng Yang, Zinan Zhao, Xinyan Tong, Zhi Hong, and Kun Qian. 2024. Llm-based robot task planning with exceptional handling for general purpose service robots. In 2024 43rd Chinese Control Conference (CCC), pages 4439–4444. IEEE. [find](https://scholar.google.com/scholar?q=Ruoyu+Wang%2C+Zhipeng+Yang%2C+Zinan+Zhao%2C+Xinyan+Tong%2C+Zhi+Hong%2C+and+Kun+Qian%2E+2024%2E+Llm%2Dbased+robot+task+planning+with+exceptional+handling+for+general+purpose+service+robots%2E+In+2024+43rd+Chinese+Contro)
42. Yiming Wang, Pei Zhang, Baosong Yang, Derek F Wong, and Rui Wang. 2025. Latent space chain-of-embedding enables output-free llm self-evaluation. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yiming+Wang%2C+Pei+Zhang%2C+Baosong+Yang%2C+Derek+F+Wong%2C+and+Rui+Wang%2E+2025%2E+Latent+space+chain%2Dof%2Dembedding+enables+output%2Dfree+llm+self%2Devaluation%2E+In+The+Thirteenth+International+Conference+on+Learning+)
43. Xinghua Zhang, Haiyang Yu, Cheng Fu, Fei Huang, and Yongbin Li. 2024. [Iopo: Empowering llms with complex instruction following via input-output preference optimization](https://md.research.elamur.ai/free/en/papers/zhang-2025-iopo-empowering-llms-complex-instruction/). arXiv preprint arXiv:2411.06208.
44. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. [Automatic chain of thought prompting in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/). In The Eleventh International Conference on Learning Representations.
45. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824– 24837.
46. Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, and Ashish Sabharwal. 2024. Answer, assemble, ace: Understanding how transformers answer multiple choice questions. arXiv preprint arXiv:2407.15018. — [arXiv:2407.15018](https://arxiv.org/abs/2407.15018) [find](https://scholar.google.com/scholar?q=Sarah+Wiegreffe%2C+Oyvind+Tafjord%2C+Yonatan+Belinkov%2C+Hannaneh+Hajishirzi%2C+and+Ashish+Sabharwal%2E+2024%2E+Answer%2C+assemble%2C+ace%3A+Understanding+how+transformers+answer+multiple+choice+questions%2E+arXiv+prepri)
47. Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. 2023. Depn: Detecting and editing privacy neurons in pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2875–2886. [find](https://scholar.google.com/scholar?q=Xinwei+Wu%2C+Junzhuo+Li%2C+Minghui+Xu%2C+Weilong+Dong%2C+Shuangzhi+Wu%2C+Chao+Bian%2C+and+Deyi+Xiong%2E+2023%2E+Depn%3A+Detecting+and+editing+privacy+neurons+in+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2023+C)
48. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024a. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+and+1+others%2E+2024a%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E1511)
49. Liu Yang, Ziqian Lin, Kangwook Lee, Dimitris Papailiopoulos, and Robert Nowak. 2025. Task vectors in in-context learning: Emergence, formation, and benefit. arXiv preprint arXiv:2501.09240. — [arXiv:2501.09240](https://arxiv.org/abs/2501.09240) [find](https://scholar.google.com/scholar?q=Liu+Yang%2C+Ziqian+Lin%2C+Kangwook+Lee%2C+Dimitris+Papailiopoulos%2C+and+Robert+Nowak%2E+2025%2E+Task+vectors+in+in%2Dcontext+learning%3A+Emergence%2C+formation%2C+and+benefit%2E+arXiv+preprint+arXiv%3A2501%2E09240%2E+%E2%80%94+%5BarXiv%3A2)
50. Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. 2024b. Do large language models latently perform multi-hop reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10210–10229. [find](https://scholar.google.com/scholar?q=Sohee+Yang%2C+Elena+Gribovskaya%2C+Nora+Kassner%2C+Mor+Geva%2C+and+Sebastian+Riedel%2E+2024b%2E+Do+large+language+models+latently+perform+multi%2Dhop+reasoning%3F+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Asso)
51. Sohee Yang, Nora Kassner, Elena Gribovskaya, Sebastian Riedel, and Mor Geva. 2024c. Do large language models perform latent multi-hop reasoning without exploiting shortcuts? arXiv preprint arXiv:2411.16679. [find](https://scholar.google.com/scholar?q=Sohee+Yang%2C+Nora+Kassner%2C+Elena+Gribovskaya%2C+Sebastian+Riedel%2C+and+Mor+Geva%2E+2024c%2E+Do+large+language+models+perform+latent+multi%2Dhop+reasoning+without+exploiting+shortcuts%3F+arXiv+preprint+arXiv%3A2411%2E)
52. Zeping Yu, Yonatan Belinkov, and Sophia Ananiadou. 2025. [Back attention: Understanding and enhancing multi-hop reasoning in large language models](https://md.research.elamur.ai/free/en/papers/yu-2025-back-attention-understanding-enhancing-multi/). arXiv preprint arXiv:2502.10835.
53. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Jiatong+Yu%2C+Tianyu+Gao%2C+Yu+Meng%2C+Tanya+Goyal%2C+and+Danqi+Chen%2E+2024%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E+In+The+Twelfth+International+Conference+on+Learn)

*Updated 2 November 2026*