---
title: "Back Attention — paper by Yu et al. (2025) — summary and ideas"
description: "The authors introduce logit flow, a neuron-level interpretability technique that tracks how contributions… 71 references, cited by 13 papers here."
authors: ["Zeping Yu", "Yonatan Belinkov", "Sophia Ananiadou"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.567/"
doi: "10.18653/v1/2025.emnlp-main.567"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yu-2025-back-attention-understanding-enhancing-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yu-2025-back-attention-understanding-enhancing-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models

# Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models

**Authors:** [Zeping Yu](https://md.research.elamur.ai/free/en/authors/zeping-yu/), [Yonatan Belinkov](https://md.research.elamur.ai/free/en/authors/yonatan-belinkov/) and [Sophia Ananiadou](https://md.research.elamur.ai/free/en/authors/sophia-ananiadou/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.567/) · [PDF](https://aclanthology.org/2025.emnlp-main.567.pdf) · [DOI 10.18653/v1/2025.emnlp-main.567](https://doi.org/10.18653/v1/2025.emnlp-main.567) · [OpenAlex](https://openalex.org/W4416036439)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [13](https://md.research.elamur.ai/free/en/papers/yu-2025-back-attention-understanding-enhancing-multi/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper examines why large language models often fail at latent multi-hop factual reasoning, where a query such as "e1's r1's r2 is" has to be answered without explicit intermediate steps. The authors introduce logit flow, a neuron-level interpretability technique that tracks how contributions to the final logits move across layers and token positions. They combine it with activation patching and apply both to filtered prompts from the TwoHop dataset. For single-hop queries they find a four-stage pipeline:

- enriching the subject representation at the entity
- pulling out the entity's attributes
- enriching the subject representation at the relation
- pulling out the relation's attributes

In failed two-hop cases, the last of these stages draws too heavily on high-layer information at the r1 position. This favours the bridge entity e2 over the correct answer e3.

To counter this, they propose back attention, which lets a lower layer attend to higher-layer hidden states at other positions. On arithmetic tasks trained from scratch, a 1-layer transformer with back attention matches a 2-layer model. Added to five pretrained LLMs, it raises accuracy on five reasoning datasets. This offers a cheaper alternative to chain-of-thought for this kind of reasoning.

## Key points
- **Data:** Analyses use the TwoHop dataset (Biran et al., 2024), with shortcut cases removed and rare triplets (\<30 occurrences) dropped, leaving a pool of 2,864 cases.
  - After capping each answer e3 at 5 examples, 889 cases remain, covering 26 triplet types. The largest type makes up only 12.7%.
  - A separate set of 568 cases, where all entities are human, is used to compare correct and incorrect two-hop predictions.
- **Single-hop mechanism:** FFN neurons do the subject enrichment at the entity, relation and last positions. Attention neurons extract entity attributes. Both attention and FFN neurons extract relation attributes.
- **Back attention in LLMs:** The method is applied to a single layer only, with multi-layer use left as future work. It raises per-token cost to about 1.8T, but because far fewer tokens need to be generated than with CoT, the authors argue total compute is lower.
- **Code and data:** Released at github.com/zepingyu0512/back-attention.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Manchester](https://md.research.elamur.ai/free/en/universities/university-of-manchester/) — University, Manchester, United Kingdom · 87 works · 44,938 citations
- [Technion – Israel Institute of Technology](https://md.research.elamur.ai/free/en/universities/technion-israel-institute-of-technology/) — University, Haifa, Israel · 49 works · 5,306 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Zeping Yu, Yonatan Belinkov and Sophia Ananiadou. 2025. Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yu2025back,
  title = {{Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models}},
  author = {Zeping Yu, Yonatan Belinkov and Sophia Ananiadou},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.567},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.567}
}
```

- [← previous Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions](https://md.research.elamur.ai/free/en/papers/zhao-2025-auto-arena-automating-llm-evaluations/)
- [next → Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction](https://md.research.elamur.ai/free/en/papers/li-2025-baichuan-audio-unified-framework/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/)

## References

71 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. 2024. [Hopping too late: Exploring the limitations of large language models on multi-hop queries](https://md.research.elamur.ai/free/en/papers/biran-2024-hopping-too-late-exploring-limitations/). arXiv preprint arXiv:2406.12775.
2. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2. [find](https://scholar.google.com/scholar?q=Trenton+Bricken%2C+Adly+Templeton%2C+Joshua+Batson%2C+Brian+Chen%2C+Adam+Jermyn%2C+Tom+Conerly%2C+Nick+Turner%2C+Cem+Anil%2C+Carson+Denison%2C+Amanda+Askell%2C+et+al%2E+2023%2E+Towards+monosemanticity%3A+Decomposing+language+m)
3. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. 2024. [Large language monkeys: Scaling inference compute with repeated sampling](https://md.research.elamur.ai/free/en/papers/brown-2024-large-language-monkeys-scaling-inference/). arXiv preprint arXiv:2407.21787.
4. Tom B Brown. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165.
5. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335. [find](https://scholar.google.com/scholar?q=Zixiang+Chen%2C+Yihe+Deng%2C+Huizhuo+Yuan%2C+Kaixuan+Ji%2C+and+Quanquan+Gu%2E+2024%2E+Self%2Dplay+fine%2Dtuning+converts+weak+language+models+to+strong+language+models%2E+arXiv+preprint+arXiv%3A2401%2E01335%2E)
6. Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712. [find](https://scholar.google.com/scholar?q=Antonia+Creswell%2C+Murray+Shanahan%2C+and+Irina+Higgins%2E+2022%2E+Selection%2Dinference%3A+Exploiting+large+language+models+for+interpretable+logical+reasoning%2E+arXiv+preprint+arXiv%3A2205%2E09712%2E)
7. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. [Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space](https://md.research.elamur.ai/free/en/papers/geva-2022-transformer-feed-forward-layers-build/). arXiv preprint arXiv:2203.14680.
8. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. [find](https://scholar.google.com/scholar?q=Hoagy+Cunningham%2C+Aidan+Ewart%2C+Logan+Riggs%2C+Robert+Huben%2C+and+Lee+Sharkey%2E+2023%2E+Sparse+autoencoders+find+highly+interpretable+features+in+language+models%2E+arXiv+preprint+arXiv%3A2309%2E08600%2E)
9. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+2021%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+)
10. Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022. [Analyzing transformers in embedding space](https://md.research.elamur.ai/free/en/papers/dar-2023-analyzing-transformers-embedding-space/). arXiv preprint arXiv:2209.02535.
11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
12. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652. [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Nicholas+Schiefer%2C+Tom+Henighan%2C+Shauna+Kravec%2C+Zac+Hatfield%2DDodds%2C+Robert+Lasenby%2C+Dawn+Drain%2C+Carol+Chen%2C+et+al%2E+2022%2E+Toy+models+of+superposition%2E+arX)
13. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12. [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+et+al%2E+2021%2E+A+mathematical+framework+for+transformer+circuits%2E)
14. Yujie Feng, Xu Chu, Yongxin Xu, Zexin Lu, Bo Liu, Philip S Yu, and Xiao-Ming Wu. 2024. Kif: Knowledge identification and fusion for language model continual learning. arXiv preprint arXiv:2408.05200. — [arXiv:2408.05200](https://arxiv.org/abs/2408.05200) [find](https://scholar.google.com/scholar?q=Yujie+Feng%2C+Xu+Chu%2C+Yongxin+Xu%2C+Zexin+Lu%2C+Bo+Liu%2C+Philip+S+Yu%2C+and+Xiao%2DMing+Wu%2E+2024%2E+Kif%3A+Knowledge+identification+and+fusion+for+language+model+continual+learning%2E+arXiv+preprint+arXiv%3A2408%2E05200%2E+)
15. Yujie Feng, Xujia Wang, Zexin Lu, Shenghong Fu, Guangyuan Shi, Yongxin Xu, Yasha Wang, Philip S Yu, Xu Chu, and Xiao-Ming Wu. 2025. [Recurrent knowledge identification and fusion for language model continual learning](https://md.research.elamur.ai/free/en/papers/feng-2025-recurrent-knowledge-identification-fusion-language/). arXiv preprint arXiv:2502.17510.
16. Jaden Fiotto-Kaufman, Alexander R Loftus, Eric Todd, Jannik Brinkmann, Caden Juang, Koyena Pal, Can Rager, Aaron Mueller, Samuel Marks, Arnab Sen Sharma, et al. 2024. Nnsight and ndif: Democratizing access to foundation model internals. arXiv preprint arXiv:2407.14561. [find](https://scholar.google.com/scholar?q=Jaden+Fiotto%2DKaufman%2C+Alexander+R+Loftus%2C+Eric+Todd%2C+Jannik+Brinkmann%2C+Caden+Juang%2C+Koyena+Pal%2C+Can+Rager%2C+Aaron+Mueller%2C+Samuel+Marks%2C+Arnab+Sen+Sharma%2C+et+al%2E+2024%2E+Nnsight+and+ndif%3A+Democratizing+a)
17. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. [Complexity-based prompting for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/). In The Eleventh International Conference on Learning Representations.
18. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Tom+Dupr%C3%A9+la+Tour%2C+Henk+Tillman%2C+Gabriel+Goh%2C+Rajan+Troll%2C+Alec+Radford%2C+Ilya+Sutskever%2C+Jan+Leike%2C+and+Jeffrey+Wu%2E+2024%2E+Scaling+and+evaluating+sparse+autoencoders%2E+arXiv+preprint+arXiv%3A2406)
19. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In The 2023 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Jasmijn+Bastings%2C+Katja+Filippova%2C+and+Amir+Globerson%2E+2023%2E+Dissecting+recall+of+factual+associations+in+auto%2Dregressive+language+models%2E+In+The+2023+Conference+on+Empirical+Methods+in+Natu)
20. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2020. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Roei+Schuster%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+2020%2E+Transformer+feed%2Dforward+layers+are+key%2Dvalue+memories%2E+arXiv+preprint+arXiv%3A2012%2E14913%2E)
21. Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. 2023. Successor heads: Recurring, interpretable attention heads in the wild. arXiv preprint arXiv:2312.09230. [find](https://scholar.google.com/scholar?q=Rhys+Gould%2C+Euan+Ong%2C+George+Ogden%2C+and+Arthur+Conmy%2E+2023%2E+Successor+heads%3A+Recurring%2C+interpretable+attention+heads+in+the+wild%2E+arXiv+preprint+arXiv%3A2312%2E09230%2E)
22. Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. 2024. [Model editing harms general abilities of large language models: Regularization to the rescue](https://md.research.elamur.ai/free/en/papers/gu-2024-model-editing-harms-general-abilities/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16801– 168…
23. Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. 2024. Model editing at scale leads to gradual and catastrophic forgetting. arXiv preprint arXiv:2401.07453. [find](https://scholar.google.com/scholar?q=Akshat+Gupta%2C+Anurag+Rao%2C+and+Gopala+Anumanchipalli%2E+2024%2E+Model+editing+at+scale+leads+to+gradual+and+catastrophic+forgetting%2E+arXiv+preprint+arXiv%3A2401%2E07453%2E)
24. Michael Hanna, Ollie Liu, and Alexandre Variengien. 2024. [How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/). Advances in Neural Information Processing Systems, 36.
25. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Yi+Gu%2C+Haodi+Ma%2C+Joshua+Jiahua+Hong%2C+Zhen+Wang%2C+Daisy+Zhe+Wang%2C+and+Zhiting+Hu%2E+2023%2E+Reasoning+with+language+model+is+planning+with+world+model%2E+arXiv+preprint+arXiv%3A2305%2E14992%2E)
26. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/). arXiv preprint arXiv:2310.01798.
27. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
28. Tianjie Ju, Yijin Chen, Xinwei Yuan, Zhuosheng Zhang, Wei Du, Yubin Zheng, and Gongshen Liu. 2024. Investigating multi-hop factual shortcuts in knowledge editing of large language models. arXiv preprint arXiv:2402.11900. [find](https://scholar.google.com/scholar?q=Tianjie+Ju%2C+Yijin+Chen%2C+Xinwei+Yuan%2C+Zhuosheng+Zhang%2C+Wei+Du%2C+Yubin+Zheng%2C+and+Gongshen+Liu%2E+2024%2E+Investigating+multi%2Dhop+factual+shortcuts+in+knowledge+editing+of+large+language+models%2E+arXiv+prepri)
29. Shahar Katz and Yonatan Belinkov. 2023. Visit: Visualizing and interpreting the semantic information flow of transformers. arXiv preprint arXiv:2305.13417. [find](https://scholar.google.com/scholar?q=Shahar+Katz+and+Yonatan+Belinkov%2E+2023%2E+Visit%3A+Visualizing+and+interpreting+the+semantic+information+flow+of+transformers%2E+arXiv+preprint+arXiv%3A2305%2E13417%2E)
30. Vedang Lad, Wes Gurnee, and Max Tegmark. 2024. The remarkable robustness of llms: Stages of inference? arXiv preprint arXiv:2406.19384. — [arXiv:2406.19384](https://arxiv.org/abs/2406.19384) [find](https://scholar.google.com/scholar?q=Vedang+Lad%2C+Wes+Gurnee%2C+and+Max+Tegmark%2E+2024%2E+The+remarkable+robustness+of+llms%3A+Stages+of+inference%3F+arXiv+preprint+arXiv%3A2406%2E19384%2E+%E2%80%94+%5BarXiv%3A2406%2E19384%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2406%2E19384%29)
31. Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. 2024a. Understanding and patching compositional reasoning in llms. arXiv preprint arXiv:2402.14328. [find](https://scholar.google.com/scholar?q=Zhaoyi+Li%2C+Gangwei+Jiang%2C+Hong+Xie%2C+Linqi+Song%2C+Defu+Lian%2C+and+Ying+Wei%2E+2024a%2E+Understanding+and+patching+compositional+reasoning+in+llms%2E+arXiv+preprint+arXiv%3A2402%2E14328%2E)
32. Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. 2024b. Chain of thought empowers transformers to solve inherently serial problems. arXiv preprint arXiv:2402.12875. [find](https://scholar.google.com/scholar?q=Zhiyuan+Li%2C+Hong+Liu%2C+Denny+Zhou%2C+and+Tengyu+Ma%2E+2024b%2E+Chain+of+thought+empowers+transformers+to+solve+inherently+serial+problems%2E+arXiv+preprint+arXiv%3A2402%2E12875%2E)
33. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). arXiv preprint arXiv:2305.20050.
34. I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=I+Loshchilov%2E+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2E)
35. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. 2024. [Improve mathematical reasoning in language models by automated process supervision](https://md.research.elamur.ai/free/en/papers/luo-2024-improve-mathematical-reasoning-language-models/). arXiv preprint arXiv:2406.06592.
36. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+gpt%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A17359%E2%80%9317372%2E)
37. AI Meta. 2024a. Introducing meta llama 3: The most capable openly available llm to date. Meta AI. [find](https://scholar.google.com/scholar?q=AI+Meta%2E+2024a%2E+Introducing+meta+llama+3%3A+The+most+capable+openly+available+llm+to+date%2E+Meta+AI%2E)
38. AI Meta. 2024b. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. Meta AI. [find](https://scholar.google.com/scholar?q=AI+Meta%2E+2024b%2E+Llama+3%2E2%3A+Revolutionizing+edge+ai+and+vision+with+open%2C+customizable+models%2E+Meta+AI%2E)
39. Ercong Nie, Helmut Schmid, and Hinrich Schütze. 2025. Mechanistic understanding and mitigation of language confusion in english-centric large language models. arXiv preprint arXiv:2505.16538. [find](https://scholar.google.com/scholar?q=Ercong+Nie%2C+Helmut+Schmid%2C+and+Hinrich+Sch%C3%BCtze%2E+2025%2E+Mechanistic+understanding+and+mitigation+of+language+confusion+in+english%2Dcentric+large+language+models%2E+arXiv+preprint+arXiv%3A2505%2E16538%2E)
40. Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. 2024. Arithmetic without algorithms: Language models solve math with a bag of heuristics. arXiv preprint arXiv:2410.21272. [find](https://scholar.google.com/scholar?q=Yaniv+Nikankin%2C+Anja+Reusch%2C+Aaron+Mueller%2C+and+Yonatan+Belinkov%2E+2024%2E+Arithmetic+without+algorithms%3A+Language+models+solve+math+with+a+bag+of+heuristics%2E+arXiv+preprint+arXiv%3A2410%2E21272%2E)
41. Nostalgebraist. 2020. Interpreting gpt: the logit lens. [find](https://scholar.google.com/scholar?q=Nostalgebraist%2E+2020%2E+Interpreting+gpt%3A+the+logit+lens%2E)
42. Chris Olah. 2022. Mechanistic interpretability, variables, and the importance of interpretable bases. In Transformer Circuits Thread. [find](https://scholar.google.com/scholar?q=Chris+Olah%2E+2022%2E+Mechanistic+interpretability%2C+variables%2C+and+the+importance+of+interpretable+bases%2E+In+Transformer+Circuits+Thread%2E)
43. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. [In-context learning and induction heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/). arXiv preprint arXiv:2209.11895.
44. OpenAI. 2024. Learning to reason with llms. Accessed: 19-09-2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024%2E+Learning+to+reason+with+llms%2E+Accessed%3A+19%2D09%2D2024%2E)
45. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are nlp models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) arXiv preprint arXiv:2103.07191.
46. Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2024. Mutual reasoning makes smaller llms stronger problem-solvers. arXiv preprint arXiv:2408.06195. [find](https://scholar.google.com/scholar?q=Zhenting+Qi%2C+Mingyuan+Ma%2C+Jiahang+Xu%2C+Li+Lyna+Zhang%2C+Fan+Yang%2C+and+Mao+Yang%2E+2024%2E+Mutual+reasoning+makes+smaller+llms+stronger+problem%2Dsolvers%2E+arXiv+preprint+arXiv%3A2408%2E06195%2E)
47. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
48. Subhro Roy and Dan Roth. 2016. [Solving general arithmetic word problems](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/). arXiv preprint arXiv:1608.01413.
49. Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. 2023. Memory injections: Correcting multi-hop reasoning failures during inference in transformer-based language models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing an… [find](https://scholar.google.com/scholar?q=Mansi+Sakarvadia%2C+Aswathy+Ajith%2C+Arham+Khan%2C+Daniel+Grzenda%2C+Nathaniel+Hudson%2C+Andr%C3%A9+Bauer%2C+Kyle+Chard%2C+and+Ian+Foster%2E+2023%2E+Memory+injections%3A+Correcting+multi%2Dhop+reasoning+failures+during+inferenc)
50. Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. 2022. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210.01892. — [arXiv:2210.01892](https://arxiv.org/abs/2210.01892) [find](https://scholar.google.com/scholar?q=Adam+Scherlis%2C+Kshitij+Sachan%2C+Adam+S+Jermyn%2C+Joe+Benton%2C+and+Buck+Shlegeris%2E+2022%2E+Polysemanticity+and+capacity+in+neural+networks%2E+arXiv+preprint+arXiv%3A2210%2E01892%2E+%E2%80%94+%5BarXiv%3A2210%2E01892%5D%28https%3A%2F%2Farxiv)
51. KaShun Shum, Shizhe Diao, and Tong Zhang. 2023. [Automatic prompt augmentation and selection with chain-of-thought from labeled data](https://md.research.elamur.ai/free/en/papers/shum-2023-automatic-prompt-augmentation-selection-chain/). arXiv preprint arXiv:2302.12822.
52. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. [Scaling llm test-time compute optimally can be more effective than scaling model parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/). arXiv preprint arXiv:2408.03314.
53. Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. arXiv preprint arXiv:2305.15054. [find](https://scholar.google.com/scholar?q=Alessandro+Stolfo%2C+Yonatan+Belinkov%2C+and+Mrinmaya+Sachan%2E+2023%2E+A+mechanistic+interpretation+of+arithmetic+reasoning+in+language+models+using+causal+mediation+analysis%2E+arXiv+preprint+arXiv%3A2305%2E15054)
54. Adly Templeton. 2024. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Anthropic. [find](https://scholar.google.com/scholar?q=Adly+Templeton%2E+2024%2E+Scaling+monosemanticity%3A+Extracting+interpretable+features+from+claude+3+sonnet%2E+Anthropic%2E)
55. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
56. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
57. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388– 12401. [find](https://scholar.google.com/scholar?q=Jesse+Vig%2C+Sebastian+Gehrmann%2C+Yonatan+Belinkov%2C+Sharon+Qian%2C+Daniel+Nevo%2C+Yaron+Singer%2C+and+Stuart+Shieber%2E+2020%2E+Investigating+gender+bias+in+language+models+using+causal+mediation+analysis%2E+Advance)
58. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022a. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593. [find](https://scholar.google.com/scholar?q=Kevin+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+2022a%2E+Interpretability+in+the+wild%3A+a+circuit+for+indirect+object+identification+in+gpt%2D2+small%2E+arXiv+preprint+a)
59. Mingyang Wang, Heike Adel, Lukas Lange, Yihong Liu, Ercong Nie, Jannik Strötgen, and Hinrich Schütze. 2025. [Lost in multilinguality: Dissecting cross-lingual factual inconsistency in transformer language models](https://md.research.elamur.ai/free/en/papers/wang-2025-lost-multilinguality-dissecting-cross-lingual/). arXiv preprint arXiv:2504.04264.
60. Zeping Yu and Sophia Ananiadou. 2023. Neuron-level knowledge attribution in large language models. arXiv preprint arXiv:2312.12141. [find](https://scholar.google.com/scholar?q=Zeping+Yu+and+Sophia+Ananiadou%2E+2023%2E+Neuron%2Dlevel+knowledge+attribution+in+large+language+models%2E+arXiv+preprint+arXiv%3A2312%2E12141%2E)
61. Zeping Yu and Sophia Ananiadou. 2024a. How do large language models learn in-context? query and key matrices of in-context heads are two towers for metric learning. arXiv preprint arXiv:2402.02872. [find](https://scholar.google.com/scholar?q=Zeping+Yu+and+Sophia+Ananiadou%2E+2024a%2E+How+do+large+language+models+learn+in%2Dcontext%3F+query+and+key+matrices+of+in%2Dcontext+heads+are+two+towers+for+metric+learning%2E+arXiv+preprint+arXiv%3A2402%2E02872%2E)
62. Zeping Yu and Sophia Ananiadou. 2024b. Interpreting arithmetic mechanism in large language models through comparative neuron analysis. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3293–3306. [find](https://scholar.google.com/scholar?q=Zeping+Yu+and+Sophia+Ananiadou%2E+2024b%2E+Interpreting+arithmetic+mechanism+in+large+language+models+through+comparative+neuron+analysis%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Nat)
63. Fred Zhang and Neel Nanda. 2023. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309.16042. [find](https://scholar.google.com/scholar?q=Fred+Zhang+and+Neel+Nanda%2E+2023%2E+Towards+best+practices+of+activation+patching+in+language+models%3A+Metrics+and+methods%2E+arXiv+preprint+arXiv%3A2309%2E16042%2E)
64. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625.
65. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022b. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
66. Xuezhi Wang and Denny Zhou. 2024. Chain-of-thought reasoning without prompting. arXiv preprint arXiv:2402.10200. [find](https://scholar.google.com/scholar?q=Xuezhi+Wang+and+Denny+Zhou%2E+2024%2E+Chain%2Dof%2Dthought+reasoning+without+prompting%2E+arXiv+preprint+arXiv%3A2402%2E10200%2E)
67. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837.
68. Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah D Goodman, Christopher D Manning, and Christopher Potts. 2024. pyvene: A library for understanding and improving pytorch models via interventions. arXiv preprint arXiv:2403.07809. [find](https://scholar.google.com/scholar?q=Zhengxuan+Wu%2C+Atticus+Geiger%2C+Aryaman+Arora%2C+Jing+Huang%2C+Zheng+Wang%2C+Noah+D+Goodman%2C+Christopher+D+Manning%2C+and+Christopher+Potts%2E+2024%2E+pyvene%3A+A+library+for+understanding+and+improving+pytorch+model)
69. Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. 2024. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402.16837. [find](https://scholar.google.com/scholar?q=Sohee+Yang%2C+Elena+Gribovskaya%2C+Nora+Kassner%2C+Mor+Geva%2C+and+Sebastian+Riedel%2E+2024%2E+Do+large+language+models+latently+perform+multi%2Dhop+reasoning%3F+arXiv+preprint+arXiv%3A2402%2E16837%2E)
70. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024a. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2024a%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Advances+in+Neural+In)
71. Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. 2024b. Knowledge circuits in pretrained transformers. Advances in Neural Information Processing Systems, 37:118571–118602. [find](https://scholar.google.com/scholar?q=Yunzhi+Yao%2C+Ningyu+Zhang%2C+Zekun+Xi%2C+Mengru+Wang%2C+Ziwen+Xu%2C+Shumin+Deng%2C+and+Huajun+Chen%2E+2024b%2E+Knowledge+circuits+in+pretrained+transformers%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A118)

*Updated 30 October 2026*