---
title: "Rethinking Interpretability in the Era of Large Language Models"
description: "The authors run no new experiments and release no resource. 159 references, cited by 43 papers here."
authors: ["Chandan Singh", "Jeevana Priya Inala", "Michel Galley", "Rich Caruana", "Jianfeng Gao"]
year: 2024
venue: "arXiv:2402.01761"
source: "https://arxiv.org/abs/2402.01761"
doi: "10.48550/arxiv.2402.01761"
arxiv: "2402.01761"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/singh-2024-rethinking-interpretability-era-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/singh-2024-rethinking-interpretability-era-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Rethinking Interpretability in the Era of Large Language Models

# Rethinking Interpretability in the Era of Large Language Models

**Authors:** [Chandan Singh](https://md.research.elamur.ai/free/en/authors/chandan-singh/), Jeevana Priya Inala, [Michel Galley](https://md.research.elamur.ai/free/en/authors/michel-galley/), Rich Caruana and [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) · **Year:** 2024 · **Venue:** arXiv:2402.01761 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.01761) · [PDF](https://arxiv.org/pdf/2402.01761) · [DOI 10.48550/arxiv.2402.01761](https://doi.org/10.48550/arxiv.2402.01761) · [arXiv 2402.01761](https://arxiv.org/abs/2402.01761) · [OpenAlex](https://openalex.org/W4391591671)

**Citations:** 44 citations (OpenAlex, 2026-10-14) · [43](https://md.research.elamur.ai/free/en/papers/singh-2024-rethinking-interpretability-era-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This position paper from Microsoft Research asks how interpretable machine learning should evolve now that large language models can describe patterns in plain language. Classic tools such as sparse linear models, decision trees, feature-importance scores and saliency maps communicate through narrow channels. LLMs, by contrast, can answer targeted follow-up questions about their own reasoning or about a dataset. The authors run no new experiments and release no resource. Instead, they survey and organize the field into two strands. The first is explaining an LLM, either one generation at a time (local) or as a whole system (global/mechanistic). The second is using an LLM to explain a new text or tabular dataset. They also discuss evaluation, favoring real outcomes with human users and measurable model improvements over simple human ratings. Their core claim is that LLMs can make interpretability far more ambitious. This depends on solving hallucinated or unfaithful explanations, prompt sensitivity, and the cost and opacity of huge models. The two research priorities they name are dataset explanation for knowledge discovery and interactive explanation.

## Key points
- **Taxonomy (Fig. 1):** The paper groups methods into three areas:
  - *Local explanation:* feature attribution such as SHAP, natural-language rationales, chain of thought, and data grounding via RAG.
  - *Global/mechanistic analysis:* attention-head importance and summaries, circuit analysis, and influence functions.
  - *Dataset explanation:* interpretable n-gram models, tree of prompts, and tabular visualizations.
- **Definition of LLM:** The authors mean transformer models with tens to hundreds of billions of parameters, such as PaLM, LLaMA and GPT-4. Prompting, including few-shot prompting, is the main interface they consider.
- **Evaluation options:**
  - Human-centered usefulness and complementarity in human–AI teams.
  - Automated checks, such as whether an explanation can simulate a component's behavior or recover a ground-truth function.
  - Gains in downstream performance, for example few-shot accuracy, removal of shortcuts, or model editing.
  - The authors caution that LLMs used as judges may score their own outputs too favorably.
- **Reliability concerns:** Cited studies show that LLM explanations can contradict the model's answers on related questions. They may also fail to entail its predictions. Proposed remedies include self-verification, iterative prompting and self-consistency methods.
- **Future directions:**
  - Generating hypotheses from data in opaque domains such as chemical compounds and DNA.
  - Conversational model auditing in the style of Talk2Model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 92 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 130 works

## Cite

Cite as: Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana and Jianfeng Gao. 2024. Rethinking Interpretability in the Era of Large Language Models. arXiv:2402.01761.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{singh2024rethinking,
  title = {{Rethinking Interpretability in the Era of Large Language Models}},
  author = {Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana and Jianfeng Gao},
  year = {2024},
  doi = {10.48550/arxiv.2402.01761},
  eprint = {2402.01761},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.01761}
}
```

- [← previous LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models](https://md.research.elamur.ai/free/en/papers/liusie-2024-llm-comparative-assessment-zero-shot/)
- [next → Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2024-self-checker-plug-play-modules/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

159 references, 50 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [1] Finale Doshi-Velez and Been Kim. A roadmap for a rigorous science of interpretability. arXiv preprint arXiv:1702.[08608, 2017](https://md.research.elamur.ai/free/en/papers/doshi-velez-2017-towards-rigorous-science-interpretable-machine/). ,→1.
2. [20] Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. [Explainability for large language models: A survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/). arXiv preprint arXiv:2309.01029, 2023. ,→1.
3. [2] W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences of the United States of America, 116(44):22071–22080, 2019. ,→2. [find](https://scholar.google.com/scholar?q=%5B2%5D+W%2E+James+Murdoch%2C+Chandan+Singh%2C+Karl+Kumbier%2C+Reza+Abbasi%2DAsl%2C+and+Bin+Yu%2E+Definitions%2C+methods%2C+and+applications+in+interpretable+machine+learning%2E+Proceedings+of+the+National+Academy+of+Science)
4. [21] Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent AI: A survey on interpreting the inner structures of deep neural networks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 464–483. [find](https://scholar.google.com/scholar?q=%5B21%5D+Tilman+Ra%CC%88uker%2C+Anson+Ho%2C+Stephen+Casper%2C+and+Dylan+Hadfield%2DMenell%2E+Toward+transparent+AI%3A+A+survey+on+interpreting+the+inner+structures+of+deep+neural+networks%2E+In+2023+IEEE+Conference+on+Secur)
5. [3] Christoph Molnar. Interpretable machine learning. Lulu. com, 2019. ,→1. [find](https://scholar.google.com/scholar?q=%5B3%5D+Christoph+Molnar%2E+Interpretable+machine+learning%2E+Lulu%2E+com%2C+2019%2E+%2C%E2%86%921%2E)
6. [22] Abeba Birhane, Atoosa Kasirzadeh, David Leslie, and Sandra Wachter. Science in the age of large language models. Nature Reviews Physics, pages 1–4, 2023. ,→1 and 7 [find](https://scholar.google.com/scholar?q=%5B22%5D+Abeba+Birhane%2C+Atoosa+Kasirzadeh%2C+David+Leslie%2C+and+Sandra+Wachter%2E+Science+in+the+age+of+large+language+models%2E+Nature+Reviews+Physics%2C+pages+1%E2%80%934%2C+2023%2E+%2C%E2%86%921+and+7)
7. [4] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. [Interpretable machine learning: Fundamental principles and 10 grand challenges](https://md.research.elamur.ai/free/en/papers/rudin-2021-interpretable-machine-learning-fundamental/). arXiv preprint arXiv:2103.11251, 2021. ,→1.
8. [23] Luca Pion-Tonachini, Kristofer Bouchard, Hector Garcia Martin, Sean Peisert, W Bradley Holtz, Anil Aswani, Dipankar Dwivedi, Haruko Wainwright, Ghan-shyam Pilania, Benjamin Nachman, et al. Learning from learning machines: a new generation of AI technology to meet the needs of science. arXiv pre… — [arXiv:2111.13786](https://arxiv.org/abs/2111.13786) [find](https://scholar.google.com/scholar?q=%5B23%5D+Luca+Pion%2DTonachini%2C+Kristofer+Bouchard%2C+Hector+Garcia+Martin%2C+Sean+Peisert%2C+W+Bradley+Holtz%2C+Anil+Aswani%2C+Dipankar+Dwivedi%2C+Haruko+Wainwright%2C+Ghan%2Dshyam+Pilania%2C+Benjamin+Nachman%2C+et+al%2E+Learni)
9. [5] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144. ACM, 2016. ,→1 and 4 — ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/)
10. [24] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [PaLM: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113, 2023. ,→3.
11. [6] Scott M Lundberg and Su-In Lee. [A unified approach to interpreting model predictions](https://md.research.elamur.ai/free/en/papers/lundberg-2017-unified-approach-interpreting-model-predictions/). In Advances in Neural Information Processing Systems, pages 4768–4777, 2017. ,→1 and 4
12. [7] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, 2015. ,→1. — [arXiv:1506.06579](https://arxiv.org/abs/1506.06579) [find](https://scholar.google.com/scholar?q=%5B7%5D+Jason+Yosinski%2C+Jeff+Clune%2C+Anh+Nguyen%2C+Thomas+Fuchs%2C+and+Hod+Lipson%2E+Understanding+neural+networks+through+deep+visualization%2E+arXiv+preprint+arXiv%3A1506%2E06579%2C+2015%2E+%2C%E2%86%921%2E+%E2%80%94+%5BarXiv%3A1506%2E06579%5D%28htt)
13. [25] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022. ,→3…
14. [8] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, and Antonio Torralba. GAN dissection: Visualizing and understanding generative adversarial networks. arXiv preprint arXiv:1811.10597, 2018. ,→1. — [arXiv:1811.10597](https://arxiv.org/abs/1811.10597) [find](https://scholar.google.com/scholar?q=%5B8%5D+David+Bau%2C+Jun%2DYan+Zhu%2C+Hendrik+Strobelt%2C+Bolei+Zhou%2C+Joshua+B+Tenenbaum%2C+William+T+Freeman%2C+and+Antonio+Torralba%2E+GAN+dissection%3A+Visualizing+and+understanding+generative+adversarial+networks%2E+ar)
15. [26] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. [Towards expert-level medical question answering with large language models](https://md.research.elamur.ai/free/en/papers/singhal-2023-towards-expert-level-medical-question/). arXiv preprint arXiv:2305.09617, 2023. ,→3.
16. [9] Sarah Tan, Rich Caruana, Giles Hooker, and Yin Lou. Distill-and-Compare: Auditing black-box models using transparent model distillation. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 303–310, 2018. ,→1. [find](https://scholar.google.com/scholar?q=%5B9%5D+Sarah+Tan%2C+Rich+Caruana%2C+Giles+Hooker%2C+and+Yin+Lou%2E+Distill%2Dand%2DCompare%3A+Auditing+black%2Dbox+models+using+transparent+model+distillation%2E+In+Proceedings+of+the+2018+AAAI%2FACM+Conference+on+AI%2C+Ethic)
17. [27] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). arXiv preprint arXiv:1711.11279, 2017. ,→3. — [arXiv:1711.11279](https://arxiv.org/abs/1711.11279) [find](https://scholar.google.com/scholar?q=%5B27%5D+Been+Kim%2C+Martin+Wattenberg%2C+Justin+Gilmer%2C+Carrie+Cai%2C+James+Wexler%2C+Fernanda+Viegas%2C+and+Rory+Sayres%2E+Interpretability+beyond+feature+attribution%3A+Quantitative+testing+with+concept+activation+v)
18. [10] Wooseok Ha, Chandan Singh, Francois Lanusse, Srigokul Upadhyayula, and Bin Yu. Adaptive wavelet distillation from neural networks through interpretations. Advances in Neural Information Processing Systems, 34:20669–20682, 2021. ,→1. [find](https://scholar.google.com/scholar?q=%5B10%5D+Wooseok+Ha%2C+Chandan+Singh%2C+Francois+Lanusse%2C+Srigokul+Upadhyayula%2C+and+Bin+Yu%2E+Adaptive+wavelet+distillation+from+neural+networks+through+interpretations%2E+Advances+in+Neural+Information+Processin)
19. [28] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pages 9505–9515, 2018. ,→3. [find](https://scholar.google.com/scholar?q=%5B28%5D+Julius+Adebayo%2C+Justin+Gilmer%2C+Michael+Muelly%2C+Ian+Goodfellow%2C+Moritz+Hardt%2C+and+Been+Kim%2E+Sanity+checks+for+saliency+maps%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+9505%E2%80%939515%2C+)
20. [11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020. ,→1 and 4
21. [29] Fateme Hashemi Chaleshtori, Atreya Ghosal, and Ana Marasović. On evaluating explanation utility for Human-AI decision-making in NLP. In XAI in Action: Past, Present, and Future Applications, 2023. ,→3. [find](https://scholar.google.com/scholar?q=%5B29%5D+Fateme+Hashemi+Chaleshtori%2C+Atreya+Ghosal%2C+and+Ana+Marasovic%CC%81%2E+On+evaluating+explanation+utility+for+Human%2DAI+decision%2Dmaking+in+NLP%2E+In+XAI+in+Action%3A+Past%2C+Present%2C+and+Future+Applications%2C+202)
22. [12] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023. ,→3 and 4
23. [30] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems… [find](https://scholar.google.com/scholar?q=%5B30%5D+Gagan+Bansal%2C+Tongshuang+Wu%2C+Joyce+Zhou%2C+Raymond+Fok%2C+Besmira+Nushi%2C+Ece+Kamar%2C+Marco+Tulio+Ribeiro%2C+and+Daniel+Weld%2E+Does+the+whole+exceed+its+parts%3F+the+effect+of+AI+explanations+on+complementa)
24. [13] OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023. ,→1, 3, and 5
25. [14] Bryce Goodman and Seth Flaxman. European union regulations on algorithmic decision-making and a” right to explanation”. arXiv preprint arXiv:1606.08813, 2016. ,→1. — [arXiv:1606.08813](https://arxiv.org/abs/1606.08813) [find](https://scholar.google.com/scholar?q=%5B14%5D+Bryce+Goodman+and+Seth+Flaxman%2E+European+union+regulations+on+algorithmic+decision%2Dmaking+and+a%E2%80%9D+right+to+explanation%E2%80%9D%2E+arXiv+preprint+arXiv%3A1606%2E08813%2C+2016%2E+%2C%E2%86%921%2E+%E2%80%94+%5BarXiv%3A1606%2E08813%5D%28https%3A%2F%2Far)
26. [31] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Thirty-seventh Conference on Neural Information Processing…
27. [15] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. [Concrete problems in AI safety](https://md.research.elamur.ai/free/en/papers/amodei-2016-concrete-problems-ai-safety/). arXiv preprint arXiv:1606.06565, 2016. (Not cited.)
28. [16] Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411–437, 2020. ,→1. [find](https://scholar.google.com/scholar?q=%5B16%5D+Iason+Gabriel%2E+Artificial+intelligence%2C+values%2C+and+alignment%2E+Minds+and+machines%2C+30%283%29%3A411%E2%80%93437%2C+2020%2E+%2C%E2%86%921%2E)
29. [32] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models, 2023. ,→3 and 5 [find](https://scholar.google.com/scholar?q=%5B32%5D+Steven+Bills%2C+Nick+Cammarata%2C+Dan+Mossing%2C+Henk+Tillman%2C+Leo+Gao%2C+Gabriel+Goh%2C+Ilya+Sutskever%2C+Jan+Leike%2C+Jeff+Wu%2C+and+William+Saunders%2E+Language+models+can+explain+neurons+in+language+models%2C+20)
30. [17] Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47– 60, 2023. ,→1 and 7 [find](https://scholar.google.com/scholar?q=%5B17%5D+Hanchen+Wang%2C+Tianfan+Fu%2C+Yuanqi+Du%2C+Wenhao+Gao%2C+Kexin+Huang%2C+Ziming+Liu%2C+Payal+Chandak%2C+Shengchao+Liu%2C+Peter+Van+Katwyk%2C+Andreea+Deac%2C+et+al%2E+Scientific+discovery+in+the+age+of+artificial+intell)
31. [33] Chandan Singh, Aliyah R Hsu, Richard Antonello, Shailee Jain, Alexander G Huth, Bin Yu, and Jianfeng Gao. Explaining black box text modules in natural language with language models. arXiv preprint arXiv:2305.09863, 2023. ,→3 and 5 — [arXiv:2305.09863](https://arxiv.org/abs/2305.09863) [find](https://scholar.google.com/scholar?q=%5B33%5D+Chandan+Singh%2C+Aliyah+R+Hsu%2C+Richard+Antonello%2C+Shailee+Jain%2C+Alexander+G+Huth%2C+Bin+Yu%2C+and+Jianfeng+Gao%2E+Explaining+black+box+text+modules+in+natural+language+with+language+models%2E+arXiv+preprin)
32. [34] Sarah Schwettmann, Tamar Rott Shaham, Joanna Materzynska, Neil Chowdhury, Shuang Li, Jacob Andreas, David Bau, and Antonio Torralba. FIND: A function description benchmark for evaluating interpretability methods. arXiv e-prints, pages arXiv–2309, 2023. ,→3. [find](https://scholar.google.com/scholar?q=%5B34%5D+Sarah+Schwettmann%2C+Tamar+Rott+Shaham%2C+Joanna+Materzynska%2C+Neil+Chowdhury%2C+Shuang+Li%2C+Jacob+Andreas%2C+David+Bau%2C+and+Antonio+Torralba%2E+FIND%3A+A+function+description+benchmark+for+evaluating+interpre)
33. [53] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8763. PML…
34. [35] Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. [Goal driven discovery of distributional differences via language descriptions](https://md.research.elamur.ai/free/en/papers/zhong-2023-goal-driven-discovery-distributional-differences/). ArXiv, abs/2302.14233, 2023. ,→3 and 6
35. [54] Himabindu Lakkaraju, Dylan Slack, Yuxin Chen, Chenhao Tan, and Sameer Singh. Rethinking explainability as a dialogue: A practitioner’s perspective. arXiv preprint arXiv:2202.01875, 2022. ,→4. — [arXiv:2202.01875](https://arxiv.org/abs/2202.01875) [find](https://scholar.google.com/scholar?q=%5B54%5D+Himabindu+Lakkaraju%2C+Dylan+Slack%2C+Yuxin+Chen%2C+Chenhao+Tan%2C+and+Sameer+Singh%2E+Rethinking+explainability+as+a+dialogue%3A+A+practitioner%E2%80%99s+perspective%2E+arXiv+preprint+arXiv%3A2202%2E01875%2C+2022%2E+%2C%E2%86%924%2E+%E2%80%94+%5B)
36. [36] Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. [Faithfulness tests for natural language explanations](https://md.research.elamur.ai/free/en/papers/atanasova-2023-faithfulness-tests-natural-language-explanations/). arXiv preprint arXiv:2305.18029, 2023. ,→3.
37. [55] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. ICML, 2017. ,→4. [find](https://scholar.google.com/scholar?q=%5B55%5D+Mukund+Sundararajan%2C+Ankur+Taly%2C+and+Qiqi+Yan%2E+Axiomatic+attribution+for+deep+networks%2E+ICML%2C+2017%2E+%2C%E2%86%924%2E)
38. [56] Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65:211–222, 2017. ,→4. [find](https://scholar.google.com/scholar?q=%5B56%5D+Gre%CC%81goire+Montavon%2C+Sebastian+Lapuschkin%2C+Alexander+Binder%2C+Wojciech+Samek%2C+and+Klaus%2DRobert+Mu%CC%88ller%2E+Explaining+nonlinear+classification+decisions+with+deep+taylor+decomposition%2E+Pattern+Recogni)
39. [37] Letitia Parcalabescu and Anette Frank. On measuring faithfulness of natural language explanations. arXiv preprint arXiv:2311.07466, 2023. (Not cited.) — [arXiv:2311.07466](https://arxiv.org/abs/2311.07466) [find](https://scholar.google.com/scholar?q=%5B37%5D+Letitia+Parcalabescu+and+Anette+Frank%2E+On+measuring+faithfulness+of+natural+language+explanations%2E+arXiv+preprint+arXiv%3A2311%2E07466%2C+2023%2E+%28Not+cited%2E%29+%E2%80%94+%5BarXiv%3A2311%2E07466%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
40. [38] Hanjie Chen, Faeze Brahman, Xiang Ren, Yangfeng Ji, Yejin Choi, and Swabha Swayamdipta. Rev: information-theoretic evaluation of free-text rationales. arXiv preprint arXiv:2210.04982, 2022. ,→3. [find](https://scholar.google.com/scholar?q=%5B38%5D+Hanjie+Chen%2C+Faeze+Brahman%2C+Xiang+Ren%2C+Yangfeng+Ji%2C+Yejin+Choi%2C+and+Swabha+Swayamdipta%2E+Rev%3A+information%2Dtheoretic+evaluation+of+free%2Dtext+rationales%2E+arXiv+preprint+arXiv%3A2210%2E04982%2C+2022%2E+%2C%E2%86%923%2E)
41. [57] Sandipan Sikdar, Parantapa Bhattacharya, and Kieran Heese. Integrated directional gradients: Feature interaction attribution for neural nlp models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural L… [find](https://scholar.google.com/scholar?q=%5B57%5D+Sandipan+Sikdar%2C+Parantapa+Bhattacharya%2C+and+Kieran+Heese%2E+Integrated+directional+gradients%3A+Feature+interaction+attribution+for+neural+nlp+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+th)
42. [39] Satyapriya Krishna, Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, and Himabindu Lakkaraju. Post hoc explanations of language models can improve language models. arXiv preprint arXiv:2305.11426, 2023. ,→3. — [arXiv:2305.11426](https://arxiv.org/abs/2305.11426) [find](https://scholar.google.com/scholar?q=%5B39%5D+Satyapriya+Krishna%2C+Jiaqi+Ma%2C+Dylan+Slack%2C+Asma+Ghandeharioun%2C+Sameer+Singh%2C+and+Himabindu+Lakkaraju%2E+Post+hoc+explanations+of+language+models+can+improve+language+models%2E+arXiv+preprint+arXiv%3A23)
43. [58] Joseph Enguehard. Sequential integrated gradients: a simple but effective method for explaining language models. arXiv preprint arXiv:2305.15853, 2023. ,→4. [find](https://scholar.google.com/scholar?q=%5B58%5D+Joseph+Enguehard%2E+Sequential+integrated+gradients%3A+a+simple+but+effective+method+for+explaining+language+models%2E+arXiv+preprint+arXiv%3A2305%2E15853%2C+2023%2E+%2C%E2%86%924%2E)
44. [40] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. [Orca: Progressive learning from complex explanation traces of GPT-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/). arXiv preprint arXiv:2306.02707, 2023. ,→3.
45. [59] Hugh Chen, Ian C Covert, Scott M Lundberg, and Su-In Lee. Algorithms to estimate shapley value feature attributions. Nature Machine Intelligence, pages 1–12, 2023. ,→4. [find](https://scholar.google.com/scholar?q=%5B59%5D+Hugh+Chen%2C+Ian+C+Covert%2C+Scott+M+Lundberg%2C+and+Su%2DIn+Lee%2E+Algorithms+to+estimate+shapley+value+feature+attributions%2E+Nature+Machine+Intelligence%2C+pages+1%E2%80%9312%2C+2023%2E+%2C%E2%86%924%2E)
46. [41] Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. [Can language models learn from explanations in context?](https://md.research.elamur.ai/free/en/papers/lampinen-2022-can-language-models-learn-explanations/) arXiv preprint arXiv:2204.02329, 2022. ,→3.
47. [60] Siwon Kim, Jihun Yi, Eunji Kim, and Sungroh Yoon. Interpretation of nlp models through input marginalization. arXiv preprint arXiv:2010.13984, 2020. ,→4. [find](https://scholar.google.com/scholar?q=%5B60%5D+Siwon+Kim%2C+Jihun+Yi%2C+Eunji+Kim%2C+and+Sungroh+Yoon%2E+Interpretation+of+nlp+models+through+input+marginalization%2E+arXiv+preprint+arXiv%3A2010%2E13984%2C+2020%2E+%2C%E2%86%924%2E)
48. [42] Xi Ye and Greg Durrett. Explanation selection using unlabeled data for chain-of-thought prompting. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 619–637, 2023. ,→3. [find](https://scholar.google.com/scholar?q=%5B42%5D+Xi+Ye+and+Greg+Durrett%2E+Explanation+selection+using+unlabeled+data+for+chain%2Dof%2Dthought+prompting%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages)
49. [61] Sarah Wiegreffe and Yuval Pinter. [Attention is not not explanation](https://md.research.elamur.ai/free/en/papers/wiegreffe-2019-attention-is-not-not-explanation/). arXiv preprint arXiv:1908.04626, 2019. ,→4.
50. [62] Sarthak Jain and Byron C Wallace. [Attention is not explanation](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/). arXiv preprint arXiv:1902.10186, 2019. ,→4.
51. [43] Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding. Communications of the ACM (CACM), 2023. ,→3. [find](https://scholar.google.com/scholar?q=%5B43%5D+Mengnan+Du%2C+Fengxiang+He%2C+Na+Zou%2C+Dacheng+Tao%2C+and+Xia+Hu%2E+Shortcut+learning+of+large+language+models+in+natural+language+understanding%2E+Communications+of+the+ACM+%28CACM%29%2C+2023%2E+%2C%E2%86%923%2E)
52. [63] Nicholas Kroeger, Dan Ley, Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. Are large language models post hoc explainers? arXiv preprint arXiv:2310.05797, 2023. ,→4. — [arXiv:2310.05797](https://arxiv.org/abs/2310.05797) [find](https://scholar.google.com/scholar?q=%5B63%5D+Nicholas+Kroeger%2C+Dan+Ley%2C+Satyapriya+Krishna%2C+Chirag+Agarwal%2C+and+Himabindu+Lakkaraju%2E+Are+large+language+models+post+hoc+explainers%3F+arXiv+preprint+arXiv%3A2310%2E05797%2C+2023%2E+%2C%E2%86%924%2E+%E2%80%94+%5BarXiv%3A2310%2E05)
53. [44] Cheongwoong Kang and Jaesik Choi. Impact of co-occurrence on factual knowledge of large language models. arXiv preprint arXiv:2310.08256, 2023. (Not cited.) [find](https://scholar.google.com/scholar?q=%5B44%5D+Cheongwoong+Kang+and+Jaesik+Choi%2E+Impact+of+co%2Doccurrence+on+factual+knowledge+of+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E08256%2C+2023%2E+%28Not+cited%2E%29)
54. [64] Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. [e-snli: Natural language inference with natural language explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/). Advances in Neural Information Processing Systems, 31, 2018. ,→4.
55. [45] Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. ”Will you find these shortcuts?” a protocol for evaluating the faithfulness of input salience methods for text classification. arXiv preprint arXiv:2111.07367, 2021. ,→3. [find](https://scholar.google.com/scholar?q=%5B45%5D+Jasmijn+Bastings%2C+Sebastian+Ebert%2C+Polina+Zablotskaia%2C+Anders+Sandholm%2C+and+Katja+Filippova%2E+%E2%80%9DWill+you+find+these+shortcuts%3F%E2%80%9D+a+protocol+for+evaluating+the+faithfulness+of+input+salience+methods+)
56. [65] Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. [Explain yourself! leveraging language models for commonsense reasoning](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/). arXiv preprint arXiv:1906.02361, 2019. ,→4.
57. [46] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual knowledge in GPT. arXiv preprint arXiv:2202.05262, 2022. ,→3 and 5 [find](https://scholar.google.com/scholar?q=%5B46%5D+Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+Locating+and+editing+factual+knowledge+in+GPT%2E+arXiv+preprint+arXiv%3A2202%2E05262%2C+2022%2E+%2C%E2%86%923+and+5)
58. [66] Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In European conference on computer vision, pages 3–19. Springer, 2016. ,→4. [find](https://scholar.google.com/scholar?q=%5B66%5D+Lisa+Anne+Hendricks%2C+Zeynep+Akata%2C+Marcus+Rohrbach%2C+Jeff+Donahue%2C+Bernt+Schiele%2C+and+Trevor+Darrell%2E+Generating+visual+explanations%2E+In+European+conference+on+computer+vision%2C+pages+3%E2%80%9319%2E+Springe)
59. [47] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing at scale, 2022. (Not cited.) [find](https://scholar.google.com/scholar?q=%5B47%5D+Eric+Mitchell%2C+Charles+Lin%2C+Antoine+Bosselut%2C+Chelsea+Finn%2C+and+Christopher+D%2E+Manning%2E+Fast+model+editing+at+scale%2C+2022%2E+%28Not+cited%2E%29)
60. [67] Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, and Huan Liu. LLMs as counterfactual explanation modules: Can ChatGPT explain black-box text classifiers? arXiv preprint arXiv:2309.13340, 2023. ,→4. — [arXiv:2309.13340](https://arxiv.org/abs/2309.13340) [find](https://scholar.google.com/scholar?q=%5B67%5D+Amrita+Bhattacharjee%2C+Raha+Moraffah%2C+Joshua+Garland%2C+and+Huan+Liu%2E+LLMs+as+counterfactual+explanation+modules%3A+Can+ChatGPT+explain+black%2Dbox+text+classifiers%3F+arXiv+preprint+arXiv%3A2309%2E13340%2C+202)
61. [48] Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models, 2023. ,→3. [find](https://scholar.google.com/scholar?q=%5B48%5D+Evan+Hernandez%2C+Belinda+Z%2E+Li%2C+and+Jacob+Andreas%2E+Inspecting+and+editing+knowledge+representations+in+language+models%2C+2023%2E+%2C%E2%86%923%2E)
62. [68] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. arXiv preprint arXiv:2306.13063, 2023. ,→4. [find](https://scholar.google.com/scholar?q=%5B68%5D+Miao+Xiong%2C+Zhiyuan+Hu%2C+Xinyang+Lu%2C+Yifei+Li%2C+Jie+Fu%2C+Junxian+He%2C+and+Bryan+Hooi%2E+Can+LLMs+express+their+uncertainty%3F+an+empirical+evaluation+of+confidence+elicitation+in+LLMs%2E+arXiv+preprint+arX)
63. [49] Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. Interpreting interpretability: understanding data scientists’ use of interpretability tools for machine learning. In Proceedings of the 2020 CHI conference on human factors in computing sys… [find](https://scholar.google.com/scholar?q=%5B49%5D+Harmanpreet+Kaur%2C+Harsha+Nori%2C+Samuel+Jenkins%2C+Rich+Caruana%2C+Hanna+Wallach%2C+and+Jennifer+Wortman+Vaughan%2E+Interpreting+interpretability%3A+understanding+data+scientists%E2%80%99+use+of+interpretability+too)
64. [69] Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju. Quantifying uncertainty in natural language explanations of large language models. arXiv preprint arXiv:2311.03533, 2023. (Not cited.) — [arXiv:2311.03533](https://arxiv.org/abs/2311.03533) [find](https://scholar.google.com/scholar?q=%5B69%5D+Sree+Harsha+Tanneru%2C+Chirag+Agarwal%2C+and+Himabindu+Lakkaraju%2E+Quantifying+uncertainty+in+natural+language+explanations+of+large+language+models%2E+arXiv+preprint+arXiv%3A2311%2E03533%2C+2023%2E+%28Not+cited%2E)
65. [50] Daniel S Weld and Gagan Bansal. The challenge of crafting intelligible intelligence. Communications of the ACM, 62(6):70–79, 2019. ,→3. [find](https://scholar.google.com/scholar?q=%5B50%5D+Daniel+S+Weld+and+Gagan+Bansal%2E+The+challenge+of+crafting+intelligible+intelligence%2E+Communications+of+the+ACM%2C+62%286%29%3A70%E2%80%9379%2C+2019%2E+%2C%E2%86%923%2E)
66. [70] Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models’ reluctance to express uncertainty, 2024. ,→4. [find](https://scholar.google.com/scholar?q=%5B70%5D+Kaitlyn+Zhou%2C+Jena+D%2E+Hwang%2C+Xiang+Ren%2C+and+Maarten+Sap%2E+Relying+on+the+unreliable%3A+The+impact+of+language+models%E2%80%99+reluctance+to+express+uncertainty%2C+2024%2E+%2C%E2%86%924%2E)
67. [51] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/). arXiv preprint arXiv:2211.09085, 2022. ,→3 and 7
68. [71] Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown. Do models explain themselves? Counterfactual simulatability of natural language explanations. arXiv preprint arXiv:2307.08678, 2023. ,→4 and 7 [find](https://scholar.google.com/scholar?q=%5B71%5D+Yanda+Chen%2C+Ruiqi+Zhong%2C+Narutatsu+Ri%2C+Chen+Zhao%2C+He+He%2C+Jacob+Steinhardt%2C+Zhou+Yu%2C+and+Kathleen+McKeown%2E+Do+models+explain+themselves%3F+Counterfactual+simulatability+of+natural+language+explanati)
69. [52] Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Anima Anandkumar. Multi-modal molecule structure-text model for text-based retrieval and editing. ArXiv, abs/2212.10789, 2022. (Not cited.) [find](https://scholar.google.com/scholar?q=%5B52%5D+Shengchao+Liu%2C+Weili+Nie%2C+Chengpeng+Wang%2C+Jiarui+Lu%2C+Zhuoran+Qiao%2C+Ling+Liu%2C+Jian+Tang%2C+Chaowei+Xiao%2C+and+Anima+Anandkumar%2E+Multi%2Dmodal+molecule+structure%2Dtext+model+for+text%2Dbased+retrieval+and+)
70. [72] Xi Ye and Greg Durrett. [The unreliability of explanations in few-shot prompting for textual reasoning](https://md.research.elamur.ai/free/en/papers/ye-2022-unreliability-explanations-few-shot-prompting/). Advances in neural information processing systems, 35:30378–30392, 2022. ,→4 and 7
71. [73] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Informatio… — [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
72. [90] John X Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M Rush. [Text embeddings reveal (almost) as much as text](https://md.research.elamur.ai/free/en/papers/morris-2023-text-embeddings-reveal-almost-as/). arXiv preprint arXiv:2310.06816, 2023. ,→5.
73. [91] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hend… — [Representation Engineering: A Top-Down Approach to AI Transparency](https://md.research.elamur.ai/free/en/papers/zou-2023-representation-engineering-top-down-approach/)
74. [74] Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.07686, 2022. ,→4. [find](https://scholar.google.com/scholar?q=%5B74%5D+Aman+Madaan+and+Amir+Yazdanbakhsh%2E+Text+and+patterns%3A+For+effective+chain+of+thought%2C+it+takes+two+to+tango%2E+arXiv+preprint+arXiv%3A2209%2E07686%2C+2022%2E+%2C%E2%86%924%2E)
75. [75] Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. [Towards understanding chain-of-thought prompting: An empirical study of what matters](https://md.research.elamur.ai/free/en/papers/wang-2023-understanding-chain-thought-prompting-empirical/). arXiv preprint arXiv:2212.10001, 2022. (Not cited.)
76. [92] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. [Eliciting latent predictions from transformers with the tuned lens](https://md.research.elamur.ai/free/en/papers/belrose-2023-eliciting-latent-predictions-transformers-tuned/). arXiv preprint arXiv:2303.08112, 2023. ,→5.
77. [76] Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson E. Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, John Kernion, Kamil.e Lukovsiut.e, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Samuel McCandlish, Sandipan K… — [Measuring Faithfulness in Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/lanham-2023-measuring-faithfulness-chain-thought-reasoning/)
78. [93] Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. PatchScope: A unifying framework for inspecting hidden representations of language models, 2024. ,→5. [find](https://scholar.google.com/scholar?q=%5B93%5D+Asma+Ghandeharioun%2C+Avi+Caciularu%2C+Adam+Pearce%2C+Lucas+Dixon%2C+and+Mor+Geva%2E+PatchScope%3A+A+unifying+framework+for+inspecting+hidden+representations+of+language+models%2C+2024%2E+%2C%E2%86%925%2E)
79. [94] Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153–17163, 2020. ,→5. [find](https://scholar.google.com/scholar?q=%5B94%5D+Jesse+Mu+and+Jacob+Andreas%2E+Compositional+explanations+of+neurons%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A17153%E2%80%9317163%2C+2020%2E+%2C%E2%86%925%2E)
80. [77] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. ,→4. [find](https://scholar.google.com/scholar?q=%5B77%5D+Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+arXiv+preprint+arX)
81. [95] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023. ,→5. [find](https://scholar.google.com/scholar?q=%5B95%5D+Wes+Gurnee%2C+Neel+Nanda%2C+Matthew+Pauly%2C+Katherine+Harvey%2C+Dmitrii+Troitskii%2C+and+Dimitris+Bertsimas%2E+Finding+neurons+in+a+haystack%3A+Case+studies+with+sparse+probing%2E+arXiv+preprint+arXiv%3A2305%2E0161)
82. [78] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of Thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687, 2023. ,→4. [find](https://scholar.google.com/scholar?q=%5B78%5D+Maciej+Besta%2C+Nils+Blach%2C+Ales+Kubicek%2C+Robert+Gerstenberger%2C+Lukas+Gianinazzi%2C+Joanna+Gajda%2C+Tomasz+Lehmann%2C+Michal+Podstawski%2C+Hubert+Niewiadomski%2C+Piotr+Nyczyk%2C+et+al%2E+Graph+of+Thoughts%3A+Solvi)
83. [96] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2022. ,→5. [find](https://scholar.google.com/scholar?q=%5B96%5D+Evan+Hernandez%2C+Sarah+Schwettmann%2C+David+Bau%2C+Teona+Bagashvili%2C+Antonio+Torralba%2C+and+Jacob+Andreas%2E+Natural+language+descriptions+of+deep+visual+features%2E+In+International+Conference+on+Learning)
84. [97] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. arXiv preprint arXiv:2211.00593, 2022. ,→5. [find](https://scholar.google.com/scholar?q=%5B97%5D+Kevin+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+Interpretability+in+the+wild%3A+a+circuit+for+indirect+object+identification+in+GPT%2D2+small%2E+arXiv+preprint+arX)
85. [79] Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). ArXiv, abs/2112.00114, 2021. ,…
86. [98] Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? arXiv preprint arXiv:2310.17191, 2023. ,→5. [find](https://scholar.google.com/scholar?q=%5B98%5D+Jiahai+Feng+and+Jacob+Steinhardt%2E+How+do+language+models+bind+entities+in+context%3F+arXiv+preprint+arXiv%3A2310%2E17191%2C+2023%2E+%2C%E2%86%925%2E)
87. [80] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models, 2022. (Not cited.) [find](https://scholar.google.com/scholar?q=%5B80%5D+Ofir+Press%2C+Muru+Zhang%2C+Sewon+Min%2C+Ludwig+Schmidt%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+Measuring+and+narrowing+the+compositionality+gap+in+language+models%2C+2022%2E+%28Not+cited%2E%29)
88. [99] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. arXiv preprint arXiv:2310.08744, 2023. ,→5. [find](https://scholar.google.com/scholar?q=%5B99%5D+Jack+Merullo%2C+Carsten+Eickhoff%2C+and+Ellie+Pavlick%2E+Circuit+component+reuse+across+tasks+in+transformer+language+models%2E+arXiv+preprint+arXiv%3A2310%2E08744%2C+2023%2E+%2C%E2%86%925%2E)
89. [81] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625, 2022. ,→4.
90. [100] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696, 2021. ,→5. [find](https://scholar.google.com/scholar?q=%5B100%5D+Damai+Dai%2C+Li+Dong%2C+Yaru+Hao%2C+Zhifang+Sui%2C+Baobao+Chang%2C+and+Furu+Wei%2E+Knowledge+neurons+in+pretrained+transformers%2E+arXiv+preprint+arXiv%3A2104%2E08696%2C+2021%2E+%2C%E2%86%925%2E)
91. [82] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/). arXiv preprint arXiv:2303.03846, 2023. ,→4.
92. [101] Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. [Does circuit analysis interpretability scale? Evidence from multiple choice capabilities in chinchilla](https://md.research.elamur.ai/free/en/papers/lieberum-2023-circuit-analysis-interpretability-scale-evidence/). arXiv preprint arXiv:2307.09458, 2023. ,→5.
93. [83] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. [REALM: Retrieval-augmented language model pre-training](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/). ArXiv, abs/2002.08909, 2020. ,→5.
94. [102] Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D. Goodman. [Interpretability at scale: Identifying causal mechanisms in Alpaca](https://md.research.elamur.ai/free/en/papers/wu-2023-interpretability-scale-identifying-causal-mechanisms/). ArXiv, abs/2305.08809, 2023. ,→5.
95. [84] Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Lidén, Zhou Yu, Weizhu Chen, and Jianfeng Gao. [Check your facts and try again: Improving large language models with external knowledge and automated feedback](https://md.research.elamur.ai/free/en/papers/peng-2023-check-facts-try-again-improving/). ArXiv, abs/2302.12813, 2023. ,→5.
96. [103] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1, 2021. ,→5. [find](https://scholar.google.com/scholar?q=%5B103%5D+Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+et+al%2E+A+mathematical+framework+for+transformer+circuits%2E)
97. [85] Theodora Worledge, Judy Hanwen Shen, Nicole Meister, Caleb Winston, and Carlos Guestrin. Unifying corroborative and contributive attributions in large language models. arXiv preprint arXiv:2311.12233, 2023. ,→5 and 6 — [arXiv:2311.12233](https://arxiv.org/abs/2311.12233) [find](https://scholar.google.com/scholar?q=%5B85%5D+Theodora+Worledge%2C+Judy+Hanwen+Shen%2C+Nicole+Meister%2C+Caleb+Winston%2C+and+Carlos+Guestrin%2E+Unifying+corroborative+and+contributive+attributions+in+large+language+models%2E+arXiv+preprint+arXiv%3A2311%2E1)
98. [104] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova Das-Sarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. [In-context learning and induction heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/). arXiv preprint arXiv:2209.11895, 2022. ,→5.
99. [86] Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:1805.[01070, 2018](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/). ,→5.
100. [105] Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Arhitectures and algorithms, 2024. ,→5. [find](https://scholar.google.com/scholar?q=%5B105%5D+Ekin+Akyu%CC%88rek%2C+Bailin+Wang%2C+Yoon+Kim%2C+and+Jacob+Andreas%2E+In%2Dcontext+language+learning%3A+Arhitectures+and+algorithms%2C+2024%2E+%2C%E2%86%925%2E)
101. [106] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? A case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583–30598, 2022. ,→5. [find](https://scholar.google.com/scholar?q=%5B106%5D+Shivam+Garg%2C+Dimitris+Tsipras%2C+Percy+S+Liang%2C+and+Gregory+Valiant%2E+What+can+transformers+learn+in%2Dcontext%3F+A+case+study+of+simple+function+classes%2E+Advances+in+Neural+Information+Processing+Syst)
102. [87] Frederick Liu and Besim Avci. Incorporating priors with feature attribution on text classification. arXiv preprint arXiv:1906.08286, 2019. ,→5. [find](https://scholar.google.com/scholar?q=%5B87%5D+Frederick+Liu+and+Besim+Avci%2E+Incorporating+priors+with+feature+attribution+on+text+classification%2E+arXiv+preprint+arXiv%3A1906%2E08286%2C+2019%2E+%2C%E2%86%925%2E)
103. [88] Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. arXiv preprint arXiv:1808.09031, 2018. ,→5. [find](https://scholar.google.com/scholar?q=%5B88%5D+Rebecca+Marvin+and+Tal+Linzen%2E+Targeted+syntactic+evaluation+of+language+models%2E+arXiv+preprint+arXiv%3A1808%2E09031%2C+2018%2E+%2C%E2%86%925%2E)
104. [107] Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? a study in length generalization. arXiv preprint arXiv:2310.16028, 2023. ,→5. [find](https://scholar.google.com/scholar?q=%5B107%5D+Hattie+Zhou%2C+Arwen+Bradley%2C+Etai+Littwin%2C+Noam+Razin%2C+Omid+Saremi%2C+Josh+Susskind%2C+Samy+Bengio%2C+and+Preetum+Nakkiran%2E+What+algorithms+can+transformers+learn%3F+a+study+in+length+generalization%2E+arX)
105. [89] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does BERT look at? An analysis of bert’s attention. arXiv preprint arXiv:1906.04341, 2019. ,→5. [find](https://scholar.google.com/scholar?q=%5B89%5D+Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D+Manning%2E+What+does+BERT+look+at%3F+An+analysis+of+bert%E2%80%99s+attention%2E+arXiv+preprint+arXiv%3A1906%2E04341%2C+2019%2E+%2C%E2%86%925%2E)
106. [108] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023. ,→5. [find](https://scholar.google.com/scholar?q=%5B108%5D+Roger+Grosse%2C+Juhan+Bae%2C+Cem+Anil%2C+Nelson+Elhage%2C+Alex+Tamkin%2C+Amirhossein+Tajdini%2C+Benoit+Steiner%2C+Dustin+Li%2C+Esin+Durmus%2C+Ethan+Perez%2C+et+al%2E+Studying+large+language+model+generalization+with+)
107. [127] Trevor Hastie and Robert Tibshirani. Generalized additive models. Statistical Science, 1(3):297–318, 1986. ,→6. [find](https://scholar.google.com/scholar?q=%5B127%5D+Trevor+Hastie+and+Robert+Tibshirani%2E+Generalized+additive+models%2E+Statistical+Science%2C+1%283%29%3A297%E2%80%93318%2C+1986%2E+%2C%E2%86%926%2E)
108. [128] Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 623–631, 2013. (Not cited.) [find](https://scholar.google.com/scholar?q=%5B128%5D+Yin+Lou%2C+Rich+Caruana%2C+Johannes+Gehrke%2C+and+Giles+Hooker%2E+Accurate+intelligible+models+with+pairwise+interactions%2E+In+Proceedings+of+the+19th+ACM+SIGKDD+international+conference+on+Knowledge+dis)
109. [109] Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. [Large language models struggle to learn long-tail knowledge](https://md.research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/). In International Conference on Machine Learning, pages 15696–15707. PMLR, 2023. ,→5.
110. [110] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. [Scaling laws and interpretability of learning from repeated data](https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/). arXiv preprint arXiv:2205.10487, 2022. ,→5.
111. [129] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1721–17… [find](https://scholar.google.com/scholar?q=%5B129%5D+Rich+Caruana%2C+Yin+Lou%2C+Johannes+Gehrke%2C+Paul+Koch%2C+Marc+Sturm%2C+and+Noemie+Elhadad%2E+Intelligible+models+for+healthcare%3A+Predicting+pneumonia+risk+and+hospital+30%2Dday+readmission%2E+In+Proceedings+o)
112. [111] Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. [Sources of hallucination by large language models on inference tasks](https://md.research.elamur.ai/free/en/papers/mckenna-2023-sources-hallucination-large-language-models/). arXiv preprint arXiv:2305.14552, 2023. ,→5.
113. [130] L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and Regression Trees. Wadsworth and Brooks, Monterey, CA, 1984. ,→6. [find](https://scholar.google.com/scholar?q=%5B130%5D+L%2E+Breiman%2C+J%2E+H%2E+Friedman%2C+R%2E+A%2E+Olshen%2C+and+C%2E+J%2E+Stone%2E+Classification+and+Regression+Trees%2E+Wadsworth+and+Brooks%2C+Monterey%2C+CA%2C+1984%2E+%2C%E2%86%926%2E)
114. [131] J. Ross Quinlan. Induction of decision trees. Machine learning, 1(1):81–106, 1986. (Not cited.) [find](https://scholar.google.com/scholar?q=%5B131%5D+J%2E+Ross+Quinlan%2E+Induction+of+decision+trees%2E+Machine+learning%2C+1%281%29%3A81%E2%80%93106%2C+1986%2E+%28Not+cited%2E%29)
115. [112] Dong-Ho Lee, Akshen Kadakia, Brihi Joshi, Aaron Chan, Ziyi Liu, Kiran Narahari, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, et al. XMD: An end-to-end framework for interactive explanation-based debugging of nlp models. arXiv preprint arXiv:2210.16978, 2022. ,→5. [find](https://scholar.google.com/scholar?q=%5B112%5D+Dong%2DHo+Lee%2C+Akshen+Kadakia%2C+Brihi+Joshi%2C+Aaron+Chan%2C+Ziyi+Liu%2C+Kiran+Narahari%2C+Takashi+Shibuya%2C+Ryosuke+Mitani%2C+Toshiyuki+Sekiya%2C+Jay+Pujara%2C+et+al%2E+XMD%3A+An+end%2Dto%2Dend+framework+for+interactive)
116. [132] Abhineet Agarwal, Yan Shuo Tan, Omer Ronen, Chandan Singh, and Bin Yu. Hierarchical shrinkage: improving the accuracy and interpretability of tree-based methods. arXiv:2202.00858, 2 2022. arXiv: 2202.00858. ,→6. — [arXiv:2202.00858](https://arxiv.org/abs/2202.00858) [find](https://scholar.google.com/scholar?q=%5B132%5D+Abhineet+Agarwal%2C+Yan+Shuo+Tan%2C+Omer+Ronen%2C+Chandan+Singh%2C+and+Bin+Yu%2E+Hierarchical+shrinkage%3A+improving+the+accuracy+and+interpretability+of+tree%2Dbased+methods%2E+arXiv%3A2202%2E00858%2C+2+2022%2E+arXiv%3A)
117. [113] Dylan Slack, Satyapriya Krishna, Himabindu Lakkaraju, and Sameer Singh. Talktomodel: Understanding machine learning models with open ended dialogues. arXiv preprint arXiv:2207.04154, 2022. ,→5 and 7 — [arXiv:2207.04154](https://arxiv.org/abs/2207.04154) [find](https://scholar.google.com/scholar?q=%5B113%5D+Dylan+Slack%2C+Satyapriya+Krishna%2C+Himabindu+Lakkaraju%2C+and+Sameer+Singh%2E+Talktomodel%3A+Understanding+machine+learning+models+with+open+ended+dialogues%2E+arXiv+preprint+arXiv%3A2207%2E04154%2C+2022%2E+%2C%E2%86%925+a)
118. [133] Chandan Singh, Keyan Nasseri, Yan Shuo Tan, Tiffany Tang, and Bin Yu. imodels: a python package for fitting interpretable models. Journal of Open Source Software, 6(61):3192, 2021. ,→6. [find](https://scholar.google.com/scholar?q=%5B133%5D+Chandan+Singh%2C+Keyan+Nasseri%2C+Yan+Shuo+Tan%2C+Tiffany+Tang%2C+and+Bin+Yu%2E+imodels%3A+a+python+package+for+fitting+interpretable+models%2E+Journal+of+Open+Source+Software%2C+6%2861%29%3A3192%2C+2021%2E+%2C%E2%86%926%2E)
119. [114] Qianli Wang, Tatiana Anikina, Nils Feldhus, Josef van Genabith, Leonhard Hennig, and Sebastian Möller. LLMCheckup: Conversational examination of large language models via interpretability tools. arXiv preprint arXiv:2401.12576, 2024. ,→5. — [arXiv:2401.12576](https://arxiv.org/abs/2401.12576) [find](https://scholar.google.com/scholar?q=%5B114%5D+Qianli+Wang%2C+Tatiana+Anikina%2C+Nils+Feldhus%2C+Josef+van+Genabith%2C+Leonhard+Hennig%2C+and+Sebastian+Mo%CC%88ller%2E+LLMCheckup%3A+Conversational+examination+of+large+language+models+via+interpretability+tools)
120. [134] Yan Shuo Tan, Chandan Singh, Keyan Nasseri, Abhineet Agarwal, and Bin Yu. Fast interpretable greedy-tree sums (figs). arXiv:2201.11931 [cs, stat], 1 2022. arXiv: 2201.11931. ,→6. — [arXiv:2201.11931](https://arxiv.org/abs/2201.11931) [find](https://scholar.google.com/scholar?q=%5B134%5D+Yan+Shuo+Tan%2C+Chandan+Singh%2C+Keyan+Nasseri%2C+Abhineet+Agarwal%2C+and+Bin+Yu%2E+Fast+interpretable+greedy%2Dtree+sums+%28figs%29%2E+arXiv%3A2201%2E11931+%5Bcs%2C+stat%5D%2C+1+2022%2E+arXiv%3A+2201%2E11931%2E+%2C%E2%86%926%2E+%E2%80%94+%5BarXiv%3A2201%2E1)
121. [115] Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. Tell your model where to attend: Post-hoc attention steering for LLMs. arXiv preprint arXiv:2311.02262, 2023. ,→5 and 7 — [arXiv:2311.02262](https://arxiv.org/abs/2311.02262) [find](https://scholar.google.com/scholar?q=%5B115%5D+Qingru+Zhang%2C+Chandan+Singh%2C+Liyuan+Liu%2C+Xiaodong+Liu%2C+Bin+Yu%2C+Jianfeng+Gao%2C+and+Tuo+Zhao%2E+Tell+your+model+where+to+attend%3A+Post%2Dhoc+attention+steering+for+LLMs%2E+arXiv+preprint+arXiv%3A2311%2E02262%2C)
122. [135] Chandan Singh, Armin Askari, Rich Caruana, and Jianfeng Gao. Augmenting interpretable models with large language models during training. Nature Communications, 14(1):7913, 2023. ,→6. [find](https://scholar.google.com/scholar?q=%5B135%5D+Chandan+Singh%2C+Armin+Askari%2C+Rich+Caruana%2C+and+Jianfeng+Gao%2E+Augmenting+interpretable+models+with+large+language+models+during+training%2E+Nature+Communications%2C+14%281%29%3A7913%2C+2023%2E+%2C%E2%86%926%2E)
123. [116] Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558, 2023. ,→5. — [arXiv:2312.13558](https://arxiv.org/abs/2312.13558) [find](https://scholar.google.com/scholar?q=%5B116%5D+Pratyusha+Sharma%2C+Jordan+T+Ash%2C+and+Dipendra+Misra%2E+The+truth+is+in+there%3A+Improving+reasoning+in+language+models+with+layer%2Dselective+rank+reduction%2E+arXiv+preprint+arXiv%3A2312%2E13558%2C+2023%2E+%2C%E2%86%925%2E)
124. [136] Denis Jered McInerney, Geoffrey Young, Jan-Willem van de Meent, and Byron C Wallace. Chill: Zero-shot custom interpretable feature extraction from clinical notes with large language models. arXiv preprint arXiv:2302.12343, 2023. ,→6. [find](https://scholar.google.com/scholar?q=%5B136%5D+Denis+Jered+McInerney%2C+Geoffrey+Young%2C+Jan%2DWillem+van+de+Meent%2C+and+Byron+C+Wallace%2E+Chill%3A+Zero%2Dshot+custom+interpretable+feature+extraction+from+clinical+notes+with+large+language+models%2E+arXi)
125. [117] Victor Dibia. Lida: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models. arXiv preprint arXiv:2303.02927, 2023. ,→6. [find](https://scholar.google.com/scholar?q=%5B117%5D+Victor+Dibia%2E+Lida%3A+A+tool+for+automatic+generation+of+grammar%2Dagnostic+visualizations+and+infographics+using+large+language+models%2E+arXiv+preprint+arXiv%3A2303%2E02927%2C+2023%2E+%2C%E2%86%926%2E)
126. [137] Madeleine Grunde-McLaughlin, Michelle S Lam, Ranjay Krishna, Daniel S Weld, and Jeffrey Heer. Designing LLM chains by adapting techniques from crowdsourcing workflows. arXiv preprint arXiv:2312.11681, 2023. ,→6. [find](https://scholar.google.com/scholar?q=%5B137%5D+Madeleine+Grunde%2DMcLaughlin%2C+Michelle+S+Lam%2C+Ranjay+Krishna%2C+Daniel+S+Weld%2C+and+Jeffrey+Heer%2E+Designing+LLM+chains+by+adapting+techniques+from+crowdsourcing+workflows%2E+arXiv+preprint+arXiv%3A2312%2E)
127. [118] Avanika Narayan, Ines Chami, Laurel Orr, Simran Arora, and Christopher Ré. Can foundation models wrangle your data? arXiv preprint arXiv:2205.09911, 2022. ,→6. — [arXiv:2205.09911](https://arxiv.org/abs/2205.09911) [find](https://scholar.google.com/scholar?q=%5B118%5D+Avanika+Narayan%2C+Ines+Chami%2C+Laurel+Orr%2C+Simran+Arora%2C+and+Christopher+Re%CC%81%2E+Can+foundation+models+wrangle+your+data%3F+arXiv+preprint+arXiv%3A2205%2E09911%2C+2022%2E+%2C%E2%86%926%2E+%E2%80%94+%5BarXiv%3A2205%2E09911%5D%28https%3A%2F%2Farxi)
128. [138] John X Morris, Chandan Singh, Alexander M Rush, Jianfeng Gao, and Yuntian Deng. Tree prompting: efficient task adaptation without fine-tuning. arXiv preprint arXiv:2310.14034, 2023. ,→6. [find](https://scholar.google.com/scholar?q=%5B138%5D+John+X+Morris%2C+Chandan+Singh%2C+Alexander+M+Rush%2C+Jianfeng+Gao%2C+and+Yuntian+Deng%2E+Tree+prompting%3A+efficient+task+adaptation+without+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2310%2E14034%2C+2023%2E+%2C%E2%86%926%2E)
129. [119] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Benchmarking large language models as AI research agents. arXiv preprint arXiv:2310.[03302, 2023](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/). ,→6.
130. [139] Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188, 2023. ,→6 and 7 [find](https://scholar.google.com/scholar?q=%5B139%5D+Liangming+Pan%2C+Michael+Saxon%2C+Wenda+Xu%2C+Deepak+Nathani%2C+Xinyi+Wang%2C+and+William+Yang+Wang%2E+Automatically+correcting+large+language+models%3A+Surveying+the+landscape+of+diverse+self%2Dcorrection+stra)
131. [120] Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-GPT: Table-tuned gpt for diverse table tasks. arXiv preprint arXiv:2310.09263, 2023. ,→6. — [arXiv:2310.09263](https://arxiv.org/abs/2310.09263) [find](https://scholar.google.com/scholar?q=%5B120%5D+Peng+Li%2C+Yeye+He%2C+Dror+Yashar%2C+Weiwei+Cui%2C+Song+Ge%2C+Haidong+Zhang%2C+Danielle+Rifinski+Fainman%2C+Dongmei+Zhang%2C+and+Surajit+Chaudhuri%2E+Table%2DGPT%3A+Table%2Dtuned+gpt+for+diverse+table+tasks%2E+arXiv+prep)
132. [140] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative refinement with… [find](https://scholar.google.com/scholar?q=%5B140%5D+Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katheri)
133. [121] Han Zhang, Xumeng Wen, Shun Zheng, Wei Xu, and Jiang Bian. Towards foundation models for learning on tabular data. arXiv preprint arXiv:2310.07338, 2023. (Not cited.) — [arXiv:2310.07338](https://arxiv.org/abs/2310.07338) [find](https://scholar.google.com/scholar?q=%5B121%5D+Han+Zhang%2C+Xumeng+Wen%2C+Shun+Zheng%2C+Wei+Xu%2C+and+Jiang+Bian%2E+Towards+foundation+models+for+learning+on+tabular+data%2E+arXiv+preprint+arXiv%3A2310%2E07338%2C+2023%2E+%28Not+cited%2E%29+%E2%80%94+%5BarXiv%3A2310%2E07338%5D%28https%3A)
134. [122] Tianping Zhang, Shaowen Wang, Shuicheng Yan, Jian Li, and Qian Liu. Generative table pre-training empowers models for tabular prediction. arXiv preprint arXiv:2305.09696, 2023. ,→6. [find](https://scholar.google.com/scholar?q=%5B122%5D+Tianping+Zhang%2C+Shaowen+Wang%2C+Shuicheng+Yan%2C+Jian+Li%2C+and+Qian+Liu%2E+Generative+table+pre%2Dtraining+empowers+models+for+tabular+prediction%2E+arXiv+preprint+arXiv%3A2305%2E09696%2C+2023%2E+%2C%E2%86%926%2E)
135. [141] Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao, and Hoifung Poon. [Self-verification improves few-shot clinical information extraction](https://md.research.elamur.ai/free/en/papers/gero-2023-self-verification-improves-few-shot/). arXiv preprint arXiv:2306.00024, 2023. ,→6.
136. [123] Benjamin J Lengerich, Sebastian Bordt, Harsha Nori, Mark E Nunnally, Yin Aphinyanaphongs, Manolis Kellis, and Rich Caruana. LLMs understand glass-box models, discover surprises, and suggest repairs. arXiv preprint arXiv:2308.01157, 2023. ,→6. — [arXiv:2308.01157](https://arxiv.org/abs/2308.01157) [find](https://scholar.google.com/scholar?q=%5B123%5D+Benjamin+J+Lengerich%2C+Sebastian+Bordt%2C+Harsha+Nori%2C+Mark+E+Nunnally%2C+Yin+Aphinyanaphongs%2C+Manolis+Kellis%2C+and+Rich+Caruana%2E+LLMs+understand+glass%2Dbox+models%2C+discover+surprises%2C+and+suggest+repa)
137. [142] Grégoire Mialon, Roberto Dessı̀, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. [Augmented language models: a survey](https://md.research.elamur.ai/free/en/papers/mialon-2023-augmented-language-models-survey/). arXiv preprint arXiv:2302.07842, 2023. ,→6.
138. [124] Rakesh R Menon, Kerem Zaman, and Shashank Srivastava. MaNtLE: Model-agnostic natural language explainer. arXiv preprint arXiv:2305.12995, 2023. ,→6. [find](https://scholar.google.com/scholar?q=%5B124%5D+Rakesh+R+Menon%2C+Kerem+Zaman%2C+and+Shashank+Srivastava%2E+MaNtLE%3A+Model%2Dagnostic+natural+language+explainer%2E+arXiv+preprint+arXiv%3A2305%2E12995%2C+2023%2E+%2C%E2%86%926%2E)
139. [143] Chandan Singh, John X. Morris, Jyoti Aneja, Alexander M. Rush, and Jianfeng Gao. Explaining patterns in data with language models via interpretable autoprompting, 2023. ,→6 and 7 [find](https://scholar.google.com/scholar?q=%5B143%5D+Chandan+Singh%2C+John+X%2E+Morris%2C+Jyoti+Aneja%2C+Alexander+M%2E+Rush%2C+and+Jianfeng+Gao%2E+Explaining+patterns+in+data+with+language+models+via+interpretable+autoprompting%2C+2023%2E+%2C%E2%86%926+and+7)
140. [125] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), pages 267–288, 1996. ,→6. [find](https://scholar.google.com/scholar?q=%5B125%5D+Robert+Tibshirani%2E+Regression+shrinkage+and+selection+via+the+lasso%2E+Journal+of+the+Royal+Statistical+Society%2E+Series+B+%28Methodological%29%2C+pages+267%E2%80%93288%2C+1996%2E+%2C%E2%86%926%2E)
141. [144] Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. Describing differences between text distributions with natural language. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Mac… [find](https://scholar.google.com/scholar?q=%5B144%5D+Ruiqi+Zhong%2C+Charlie+Snell%2C+Dan+Klein%2C+and+Jacob+Steinhardt%2E+Describing+differences+between+text+distributions+with+natural+language%2E+In+Kamalika+Chaudhuri%2C+Stefanie+Jegelka%2C+Le+Song%2C+Csaba+Szep)
142. [126] Berk Ustun and Cynthia Rudin. Supersparse linear integer models for optimized medical scoring systems. Machine Learning, 102:349–391, 2016. ,→6. [find](https://scholar.google.com/scholar?q=%5B126%5D+Berk+Ustun+and+Cynthia+Rudin%2E+Supersparse+linear+integer+models+for+optimized+medical+scoring+systems%2E+Machine+Learning%2C+102%3A349%E2%80%93391%2C+2016%2E+%2C%E2%86%926%2E)
143. [145] Zhiying Zhu, Weixin Liang, and James Zou. Gsclip: A framework for explaining distribution shifts in natural language. arXiv preprint arXiv:2206.15007, 2022. ,→6. — [arXiv:2206.15007](https://arxiv.org/abs/2206.15007) [find](https://scholar.google.com/scholar?q=%5B145%5D+Zhiying+Zhu%2C+Weixin+Liang%2C+and+James+Zou%2E+Gsclip%3A+A+framework+for+explaining+distribution+shifts+in+natural+language%2E+arXiv+preprint+arXiv%3A2206%2E15007%2C+2022%2E+%2C%E2%86%926%2E+%E2%80%94+%5BarXiv%3A2206%2E15007%5D%28https%3A%2F%2Farx)
144. [146] Zihan Wang, Jingbo Shang, and Ruiqi Zhong. Goal-driven explainable clustering via language descriptions. arXiv preprint arXiv:2305.13749, 2023. ,→6. [find](https://scholar.google.com/scholar?q=%5B146%5D+Zihan+Wang%2C+Jingbo+Shang%2C+and+Ruiqi+Zhong%2E+Goal%2Ddriven+explainable+clustering+via+language+descriptions%2E+arXiv+preprint+arXiv%3A2305%2E13749%2C+2023%2E+%2C%E2%86%926%2E)
145. [147] Chau Minh Pham, Alexander Hoyle, Simeng Sun, and Mohit Iyyer. [TopicGPT: A prompt-based topic modeling framework](https://md.research.elamur.ai/free/en/papers/pham-2024-topicgpt-prompt-based-topic-modeling/). arXiv preprint arXiv:2311.01449, 2023. ,→6.
146. [148] SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. [A comprehensive survey of hallucination mitigation techniques in large language models](https://md.research.elamur.ai/free/en/papers/tonmoy-2024-comprehensive-survey-hallucination-mitigation-techniques/). arXiv preprint arXiv:2401.01313, 2024. ,→6.
147. [149] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324, 2023. ,→6. [find](https://scholar.google.com/scholar?q=%5B149%5D+Melanie+Sclar%2C+Yejin+Choi%2C+Yulia+Tsvetkov%2C+and+Alane+Suhr%2E+Quantifying+language+models%E2%80%99+sensitivity+to+spurious+features+in+prompt+design+or%3A+How+i+learned+to+start+worrying+about+prompt+formatt)
148. [150] Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.[04388, 2023](https://md.research.elamur.ai/free/en/papers/turpin-2023-language-models-don-t-always/). ,→6.
149. [151] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). ArXiv, abs/2307.03172, 2023. ,→7.
150. [152] Yanda Chen, Chandan Singh, Xiaodong Liu, Simiao Zuo, Bin Yu, He He, and Jianfeng Gao. Towards consistent natural-language explanations via explanation-consistency finetuning, 2024. ,→7. [find](https://scholar.google.com/scholar?q=%5B152%5D+Yanda+Chen%2C+Chandan+Singh%2C+Xiaodong+Liu%2C+Simiao+Zuo%2C+Bin+Yu%2C+He+He%2C+and+Jianfeng+Gao%2E+Towards+consistent+natural%2Dlanguage+explanations+via+explanation%2Dconsistency+finetuning%2C+2024%2E+%2C%E2%86%927%2E)
151. [153] Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto, and Percy Liang. Benchmarking and improving generator-validator consistency of language models. arXiv preprint arXiv:2310.01846, 2023. (Not cited.) [find](https://scholar.google.com/scholar?q=%5B153%5D+Xiang+Lisa+Li%2C+Vaishnavi+Shrivastava%2C+Siyan+Li%2C+Tatsunori+Hashimoto%2C+and+Percy+Liang%2E+Benchmarking+and+improving+generator%2Dvalidator+consistency+of+language+models%2E+arXiv+preprint+arXiv%3A2310%2E018)
152. [154] Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, and Jacob Andreas. Deductive closure training of language models for coherence, accuracy, and updatability. arXiv preprint arXiv:2401.08574, 2024. ,→7. [find](https://scholar.google.com/scholar?q=%5B154%5D+Afra+Feyza+Akyu%CC%88rek%2C+Ekin+Akyu%CC%88rek%2C+Leshem+Choshen%2C+Derry+Wijaya%2C+and+Jacob+Andreas%2E+Deductive+closure+training+of+language+models+for+coherence%2C+accuracy%2C+and+updatability%2E+arXiv+preprint+arXiv)
153. [155] Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. Large language models for automated open-domain scientific hypotheses discovery. arXiv preprint arXiv:2309.02726, 2023. ,→7. [find](https://scholar.google.com/scholar?q=%5B155%5D+Zonglin+Yang%2C+Xinya+Du%2C+Junxian+Li%2C+Jie+Zheng%2C+Soujanya+Poria%2C+and+Erik+Cambria%2E+Large+language+models+for+automated+open%2Ddomain+scientific+hypotheses+discovery%2E+arXiv+preprint+arXiv%3A2309%2E02726%2C)
154. [156] Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Animashree Anandkumar. Multi-modal molecule structure–text model for text-based retrieval and editing. Nature Machine Intelligence, 5(12):1447–1457, 2023. ,→7. [find](https://scholar.google.com/scholar?q=%5B156%5D+Shengchao+Liu%2C+Weili+Nie%2C+Chengpeng+Wang%2C+Jiarui+Lu%2C+Zhuoran+Qiao%2C+Ling+Liu%2C+Jian+Tang%2C+Chaowei+Xiao%2C+and+Animashree+Anandkumar%2E+Multi%2Dmodal+molecule+structure%E2%80%93text+model+for+text%2Dbased+retrieva)
155. [157] Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. [Mathematical discoveries from program search with large language models](https://md.research.elamur.ai/free/en/papers/paredes-2023-mathematical-discoveries-program-search-large/). N…
156. [158] Lisa Schut, Nenad Tomasev, Tom McGrath, Demis Hassabis, Ulrich Paquet, and Been Kim. Bridging the Human-AI knowledge gap: Concept discovery and transfer in alphazero. arXiv preprint arXiv:2310.16410, 2023. ,→7. — [arXiv:2310.16410](https://arxiv.org/abs/2310.16410) [find](https://scholar.google.com/scholar?q=%5B158%5D+Lisa+Schut%2C+Nenad+Tomasev%2C+Tom+McGrath%2C+Demis+Hassabis%2C+Ulrich+Paquet%2C+and+Been+Kim%2E+Bridging+the+Human%2DAI+knowledge+gap%3A+Concept+discovery+and+transfer+in+alphazero%2E+arXiv+preprint+arXiv%3A2310%2E1)
157. [159] Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, et al. Towards conversational diagnostic ai. arXiv preprint arXiv:2401.05654, 2024. ,→7. — [arXiv:2401.05654](https://arxiv.org/abs/2401.05654) [find](https://scholar.google.com/scholar?q=%5B159%5D+Tao+Tu%2C+Anil+Palepu%2C+Mike+Schaekermann%2C+Khaled+Saab%2C+Jan+Freyberg%2C+Ryutaro+Tanno%2C+Amy+Wang%2C+Brenna+Li%2C+Mohamed+Amin%2C+Nenad+Tomasev%2C+et+al%2E+Towards+conversational+diagnostic+ai%2E+arXiv+preprint+ar)
158. [160] Belinda Z Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. Eliciting human preferences with language models. arXiv preprint arXiv:2310.11589, 2023. ,→7. [find](https://scholar.google.com/scholar?q=%5B160%5D+Belinda+Z+Li%2C+Alex+Tamkin%2C+Noah+Goodman%2C+and+Jacob+Andreas%2E+Eliciting+human+preferences+with+language+models%2E+arXiv+preprint+arXiv%3A2310%2E11589%2C+2023%2E+%2C%E2%86%927%2E)
159. [161] Xu Huang, Jianxun Lian, Yuxuan Lei, Jing Yao, Defu Lian, and Xing Xie. Recommender AI agent: Integrating large language models for interactive recommendations. arXiv preprint arXiv:2308.16505, 2023. ,→7. — [arXiv:2308.16505](https://arxiv.org/abs/2308.16505) [find](https://scholar.google.com/scholar?q=%5B161%5D+Xu+Huang%2C+Jianxun+Lian%2C+Yuxuan+Lei%2C+Jing+Yao%2C+Defu+Lian%2C+and+Xing+Xie%2E+Recommender+AI+agent%3A+Integrating+large+language+models+for+interactive+recommendations%2E+arXiv+preprint+arXiv%3A2308%2E16505%2C+2)

*Updated 29 October 2026*