--- title: "Amnesic Probing — Elazar 2021" description: "Probing is a common way to inspect neural language models: a small classifier is trained to recover a linguistic property from hidden representations." authors: ["Yanai Elazar", "Shauli Ravfogel", "Alon Jacovi", "Yoav Goldberg"] year: 2021 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2021.tacl-1.10/" doi: "10.1162/tacl_a_00359" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/elazar-2021-amnesic-probing-behavioral-explanation-amnesic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elazar-2021-amnesic-probing-behavioral-explanation-amnesic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/) / Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals # Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals **Authors:** [Yanai Elazar](https://md.research.elamur.ai/free/en/authors/yanai-elazar/), [Shauli Ravfogel](https://md.research.elamur.ai/free/en/authors/shauli-ravfogel/), [Alon Jacovi](https://md.research.elamur.ai/free/en/authors/alon-jacovi/), [Yoav Goldberg](https://md.research.elamur.ai/free/en/authors/yoav-goldberg/) · **Year:** 2021 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.tacl-1.10/) · [PDF](https://aclanthology.org/2021.tacl-1.10.pdf) · [DOI 10.1162/tacl\_a\_00359](https://doi.org/10.1162/tacl_a_00359) · [OpenAlex](https://openalex.org/W3159900299) **Citations:** 123 citations (openalex-snapshot · 2026-10-14) · [118](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Probing is a common way to inspect neural language models: a small classifier is trained to recover a linguistic property from hidden representations. However, a successful probe only shows that the information can be extracted. It says nothing about whether the model actually relies on it. Elazar et al. propose Amnesic Probing, a counterfactual test of usage. The target property is erased from the representation, and the authors measure how much the model's behavior on its main task degrades. The erasure uses Iterative Nullspace Projection (INLP), which removes linearly decodable information without retraining the encoder. Applying this to BERT-base for word prediction, they study properties such as part-of-speech and dependency labels. Their central finding is that high probing accuracy does not predict how important a property is for the task. Usage also differs between masked and non-masked inputs, and individual labels within a property matter to different degrees. The work is a warning against drawing causal or behavioral conclusions from probing scores alone. It also offers a reusable intervention-based analysis tool. ## Key points - **How INLP erases a property:** It repeatedly trains linear classifiers (linear SVM) to predict the property and projects the representations onto each classifier's nullspace. This continues until a probe scores within about one point of majority accuracy on the development set. - **Two controls:** - *Rand* removes the same number of directions, chosen at random, to account for generic information loss. - *Selectivity* reattaches gold property values as learned 32-dimensional vectors and fine-tunes the following layers to check whether performance can be restored. - **Masked vs. non-masked:** Properties like POS and dependency labels are used differently when the token is masked (the LM training regime) than when it is visible (the regime typical for probing and fine-tuning). - **NLP pipeline claim revisited:** The authors re-examine the claim of Tenney et al. (2019a) that BERT's layers mirror a classical NLP pipeline. Their layer-wise amnesic analysis gives a new interpretation, but it covers BERT-base, whereas Tenney et al. studied BERT-Large. - **Scope and availability:** Published in TACL vol. 9 (2021), pp. 160–175. The method removes only linear information, and the authors call for non-linear removal methods. Code is at github.com/yanaiela/amnesic_probing. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 118 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 111 not yet. - [Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations](https://md.research.elamur.ai/free/en/papers/agarwal-2025-mechanisms-vs-outcomes-probing-syntax/) - [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://md.research.elamur.ai/free/en/papers/belrose-2023-eliciting-latent-predictions-transformers-tuned/) - [Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/dankers-2022-can-transformer-be-too-compositional/) - [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/) - [Compositional Meaning Representations in LLMs: a Critical Review of Probing Studies](https://md.research.elamur.ai/free/en/papers/marro-2026-compositional-meaning-representations-llms-critical/) - [Toward Cognitive Alignment in Large Language Models: Integrating Linguistic Theory and Human Data](https://md.research.elamur.ai/free/en/papers/zaghouani-2026-cognitive-alignment-large-language-models/) - [Representation Engineering: A Top-Down Approach to AI Transparency](https://md.research.elamur.ai/free/en/papers/zou-2023-representation-engineering-top-down-approach/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{elazar2021amnesic, title = {{Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals}}, author = {Elazar, Yanai and Ravfogel, Shauli and Jacovi, Alon and Goldberg, Yoav}, journal = {Transactions of the Association for Computational Linguistics}, year = {2021}, doi = {10.1162/tacl\_a\_00359}, url = {https://doi.org/10.1162/tacl_a_00359} } ``` - [← previous AlephBERT: A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With](https://md.research.elamur.ai/free/en/papers/langstream-2021-seker-alephbert-hebrew-large-pre-trained/) - [next → An Amharic News Text classification Dataset](https://md.research.elamur.ai/free/en/papers/azime-2021-amharic-news-text-classification-dataset/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. Probing linguistic features of sentence-level representations in relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1534–1545, Online. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Christoph+Alt%2C+Aleksandra+Gabryszak%2C+and+Leonhard+Hennig%2E+2020%2E+Probing+linguistic+features+of+sentence%2Dlevel+representations+in+relation+extraction%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+A) 2. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. CoRR, abs/1608 .04207. \#\# Conclusions [find](https://scholar.google.com/scholar?q=Yossi+Adi%2C+Einat+Kermany%2C+Yonatan+Belinkov%2C+Ofer+Lavi%2C+and+Yoav+Goldberg%2E+2016%2E+Fine%2Dgrained+analysis+of+sentence+embeddings+using+auxiliary+prediction+tasks%2E+CoRR%2C+abs%2F1608+%2E04207%2E+%5C%23%5C%23+Conclusions) [our card](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/)