---
title: "Probing Across Time: What Does RoBERTa Know and When?"
description: "The authors release their code and checkpoints. 37 references, cited by 59 papers here. Among them: 35."
authors: ["Leo Z. Liu", "Yizhong Wang", "Jungo Kasai", "Hannaneh Hajishirzi", "Noah A. Smith"]
year: 2021
venue: "Findings of the Association for Computational Linguistics: EMNLP 2021"
source: "https://aclanthology.org/2021.findings-emnlp.71/"
doi: "10.18653/v1/2021.findings-emnlp.71"
arxiv: "2104.07885"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2021-probing-across-time-what-does/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2021-probing-across-time-what-does/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Probing Across Time: What Does RoBERTa Know and When?

# Probing Across Time: What Does RoBERTa Know and When?

**Authors:** Leo Z. Liu, [Yizhong Wang](https://md.research.elamur.ai/free/en/authors/yizhong-wang/), [Jungo Kasai](https://md.research.elamur.ai/free/en/authors/jungo-kasai/), [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/en/authors/hannaneh-hajishirzi/) and [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) · **Year:** 2021 · **Venue:** [Findings of the Association for Computational Linguistics: EMNLP 2021](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2021](https://md.research.elamur.ai/free/en/venues/findings/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.findings-emnlp.71/) · [PDF](https://aclanthology.org/2021.findings-emnlp.71.pdf) · [DOI 10.18653/v1/2021.findings-emnlp.71](https://doi.org/10.18653/v1/2021.findings-emnlp.71) · [arXiv 2104.07885](https://arxiv.org/abs/2104.07885) · [OpenAlex](https://openalex.org/W3213280751)

**Citations:** 35 citations (OpenAlex, 2026-10-14) · [59](https://md.research.elamur.ai/free/en/papers/liu-2021-probing-across-time-what-does/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks when, during pretraining, a masked language model acquires different kinds of knowledge, rather than only what a finished model knows. The authors reproduce RoBERTa-base pretraining from scratch, save checkpoints across roughly 1M training steps, and evaluate each checkpoint with five public probe packages. Structural probes (linear classifiers on frozen representations) and behavioral probes (zero-training masked-LM or sentence-scoring tests) cover linguistic, factual, commonsense and reasoning knowledge. They also pretrain variants on corpora that differ in domain and size, and fine-tune checkpoints on standard benchmarks. Linguistic knowledge appears early, stays stable and is robust to corpus domain. Factual and commonsense knowledge grow more slowly and depend more on the domain. Reasoning is largely not learned in a stable way. Corpus diversity seems to matter more than raw corpus size. These results can guide more efficient pretraining schedules and help explain how different capabilities emerge. The authors release their code and checkpoints.

## Key points
- Probe suite: LKT (POS tagging, chunking, NER, arc prediction/classification), BLiMP (minimal pairs), LAMA (T-REx, Google RE, ConceptNet, SQuAD), CAT (commonsense) and oLMpics (reasoning).
- The order in which knowledge types are learned holds across data variations. Corpus domain and size mainly change learning speed and final scores.
- Most benchmarks, such as SST-2 and SQuAD, reach high fine-tuning performance in under half of the pretraining steps. Tasks that need complex knowledge, such as ReCoRD and WSC, benefit from longer pretraining.
- Unlike Chiang et al. (2020), the authors find that factual and commonsense knowledge keep improving during pretraining. They attribute this to their broader probe selection, since a T-REx subset alone plateaus early.
- MLP probes scored highly even on randomly initialized RoBERTa, so the authors use linear probes instead. Checkpoints are available at github.com/leo-liuzy/probe-across-time.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 263 works · 180,581 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 219 works · 89,712 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 58 works

## Cite

Cite as: Leo Z. Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi and Noah A. Smith. 2021. Probing Across Time: What Does RoBERTa Know and When?. Findings of the Association for Computational Linguistics: EMNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2021probing,
  title = {{Probing Across Time: What Does RoBERTa Know and When?}},
  author = {Leo Z. Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi and Noah A. Smith},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.71},
  eprint = {2104.07885},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.findings-emnlp.71}
}
```

- [← previous Cross-Domain Label-Adaptive Stance Detection](https://md.research.elamur.ai/free/en/papers/hardalov-2021-cross-domain-label-adaptive-stance/)
- [next → Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models](https://md.research.elamur.ai/free/en/papers/khot-2021-text-modular-networks-learning-decompose/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

37 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yonatan Belinkov, Sebastian Gehrmann, and Ellie Pavlick. 2020. Interpretability and analysis in neural NLP. In Proc. of ACL: Tutorial Abstracts. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2C+Sebastian+Gehrmann%2C+and+Ellie+Pavlick%2E+2020%2E+Interpretability+and+analysis+in+neural+NLP%2E+In+Proc%2E+of+ACL%3A+Tutorial+Abstracts%2E)
2. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. [Abductive commonsense reasoning](https://md.research.elamur.ai/free/en/papers/bhagavatula-2019-abductive-commonsense-reasoning/). In Proc. of ICLR.
3. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Cheng-Han Chiang, Sung-Feng Huang, and Hung-yi Lee. 2020. Pretrained language model embryology: The birth of ALBERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6813–6828, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang%2C+Sung%2DFeng+Huang%2C+and+Hung%2Dyi+Lee%2E+2020%2E+Pretrained+language+model+embryology%3A+The+birth+of+ALBERT%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
5. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proc. of BlackboxNLP. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D+Manning%2E+2019%2E+What+does+BERT+look+at%3F+an+analysis+of+BERT%E2%80%99s+attention%2E+In+Proc%2E+of+BlackboxNLP%2E)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proc. of NAACL.
7. William B. Dolan and Chris Brockett. 2005. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proc. of IWP. Asian Federation of Natural Language Processing.
8. Aaron Gokaslan and Vanya Cohen. 2019. OpenWeb-Text corpus. [http://Skylion007.github.io/](http://Skylion007.github.io/) OpenWebTextCorpus . \#\# Acknowledgments Suchin Gururangan, Ana Marasović, Swabha In random order, we appreciate valuable discus-Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, sion and feedback from Noah’s ARK gro… [↗](http://Skylion007.github.io/)
9. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proc. of EMNLP. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Percy+Liang%2E+2019%2E+Designing+and+interpreting+probes+with+control+tasks%2E+In+Proc%2E+of+EMNLP%2E+Association+for+Computational+Linguistics%2E)
10. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2018%2E+Language+models+are+unsupervised+multitask+learners%2E)
11. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. [Revealing the dark secrets of bert](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/). In Proc. of EMNLP.
12. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). JMLR.
13. Loïc Lannelongue, Jason Grealey, and Michael Inouye. 2020. Green algorithms: Quantifying the carbon emissions of computation. CoRR, abs/2007.07610. [find](https://scholar.google.com/scholar?q=Lo%C3%AFc+Lannelongue%2C+Jason+Grealey%2C+and+Michael+Inouye%2E+2020%2E+Green+algorithms%3A+Quantifying+the+carbon+emissions+of+computation%2E+CoRR%2C+abs%2F2007%2E07610%2E)
14. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd schema challenge. In Proc. of KR&R. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+Winograd+schema+challenge%2E+In+Proc%2E+of+KR%26R%2E)
15. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. [Linguistic knowledge and transferability of contextual representations](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/). In Proc. of NAACL.
16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A Robustly Optimized BERT Pretraining Approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019b%2E+RoBERTa%3A+A+Robustly+Optimized+BERT+Pretraining+Approach%2E)
17. James L. McClelland and David E. Rumelhart. 1986. Parallel Distributed Processing, volume 2. [find](https://scholar.google.com/scholar?q=James+L%2E+McClelland+and+David+E%2E+Rumelhart%2E+1986%2E+Parallel+Distributed+Processing%2C+volume+2%2E)
18. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James F. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+F%2E+Allen%2E+2016%2E+A+corpus+and+cloze+evaluation+for+deeper+understanding+of+co)
19. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100, 000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proc. of EMNLP.
20. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. [Masked language model scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/). In Proc. of ACL.
21. Naomi Saphra and Adam Lopez. 2019. [Understanding learning dynamics of language models with SVCCA](https://md.research.elamur.ai/free/en/papers/saphra-2019-understanding-learning-dynamics-language-models/). In Proc. of NAACL.
22. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E)
23. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2019. [oLMpics - on what language model pre-training captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/). TACL.
24. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. [http://web.archive.org/save/http://commoncrawl](http://web.archive.org/save/http://commoncrawl). org/2016/10/news-dataset-availabl . [↗](http://web.archive.org/save/http://commoncrawl)
25. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. [BLiMP: The benchmark of linguistic minimal pairs for english](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). TACL.
26. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [GloVe: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proc. of EMNLP.
27. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. TACL. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+TACL%2E)
28. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proc. of NAACL.
29. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proc%2E+of+NAACL%2E)
30. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. [Defending against neural fake news](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/). In Proc. of NeurIPS.
31. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proc. of EMNLP.
32. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. [Information-theoretic probing for linguistic structure](https://md.research.elamur.ai/free/en/papers/pimentel-2020-information-theoretic-probing-linguistic-structure/). In Proc. of ACL.
33. Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D. Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. 2020. [Back to the future: Unsupervised backprop-based decoding for counterfactual and abductive commonsense reasoning](https://md.research.elamur.ai/free/en/papers/qin-2020-back-future-unsupervised-backprop-based/). In Proc. of EMNLP.
34. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. [ReCoRD: Bridging the gap between human and machine commonsense reading comprehension](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/).
35. Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R. Bowman. 2020. When do you need billions of words of pretraining data? [find](https://scholar.google.com/scholar?q=Yian+Zhang%2C+Alex+Warstadt%2C+Haau%2DSing+Li%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+When+do+you+need+billions+of+words+of+pretraining+data%3F)
36. Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. Evaluating commonsense in pre-trained language models. In Proc. of AAAI. [find](https://scholar.google.com/scholar?q=Xuhui+Zhou%2C+Yue+Zhang%2C+Leyang+Cui%2C+and+Dandan+Huang%2E+2020%2E+Evaluating+commonsense+in+pre%2Dtrained+language+models%2E+In+Proc%2E+of+AAAI%2E)
37. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In Proc. of ICCV.

*Updated 24 October 2026*