---
title: "BERT Rediscovers the Classical NLP Pipeline — Tenney 2019"
description: "Pre-trained encoders like BERT perform very well, but it is unclear where and how they store linguistic knowledge."
authors: ["Ian Tenney", "Dipanjan Das", "Ellie Pavlick"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1452/"
doi: "10.18653/v1/p19-1452"
arxiv: "1905.05950"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / BERT Rediscovers the Classical NLP Pipeline

# BERT Rediscovers the Classical NLP Pipeline

**Authors:** [Ian Tenney](https://md.research.elamur.ai/free/en/authors/ian-tenney/), [Dipanjan Das](https://md.research.elamur.ai/free/en/authors/dipanjan-das/), [Ellie Pavlick](https://md.research.elamur.ai/free/en/authors/ellie-pavlick/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1452/) · [PDF](https://aclanthology.org/P19-1452.pdf) · [DOI 10.18653/v1/p19-1452](https://doi.org/10.18653/v1/p19-1452) · [arXiv 1905.05950](https://arxiv.org/abs/1905.05950) · [OpenAlex](https://openalex.org/W2946417913)

**Citations:** 1259 citations (openalex-snapshot · 2026-10-14) · [733](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Pre-trained encoders like BERT perform very well, but it is unclear where and how they store linguistic knowledge. The authors ask whether BERT's layers mirror the stages of a classical NLP pipeline. They freeze the uncased BERT-base and BERT-large models and apply the "edge probing" suite of Tenney et al. (2019). This suite covers eight span-labeling tasks drawn from OntoNotes 5.0, the English Web Treebank, SPR1 and SemEval 2010 Task 8, all scored with micro-averaged F1. They introduce two complementary layer-wise measures. Scalar mixing weights, learned on training data, reveal which layers a probe relies on. Cumulative scoring, computed on evaluation data, estimates the layer at which an example first becomes solvable. The tasks emerge in the traditional order: POS, constituents, dependencies, semantic roles, then coreference. Syntactic information sits in a few layers, while semantic information is spread across the whole network. Per-sentence analysis shows that BERT can revise early decisions once higher-level evidence appears. The work offers an interpretable picture of how deep language models organize hierarchical linguistic structure.

## Key points
- The probes cover eight tasks: POS, Consts., Deps., entities, SRL, Coref., SPR and SemEval relations. Their classifiers can see only the token vectors inside the target spans.
- In BERT-large (24 layers), differential scores peak early, around layers 1–7, because many cases have shallow shortcuts. Mixing weights concentrate later, around layers 9–20.
- At layer 0 (non-contextual embeddings), 89% of POS tags are already correct. SPR and relation classification show near-uniform weights and improve gradually through almost all layers.
- BERT-base (12 layers) shows the same task ordering, with a "stretching effect": each task occupies similar positions relative to the top of the model.
- In one example, "Toronto" is first tagged GPE and then corrected to ORG (a sports team) once its semantic role (ARG1 of "smoked") is resolved.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Semantic role labeling](https://md.research.elamur.ai/free/en/topics/semantic-role-labeling/)

## Cited by

Cited by

733 papers of our library cite it (of the 140673 whose bibliographies we read); 55 of them have a page here, 678 not yet.

- [BEReshiT: an Ancient Hebrew Model based on DictaBERT](https://md.research.elamur.ai/free/en/papers/aclbulk-2026-stoupak-bereshit-ancient-hebrew-model-based/)
- [Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations](https://md.research.elamur.ai/free/en/papers/agarwal-2025-mechanisms-vs-outcomes-probing-syntax/)
- [Unsupervised Domain Clusters in Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/aharoni-2020-unsupervised-domain-clusters-pretrained-language/)
- [Bridging Linguistic Structure and Mechanistic Interpretability for Conceptual Interpretation in Language Models](https://md.research.elamur.ai/free/en/papers/aljaafari-2026-bridging-linguistic-structure-mechanistic-interpretability/)
- [Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/anikina-2025-reverse-probing-evaluating-knowledge-transfer/)
- [Comparing Native and Learner Englishes Using a Large Pre-trained Language Model](https://md.research.elamur.ai/free/en/papers/aoyama-2022-comparing-native-learner-englishes-large/)
- [Experience Grounds Language](https://md.research.elamur.ai/free/en/papers/bisk-2020-experience-grounds-language/)
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://md.research.elamur.ai/free/en/papers/chuang-2023-dola-decoding-contrasting-layers-improves/)
- [Rethinking embedding coupling in pre-trained language models](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/)
- [Visualizing and Measuring the Geometry of BERT](https://md.research.elamur.ai/free/en/papers/coenen-2019-visualizing-measuring-geometry-bert/)
- [BERTje: A Dutch BERT Model](https://md.research.elamur.ai/free/en/papers/de-2019-vries-bertje-dutch-bert-model/)
- [Multi-Sentence Argument Linking](https://md.research.elamur.ai/free/en/papers/ebner-2020-multi-sentence-argument-linking/)
- [Joint Universal Syntactic and Semantic Parsing](https://md.research.elamur.ai/free/en/papers/eskin-2021-joint-universal-syntactic-semantic-parsing/)
- [Modular Neural Machine Translation with a Semantic Pivot - Pilot Study Using AMR](https://md.research.elamur.ai/free/en/papers/gao-2026-modular-neural-machine-translation-semantic/)
- [Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation](https://md.research.elamur.ai/free/en/papers/geiger-2020-neural-natural-language-inference-models/)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://md.research.elamur.ai/free/en/papers/geva-2022-transformer-feed-forward-layers-build/)
- [Applying Occam's Razor to Transformer-Based Dependency Parsing: What Works, What Doesn't, and What is Really Necessary](https://md.research.elamur.ai/free/en/papers/grunewald-2021-applying-occam-s-razor-transformer/)
- [Coordinate Constructions in English Enhanced Universal Dependencies: Analysis and Computational Modeling](https://md.research.elamur.ai/free/en/papers/grunewald-2021-coordinate-constructions-english-enhanced-universal/)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/)
- [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/)
- [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/)
- [How Can We Know What Language Models Know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://md.research.elamur.ai/free/en/papers/li-2023-inference-time-intervention-eliciting-truthful/)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models](https://md.research.elamur.ai/free/en/papers/lin-2020-birds-have-four-legs-numersense/)
- [Evaluating statistical language models as pragmatic reasoners](https://md.research.elamur.ai/free/en/papers/lipkin-2023-evaluating-statistical-language-models-pragmatic/)
- [On Quick Kisses and How to Make Them Count: A Study on Event Construal in Light Verb Constructions with BERT](https://md.research.elamur.ai/free/en/papers/liu-2023-quick-kisses-how-make-them/)
- [Evaluating large language models for the tasks of PoS tagging within the Universal Dependency framework](https://md.research.elamur.ai/free/en/papers/machado-2024-evaluating-large-language-models-tasks/)
- [CamemBERT: a Tasty French Language Model](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
- [Rethinking the Idiomaticity Decomposability Hypothesis: Evidence from Distributional Learning](https://md.research.elamur.ai/free/en/papers/mi-2026-rethinking-idiomaticity-decomposability-hypothesis-evidence/)
- [Discursive Socratic Questioning: Evaluating the Faithfulness of Language Models' Understanding of Discourse Relations](https://md.research.elamur.ai/free/en/papers/miao-2024-discursive-socratic-questioning-evaluating-faithfulness/)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
- [How Context Affects Language Models' Factual Predictions](https://md.research.elamur.ai/free/en/papers/petroni-2020-context-affects-language-models-factual/)
- [Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://md.research.elamur.ai/free/en/papers/ravfogel-2020-null-it-out-guarding-protected/)
- [COMET: A Neural Framework for MT Evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/)
- [The Persian Dependency Treebank Made Universal](https://md.research.elamur.ai/free/en/papers/safari-2022-persian-dependency-treebank-made-universal/)
- [Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task](https://md.research.elamur.ai/free/en/papers/sellam-2020-learning-evaluate-translation-beyond-english/)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://md.research.elamur.ai/free/en/papers/sinha-2021-masked-language-modeling-distributional-hypothesis/)
- [oLMpics -- On what Language Model Pre-training Captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/)
- [AuthorMix: Modular Authorship Style Transfer via Layer-wise Adapter Mixing](https://md.research.elamur.ai/free/en/papers/thillainathan-2026-authormix-modular-authorship-style-transfer/)
- [How Are Idioms Processed Inside Transformer Language Models?](https://md.research.elamur.ai/free/en/papers/tian-2023-how-are-idioms-processed-inside/)
- [TDMulti: A Tunisian Dialect-Modern Standard Arabic Multitask Corpus with a Context-Aware Cross-Attention BERT Model](https://md.research.elamur.ai/free/en/papers/torjmen-2026-tdmulti-tunisian-dialect-modern-standard/)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://md.research.elamur.ai/free/en/papers/ud-2019-kondratyuk-75-languages-1-model-parsing/)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://md.research.elamur.ai/free/en/papers/ud-2021-nguyen-trankit-light-weight-transformer-based/)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://md.research.elamur.ai/free/en/papers/vig-2019-analyzing-structure-attention-transformer-language/)
- [Probing Pretrained Language Models for Lexical Semantics](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/)
- [UniTE: Unified Translation Evaluation](https://md.research.elamur.ai/free/en/papers/wan-2022-unite-unified-translation-evaluation/)
- [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
- [Coreference Reasoning in Machine Reading Comprehension](https://md.research.elamur.ai/free/en/papers/wu-2021-coreference-reasoning-machine-reading-comprehension/)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/)
- [Revisiting Few-sample BERT Fine-tuning](https://md.research.elamur.ai/free/en/papers/zhang-2020-revisiting-few-sample-bert-fine/)
- [Explainability for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tenney2019bert,
  title = {{BERT Rediscovers the Classical NLP Pipeline}},
  author = {Tenney, Ian and Das, Dipanjan and Pavlick, Ellie},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1452},
  eprint = {1905.05950},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1452}
}
```

- [← previous BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/xu-2019-bert-post-training-review-reading/)
- [next → BERTje: A Dutch BERT Model](https://md.research.elamur.ai/free/en/papers/de-2019-vries-bertje-dutch-bert-model/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yonatan Belinkov. 2018. On internal language representations in deep learning: An analysis of machine translation and speech recognition. Ph.D. thesis, Massachusetts Institute of Technology. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2E+2018%2E+On+internal+language+representations+in+deep+learning%3A+An+analysis+of+machine+translation+and+speech+recognition%2E+Ph%2ED%2E+thesis%2C+Massachusetts+Institute+of+Technology%2E)
2. Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs encode soft hierarchical syntax. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Terra+Blevins%2C+Omer+Levy%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+RNNs+encode+soft+hierarchical+syntax%2E+In+Proceedings+of+ACL%2E)
3. Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single $&\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Germa%CC%81n+Kruszewski%2C+Guillaume+Lample%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Marco+Baroni%2E+2018%2E+What+you+can+cram+into+a+single+%24%26%5C%23%2A+vector%3A+Probing+sentence+embeddings+for+linguistic+properties%2E+In+Pr) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+NAACL%2E)
5. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Seman… [find](https://scholar.google.com/scholar?q=Iris+Hendrickx%2C+Su+Nam+Kim%2C+Zornitsa+Kozareva%2C+Preslav+Nakov%2C+Diarmuid+O%CC%81+Se%CC%81aghdha%2C+Sebastian+Pado%CC%81%2C+Marco+Pennacchiotti%2C+Lorenza+Romano%2C+and+Stan+Szpakowicz%2E+2009%2E+Semeval%2D2010+task+8%3A+Multi%2Dway+cla) [our card](https://md.research.elamur.ai/free/en/papers/hendrickx-2010-semeval-2010-task-8-multi/)
6. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. arXiv preprint 1901.11504. [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+2019%2E+Multi%2Dtask+deep+neural+networks+for+natural+language+understanding%2E+arXiv+preprint+1901%2E11504%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/)
7. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of ACL: System Demonstrations. [find](https://scholar.google.com/scholar?q=Christopher+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+Bethard%2C+and+David+McClosky%2E+2014%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+Proceedings+of+ACL%3A+System+Demonst) [our card](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/)
8. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+EMNLP%2E)
9. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S%2E+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Proceedings+of+NIPS%2E)
10. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Proceedings+of+NIPS%2E)
11. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. OntoNotes release 5.0 LDC2013T19. Linguistic Data Consortium, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Ralph+Weischedel%2C+Martha+Palmer%2C+Mitchell+Marcus%2C+Eduard+Hovy%2C+Sameer+Pradhan%2C+Lance+Ramshaw%2C+Nianwen+Xue%2C+Ann+Taylor%2C+Jeff+Kaufman%2C+Michelle+Franchini%2C+et+al%2E+2013%2E+OntoNotes+release+5%2E0+LDC2013T19%2E+)
12. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018a%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+NAACL%2E)
13. Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Luke+Zettlemoyer%2C+and+Wen%2Dtau+Yih%2E+2018b%2E+Dissecting+contextual+word+embeddings%3A+Architecture+and+representation%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/)
14. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+J%2E+Edward+Hu%2C+Ellie+Pavlick%2C+Aaron+Steven+White%2C+and+Benjamin+Van+Durme%2E+2018%2E+Collecting+diverse+natural+language+inference+problems+for+sentence+repre) [our card](https://md.research.elamur.ai/free/en/papers/poliak-2018-collecting-diverse-natural-language-inference/)
15. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [https://blog.openai.com/language-unsupervised](https://blog.openai.com/language-unsupervised). [↗](https://blog.openai.com/language-unsupervised)
16. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [https://blog.openai.com/better-language-models](https://blog.openai.com/better-language-models). [↗](https://blog.openai.com/better-language-models)
17. Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. Transactions of the Association of Computational Linguistics. [find](https://scholar.google.com/scholar?q=Drew+Reisinger%2C+Rachel+Rudinger%2C+Francis+Ferraro%2C+Craig+Harman%2C+Kyle+Rawlins%2C+and+Benjamin+Van+Durme%2E+2015%2E+Semantic+proto%2Droles%2E+Transactions+of+the+Association+of+Computational+Linguistics%2E)
18. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Xing+Shi%2C+Inkit+Padhi%2C+and+Kevin+Knight%2E+2016%2E+Does+string%2Dbased+neural+MT+learn+source+syntax%3F+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/shi-2016-does-string-based-neural-mt/)
19. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Christopher D. Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation. [find](https://scholar.google.com/scholar?q=Natalia+Silveira%2C+Timothy+Dozat%2C+Marie%2DCatherine+de+Marneffe%2C+Samuel+Bowman%2C+Miriam+Connor%2C+John+Bauer%2C+and+Christopher+D%2E+Manning%2E+2014%2E+A+gold+standard+dependency+corpus+for+English%2E+In+Proceedings+)
20. Adam Teichert, Adam Poliak, Benjamin Van Durme, and Matthew Gormley. 2017. Semantic proto-role labeling. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Adam+Teichert%2C+Adam+Poliak%2C+Benjamin+Van+Durme%2C+and+Matthew+Gormley%2E+2017%2E+Semantic+proto%2Drole+labeling%2E+In+Proceedings+of+AAAI%2E)
21. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Lear… [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Patrick+Xia%2C+Berlin+Chen%2C+Alex+Wang%2C+Adam+Poliak%2C+R+Thomas+McCoy%2C+Najoung+Kim%2C+Benjamin+Van+Durme%2C+Sam+Bowman%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019%2E+What+do+you+learn+from+context%3F+probin) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/)