---
title: "Frequency Effects on Syntactic Rule Learning in Transformers"
description: "The paper asks whether pre-trained transformers like BERT truly acquire abstract symbols and rules… 29 references, cited by 49 papers here."
authors: ["Jason Wei", "Dan Garrette", "Tal Linzen", "Ellie Pavlick"]
year: 2021
venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2021.emnlp-main.72/"
doi: "10.18653/v1/2021.emnlp-main.72"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wei-2021-frequency-effects-syntactic-rule-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wei-2021-frequency-effects-syntactic-rule-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Frequency Effects on Syntactic Rule Learning in Transformers

# Frequency Effects on Syntactic Rule Learning in Transformers

**Authors:** [Jason Wei](https://md.research.elamur.ai/free/en/authors/jason-wei/), [Dan Garrette](https://md.research.elamur.ai/free/en/authors/dan-garrette/), [Tal Linzen](https://md.research.elamur.ai/free/en/authors/tal-linzen/) and [Ellie Pavlick](https://md.research.elamur.ai/free/en/authors/ellie-pavlick/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.72/) · [PDF](https://aclanthology.org/2021.emnlp-main.72.pdf) · [DOI 10.18653/v1/2021.emnlp-main.72](https://doi.org/10.18653/v1/2021.emnlp-main.72) · [OpenAlex](https://openalex.org/W3200809495)

**Citations:** 42 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/wei-2021-frequency-effects-syntactic-rule-learning/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether pre-trained transformers like BERT truly acquire abstract symbols and rules, using English subject–verb agreement (SVA) as a test case. Instead of probing a released checkpoint, the authors pre-train several BERT models from scratch on English Wikipedia. This gives them control over the vocabulary and lets them deliberately change how often specific verb forms appear in the pre-training data. Evaluation uses targeted syntactic evaluation: a verb is masked, and the MLM head must score the correct inflection above the wrong one. Three hypotheses are compared: an idealized symbolic learner, a purely co-occurrence-based item-specific learner, and a symbolic learner with noisy mapping from words to features.

BERT handles subject–verb pairs never seen together in training fairly well, which argues against pure memorization. However, its accuracy is causally affected by both a verb form's absolute frequency and its frequency relative to the competing inflection. Probing classifiers indicate that rare-verb errors come from not learning the verb's number feature. Large frequency imbalances, by contrast, push BERT toward the more common form even against the rule. The study clarifies how training statistics shape apparent rule learning.

## Key points
- Multiple BERT instances were pre-trained on Wikipedia only (2.3 billion tokens), since BooksCorpus is not publicly available.
- Training their own models let the authors add any verb form to the vocabulary as a single wordpiece. This avoids the restriction to in-vocabulary verbs used by Goldberg (2019).
- Frequency interventions targeted 60 "verbs of interest" (e.g., *emphasize*, *threaten*, *propose*, *reveal*) whose inflection frequencies were manipulated in the pre-training corpus.
- Of the three hypotheses, the "symbolic learner with noisy observations" (H3) fits best. This hypothesis does not fully account for BERT's added sensitivity to relative frequency between competing word forms.
- The authors describe their study as the first to test these questions through controlled interventions on pre-training data, rather than through post-hoc analysis.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 136 works · 86,390 citations
- [Brown University](https://md.research.elamur.ai/free/en/universities/brown-university/) — University, Providence, United States · 58 works · 25,310 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Jason Wei, Dan Garrette, Tal Linzen and Ellie Pavlick. 2021. Frequency Effects on Syntactic Rule Learning in Transformers. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wei2021frequency,
  title = {{Frequency Effects on Syntactic Rule Learning in Transformers}},
  author = {Jason Wei, Dan Garrette, Tal Linzen and Ellie Pavlick},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.72},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.72}
}
```

- [← previous As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages](https://md.research.elamur.ai/free/en/papers/vries-2021-as-good-as-new-how/)
- [next → QED: A Framework and Dataset for Explanations in Question Answering](https://md.research.elamur.ai/free/en/papers/lamm-2021-qed-framework-dataset-explanations-question/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. 2021. Predicting inductive biases of pre-trained models. In Proc. of ICLR. [find](https://scholar.google.com/scholar?q=Charles+Lovering%2C+Rohan+Jha%2C+Tal+Linzen%2C+and+Ellie+Pavlick%2E+2021%2E+Predicting+inductive+biases+of+pre%2Dtrained+models%2E+In+Proc%2E+of+ICLR%2E)
2. Rui Chaves and Stephanie Richter. 2021. Look at that! BERT can be easily distracted from paying attention to morphosyntax. In Proceedings of the Society for Computation in Linguistics. [find](https://scholar.google.com/scholar?q=Rui+Chaves+and+Stephanie+Richter%2E+2021%2E+Look+at+that%21+BERT+can+be+easily+distracted+from+paying+attention+to+morphosyntax%2E+In+Proceedings+of+the+Society+for+Computation+in+Linguistics%2E)
3. Noam Chomsky. 1956. Three models for the description of language. IRE Transactions on Information Theory, 2(3):113–124. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1956%2E+Three+models+for+the+description+of+language%2E+IRE+Transactions+on+Information+Theory%2C+2%283%29%3A113%E2%80%93124%2E)
4. Alec Marantz. 2013. Words and rules revisited: Reassessing the role of construction and memory in language. In 27th Pacific Asia Conference on Language, Information, and Computation, PACLIC 2013. National Chengchi University. [find](https://scholar.google.com/scholar?q=Alec+Marantz%2E+2013%2E+Words+and+rules+revisited%3A+Reassessing+the+role+of+construction+and+memory+in+language%2E+In+27th+Pacific+Asia+Conference+on+Language%2C+Information%2C+and+Computation%2C+PACLIC+2013%2E+Nati)
5. Gabriella Chronis and Katrin Erk. 2020. When is a bishop not like a rook? When it’s like a rabbi! Multi-prototype BERT embeddings for estimating semantic relationships. In Proc. of CoNLL. [find](https://scholar.google.com/scholar?q=Gabriella+Chronis+and+Katrin+Erk%2E+2020%2E+When+is+a+bishop+not+like+a+rook%3F+When+it%E2%80%99s+like+a+rabbi%21+Multi%2Dprototype+BERT+embeddings+for+estimating+semantic+relationships%2E+In+Proc%2E+of+CoNLL%2E)
6. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proc%2E+of+EMNLP%2E)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proc. of NAACL.
8. Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016. [Probing for semantic evidence of composition by means of simple classification tasks](https://md.research.elamur.ai/free/en/papers/ettinger-2016-probing-semantic-evidence-composition-means/). In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP.
9. Jerry A. Fodor and Zenon W. Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3–71. [find](https://scholar.google.com/scholar?q=Jerry+A%2E+Fodor+and+Zenon+W%2E+Pylyshyn%2E+1988%2E+Connectionism+and+cognitive+architecture%3A+A+critical+analysis%2E+Cognition%2C+28%281%2D2%29%3A3%E2%80%9371%2E)
10. Yoav Goldberg. 2019. [Assessing bert’s syntactic abilities](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/). arXiv preprint arXiv:1901.05287.
11. Emily Goodwin, Koustuv Sinha, and Timothy J. O’Donnell. 2020. Probing linguistic systematicity. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Emily+Goodwin%2C+Koustuv+Sinha%2C+and+Timothy+J%2E+O%E2%80%99Donnell%2E+2020%2E+Probing+linguistic+systematicity%2E+In+Proc%2E+of+ACL%2E)
12. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. [Colorless green recurrent networks dream hierarchically](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/). In Proc. of NAACL.
13. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Jon+Gauthier%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2020%2E+A+systematic+assessment+of+syntactic+generalization+in+neural+language+models%2E+In+Proc%2E+of+NAACL%2E)
14. Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Najoung+Kim+and+Tal+Linzen%2E+2020%2E+COGS%3A+A+compositional+generalization+challenge+based+on+semantic+interpretation%2E+In+Proc%2E+of+EMNLP%2E)
15. Najoung Kim and Paul Smolensky. 2021. Testing for grammatical category abstraction in neural language models. Proceedings of the Society for Computation in Linguistics. [find](https://scholar.google.com/scholar?q=Najoung+Kim+and+Paul+Smolensky%2E+2021%2E+Testing+for+grammatical+category+abstraction+in+neural+language+models%2E+Proceedings+of+the+Society+for+Computation+in+Linguistics%2E)
16. Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proc. of ICML. [find](https://scholar.google.com/scholar?q=Brenden+Lake+and+Marco+Baroni%2E+2018%2E+Generalization+without+systematicity%3A+On+the+compositional+skills+of+sequence%2Dto%2Dsequence+recurrent+networks%2E+In+Proc%2E+of+ICML%2E)
17. Tal Linzen and Marco Baroni. 2020. Syntactic structure from deep learning. Annual Review of Linguistics. [find](https://scholar.google.com/scholar?q=Tal+Linzen+and+Marco+Baroni%2E+2020%2E+Syntactic+structure+from+deep+learning%2E+Annual+Review+of+Linguistics%2E)
18. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. [Assessing the ability of LSTMs to learn syntax-sensitive dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). TACL.
19. Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. 2021. Refining targeted syntactic evaluation of language models. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Benjamin+Newman%2C+Kai%2DSiang+Ang%2C+Julia+Gong%2C+and+John+Hewitt%2E+2021%2E+Refining+targeted+syntactic+evaluation+of+language+models%2E+In+Proc%2E+of+NAACL%2E)
20. Judea Pearl. 1988. Probabilistic reasoning in intelligent systems: Networks of plausible inference. [find](https://scholar.google.com/scholar?q=Judea+Pearl%2E+1988%2E+Probabilistic+reasoning+in+intelligent+systems%3A+Networks+of+plausible+inference%2E)
21. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. [A primer in BERTology: What we know about how BERT works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). TACL.
22. Paul Smolensky. 1988. On the proper treatment of connectionism. Behavioral and Brain Sciences, 11(1):1–23. [find](https://scholar.google.com/scholar?q=Paul+Smolensky%2E+1988%2E+On+the+proper+treatment+of+connectionism%2E+Behavioral+and+Brain+Sciences%2C+11%281%29%3A1%E2%80%9323%2E)
23. Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and practical BERT models for sequence labeling. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Henry+Tsai%2C+Jason+Riesa%2C+Melvin+Johnson%2C+Naveen+Arivazhagan%2C+Xin+Li%2C+and+Amelia+Archer%2E+2019%2E+Small+and+practical+BERT+models+for+sequence+labeling%2E+In+Proc%2E+of+EMNLP%2E)
24. Sara Veldhoen, Dieuwke Hupkes, and Willem Zuidema. 2016. Diagnostic Classifiers: Revealing how Neural Networks Process Hierarchical Structure. In CEUR Workshop Proceedings. [find](https://scholar.google.com/scholar?q=Sara+Veldhoen%2C+Dieuwke+Hupkes%2C+and+Willem+Zuidema%2E+2016%2E+Diagnostic+Classifiers%3A+Revealing+how+Neural+Networks+Process+Hierarchical+Structure%2E+In+CEUR+Workshop+Proceedings%2E)
25. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP.
26. Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Yian+Zhang%2C+Xiaocheng+Li%2C+Haokun+Liu%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+Learning+which+features+matter%3A+RoBERTa+acquires+a+preference+for+linguistic+generalizations+%28eventually%29%2E+In+Proc%2E+of+)
27. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP. [find](https://scholar.google.com/scholar?q=Hitomi+Yanaka%2C+Koji+Mineshima%2C+Daisuke+Bekki%2C+Kentaro+Inui%2C+Satoshi+Sekine%2C+Lasha+Abzianidze%2C+and+Johan+Bos%2E+2019%2E+Can+neural+networks+understand+monotonicity+reasoning%3F+In+Proceedings+of+the+2019+ACL)
28. Charles Yu, Ryan Sie, Nicolas Tedeschi, and Leon Bergen. 2020. Word frequency does not predict grammatical knowledge in language models. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Charles+Yu%2C+Ryan+Sie%2C+Nicolas+Tedeschi%2C+and+Leon+Bergen%2E+2020%2E+Word+frequency+does+not+predict+grammatical+knowledge+in+language+models%2E+In+Proc%2E+of+EMNLP%2E)
29. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In Proc. of ICCV.

*Updated 24 October 2026*