---
title: "Targeted Syntactic Evaluation on the Chomsky Hierarchy — Someya 2024"
description: "The authors propose a new approach to Targeted Syntactic Evaluation. 46 references, cited by 10 papers here."
authors: ["Taiga Someya", "Ryo Yoshida", "Yohei Oseki"]
year: 2024
venue: "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)"
source: "https://aclanthology.org/2024.lrec-main.1356/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/someya-2024-targeted-syntactic-evaluation-chomsky-hierarchy/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/someya-2024-targeted-syntactic-evaluation-chomsky-hierarchy/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Targeted Syntactic Evaluation on the Chomsky Hierarchy

# Targeted Syntactic Evaluation on the Chomsky Hierarchy

**Authors:** [Taiga Someya](https://md.research.elamur.ai/free/en/authors/taiga-someya/), [Ryo Yoshida](https://md.research.elamur.ai/free/en/authors/ryo-yoshida/) and [Yohei Oseki](https://md.research.elamur.ai/free/en/authors/yohei-oseki/) · **Year:** 2024 · **Venue:** Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.lrec-main.1356/) · [PDF](https://aclanthology.org/2024.lrec-main.1356.pdf)

**Citations:** [10](https://md.research.elamur.ai/free/en/papers/someya-2024-targeted-syntactic-evaluation-chomsky-hierarchy/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The authors propose a new approach to Targeted Syntactic Evaluation. Rather than testing individual grammatical phenomena, they examine how language models handle constructions of varying complexity according to the Chomsky hierarchy. To this end, four artificial formal languages are constructed, each modeling a syntactic phenomenon of natural language. The first imitates the arbitrary repetition of adjectives before a noun. The second imitates nested clauses without agreement, as in Japanese, and the third, nested dependencies with agreement, as in English. The fourth reproduces cross-serial dependencies characteristic of Swiss German. Each nonterminal symbol can be realized by several terminals, so each language has its own vocabulary. Five variants with different vocabulary sizes were created for each language. LSTM, Transformer LM and Stack-RNN were trained as language models. They were then tested on minimal pairs: the model had to assign a higher probability to the grammatical string than to the ungrammatical one. All architectures learned the simplest, regular language, while none of them coped with either type of dependency. The authors conclude that the models do not acquire the true dependency structure but merely approximate it with rules at the level of regular grammars.

## Key points
- Four formal languages: (Adj)ⁿ NP (regular, recognized by a finite-state automaton), NPⁿ VPⁿ, Nested Dependency and Cross Serial Dependency. Each has five variants with different vocabulary sizes.
- Three architectures are compared: LSTM, Transformer LM and Stack-RNN. The evaluation compares each grammatical sentence with an ungrammatical counterpart, and success means a higher probability for the correct sentence.
- All models mastered (Adj)ⁿ NP. LSTM and Transformer LM failed on NPⁿ VPⁿ. No architecture correctly recognized Nested Dependency or Cross Serial Dependency.
- This approach differs from works such as Delétang et al. (2023) in two respects. First, the models are trained specifically as language models. Second, it accounts for a diversity of terminals closer to the vocabulary of natural language.
- The code is open: github.com/osekilab/TSE-Chomsky.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/)

## Cite

Cite as: Taiga Someya, Ryo Yoshida and Yohei Oseki. 2024. Targeted Syntactic Evaluation on the Chomsky Hierarchy. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{someya2024targeted,
  title = {{Targeted Syntactic Evaluation on the Chomsky Hierarchy}},
  author = {Taiga Someya, Ryo Yoshida and Yohei Oseki},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  url = {https://aclanthology.org/2024.lrec-main.1356/}
}
```

- [← previous How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?](https://md.research.elamur.ai/free/en/papers/srikanth-2024-how-often-are-errors-natural/)
- [next → The Gricean Maxims in NLP — A Survey](https://md.research.elamur.ai/free/en/papers/krause-2024-gricean-maxims-nlp-survey/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

46 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alfred V Aho and Jeffrey D Ullman. 1972. The Theory of Parsing, Translation and Compiling. Parsing, vol. I. Prentice-Hall, Englewood Cliffs. [find](https://scholar.google.com/scholar?q=Alfred+V+Aho+and+Jeffrey+D+Ullman%2E+1972%2E+The+Theory+of+Parsing%2C+Translation+and+Compiling%2E+Parsing%2C+vol%2E+I%2E+Prentice%2DHall%2C+Englewood+Cliffs%2E)
2. Aixiu An, Peng Qian, Ethan Wilcox, and Roger Levy. 2019. Representation of constituents in neural language models: Coordination phrase as a case study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lang… [find](https://scholar.google.com/scholar?q=Aixiu+An%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2019%2E+Representation+of+constituents+in+neural+language+models%3A+Coordination+phrase+as+a+case+study%2E+In+Proceedings+of+the+2019+Conference+on+Empiric)
3. Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. 2020. On the Ability and Limitations of Transformers to Recognize Formal Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7096–7116, Online. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Satwik+Bhattamishra%2C+Kabir+Ahuja%2C+and+Navin+Goyal%2E+2020%2E+On+the+Ability+and+Limitations+of+Transformers+to+Recognize+Formal+Languages%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Nat)
4. Mikael Boden and Janet Wiles. 2002. On learning context free and context sensitive languages. IEEE Transactions on Neural Networks, 13:491– 493. [find](https://scholar.google.com/scholar?q=Mikael+Boden+and+Janet+Wiles%2E+2002%2E+On+learning+context+free+and+context+sensitive+languages%2E+IEEE+Transactions+on+Neural+Networks%2C+13%3A491%E2%80%93+493%2E)
5. Mikael Bodón and Janet Wiles. 2000. Context-free and context-sensitive dynamics in recurrent neural networks. Connection Science, 12(3-4):197– 210. [find](https://scholar.google.com/scholar?q=Mikael+Bod%C3%B3n+and+Janet+Wiles%2E+2000%2E+Context%2Dfree+and+context%2Dsensitive+dynamics+in+recurrent+neural+networks%2E+Connection+Science%2C+12%283%2D4%29%3A197%E2%80%93+210%2E)
6. Rui P Chaves. 2020. What don’t RNN language models learn about Filler-Gap dependencies? Proceedings of the Society for Computation in Linguistics, 3(1):20–30. [find](https://scholar.google.com/scholar?q=Rui+P+Chaves%2E+2020%2E+What+don%E2%80%99t+RNN+language+models+learn+about+Filler%2DGap+dependencies%3F+Proceedings+of+the+Society+for+Computation+in+Linguistics%2C+3%281%29%3A20%E2%80%9330%2E)
7. Noam Chomsky. 1957. Syntactic structures. Mouton. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1957%2E+Syntactic+structures%2E+Mouton%2E)
8. Shammur Absar Chowdhury and Roberto Zamparelli. 2018. RNN simulations of grammaticality judgments on long-distance dependencies. In Proceedings of the 27th International Conference on Computational Linguistics, pages 133– 144, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shammur+Absar+Chowdhury+and+Roberto+Zamparelli%2E+2018%2E+RNN+simulations+of+grammaticality+judgments+on+long%2Ddistance+dependencies%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Li)
9. Jillian Da Costa and Rui Chaves. 2020. Assessing the ability of transformer-based neural models to represent structurally unbounded dependencies. In Proceedings of the Society for Computation in Linguistics 2020, pages 12–21, New York, New York. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jillian+Da+Costa+and+Rui+Chaves%2E+2020%2E+Assessing+the+ability+of+transformer%2Dbased+neural+models+to+represent+structurally+unbounded+dependencies%2E+In+Proceedings+of+the+Society+for+Computation+in+Lingu)
10. Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A Ortega. 2023. Neural networks and the chomsky hierarchy. In The Eleventh International Conference on Learning Representations. [our card](https://md.research.elamur.ai/free/en/papers/deletang-2022-neural-networks-chomsky-hierarchy/)
11. Javid Ebrahimi, Dhruv Gelda, and Wei Zhang. 2020. How can self-attention networks recognize dyck-n languages? CoRR, abs/2010.04303. [find](https://scholar.google.com/scholar?q=Javid+Ebrahimi%2C+Dhruv+Gelda%2C+and+Wei+Zhang%2E+2020%2E+How+can+self%2Dattention+networks+recognize+dyck%2Dn+languages%3F+CoRR%2C+abs%2F2010%2E04303%2E)
12. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui… [our card](https://md.research.elamur.ai/free/en/papers/futrell-2019-neural-language-models-as-psycholinguistic/)
13. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… [our card](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/)
14. Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156–171. [find](https://scholar.google.com/scholar?q=Michael+Hahn%2E+2020%2E+Theoretical+limitations+of+self%2Dattention+in+neural+sequence+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A156%E2%80%93171%2E)
15. S Hochreiter and J Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=S+Hochreiter+and+J+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Comput%2E%2C+9%288%29%3A1735%E2%80%931780%2E)
16. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the Association of Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Jon+Gauthier%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2020%2E+A+systematic+assessment+of+syntactic+generalization+in+neural+language+models%2E+In+Proceedings+of+the+Association+of+Computatio)
17. Armand Joulin and Tomas Mikolov. 2015. Inferring algorithmic patterns with stack-augmented recurrent nets. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 190â��198, Cambridge, MA, USA. MIT Press. [find](https://scholar.google.com/scholar?q=Armand+Joulin+and+Tomas+Mikolov%2E+2015%2E+Inferring+algorithmic+patterns+with+stack%2Daugmented+recurrent+nets%2E+In+Proceedings+of+the+28th+International+Conference+on+Neural+Information+Processing+Systems+)
18. Katharina Kann, Alex Warstadt, Adina Williams, and Samuel R Bowman. 2019. Verb argument structure alternations in word and sentence embeddings. In Proceedings of the Society for Computation in Linguistics (SCiL) 2019, pages 287–297. [find](https://scholar.google.com/scholar?q=Katharina+Kann%2C+Alex+Warstadt%2C+Adina+Williams%2C+and+Samuel+R+Bowman%2E+2019%2E+Verb+argument+structure+alternations+in+word+and+sentence+embeddings%2E+In+Proceedings+of+the+Society+for+Computation+in+Linguis)
19. D. Terence Langendoen. 1981. The generative capacity of word-formation components. Linguistic Inquiry, 12(2):320–322. [find](https://scholar.google.com/scholar?q=D%2E+Terence+Langendoen%2E+1981%2E+The+generative+capacity+of+word%2Dformation+components%2E+Linguistic+Inquiry%2C+12%282%29%3A320%E2%80%93322%2E)
20. Sabine Lehmann, Stephan Oepen, Sylvie Regnier-Prost, Klaus Netter, Veronika Lux, Judith Klein, Kirsten Falkedal, Frederik Fouvry, Dominique Estival, Eva Dauphin, Herve Compagnion, Judith Baur, Lorna Balkan, and Doug Arnold. 1996. TSNLP - test suites for natural language processing. In COLING 1996 Vo… [find](https://scholar.google.com/scholar?q=Sabine+Lehmann%2C+Stephan+Oepen%2C+Sylvie+Regnier%2DProst%2C+Klaus+Netter%2C+Veronika+Lux%2C+Judith+Klein%2C+Kirsten+Falkedal%2C+Frederik+Fouvry%2C+Dominique+Estival%2C+Eva+Dauphin%2C+Herve+Compagnion%2C+Judith+Baur%2C+Lorna+B)
21. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn Syntax-Sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. [our card](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/)
22. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E+OpenReview%2En)
23. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1192%E2%80%931202%2C+Bruss)
24. Rowan Hall Maudslay and Ryan Cotterell. 2021. Do syntactic probes probe syntax? experiments with jabberwocky probing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 124–131, Online. Association… [find](https://scholar.google.com/scholar?q=Rowan+Hall+Maudslay+and+Ryan+Cotterell%2E+2021%2E+Do+syntactic+probes+probe+syntax%3F+experiments+with+jabberwocky+probing%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Associa)
25. Vladislav Mikhailov, Ekaterina Taktasheva, Elina Sigdel, and Ekaterina Artemova. 2021. RuSen-tEval: Linguistic source, encoder force! In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 43–65, Kiyv, Ukraine. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vladislav+Mikhailov%2C+Ekaterina+Taktasheva%2C+Elina+Sigdel%2C+and+Ekaterina+Artemova%2E+2021%2E+RuSen%2DtEval%3A+Linguistic+source%2C+encoder+force%21+In+Proceedings+of+the+8th+Workshop+on+Balto%2DSlavic+Natural+Languag)
26. Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. 2020. Cross-linguistic syntactic evaluation of word prediction models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA. Association for Computati… [find](https://scholar.google.com/scholar?q=Aaron+Mueller%2C+Garrett+Nicolai%2C+Panayiota+Petrou%2DZeniou%2C+Natalia+Talmina%2C+and+Tal+Linzen%2E+2020%2E+Cross%2Dlinguistic+syntactic+evaluation+of+word+prediction+models%2E+In+Proceedings+of+the+58th+Annual+Meeti)
27. Shauli Ravfogel, Yoav Goldberg, and Francis Tyers. 2018. Can LSTM learn to capture agreement? the case of basque. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 98–107, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shauli+Ravfogel%2C+Yoav+Goldberg%2C+and+Francis+Tyers%2E+2018%2E+Can+LSTM+learn+to+capture+agreement%3F+the+case+of+basque%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Analyzing+and+Interpreting+Neur)
28. Paul Rodriguez and Janet Wiles. 1997. Recurrent neural networks can learn to implement symbol-sensitive counting. In Advances in Neural Information Processing Systems, volume 10. MIT Press. [find](https://scholar.google.com/scholar?q=Paul+Rodriguez+and+Janet+Wiles%2E+1997%2E+Recurrent+neural+networks+can+learn+to+implement+symbol%2Dsensitive+counting%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+10%2E+MIT+Press%2E)
29. Stuart M. Shieber. 1985. Evidence against the context-freeness of natural language. Linguistics and Philosophy, 8(3):333–343. [find](https://scholar.google.com/scholar?q=Stuart+M%2E+Shieber%2E+1985%2E+Evidence+against+the+context%2Dfreeness+of+natural+language%2E+Linguistics+and+Philosophy%2C+8%283%29%3A333%E2%80%93343%2E)
30. Natalia Skachkova, Thomas Trost, and Dietrich Klakow. 2018. Closing brackets with recurrent neural networks. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 232–239, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Natalia+Skachkova%2C+Thomas+Trost%2C+and+Dietrich+Klakow%2E+2018%2E+Closing+brackets+with+recurrent+neural+networks%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Analyzing+and+Interpreting+Neural+Ne)
31. Taiga Someya and Yohei Oseki. 2023. JBLiMP: Japanese benchmark of linguistic minimal pairs. [find](https://scholar.google.com/scholar?q=Taiga+Someya+and+Yohei+Oseki%2E+2023%2E+JBLiMP%3A+Japanese+benchmark+of+linguistic+minimal+pairs%2E)
32. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1581– 1594, Dubrovnik, Croatia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=In+Findings+of+the+Association+for+Computational+Linguistics%3A+EACL+2023%2C+pages+1581%E2%80%93+1594%2C+Dubrovnik%2C+Croatia%2E+Association+for+Computational+Linguistics%2E)
33. Mirac Suzgun, Yonatan Belinkov, Stuart Shieber, and Sebastian Gehrmann. 2019. LSTM networks can perform dynamic counting. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges, pages 44–54, Florence. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mirac+Suzgun%2C+Yonatan+Belinkov%2C+Stuart+Shieber%2C+and+Sebastian+Gehrmann%2E+2019%2E+LSTM+networks+can+perform+dynamic+counting%2E+In+Proceedings+of+the+Workshop+on+Deep+Learning+and+Formal+Languages%3A+Building)
34. Daniela Trotta, Raffaele Guarasci, Elisa Leonardelli, and Sara Tonelli. 2021. Monolingual and cross-lingual acceptability judgments with the Italian CoLA corpus. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2929–2940, Punta Cana, Dominican Republic. Association for… [find](https://scholar.google.com/scholar?q=Daniela+Trotta%2C+Raffaele+Guarasci%2C+Elisa+Leonardelli%2C+and+Sara+Tonelli%2E+2021%2E+Monolingual+and+cross%2Dlingual+acceptability+judgments+with+the+Italian+CoLA+corpus%2E+In+Findings+of+the+Association+for+Com)
35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł Ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
36. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
37. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. Superglue: A stickier benchmark for general-purpose language understanding systems. CoRR, abs/1905.00537. [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
38. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
39. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019c. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
40. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Alicia+Parrish%2C+Haokun+Liu%2C+Anhad+Mohananey%2C+Wei+Peng%2C+Sheng%2DFu+Wang%2C+and+Samuel+R+Bowman%2E+2020%2E+BLiMP%3A+The+benchmark+of+linguistic+minimal+pairs+for+english%2E+Transactions+of+the+Associ)
41. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2E)
42. Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision RNNs for language recognition. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 740–745, Melbourne, Australia. Associati… [our card](https://md.research.elamur.ai/free/en/papers/weiss-2018-practical-computational-power-finite-precision/)
43. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about Filler–Gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, Brussels, Belgium. Association for Computati… [our card](https://md.research.elamur.ai/free/en/papers/wilcox-2018-rnn-language-models-learn-about/)
44. Ethan Wilcox, Roger P. Levy, and Richard Futrell. 2019. Hierarchical representation in neural language models: Suppression and recovery of expectations. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 181– 190, Florence, Italy. Associati… [find](https://scholar.google.com/scholar?q=Ethan+Wilcox%2C+Roger+P%2E+Levy%2C+and+Richard+Futrell%2E+2019%2E+Hierarchical+representation+in+neural+language+models%3A+Suppression+and+recovery+of+expectations%2E+In+Proceedings+of+the+2019+ACL+Workshop+Blackbo)
45. Janet Wiles and Jeffrey Elman. 1995. Learning to count without a counter: A case study of dynamics and activation landscapes in recurrent networks. [find](https://scholar.google.com/scholar?q=Janet+Wiles+and+Jeffrey+Elman%2E+1995%2E+Learning+to+count+without+a+counter%3A+A+case+study+of+dynamics+and+activation+landscapes+in+recurrent+networks%2E)
46. Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. CLiMP: A benchmark for Chinese language model evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2784–2790, Online. Association fo… [find](https://scholar.google.com/scholar?q=Beilei+Xiang%2C+Changbing+Yang%2C+Yu+Li%2C+Alex+Warstadt%2C+and+Katharina+Kann%2E+2021%2E+CLiMP%3A+A+benchmark+for+Chinese+language+model+evaluation%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of)