---
title: "SyntaxGym: An Online Platform for Targeted — Gauthier 2020"
description: "Targeted syntactic evaluation tests whether neural language models have learned specific grammatical generalizations."
authors: ["Jon Gauthier", "Jennifer Hu", "Ethan Wilcox", "Peng Qian", "Roger Levy"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations"
source: "https://aclanthology.org/2020.acl-demos.10/"
doi: "10.18653/v1/2020.acl-demos.10"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / SyntaxGym: An Online Platform for Targeted Evaluation of Language Models

# SyntaxGym: An Online Platform for Targeted Evaluation of Language Models

**Authors:** Jon Gauthier, [Jennifer Hu](https://md.research.elamur.ai/free/en/authors/jennifer-hu/), [Ethan Wilcox](https://md.research.elamur.ai/free/en/authors/ethan-wilcox/), [Peng Qian](https://md.research.elamur.ai/free/en/authors/peng-qian/), [Roger Levy](https://md.research.elamur.ai/free/en/authors/roger-levy/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-demos.10/) · [PDF](https://aclanthology.org/2020.acl-demos.10.pdf) · [DOI 10.18653/v1/2020.acl-demos.10](https://doi.org/10.18653/v1/2020.acl-demos.10) · [OpenAlex](https://openalex.org/W3037115370)

**Citations:** 60 citations (openalex-snapshot · 2026-10-14) · [85](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Targeted syntactic evaluation tests whether neural language models have learned specific grammatical generalizations. Broad metrics such as perplexity can hide large differences in syntactic knowledge between models that score similarly. This kind of work, however, requires both psycholinguistic design expertise and the engineering capacity to run large models. The authors close this gap with SyntaxGym, a website (syntaxgym.org) backed by open-source tooling.

Linguists contribute test suites written in a standardized JSON format. NLP researchers contribute models packaged as Docker images that follow a shared API. The backend validates submissions, scores every model against every suite in the cloud, stores raw surprisals and accuracies, and presents interactive visualizations. Two command-line tools, syntaxgym and lm-zoo, let anyone reproduce these evaluations on their own machine.

Compared with BLiMP, the framework supports stricter predictions, such as conjunctions of inequalities across conditions. It also measures surprisal only within pre-specified critical regions. The contribution is infrastructure rather than a new empirical finding: it makes targeted evaluation more standardized, reproducible and accessible to both communities.

## Key points
- **Test suite components:** suites are built from regions, conditions, items and predictions.
  - Predictions are inequalities over model surprisal within chosen regions, for example the matrix verb in subject–verb agreement items.
  - A model's accuracy is the share of items for which the prediction holds.
- **lm-zoo API:** each model is wrapped in a Docker image exposing three standard commands: `tokenize`, `get-surprisals` and `unkify`. Documentation is at cpllab.github.io/lm-zoo.
- **syntaxgym CLI:** it lists hosted models and suites and evaluates either hosted or user-supplied ones.
  - Example models: gpt-2, gpt-2-xl, transformer-xl.
  - Example suites: number-orc, number-src, mvrr.
  - In a demo run, gpt-2 scores 0.7857 (22/28) on mvrr.
- **Related datasets:** the paper positions itself against Marvin and Linzen (2018) and BLiMP (Warstadt et al., 2020).
  - Marvin and Linzen covers few phenomena.
  - BLiMP sentences are template-generated and complementary to SyntaxGym's hand-designed suites.
- **Planned extensions:** human performance as a reference metric, and broader lm-zoo features such as next-word distributions and internal representations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

85 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 78 not yet.

- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://md.research.elamur.ai/free/en/papers/dhole-2023-nl-augmenter-framework-task-sensitive/)
- [Expectations over Unspoken Alternatives Predict Pragmatic Inferences](https://md.research.elamur.ai/free/en/papers/hu-2023-expectations-over-unspoken-alternatives-predict/)
- [Prompting is not a substitute for probability measurements in large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/)
- [Mechanistic Interpretability Meets Cognitive Linguistics: Modelling Locative Image Schemas in the Circuit Framework](https://md.research.elamur.ai/free/en/papers/proietti-2026-mechanistic-interpretability-meets-cognitive-linguistics/)
- [Are Frequent Phrases Directly Retrieved like Idioms? An Investigation with Self-Paced Reading and Language Models](https://md.research.elamur.ai/free/en/papers/rambelli-2023-are-frequent-phrases-directly-retrieved/)
- [Open-access Dataset on Acceptability Ratings of Korean Clausal Constructions by Humans and GPT Models](https://md.research.elamur.ai/free/en/papers/shin-2026-open-access-dataset-acceptability-ratings/)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gauthier2020syntaxgym,
  title = {{SyntaxGym: An Online Platform for Targeted Evaluation of Language Models}},
  author = {Gauthier, Jon and Hu, Jennifer and Wilcox, Ethan and Qian, Peng and Levy, Roger},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations},
  year = {2020},
  doi = {10.18653/v1/2020.acl-demos.10},
  url = {https://doi.org/10.18653/v1/2020.acl-demos.10}
}
```

- [← previous Syntax-Guided Controlled Generation of Paraphrases](https://md.research.elamur.ai/free/en/papers/kumar-2020-syntax-guided-controlled-generation-paraphrases/)
- [next → Synthetic Data for English Lexical Normalization: How Close Can We Get to Manually Annotated Data?](https://md.research.elamur.ai/free/en/papers/dekker-2020-synthetic-data-english-lexical-normalization/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marten van Schijndel, Aaron Mueller, and Tal Linzen. 2019. Quantity doesn’t buy quality syntax with neural language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL… [find](https://scholar.google.com/scholar?q=Marten+van+Schijndel%2C+Aaron+Mueller%2C+and+Tal+Linzen%2E+2019%2E+Quantity+doesn%E2%80%99t+buy+quality+syntax+with+neural+language+models%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Langua)
2. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+G+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+An)
3. Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yikang+Shen%2C+Shawn+Tan%2C+Alessandro+Sordoni%2C+and+Aaron+Courville%2E+2019%2E+Ordered+neurons%3A+Integrating+tree+structures+into+recurrent+neural+networks%2E+In+International+Conference+on+Learning+Representati)
4. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. In Proceedings of NAACL-HLT, pages 199–209. [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Adhiguna+Kuncoro%2C+Miguel+Ballesteros%2C+and+Noah+A+Smith%2E+2016%2E+Recurrent+neural+network+grammars%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+199%E2%80%93209%2E)
5. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Richard+Futrell%2C+Ethan+Wilcox%2C+Takashi+Morita%2C+Peng+Qian%2C+Miguel+Ballesteros%2C+and+Roger+Levy%2E+2019%2E+Neural+language+models+as+psycholinguistic+subjects%3A+Representations+of+syntactic+state%2E+In+Proceedi) [our card](https://md.research.elamur.ai/free/en/papers/futrell-2019-neural-language-models-as-psycholinguistic/)
6. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: A benchmark of linguistic minimal pairs for English. In Proceedings of the Society for Computation in Linguistics. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Alicia+Parrish%2C+Haokun+Liu%2C+Anhad+Mohananey%2C+Wei+Peng%2C+Sheng%2DFu+Wang%2C+and+Samuel+R+Bowman%2E+2020%2E+BLiMP%3A+A+benchmark+of+linguistic+minimal+pairs+for+English%2E+In+Proceedings+of+the+Societ)
7. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+R%E2%80%99emi+Louf%2C+Morgan+Funtow%2Dicz%2C+and+Jamie+Brew%2E+2019%2E+Huggingface%E2%80%99s+transformers%3A+S) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
8. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of NAACL-HLT, pages 1195–1205. [find](https://scholar.google.com/scholar?q=Kristina+Gulordava%2C+Piotr+Bojanowski%2C+Edouard+Grave%2C+Tal+Linzen%2C+and+Marco+Baroni%2E+2018%2E+Colorless+green+recurrent+networks+dream+hierarchically%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+1195%E2%80%931205%2E) [our card](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/)
9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2E)
10. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the Association of Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Jon+Gauthier%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2020%2E+A+systematic+assessment+of+syntactic+generalization+in+neural+language+models%2E+In+Proceedings+of+the+Association+of+Computatio)
11. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+2016%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2E)
12. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 5:1202–1247. [find](https://scholar.google.com/scholar?q=Jey+Han+Lau%2C+Alexander+Clark%2C+and+Shalom+Lappin%2E+2017%2E+Grammaticality%2C+acceptability%2C+and+probability%3A+A+probabilistic+view+of+linguistic+knowledge%2E+Cognitive+Science%2C+5%3A1202%E2%80%931247%2E)
13. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. In Transactions of the Association for Computational Linguistics, volume 4, pages 521–535. [find](https://scholar.google.com/scholar?q=Tal+Linzen%2C+Emmanuel+Dupoux%2C+and+Yoav+Goldberg%2E+2016%2E+Assessing+the+ability+of+LSTMs+to+learn+syntax%2Dsensitive+dependencies%2E+In+Transactions+of+the+Association+for+Computational+Linguistics%2C+volume+4%2C) [our card](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/)
14. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1192%E2%80%931202%2C+Bruss)
15. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%3A9%2E)