---
title: "Multilingual Nonce Dependency Treebanks — Arps 2024"
description: "The authors test whether probes that extract syntax from language model representations actually capture sentence structure rather than relying on…"
authors: ["David Arps", "Laura Kallmeyer", "Younes Samih", "Hassan Sajjad"]
year: 2024
venue: "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.naacl-long.433/"
doi: "10.18653/v1/2024.naacl-long.433"
arxiv: "2311.07497"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/arps-2024-multilingual-nonce-dependency-treebanks-understanding/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/arps-2024-multilingual-nonce-dependency-treebanks-understanding/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2024](https://md.research.elamur.ai/free/papers/2024/page/3/) / Multilingual Nonce Dependency Treebanks: Understanding how Language Models Represent and Process Syntactic Structure

# Multilingual Nonce Dependency Treebanks: Understanding how Language Models Represent and Process Syntactic Structure

**Authors:** David Arps, [Laura Kallmeyer](https://md.research.elamur.ai/free/authors/laura-kallmeyer/), [Younes Samih](https://md.research.elamur.ai/free/authors/younes-samih/), [Hassan Sajjad](https://md.research.elamur.ai/free/authors/hassan-sajjad/) · **Year:** 2024 · **Venue:** Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.naacl-long.433/) · [PDF](https://aclanthology.org/2024.naacl-long.433.pdf) · [DOI 10.18653/v1/2024.naacl-long.433](https://doi.org/10.18653/v1/2024.naacl-long.433) · [arXiv 2311.07497](https://arxiv.org/abs/2311.07497) · [OpenAlex](https://openalex.org/W4401042796)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · 3 in our library, of the 140673 papers whose bibliographies we read

## Summary
The authors test whether probes that extract syntax from language model representations actually capture sentence structure rather than relying on semantic cues. To do this, they created SPUD (Semantically Perturbed Universal Dependencies). This framework turns a UD treebank into a corpus of nonsensical but grammatically correct sentences. In each sentence, content words are replaced with other lemmas that occurred in the same syntactic context in the corpus. The dependency tree stays the same, and language-specific rules control morphology and word order. Such data were built for Arabic, English, French, German and Russian. Two experiments were run on them: sentence evaluation via the perplexity of autoregressive (ALM) and masked (MLM) models, and structural probing of dependency trees. On SPUD, ALM perplexity increases noticeably more than MLM pseudo-perplexity. Probe accuracy also drops but is largely preserved. Thus, the models' syntactic knowledge is to a large extent independent of lexical semantics. Applying the method to a new language, including Ukrainian, requires only a UD corpus and a native speaker.

## Key points
- The corpora used are PADT (Arabic), EWT (English), GSD (French), HDT (German) and SynTagRus (Russian).
- Only words with UPOS ADJ, ADV, NOUN, PROPN and VERB are replaced. A word's syntactic context consists of its UPOS, its relation to the head and its relations to its dependents. The required word forms are taken from UDLexicon and Wiktextract. According to human evaluation, SPUD is more grammatical than the nonce data from previous work.
- PPL was computed for ALMs, and PPPL and subword-aware PPPL l2r for MLMs. On SPUD, all metrics show lower sentence probability; this is most pronounced in PPL for ALMs. PPPL is also affected by whether subword information is available to the model.
- Probing was performed with DepProbe (Müller-Eberstein et al., 2022) on mBERT and mGPT. mGPT has lower overall quality and a larger drop on SPUD. For both models, the drop is related more to edge attachment than to relation type labeling.
- The code and a step-by-step guide for creating SPUD for other languages are published here: github.com/davidarps/spud.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Treebanks](https://md.research.elamur.ai/free/topics/treebanks/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{arps2024multilingual,
  title = {{Multilingual Nonce Dependency Treebanks: Understanding how Language Models Represent and Process Syntactic Structure}},
  author = {Arps, David and Kallmeyer, Laura and Samih, Younes and Sajjad, Hassan},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.433},
  eprint = {2311.07497},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.433}
}
```

- [← previous Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://md.research.elamur.ai/free/papers/zhu-2024-multilingual-machine-translation-large-language/)
- [next → Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles](https://md.research.elamur.ai/free/papers/nath-2024-multimodal-cross-document-event-coreference/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/papers/2024/page/3/)

## References

53 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Langu… [find](https://scholar.google.com/scholar?q=Wissam+Antoun%2C+Fady+Baly%2C+and+Hazem+Hajj%2E+2020%2E+AraBERT%3A+Transformer%2Dbased+model+for+Arabic+language+understanding%2E+In+Proceedings+of+the+4th+Workshop+on+Open%2DSource+Arabic+Corpora+and+Processing+Tool) [our card](https://md.research.elamur.ai/free/papers/antoun-2020-arabert-transformer-based-model-arabic/)
2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraGPT2: Pre-trained transformer for Arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196–207, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wissam+Antoun%2C+Fady+Baly%2C+and+Hazem+Hajj%2E+2021%2E+AraGPT2%3A+Pre%2Dtrained+transformer+for+Arabic+language+generation%2E+In+Proceedings+of+the+Sixth+Arabic+Natural+Language+Processing+Workshop%2C+pages+196%E2%80%93207%2C)
3. David Arps, Younes Samih, Laura Kallmeyer, and Hassan Sajjad. 2022. Probing for constituency structure in neural language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6738–6757, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Arps%2C+Younes+Samih%2C+Laura+Kallmeyer%2C+and+Hassan+Sajjad%2E+2022%2E+Probing+for+constituency+structure+in+neural+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+20)
4. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2E+2022%2E+Probing+classifiers%3A+Promises%2C+shortcomings%2C+and+advances%2E+Computational+Linguistics%2C+48%281%29%3A207%E2%80%93219%2E) [our card](https://md.research.elamur.ai/free/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/)
5. Emanuel Borges Völker, Maximilian Wendt, Felix Hennig, and Arne Köhn. 2019. HDT-UD: A very large Universal Dependencies treebank for German. In Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019), pages 46–57, Paris, France. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emanuel+Borges+V%C3%B6lker%2C+Maximilian+Wendt%2C+Felix+Hennig%2C+and+Arne+K%C3%B6hn%2E+2019%2E+HDT%2DUD%3A+A+very+large+Universal+Dependencies+treebank+for+German%2E+In+Proceedings+of+the+Third+Workshop+on+Universal+Dependenc)
6. Tyler A. Chang and Benjamin K. Bergen. 2023. Language Model Behavior: A Comprehensive Survey. Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tyler+A%2E+Chang+and+Benjamin+K%2E+Bergen%2E+2023%2E+Language+Model+Behavior%3A+A+Comprehensive+Survey%2E+Computational+Linguistics%2E)
7. Ethan A. Chi, John Hewitt, and Christopher D. Manning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564–5577, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ethan+A%2E+Chi%2C+John+Hewitt%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Finding+universal+grammatical+relations+in+multilingual+BERT%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computationa)
8. Noam Chomsky. 1957. Syntactic Structures. Mouton and Co., The Hague. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1957%2E+Syntactic+Structures%2E+Mouton+and+Co%2E%2C+The+Hague%2E)
9. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate Arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074, Portorož, Slovenia. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Kareem+Darwish+and+Hamdy+Mubarak%2E+2016%2E+Farasa%3A+A+new+fast+and+accurate+Arabic+word+segmenter%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9916%29%2C+page)
10. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+de+Marneffe%2C+Christopher+D%2E+Manning%2C+Joakim+Nivre%2C+and+Daniel+Zeman%2E+2021%2E+Universal+Dependencies%2E+Computational+Linguistics%2C+47%282%29%3A255%E2%80%93308%2E) [our card](https://md.research.elamur.ai/free/papers/demarneffe-2021-universal-dependencies/)
11. Mathieu Dehouck and Carlos Gómez-Rodríguez. 2020. Data augmentation via subtree swapping for dependency parsing of low-resource languages. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3818–3830, Barcelona, Spain (Online). International Committee on Computat… [find](https://scholar.google.com/scholar?q=Mathieu+Dehouck+and+Carlos+G%C3%B3mez%2DRodr%C3%ADguez%2E+2020%2E+Data+augmentation+via+subtree+swapping+for+dependency+parsing+of+low%2Dresource+languages%2E+In+Proceedings+of+the+28th+International+Conference+on+Comput)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
13. Kira Droganova, Olga Lyashevskaya, and Daniel Zeman. 2018. Data Conversion and Consistency of Monolingual Corpora: Russian UD Treebanks. In Proceedings of the 17th international workshop on treebanks and linguistic theories (tlt 2018), 155, pages 53–66. Linköping University Electronic Press Linköpin… [find](https://scholar.google.com/scholar?q=Kira+Droganova%2C+Olga+Lyashevskaya%2C+and+Daniel+Zeman%2E+2018%2E+Data+Conversion+and+Consistency+of+Monolingual+Corpora%3A+Russian+UD+Treebanks%2E+In+Proceedings+of+the+17th+international+workshop+on+treebanks+)
14. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. [find](https://scholar.google.com/scholar?q=Matthew+S%2E+Dryer+and+Martin+Haspelmath%2C+editors%2E+2013%2E+WALS+Online%2E+Max+Planck+Institute+for+Evolutionary+Anthropology%2C+Leipzig%2E)
15. Tiwalayo Eisape, Vineet Gangireddy, Roger Levy, and Yoon Kim. 2022. Probing for incremental parse states in autoregressive language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2801–2813, Abu Dhabi, United Arab Emirates. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Tiwalayo+Eisape%2C+Vineet+Gangireddy%2C+Roger+Levy%2C+and+Yoon+Kim%2E+2022%2E+Probing+for+incremental+parse+states+in+autoregressive+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics)
16. Bruno Guillaume, Marie-Catherine de Marneffe, and Guy Perrier. 2019. Conversion et améliorations de corpus du français annotés en Universal Dependencies [conversion and improvement of Universal Dependencies French corpora]. Traitement Automatique des Langues, 60(2):71–95. [find](https://scholar.google.com/scholar?q=Bruno+Guillaume%2C+Marie%2DCatherine+de+Marneffe%2C+and+Guy+Perrier%2E+2019%2E+Conversion+et+am%C3%A9liorations+de+corpus+du+fran%C3%A7ais+annot%C3%A9s+en+Universal+Dependencies+%5Bconversion+and+improvement+of+Universal+Depend)
17. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… [find](https://scholar.google.com/scholar?q=Kristina+Gulordava%2C+Piotr+Bojanowski%2C+Edouard+Grave%2C+Tal+Linzen%2C+and+Marco+Baroni%2E+2018%2E+Colorless+green+recurrent+networks+dream+hierarchically%2E+In+Proceedings+of+the+2018+Conference+of+the+North+Ame) [our card](https://md.research.elamur.ai/free/papers/gulordava-2018-colorless-green-recurrent-networks-dream/)
18. Jan Hajic, Otakar Smrz, Petr Zemánek, Petr Pajas, Jan Šnaidauf, Emanuel Beška, Jakub Kráčmar, and Kamila Hassanová. 2009. Prague Arabic Dependency Treebank 1.0. [find](https://scholar.google.com/scholar?q=Jan+Hajic%2C+Otakar+Smrz%2C+Petr+Zem%C3%A1nek%2C+Petr+Pajas%2C+Jan+%C5%A0naidauf%2C+Emanuel+Be%C5%A1ka%2C+Jakub+Kr%C3%A1c%CC%8Cmar%2C+and+Kamila+Hassanov%C3%A1%2E+2009%2E+Prague+Arabic+Dependency+Treebank+1%2E0%2E)
19. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129… [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Christopher+D%2E+Manning%2E+2019%2E+A+structural+probe+for+finding+syntax+in+word+representations%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/papers/hewitt-2019-structural-probe-finding-syntax-word/)
20. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and ’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907–926. [find](https://scholar.google.com/scholar?q=Dieuwke+Hupkes%2C+Sara+Veldhoen%2C+and+Willem+Zuidema%2E+2018%2E+Visualisation+and+%E2%80%99diagnostic+classifiers%E2%80%99+reveal+how+recurrent+and+recursive+neural+networks+process+hierarchical+structure%2E+Journal+of+Artifi)
21. Carina Kauf and Anna Ivanova. 2023. A better way to do masked language model scoring. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 925–935, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Carina+Kauf+and+Anna+Ivanova%2E+2023%2E+A+better+way+to+do+masked+language+model+scoring%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short+Papers)
22. Carina Kauf, Greta Tuckute, Roger Levy, Jacob Andreas, and Evelina Fedorenko. 2023. Lexical-Semantic Content, Not Syntactic Structure, Is the Main Contributor to ANN-Brain Similarity of fMRI Responses in the Language Network. Neurobiology of Language. [find](https://scholar.google.com/scholar?q=Carina+Kauf%2C+Greta+Tuckute%2C+Roger+Levy%2C+Jacob+Andreas%2C+and+Evelina+Fedorenko%2E+2023%2E+Lexical%2DSemantic+Content%2C+Not+Syntactic+Structure%2C+Is+the+Main+Contributor+to+ANN%2DBrain+Similarity+of+fMRI+Responses)
23. Reinhard Köhler. 2003. Zur Type-Token-Ratio syntaktischer Einheiten: Eine quantitativ-korpuslinguistische Studie, pages 93–101. Deutscher Universitätsverlag, Wiesbaden. [find](https://scholar.google.com/scholar?q=Reinhard+K%C3%B6hler%2E+2003%2E+Zur+Type%2DToken%2DRatio+syntaktischer+Einheiten%3A+Eine+quantitativ%2Dkorpuslinguistische+Studie%2C+pages+93%E2%80%93101%2E+Deutscher+Universit%C3%A4tsverlag%2C+Wiesbaden%2E)
24. Artur Kulmizev and Joakim Nivre. 2022. Schrödinger’s tree—On syntax and neural language models. Frontiers in Artificial Intelligence, 5. [find](https://scholar.google.com/scholar?q=Artur+Kulmizev+and+Joakim+Nivre%2E+2022%2E+Schr%C3%B6dinger%E2%80%99s+tree%E2%80%94On+syntax+and+neural+language+models%2E+Frontiers+in+Artificial+Intelligence%2C+5%2E)
25. Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. 2020. Do neural language models show preferences for syntactic formalisms? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4077– 4091, Online. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Artur+Kulmizev%2C+Vinit+Ravishankar%2C+Mostafa+Abdou%2C+and+Joakim+Nivre%2E+2020%2E+Do+neural+language+models+show+preferences+for+syntactic+formalisms%3F+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associat)
26. Jenny Kunz and Marco Kuhlmann. 2020. Classifier probes may just learn from linear context features. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5136–5146, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jenny+Kunz+and+Marco+Kuhlmann%2E+2020%2E+Classifier+probes+may+just+learn+from+linear+context+features%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+pages+5136%E2%80%935146%2C+B)
27. Jenny Kunz and Marco Kuhlmann. 2021. Test harder than you train: Probing with extrapolation splits. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 15–25, Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jenny+Kunz+and+Marco+Kuhlmann%2E+2021%2E+Test+harder+than+you+train%3A+Probing+with+extrapolation+splits%2E+In+Proceedings+of+the+Fourth+BlackboxNLP+Workshop+on+Analyzing+and+Interpreting+Neural+Networks+for+)
28. Karim Lasri, Alessandro Lenci, and Thierry Poibeau. 2022a. Does BERT really agree? fine-grained analysis of lexical dependence on a syntactic task. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2309–2315, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Karim+Lasri%2C+Alessandro+Lenci%2C+and+Thierry+Poibeau%2E+2022a%2E+Does+BERT+really+agree%3F+fine%2Dgrained+analysis+of+lexical+dependence+on+a+syntactic+task%2E+In+Findings+of+the+Association+for+Computational+Lin)
29. Karim Lasri, Olga Seminck, Alessandro Lenci, and Thierry Poibeau. 2022b. Subject verb agreement error patterns in meaningless sentences: Humans vs. BERT. In Proceedings of the 29th International Conference on Computational Linguistics, pages 37–43, Gyeongju, Republic of Korea. International Committe… [find](https://scholar.google.com/scholar?q=Karim+Lasri%2C+Olga+Seminck%2C+Alessandro+Lenci%2C+and+Thierry+Poibeau%2E+2022b%2E+Subject+verb+agreement+error+patterns+in+meaningless+sentences%3A+Humans+vs%2E+BERT%2E+In+Proceedings+of+the+29th+International+Confe)
30. Jey Han Lau, Carlos Armendariz, Shalom Lappin, Matthew Purver, and Chang Shu. 2020. How furiously can colorless green ideas sleep? sentence acceptability in context. Transactions of the Association for Computational Linguistics, 8:296–310. [find](https://scholar.google.com/scholar?q=Jey+Han+Lau%2C+Carlos+Armendariz%2C+Shalom+Lappin%2C+Matthew+Purver%2C+and+Chang+Shu%2E+2020%2E+How+furiously+can+colorless+green+ideas+sleep%3F+sentence+acceptability+in+context%2E+Transactions+of+the+Association+fo)
31. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge. Cognitive Science, 41(5):1202–1241. [find](https://scholar.google.com/scholar?q=Jey+Han+Lau%2C+Alexander+Clark%2C+and+Shalom+Lappin%2E+2017%2E+Grammaticality%2C+Acceptability%2C+and+Probability%3A+A+Probabilistic+View+of+Linguistic+Knowledge%2E+Cognitive+Science%2C+41%285%29%3A1202%E2%80%931241%2E)
32. Tal Linzen and Marco Baroni. 2021. Syntactic structure from deep learning. Annual Review of Linguistics, 7(1):195–212. [find](https://scholar.google.com/scholar?q=Tal+Linzen+and+Marco+Baroni%2E+2021%2E+Syntactic+structure+from+deep+learning%2E+Annual+Review+of+Linguistics%2C+7%281%29%3A195%E2%80%93212%2E)
33. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+Robustly+Optimized+BERT+Pretraining+Approach%2E)
34. Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. 2024. Towards Faithful Model Explanation in NLP: A Survey. Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qing+Lyu%2C+Marianna+Apidianaki%2C+and+Chris+Callison%2DBurch%2E+2024%2E+Towards+Faithful+Model+Explanation+in+NLP%3A+A+Survey%2E+Computational+Linguistics%2E)
35. Kyle Mahowald, Anna A. Ivanova, Idan Asher Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective. CoRR, abs/2301.06627. [find](https://scholar.google.com/scholar?q=Kyle+Mahowald%2C+Anna+A%2E+Ivanova%2C+Idan+Asher+Blank%2C+Nancy+Kanwisher%2C+Joshua+B%2E+Tenenbaum%2C+and+Evelina+Fedorenko%2E+2023%2E+Dissociating+language+and+thought+in+large+language+models%3A+a+cognitive+perspective)
36. Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48):30046–30054. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Kevin+Clark%2C+John+Hewitt%2C+Urvashi+Khandelwal%2C+and+Omer+Levy%2E+2020%2E+Emergent+linguistic+structure+in+artificial+neural+networks+trained+by+self%2Dsupervision%2E+Proceedings+of+the+N)
37. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=Mitchell+P%2E+Marcus%2C+Beatrice+Santorini%2C+and+Mary+Ann+Marcinkiewicz%2E+1993%2E+Building+a+large+annotated+corpus+of+English%3A+The+Penn+Treebank%2E+Computational+Linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.research.elamur.ai/free/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
38. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1192%E2%80%931202%2C+Bruss)
39. Rowan Hall Maudslay and Ryan Cotterell. 2021. Do syntactic probes probe syntax? experiments with jabberwocky probing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 124–131, Online. Association… [find](https://scholar.google.com/scholar?q=Rowan+Hall+Maudslay+and+Ryan+Cotterell%2E+2021%2E+Do+syntactic+probes+probe+syntax%3F+experiments+with+jabberwocky+probing%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Associa)
40. Alessio Miaschi, Dominique Brunato, Felice Dell’Orletta, and Giulia Venturi. 2021. What makes my model perplexed? a linguistic investigation on neural language models perplexity. In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Le… [find](https://scholar.google.com/scholar?q=Alessio+Miaschi%2C+Dominique+Brunato%2C+Felice+Dell%E2%80%99Orletta%2C+and+Giulia+Venturi%2E+2021%2E+What+makes+my+model+perplexed%3F+a+linguistic+investigation+on+neural+language+models+perplexity%2E+In+Proceedings+of+Dee)
41. Max Müller-Eberstein, Rob van der Goot, and Barbara Plank. 2022. Probing for labeled dependency trees. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7711–7726, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Max+M%C3%BCller%2DEberstein%2C+Rob+van+der+Goot%2C+and+Barbara+Plank%2E+2022%2E+Probing+for+labeled+dependency+trees%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
42. Attila Nagy, Dorina Lakatos, Botond Barta, and Judit Ács. 2023. TreeSwap: Data augmentation for machine translation via dependency subtree swapping. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 759–768, Varna, Bulgaria. INCOMA Ltd., Sho… [find](https://scholar.google.com/scholar?q=Attila+Nagy%2C+Dorina+Lakatos%2C+Botond+Barta%2C+and+Judit+%C3%81cs%2E+2023%2E+TreeSwap%3A+Data+augmentation+for+machine+translation+via+dependency+subtree+swapping%2E+In+Proceedings+of+the+14th+International+Conference)
43. Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. 2021. Refining targeted syntactic evaluation of language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3710–3723, Online. As… [find](https://scholar.google.com/scholar?q=Benjamin+Newman%2C+Kai%2DSiang+Ang%2C+Julia+Gong%2C+and+John+Hewitt%2E+2021%2E+Refining+targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the)
44. Isabel Papadimitriou, Richard Futrell, and Kyle Mahowald. 2022. When classifying grammatical role, BERT doesn’t care about word order... except when it matters. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 636–643, Dublin,… [find](https://scholar.google.com/scholar?q=Isabel+Papadimitriou%2C+Richard+Futrell%2C+and+Kyle+Mahowald%2E+2022%2E+When+classifying+grammatical+role%2C+BERT+doesn%E2%80%99t+care+about+word+order%2E%2E%2E+except+when+it+matters%2E+In+Proceedings+of+the+60th+Annual+Meeti)
45. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2E)
46. Shauli Ravfogel, Yanai Elazar, Jacob Goldberger, and Yoav Goldberg. 2020. Unsupervised distillation of syntactic information from contextualized word representations. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 91–106, Online. Associa… [find](https://scholar.google.com/scholar?q=Shauli+Ravfogel%2C+Yanai+Elazar%2C+Jacob+Goldberger%2C+and+Yoav+Goldberg%2E+2020%2E+Unsupervised+distillation+of+syntactic+information+from+contextualized+word+representations%2E+In+Proceedings+of+the+Third+Black)
47. Benoît Sagot. 2018. A multilingual collection of CoNLL-U-compatible morphological lexicons. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). synthesizing new languages. Transacti… [find](https://scholar.google.com/scholar?q=Beno%C3%AEt+Sagot%2E+2018%2E+A+multilingual+collection+of+CoNLL%2DU%2Dcompatible+morphological+lexicons%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Evaluation+%28LREC+2018%29%2C+Mi)
48. Gözde Gül Şahin and Mark Steedman. 2018. Data augmentation via dependency tree morphing for low-resource languages. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5004–5009, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=G%C3%B6zde+G%C3%BCl+S%CC%A7ahin+and+Mark+Steedman%2E+2018%2E+Data+augmentation+via+dependency+tree+morphing+for+low%2Dresource+languages%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Proc) [our card](https://md.research.elamur.ai/free/papers/ahin-2018-data-augmentation-dependency-tree-morphing/)
49. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: A benchmark of linguistic minimal pairs for English. In Proceedings of the Society for Computation in Linguistics 2020, pages 409–410, New York, New York. Association for Computati… [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Alicia+Parrish%2C+Haokun+Liu%2C+Anhad+Mohananey%2C+Wei+Peng%2C+Sheng%2DFu+Wang%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+BLiMP%3A+A+benchmark+of+linguistic+minimal+pairs+for+English%2E+In+Proceedings+of+the+Socie)
50. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Julian+Salazar%2C+Davis+Liang%2C+Toan+Q%2E+Nguyen%2C+and+Katrin+Kirchhoff%2E+2020%2E+Masked+language+model+scoring%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+page) [our card](https://md.research.elamur.ai/free/papers/salazar-2020-masked-language-model-scoring/)
51. Tatu Ylonen. 2022. Wiktextract: Wiktionary as machine-readable structured data. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1317–1325, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Tatu+Ylonen%2E+2022%2E+Wiktextract%3A+Wiktionary+as+machine%2Dreadable+structured+data%2E+In+Proceedings+of+the+Thirteenth+Language+Resources+and+Evaluation+Conference%2C+pages+1317%E2%80%931325%2C+Marseille%2C+France%2E+Europ)
52. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, and Tatiana Shavrina. 2024. mGPT: Few-Shot Learners Go Multilingual. Transactions of the Association for Computational Linguistics, 12:58–79. [find](https://scholar.google.com/scholar?q=Oleh+Shliazhko%2C+Alena+Fenogenova%2C+Maria+Tikhonova%2C+Anastasia+Kozlova%2C+Vladislav+Mikhailov%2C+and+Tatiana+Shavrina%2E+2024%2E+mGPT%3A+Few%2DShot+Learners+Go+Multilingual%2E+Transactions+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/papers/shliazhko-2022-mgpt-few-shot-learners-go/)
53. Taha Zerrouki. 2010. Pyarabic, an arabic language library for python. [https://pypi.python.org/](https://pypi.python.org/) pypi/pyarabic. [↗](https://pypi.python.org/)