---
title: "It's All in the Heads — paper by Tikhonov and Ryabinin (2021)"
description: "This makes it hard to compare methods across languages. 47 references, cited by 54 papers here."
authors: ["Alexey Tikhonov", "Max Ryabinin"]
year: 2021
venue: "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021"
source: "https://aclanthology.org/2021.findings-acl.310/"
doi: "10.18653/v1/2021.findings-acl.310"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tikhonov-2021-it-s-all-heads-attention/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tikhonov-2021-it-s-all-heads-attention/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning

# It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning

**Authors:** Alexey Tikhonov and [Max Ryabinin](https://md.research.elamur.ai/free/en/authors/max-ryabinin/) · **Year:** 2021 · **Venue:** [Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2021](https://md.research.elamur.ai/free/en/venues/findings/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.findings-acl.310/) · [PDF](https://aclanthology.org/2021.findings-acl.310.pdf) · [DOI 10.18653/v1/2021.findings-acl.310](https://doi.org/10.18653/v1/2021.findings-acl.310) · [OpenAlex](https://openalex.org/W3173316661)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [54](https://md.research.elamur.ai/free/en/papers/tikhonov-2021-it-s-all-heads-attention/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Commonsense reasoning benchmarks such as the Winograd Schema Challenge are mostly in English, and versions in other languages define the task in slightly different ways. This makes it hard to compare methods across languages. Tikhonov and Ryabinin address both issues. They merge previously released Winograd-style collections covering six languages (English, French, Japanese, Russian, Portuguese and Chinese) into XWINO, a benchmark with one strict format: a sentence, a pronoun, and two candidate noun phrases that all appear in the text. Their method is a lightweight supervised baseline. It keeps the pretrained encoder (mBERT or XLM-R) frozen and fits a linear classifier on the self-attention weights linking the pronoun to each candidate. Even in zero-shot transfer to unseen languages, it competes with supervised fine-tuning and unsupervised approaches. Analysis shows that a handful of upper-layer heads carry most of the useful signal, and these heads are the same across languages. This suggests that multilingual encoders contain language-independent reasoning mechanisms, and that unsupervised methods should focus on choosing the right heads.

## Key points
- XWINO unifies Winograd Schema datasets in six languages (en, fr, ja, ru, pt, zh) under a single strict task definition. The data and code are available at github.com/yandex-research/crosslingual_winograd.
- The baselines include Maximum Attention Score (MAS), LM-based answer ranking, [CLS]-based classification (Sakaguchi et al.) and the margin loss of Kocijan et al. The fine-tuning baselines overfit when only a few hundred examples are available.
- Five shared XLM-R heads (about 1.3% of all heads) match or beat the full head set in every language except Chinese. Five random heads do noticeably worse.
- Restricting MAS to the heads chosen by the classifiers largely closes its gap with the supervised method.
- Training accuracy keeps rising as more heads are added, but cross-lingual accuracy peaks at roughly 16 heads.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Research University Higher School of Economics](https://md.research.elamur.ai/free/en/universities/national-research-university-higher-school-of-economics/) — University, Moscow, Russia · 16 works · 39,548 citations
- [Yandex](https://md.research.elamur.ai/free/en/companies/yandex/) — Company, Moscow, Russia · 9 works · 3,341 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (3): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 58 works

## Cite

Cite as: Alexey Tikhonov and Max Ryabinin. 2021. It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tikhonov2021it,
  title = {{It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning}},
  author = {Alexey Tikhonov and Max Ryabinin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.310},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.310}
}
```

- [← previous A Rate–Distortion view of human pragmatic reasoning?](https://md.research.elamur.ai/free/en/papers/zaslavsky-2021-rate-distortion-view-human-pragmatic/)
- [next → HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition](https://md.research.elamur.ai/free/en/papers/chriqui-2021-hebert-hebemo-hebrew-bert-model/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

47 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. [Towards a human-like open-domain chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/).
2. Pascal Amsili and Olga Seminck. 2017. A Google-proof collection of French Winograd schemas. In Proceedings of the 2nd Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2017), pages 24–29, Valencia, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pascal+Amsili+and+Olga+Seminck%2E+2017%2E+A+Google%2Dproof+collection+of+French+Winograd+schemas%2E+In+Proceedings+of+the+2nd+Workshop+on+Coreference+Resolution+Beyond+OntoNotes+%28CORBON+2017%29%2C+pages+24%E2%80%9329%2C+Va)
3. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. [Massively multilingual neural machine translation in the wild: Findings and challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/).
4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473.
5. Timothée Bernard and Ting Han. 2020. Mandarino-grad: A Chinese collection of Winograd schemas. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 21–26, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Timothe%CC%81e+Bernard+and+Ting+Han%2E+2020%2E+Mandarino%2Dgrad%3A+A+Chinese+collection+of+Winograd+schemas%2E+In+Proceedings+of+the+12th+Language+Resources+and+Evaluation+Conference%2C+pages+21%E2%80%9326%2C+Marseille%2C+France%2E)
6. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Ethan A. Chi, John Hewitt, and Christopher D. Manning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564–5577, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ethan+A%2E+Chi%2C+John+Hewitt%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Finding+universal+grammatical+relations+in+multilingual+BERT%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computationa)
8. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D%2E+Manning%2E+2019%2E+What+does+BERT+look+at%3F+an+analysis+of+BERT%E2%80%99s+attention%2E+In+Proceedings+of+the+2019+ACL+Workshop+BlackboxNLP%3A+Analyzing+an)
9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling)
10. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E)
11. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
13. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. [Deberta: Decoding-enhanced bert with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In International Conference on Learning Representations.
14. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
15. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+Xtreme%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
16. Jae-young Jo and Sung-Hyon Myaeng. 2020. Roles and utilization of attention heads in transformer-based neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3404–3417, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jae%2Dyoung+Jo+and+Sung%2DHyon+Myaeng%2E+2020%2E+Roles+and+utilization+of+attention+heads+in+transformer%2Dbased+neural+language+models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computat)
17. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/).
18. Taeuk Kim, Bowen Li, and Sang goo Lee. 2020. Chart-based zero-shot constituency parsing on multiple languages. [find](https://scholar.google.com/scholar?q=Taeuk+Kim%2C+Bowen+Li%2C+and+Sang+goo+Lee%2E+2020%2E+Chart%2Dbased+zero%2Dshot+constituency+parsing+on+multiple+languages%2E)
19. Tassilo Klein and Moin Nabi. 2019. Attention is (not) all you need for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4831– 4836, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tassilo+Klein+and+Moin+Nabi%2E+2019%2E+Attention+is+%28not%29+all+you+need+for+commonsense+reasoning%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4831%E2%80%93+48)
20. Tassilo Klein and Moin Nabi. 2020. Contrastive self-supervised learning for commonsense reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7517– 7523, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tassilo+Klein+and+Moin+Nabi%2E+2020%2E+Contrastive+self%2Dsupervised+learning+for+commonsense+reasoning%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+751)
21. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for the Winograd schema challenge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4837–4842, Florence, Italy. Association fo… [find](https://scholar.google.com/scholar?q=Vid+Kocijan%2C+Ana%2DMaria+Cretu%2C+Oana%2DMaria+Camburu%2C+Yordan+Yordanov%2C+and+Thomas+Lukasiewicz%2E+2019%2E+A+surprisingly+robust+trick+for+the+Winograd+schema+challenge%2E+In+Proceedings+of+the+57th+Annual+Meetin)
22. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E+Citeseer%2E)
23. Bowen Li, Taeuk Kim, Reinald Kim Amplayo, and Frank Keller. 2020. Heads-up! unsupervised constituency parsing via self-attention heads. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Nat… [find](https://scholar.google.com/scholar?q=Bowen+Li%2C+Taeuk+Kim%2C+Reinald+Kim+Amplayo%2C+and+Frank+Keller%2E+2020%2E+Heads%2Dup%21+unsupervised+constituency+parsing+via+self%2Dattention+heads%2E+In+Proceedings+of+the+1st+Conference+of+the+Asia%2DPacific+Chapter)
24. Haokun Liu, William Huang, Dhara Mungra, and Samuel R. Bowman. 2020. Precise task formalization matters in Winograd schema evaluations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8275–8280, Online. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Haokun+Liu%2C+William+Huang%2C+Dhara+Mungra%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+Precise+task+formalization+matters+in+Winograd+schema+evaluations%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Na)
25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E)
26. Gabriela Melo, Vinicius Imaizumi, and Fábio Cozman. 2019. Winograd schemas in portuguese. In Anais do XVI Encontro Nacional de Inteligência Artificial e Computacional, pages 787–798. SBC. [find](https://scholar.google.com/scholar?q=Gabriela+Melo%2C+Vinicius+Imaizumi%2C+and+Fa%CC%81bio+Cozman%2E+2019%2E+Winograd+schemas+in+portuguese%2E+In+Anais+do+XVI+Encontro+Nacional+de+Intelige%CC%82ncia+Artificial+e+Computacional%2C+pages+787%E2%80%93798%2E+SBC%2E)
27. Leora Morgenstern, Ernest Davis, and Charles Ortiz. 2016. Planning, executing, and evaluating the winograd schema challenge. AI Magazine, 37:50–54. [find](https://scholar.google.com/scholar?q=Leora+Morgenstern%2C+Ernest+Davis%2C+and+Charles+Ortiz%2E+2016%2E+Planning%2C+executing%2C+and+evaluating+the+winograd+schema+challenge%2E+AI+Magazine%2C+37%3A50%E2%80%9354%2E)
28. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc)
29. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. [How multilingual is multilingual BERT?](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics.
30. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. [XCOPA: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Associatio…
31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/).
32. Alessandro Raganato and Jörg Tiedemann. 2018. [An analysis of encoder representations in transformer-based machine translation](https://md.research.elamur.ai/free/en/papers/raganato-2018-analysis-encoder-representations-transformer-based/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287–297, Brussels, Belgium. Association for Computational L…
33. Altaf Rahman and Vincent Ng. 2012. [Resolving complex cases of definite pronouns: The Winograd schema challenge](https://md.research.elamur.ai/free/en/papers/rahman-2012-resolving-complex-cases-definite-pronouns/). In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789, Jeju Island, Korea. Association…
34. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E)
35. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston. 2020. [Recipes for building an open-domain chatbot](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/).
36. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569–631. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Ivan+Vulic%CC%81%2C+and+Anders+S%C3%B8gaard%2E+2019%2E+A+survey+of+cross%2Dlingual+word+embedding+models%2E+Journal+of+Artificial+Intelligence+Research%2C+65%3A569%E2%80%93631%2E)
37. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732–8740. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligenc)
38. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. [Masked language model scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics.
39. Tatiana Shavrina, Alena Fenogenova, Emelyanov Anton, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020. RussianSuperGLUE: A Russian language understanding evaluation benchmark. In Proceedings of the 2020 Conference o… [find](https://scholar.google.com/scholar?q=Tatiana+Shavrina%2C+Alena+Fenogenova%2C+Emelyanov+Anton%2C+Denis+Shevelev%2C+Ekaterina+Artemova%2C+Valentin+Malykh%2C+Vladislav+Mikhailov%2C+Maria+Tikhonova%2C+Andrey+Chertok%2C+and+Andrey+Evlampiev%2E+2020%2E+RussianSuper)
40. Tomohide Shibata, Shotaro Obama, and Sadao Kurohashi. 2015. Building and analyzing the winograd schema challenge. Proceedings of the Twenty-first Annual Meeting of the Association for Natural Language Processing, 57(1):22–23. [find](https://scholar.google.com/scholar?q=Tomohide+Shibata%2C+Shotaro+Obama%2C+and+Sadao+Kurohashi%2E+2015%2E+Building+and+analyzing+the+winograd+schema+challenge%2E+Proceedings+of+the+Twenty%2Dfirst+Annual+Meeting+of+the+Association+for+Natural+Language)
41. Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, and Yonghui Wu. 2020. [Leveraging monolingual data with self-supervision for multilingual neural machine translation](https://md.research.elamur.ai/free/en/papers/siddhant-2020-leveraging-monolingual-data-self-supervision/). In Proceedings of the 58th Annual Meeting of the Association for Computational Ling…
42. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [BERT rediscovers the classical NLP pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics.
43. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. [find](https://scholar.google.com/scholar?q=Trieu+H%2E+Trinh+and+Quoc+V%2E+Le%2E+2018%2E+A+simple+method+for+commonsense+reasoning%2E)
44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc.
45. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. [Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, I…
46. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/).
47. Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019b. [Cross-lingual BERT transformation for zero-shot dependency parsing](https://md.research.elamur.ai/free/en/papers/wang-2019-cross-lingual-bert-transformation-zero/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin…

*Updated 24 October 2026*