---
title: "All Bark and No Bite — paper by Timkey and Schijndel (2021)"
description: "Researchers often use cosine similarity and Euclidean distance to probe contextual embeddings from… 43 references, cited by 74 papers here."
authors: ["William Timkey", "Marten van Schijndel"]
year: 2021
venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2021.emnlp-main.372/"
doi: "10.18653/v1/2021.emnlp-main.372"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/timkey-2021-all-bark-no-bite-rogue/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/timkey-2021-all-bark-no-bite-rogue/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality

# All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality

**Authors:** William Timkey and Marten van Schijndel · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.372/) · [PDF](https://aclanthology.org/2021.emnlp-main.372.pdf) · [DOI 10.18653/v1/2021.emnlp-main.372](https://doi.org/10.18653/v1/2021.emnlp-main.372) · [OpenAlex](https://openalex.org/W3196813608)

**Citations:** 65 citations (OpenAlex, 2026-10-14) · [74](https://md.research.elamur.ai/free/en/papers/timkey-2021-all-bark-no-bite-rogue/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Researchers often use cosine similarity and Euclidean distance to probe contextual embeddings from transformer language models. These probes have produced puzzling findings: the spaces look highly anisotropic, self-similarity is low, and later layers seem to lose semantic quality. Timkey and van Schijndel trace these findings to a measurement artifact. They break cosine similarity down into per-dimension contributions and apply this to every layer of BERT, RoBERTa, GPT-2 and XLNet, using text sampled from English Wikipedia. Static Word2Vec and GloVe vectors serve as baselines. In the transformers, a handful of "rogue" dimensions (often only one to three) account for most of the similarity score. These dimensions have very large means and high variance. They track absolute position and punctuation rather than whatever drives model behavior. Standardizing each dimension removes their outsized influence. After this step, cosine similarity agrees better with human word-similarity judgments, and representational quality no longer appears to collapse in later layers. The authors argue that any similarity-based interpretability study of these models should correct for rogue dimensions.

## Key points
- **Setup:** an 85k-token random sample of English Wikipedia was split into 128-token inputs, and 500k random token pairs were drawn from it. All four transformers have 768 dimensions and 12 layers; Word2Vec and GloVe (300d) serve as controls.
- **XLNet extreme case:** in layers 10 and 11 of XLNet, one dimension supplies over 99% of the expected cosine similarity between random tokens. In the final layer, dimension 667 averages about 180.0, while the other dimensions average roughly −0.084 (SD 0.77).
- **Static baselines:** Word2Vec and GloVe remain nearly isotropic, and no individual dimension controls their similarity scores.
- **Behavioral mismatch:** rogue dimensions can make the similarity measures depend on under 1% of the embedding space, yet they are not the dimensions that matter most for model behavior.
- **Fix and code:** standardization is enough to reveal the underlying representational quality, measured as Spearman's ρ against human similarity datasets. Code is available at github.com/wtimkey/rogue-dimensions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 89 works · 58,422 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: William Timkey and Marten van Schijndel. 2021. All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{timkey2021all,
  title = {{All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality}},
  author = {William Timkey and Marten {van Schijndel}},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.372},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.372}
}
```

- [← previous TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation](https://md.research.elamur.ai/free/en/papers/uchendu-2021-turingbench-benchmark-environment-turing-test/)
- [next → Bot-Adversarial Dialogue for Safe Conversational Agents](https://md.research.elamur.ai/free/en/papers/xu-2021-bot-adversarial-dialogue-safe-conversational/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

43 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. In this work, we showed that similarity measures in contextual language models are largely reflective of a small number of rogue dimensions, not the entire embedding space. Consequently, a few dimensions can drastically change the conclusions we draw about the linguistic phenomena a model References… [find](https://scholar.google.com/scholar?q=In+this+work%2C+we+showed+that+similarity+measures+in+contextual+language+models+are+largely+reflective+of+a+small+number+of+rogue+dimensions%2C+not+the+entire+embedding+space%2E+Consequently%2C+a+few+dimensi)
2. Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Willem Zuidema. 2019. Blackbox meets blackbox: Representational similarity & stability analysis of neural language models and brains. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 191–… [find](https://scholar.google.com/scholar?q=Samira+Abnar%2C+Lisa+Beinborn%2C+Rochelle+Choenni%2C+and+Willem+Zuidema%2E+2019%2E+Blackbox+meets+blackbox%3A+Representational+similarity+%26+stability+analysis+of+neural+language+models+and+brains%2E+In+Proceedings+)
3. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. [A study on similarity and relatedness using distributional and WordNet-based approaches](https://md.research.elamur.ai/free/en/papers/agirre-2009-study-similarity-relatedness-distributional-wordnet/). In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the A…
4. Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1–46. [find](https://scholar.google.com/scholar?q=Sebastian+Bach%2C+Alexander+Binder%2C+Gr%C3%A9goire+Montavon%2C+Frederick+Klauschen%2C+Klaus%2DRobert+M%C3%BCller%2C+and+Wojciech+Samek%2E+2015%2E+On+pixel%2Dwise+explanations+for+non%2Dlinear+classifier+decisions+by+layer%2Dwise+re)
5. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. [Don’t count, predict!](https://md.research.elamur.ai/free/en/papers/baroni-2014-don-t-count-predict-systematic/) a systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 238–247, Bal…
6. Daniel Biś, Maksim Podkorytov, and Xiuwen Liu. 2021. Too much in common: Shifting of embeddings in transformer language models and its implications. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag… [find](https://scholar.google.com/scholar?q=Daniel+Bis%CC%81%2C+Maksim+Podkorytov%2C+and+Xiuwen+Liu%2E+2021%2E+Too+much+in+common%3A+Shifting+of+embeddings+in+transformer+language+models+and+its+implications%2E+In+Proceedings+of+the+2021+Conference+of+the+North)
7. Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. [Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings](https://md.research.elamur.ai/free/en/papers/bommasani-2020-interpreting-pretrained-contextualized-representations-reductions/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4758– 4781, Online. Association for Computational Lingu…
8. Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xingyu+Cai%2C+Jiaji+Huang%2C+Yuchen+Bian%2C+and+Kenneth+Church%2E+2021%2E+Isotropy+in+the+contextual+embedding+space%3A+Clusters+and+manifolds%2E+In+International+Conference+on+Learning+Representations%2E)
9. Gabriella Chronis and Katrin Erk. 2020. When is a bishop not like a rook? when it’s like a rabbi! multi-prototype BERT embeddings for estimating semantic relationships. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 227–244, Online. Association for Computatio… [find](https://scholar.google.com/scholar?q=Gabriella+Chronis+and+Katrin+Erk%2E+2020%2E+When+is+a+bishop+not+like+a+rook%3F+when+it%E2%80%99s+like+a+rabbi%21+multi%2Dprototype+BERT+embeddings+for+estimating+semantic+relationships%2E+In+Proceedings+of+the+24th+Conf)
10. Grzegorz Chrupała and Afra Alishahi. 2019. Correlating neural and symbolic representations of language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2952–2962, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Grzegorz+Chrupa%C5%82a+and+Afra+Alishahi%2E+2019%2E+Correlating+neural+and+symbolic+representations+of+language%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+page)
11. Forrest Davis and Marten van Schijndel. 2020. Discourse structure interacts with reference but not syntax in neural language models. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 396–407, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Forrest+Davis+and+Marten+van+Schijndel%2E+2020%2E+Discourse+structure+interacts+with+reference+but+not+syntax+in+neural+language+models%2E+In+Proceedings+of+the+24th+Conference+on+Computational+Natural+Lang)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
13. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2019%2E+How+contextual+are+contextualized+word+representations%3F+Comparing+the+geometry+of+BERT%2C+ELMo%2C+and+GPT%2D2+embeddings%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+i)
14. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. [Representation degeneration problem in training natural language generation models](https://md.research.elamur.ai/free/en/papers/gao-2019-representation-degeneration-problem-training-natural/). In International Conference on Learning Representations.
15. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021. [Probing for idiomaticity in vector space models](https://md.research.elamur.ai/free/en/papers/garcia-2021-probing-idiomaticity-vector-space-models/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551–3564, Online. As…
16. Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. [SimVerb-3500: A large-scale evaluation set of verb similarity](https://md.research.elamur.ai/free/en/papers/gerz-2016-simverb-3500-large-scale-evaluation/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2173–2182, Austin, Texas. Association for Computational Lingui…
17. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. [SimLex-999: Evaluating semantic models with (genuine) similarity estimation](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/). Computational Linguistics, 41(4):665–695.
18. Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. 2021. [BERT busters: Outlier dimensions that disrupt transformers](https://md.research.elamur.ai/free/en/papers/kovaleva-2021-bert-busters-outlier-dimensions-that/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3392–3405, Online. Association for Computational Linguistics.
19. Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. [On the sentence embeddings from pre-trained language models](https://md.research.elamur.ai/free/en/papers/li-2020-sentence-embeddings-pre-trained-language/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119–9130, Online. Association for Computational Linguis…
20. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. [Linguistic knowledge and transferability of contextual representations](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech…
21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Roberta: A robustly optimized bert pretraining approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E)
22. Ziyang Luo, Artur Kulmizev, and Xiaoxi Mao. 2021. Positional artefacts propagate through masked language model embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:… [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Artur+Kulmizev%2C+and+Xiaoxi+Mao%2E+2021%2E+Positional+artefacts+propagate+through+masked+language+model+embeddings%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computationa)
23. Matej Martinc, Petra Kralj Novak, and Senja Pollak. 2020. Leveraging contextual embeddings for detecting diachronic semantic shift. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4811–4819, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Matej+Martinc%2C+Petra+Kralj+Novak%2C+and+Senja+Pollak%2E+2020%2E+Leveraging+contextual+embeddings+for+detecting+diachronic+semantic+shift%2E+In+Proceedings+of+the+12th+Language+Resources+and+Evaluation+Confere)
24. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. [On measuring social biases in sentence encoders](https://md.research.elamur.ai/free/en/papers/may-2019-measuring-social-biases-sentence-encoders/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and…
25. Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. [What happens to BERT embeddings during fine-tuning?](https://md.research.elamur.ai/free/en/papers/merchant-2020-what-happens-bert-embeddings-during/) In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 33–44, Online. Association for Computational Linguistics.
26. Timothee Mickus, Denis Paperno, Mathieu Constant, and Kees van Deemter. 2020. What do you mean, BERT? In Proceedings of the Society for Computation in Linguistics 2020, pages 279–290, New York, New York. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Timothee+Mickus%2C+Denis+Paperno%2C+Mathieu+Constant%2C+and+Kees+van+Deemter%2E+2020%2E+What+do+you+mean%2C+BERT%3F+In+Proceedings+of+the+Society+for+Computation+in+Linguistics+2020%2C+pages+279%E2%80%93290%2C+New+York%2C+New+Yo)
27. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
28. Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. 2018. On the importance of single directions for generalization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ari+S%2E+Morcos%2C+David+G%2ET%2E+Barrett%2C+Neil+C%2E+Rabinowitz%2C+and+Matthew+Botvinick%2E+2018%2E+On+the+importance+of+single+directions+for+generalization%2E+In+International+Conference+on+Learning+Representations%2E)
29. Jiaqi Mu and Pramod Viswanath. 2018. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jiaqi+Mu+and+Pramod+Viswanath%2E+2018%2E+All%2Dbut%2Dthe%2Dtop%3A+Simple+and+effective+postprocessing+for+word+representations%2E+In+International+Conference+on+Learning+Representations%2E)
30. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [GloVe: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
31. A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+Jeffrey+Wu%2C+R%2E+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
32. Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. [Visualizing and measuring the geometry of bert](https://md.research.elamur.ai/free/en/papers/coenen-2019-visualizing-measuring-geometry-bert/). In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
33. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Commun. ACM, 8(10):627–633. [find](https://scholar.google.com/scholar?q=Herbert+Rubenstein+and+John+B%2E+Goodenough%2E+1965%2E+Contextual+correlates+of+synonymy%2E+Commun%2E+ACM%2C+8%2810%29%3A627%E2%80%93633%2E)
34. Tobias Schnabel, Igor Labutov, David Mimno, and Thorsten Joachims. 2015. [Evaluation methods for unsupervised word embeddings](https://md.research.elamur.ai/free/en/papers/schnabel-2015-evaluation-methods-unsupervised-word-embeddings/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 298–307, Lisbon, Portugal. Association for Computational Linguistics.
35. Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, Ian Tenney, and Ellie Pavlick. 2021. The multiberts: Bert reproductions for robustness analysis. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Steve+Yadlowsky%2C+Jason+Wei%2C+Naomi+Saphra%2C+Alexander+D%E2%80%99Amour%2C+Tal+Linzen%2C+Jasmijn+Bastings%2C+Iulia+Turc%2C+Jacob+Eisenstein%2C+Dipanjan+Das%2C+Ian+Tenney%2C+and+Ellie+Pavlick%2E+2021%2E+The+multibe)
36. Aina Garí Soler and Marianna Apidianaki. 2021. Let’s play mono-poly: Bert can reveal words’ polysemy level and partitionability into senses. [find](https://scholar.google.com/scholar?q=Aina+Gar%C3%AD+Soler+and+Marianna+Apidianaki%2E+2021%2E+Let%E2%80%99s+play+mono%2Dpoly%3A+Bert+can+reveal+words%E2%80%99+polysemy+level+and+partitionability+into+senses%2E)
37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). CoRR, abs/1706.03762.
38. Elena Voita, Rico Sennrich, and Ivan Titov. 2020. Analyzing the source and target contributions to predictions in neural machine translation. [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2020%2E+Analyzing+the+source+and+target+contributions+to+predictions+in+neural+machine+translation%2E)
39. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. [Probing pretrained language models for lexical semantics](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computationa…
40. Bin Wang, Angela Wang, Fenxiao Chen, Yuncheng Wang, and C.-C. Jay Kuo. 2019. Evaluating word embedding models: methods and experimental results. APSIPA Transactions on Signal and Information Processing, 8. [find](https://scholar.google.com/scholar?q=Bin+Wang%2C+Angela+Wang%2C+Fenxiao+Chen%2C+Yuncheng+Wang%2C+and+C%2E%2DC%2E+Jay+Kuo%2E+2019%2E+Evaluating+word+embedding+models%3A+methods+and+experimental+results%2E+APSIPA+Transactions+on+Signal+and+Information+Processin)
41. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
42. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
43. Vitalii Zhelezniak, Aleksandar Savkov, April Shen, and Nils Hammerla. 2019. Correlation coefficients and semantic textual similarity. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… [find](https://scholar.google.com/scholar?q=Vitalii+Zhelezniak%2C+Aleksandar+Savkov%2C+April+Shen%2C+and+Nils+Hammerla%2E+2019%2E+Correlation+coefficients+and+semantic+textual+similarity%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapte)

*Updated 24 October 2026*