---
title: "Interpreting Pretrained Contextualized — Bommasani 2020"
description: "Tools for interpreting static word vectors are mature, yet they are rarely used to study contextual encoders such as BERT."
authors: ["Rishi Bommasani", "Kelly Davis", "Claire Cardie"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.431/"
doi: "10.18653/v1/2020.acl-main.431"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bommasani-2020-interpreting-pretrained-contextualized-representations-reductions/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bommasani-2020-interpreting-pretrained-contextualized-representations-reductions/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings

# Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings

**Authors:** [Rishi Bommasani](https://md.research.elamur.ai/free/en/authors/rishi-bommasani/), [Kelly Davis](https://md.research.elamur.ai/free/en/authors/kelly-davis/), [Claire Cardie](https://md.research.elamur.ai/free/en/authors/claire-cardie/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.431/) · [PDF](https://aclanthology.org/2020.acl-main.431.pdf) · [DOI 10.18653/v1/2020.acl-main.431](https://doi.org/10.18653/v1/2020.acl-main.431) · [OpenAlex](https://openalex.org/W3035102548)

**Citations:** 139 citations (openalex-snapshot · 2026-10-14) · [109](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

Tools for interpreting static word vectors are mature, yet they are rarely used to study contextual encoders such as BERT. The authors close this gap by reducing a contextual model to an ordinary lookup table. Their procedure has no parameters and only assumes that the encoder turns a word sequence into vectors. It works in two steps. First, the subword vectors of a word are merged using min, max, mean or last. Second, the word's vectors from many sampled sentences are combined using min, max or mean. Feeding the word alone serves as a baseline.

They apply this to five architectures and nine checkpoints. The resulting vectors are tested on word similarity and relatedness benchmarks. They are also checked for gender, racial/ethnic and religious bias using several established estimators.

Averaging over many contexts yields the strongest vectors, which clearly beat Word2Vec and GloVe. Bias varies across models and layers, even when models share training data. The bias estimators often contradict one another. The work therefore offers a cheap source of strong static embeddings and a warning about how reliable bias measurement is.

## Key points

- Models covered: BERT, GPT-2, XLNet, RoBERTa and DistilBERT (9 pretrained weight sets in total). The main reported results use bert-base-uncased.
- Evaluation uses RG65, WS353, SimLex-999 and SimVerb-3500: 4917 word pairs over 2005 unique words, scored by Spearman ρ on cosine similarities.
- Mean pooling works best at both the subword level and the context level, and it beats the isolated-word (Decontextualized) setup by a wide margin. Quality rises steadily as the number of sampled contexts grows from 10K to 1M.
- BERT-24 with 1M contexts reaches 0.7768 on RG65, 0.7301 on WS353, 0.5244 on SimLex-999 and 0.4280 on SimVerb-3500. Word2Vec scores 0.6787, 0.6838, 0.4420 and 0.3636 on the same sets. GloVe trained on 6B tokens does even worse, although BERT saw only about 3.3B tokens.
- Bias was measured with the methods of Bolukbasi et al., Garg et al. and Manzini et al. All data and code are publicly released.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

109 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 104 not yet.

- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [Debiasing Pre-trained Contextualised Embeddings](https://md.research.elamur.ai/free/en/papers/kaneko-2021-debiasing-pre-trained-contextualised-embeddings/)
- [One Word Is Not Enough: Simple Prompts Improve Word Embeddings](https://md.research.elamur.ai/free/en/papers/ranjan-2026-one-word-is-not-enough/)
- [From sunblock to softblock: Analyzing the correlates of neology in published writing and on social media](https://md.research.elamur.ai/free/en/papers/ryskina-2026-sunblock-softblock-analyzing-correlates-neology/)
- [Lost in Translation: Repurposing semantic similarity benchmarks for evaluating lexical-semantic consistency in LLM-based machine translation](https://md.research.elamur.ai/free/en/papers/ye-2026-lost-translation-repurposing-semantic-similarity/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bommasani2020interpreting,
  title = {{Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings}},
  author = {Bommasani, Rishi and Davis, Kelly and Cardie, Claire},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.431},
  url = {https://doi.org/10.18653/v1/2020.acl-main.431}
}
```

- [← previous Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/)
- [next → Interpreting Verbal Irony: Linguistic Strategies and the Connection to the Type of Semantic Incongruity](https://md.research.elamur.ai/free/en/papers/ghosh-2020-interpreting-verbal-irony-linguistic-strategies/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

78 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the A… [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Enrique+Alfonseca%2C+Keith+Hall%2C+Jana+Kravalova%2C+Marius+Pas%CC%A7ca%2C+and+Aitor+Soroa%2E+2009%2E+A+study+on+similarity+and+relatedness+using+distributional+and+WordNet%2Dbased+approaches%2E+In+Proceedin) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2009-study-similarity-relatedness-distributional-wordnet/)
2. Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named 10 [https://github.com/rishibommasani/](https://github.com/rishibommasani/) Contextual2Static entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… [↗](https://github.com/rishibommasani/)
3. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Sanjeev+Arora%2C+Yingyu+Liang%2C+and+Tengyu+Ma%2E+2017%2E+A+simple+but+tough%2Dto%2Dbeat+baseline+for+sentence+embeddings%2E+In+International+Conference+on+Learning+Representations%2E)
4. Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2654–2662. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Jimmy+Ba+and+Rich+Caruana%2E+2014%2E+Do+deep+nets+really+need+to+be+deep%3F+In+Z%2E+Ghahramani%2C+M%2E+Welling%2C+C%2E+Cortes%2C+N%2E+D%2E+Lawrence%2C+and+K%2E+Q%2E+Weinberger%2C+editors%2C+Advances+in+Neural+Information+Processing+)
5. Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: from allocative to representational harms in machine learning. Special Interest Group for Computing, Information and Society (SIGCIS). [find](https://scholar.google.com/scholar?q=Solon+Barocas%2C+Kate+Crawford%2C+Aaron+Shapiro%2C+and+Hanna+Wallach%2E+2017%2E+The+problem+with+bias%3A+from+allocative+to+representational+harms+in+machine+learning%2E+Special+Interest+Group+for+Computing%2C+Inform)
6. Lawrence W Barsalou. 1982. Context-independent and context-dependent information in concepts. Memory & cognition, 10(1):82–93. [find](https://scholar.google.com/scholar?q=Lawrence+W+Barsalou%2E+1982%2E+Context%2Dindependent+and+context%2Ddependent+information+in+concepts%2E+Memory+%26+cognition%2C+10%281%29%3A82%E2%80%9393%2E)
7. Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 33–39, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Christine+Basta%2C+Marta+R%2E+Costa%2Djussa%CC%80%2C+and+Noe+Casas%2E+2019%2E+Evaluating+the+underlying+gender+bias+in+contextualized+word+embeddings%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+in+Natural+Lan) [our card](https://md.research.elamur.ai/free/en/papers/basta-2019-evaluating-underlying-gender-bias-contextualized/)
8. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res., 3:1137–1155. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Re%CC%81jean+Ducharme%2C+Pascal+Vincent%2C+and+Christian+Janvin%2E+2003%2E+A+neural+probabilistic+language+model%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+3%3A1137%E2%80%931155%2E)
9. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. [find](https://scholar.google.com/scholar?q=Piotr+Bojanowski%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2017%2E+Enriching+word+vectors+with+subword+information%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+5%3A135%E2%80%93146%2E)
10. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems… [find](https://scholar.google.com/scholar?q=Tolga+Bolukbasi%2C+Kai%2DWei+Chang%2C+James+Y+Zou%2C+Venkatesh+Saligrama%2C+and+Adam+T+Kalai%2E+2016%2E+Man+is+to+computer+programmer+as+woman+is+to+homemaker%3F+debiasing+word+embeddings%2E+In+D%2E+D%2E+Lee%2C+M%2E+Sugiyama%2C+) [our card](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/)
11. Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. 2016. An analysis of deep neural network models for practical applications. CoRR, abs/1605.07678. — [arXiv:1605.07678](https://arxiv.org/abs/1605.07678) [find](https://scholar.google.com/scholar?q=Alfredo+Canziani%2C+Adam+Paszke%2C+and+Eugenio+Culurciello%2E+2016%2E+An+analysis+of+deep+neural+network+models+for+practical+applications%2E+CoRR%2C+abs%2F1605%2E07678%2E+%E2%80%94+%5BarXiv%3A1605%2E07678%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F160)
12. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D%2E+Manning%2E+2019%2E+What+does+BERT+look+at%3F+an+analysis+of+BERT%E2%80%99s+attention%2E+In+Proceedings+of+the+2019+ACL+Workshop+BlackboxNLP%3A+Analyzing+an)
13. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, pages 160–167, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+2008%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+Proceedings+of+the+25th+International+Conference+on+Mac)
14. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12:2493–2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+12%3A2493%E2%80%932537%2E) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
15. Ryan Cotterell and Hinrich Schütze. 2015. Morphological word-embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1287–1292, Denver, Colorado. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ryan+Cotterell+and+Hinrich+Schu%CC%88tze%2E+2015%2E+Morphological+word%2Dembeddings%2E+In+Proceedings+of+the+2015+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+La)
16. Ilse Depraetere. 2014. Modals and lexically-regulated saturation. Journal of Pragmatics, 71:160–177. [find](https://scholar.google.com/scholar?q=Ilse+Depraetere%2E+2014%2E+Modals+and+lexically%2Dregulated+saturation%2E+Journal+of+Pragmatics%2C+71%3A160%E2%80%93177%2E)
17. Ilse Depraetere. 2019. Meaning in context and contextual meaning: A perspective on the semantics-pragmatics interface applied to modal verbs. Anglophonia, 28. [find](https://scholar.google.com/scholar?q=Ilse+Depraetere%2E+2019%2E+Meaning+in+context+and+contextual+meaning%3A+A+perspective+on+the+semantics%2Dpragmatics+interface+applied+to+modal+verbs%2E+Anglophonia%2C+28%2E)
18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
19. Rishi Bommasani, Arzoo Katiyar, and Claire Cardie. 2019. SPARSE: Structured prediction using argument-relative structured encoding. In Proceedings of the Third Workshop on Structured Prediction for NLP, pages 13–17, Minneapolis, Minnesota. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rishi+Bommasani%2C+Arzoo+Katiyar%2C+and+Claire+Cardie%2E+2019%2E+SPARSE%3A+Structured+prediction+using+argument%2Drelative+structured+encoding%2E+In+Proceedings+of+the+Third+Workshop+on+Structured+Prediction+for+NL)
20. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2019%2E+How+contextual+are+contextualized+word+representations%3F+comparing+the+geometry+of+BERT%2C+ELMo%2C+and+GPT%2D2+embeddings%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+i)
21. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. [find](https://scholar.google.com/scholar?q=Aylin+Caliskan%2C+Joanna+J%2E+Bryson%2C+and+Arvind+Narayanan%2E+2017%2E+Semantics+derived+automatically+from+language+corpora+contain+human%2Dlike+biases%2E+Science%2C+356%286334%29%3A183%E2%80%93186%2E)
22. Allyson Ettinger. 2020. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2E+2020%2E+What+bert+is+not%3A+Lessons+from+a+new+suite+of+psycholinguistic+diagnostics+for+language+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A34%E2%80%9348%2E) [our card](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/)
23. Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. 2016. Problems with evaluation of word embeddings using word similarity tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 30– 35, Berlin, Germany. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Manaal+Faruqui%2C+Yulia+Tsvetkov%2C+Pushpendre+Rastogi%2C+and+Chris+Dyer%2E+2016%2E+Problems+with+evaluation+of+word+embeddings+using+word+similarity+tasks%2E+In+Proceedings+of+the+1st+Workshop+on+Evaluating+Vect) [our card](https://md.research.elamur.ai/free/en/papers/faruqui-2016-problems-evaluation-word-embeddings-word/)
24. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635–E3644. [find](https://scholar.google.com/scholar?q=Nikhil+Garg%2C+Londa+Schiebinger%2C+Dan+Jurafsky%2C+and+James+Zou%2E+2018%2E+Word+embeddings+quantify+100+years+of+gender+and+ethnic+stereotypes%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+115%2816%29%3AE3635%E2%80%93E)
25. Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2173–2182, Austin, Texas. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Daniela+Gerz%2C+Ivan+Vulic%CC%81%2C+Felix+Hill%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2016%2E+SimVerb%2D3500%3A+A+large%2Dscale+evaluation+set+of+verb+similarity%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+) [our card](https://md.research.elamur.ai/free/en/papers/gerz-2016-simverb-3500-large-scale-evaluation/)
26. Anna Gladkova and Aleksandr Drozd. 2016. Intrinsic evaluations of word embeddings: What can we do better? In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 36–42, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anna+Gladkova+and+Aleksandr+Drozd%2E+2016%2E+Intrinsic+evaluations+of+word+embeddings%3A+What+can+we+do+better%3F+In+Proceedings+of+the+1st+Workshop+on+Evaluating+Vector%2DSpace+Representations+for+NLP%2C+pages+3)
27. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol… [find](https://scholar.google.com/scholar?q=Hila+Gonen+and+Yoav+Goldberg%2E+2019%2E+Lipstick+on+a+pig%3A+Debiasing+methods+cover+up+systematic+gender+biases+in+word+embeddings+but+do+not+remove+them%2E+In+Proceedings+of+the+2019+Conference+of+the+North) [our card](https://md.research.elamur.ai/free/en/papers/gonen-2019-lipstick-pig-debiasing-methods-cover/)
28. Souleiman Hasan and Edward Curry. 2017. Word re-embedding via manifold dimensionality retention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 321–326, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Souleiman+Hasan+and+Edward+Curry%2E+2017%2E+Word+re%2Dembedding+via+manifold+dimensionality+retention%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+321%E2%80%9332)
29. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, Chi… [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Percy+Liang%2E+2019%2E+Designing+and+interpreting+probes+with+control+tasks%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Internati)
30. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129… [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Christopher+D%2E+Manning%2E+2019%2E+A+structural+probe+for+finding+syntax+in+word+representations%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/)
31. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2015%2E+SimLex%2D999%3A+Evaluating+semantic+models+with+%28genuine%29+similarity+estimation%2E+Computational+Linguistics%2C+41%284%29%3A665%E2%80%93695%2E) [our card](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/)
32. Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeffrey+Dean%2E+2015%2E+Distilling+the+knowledge+in+a+neural+network%2E+In+NIPS+Deep+Learning+and+Representation+Learning+Workshop%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
33. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166–172, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Keita+Kurita%2C+Nidhi+Vyas%2C+Ayush+Pareek%2C+Alan+W+Black%2C+and+Yulia+Tsvetkov%2E+2019%2E+Measuring+bias+in+contextualized+word+representations%2E+In+Proceedings+of+the+First+Workshop+on+Gender+Bias+in+Natural+La) [our card](https://md.research.elamur.ai/free/en/papers/kurita-2019-measuring-bias-contextualized-word-representations/)
34. Omer Levy and Yoav Goldberg. 2014a. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302–308, Baltimore, Maryland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+2014a%2E+Dependency%2Dbased+word+embeddings%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short+Papers%29%2C+pages+302%E2%80%93308)
35. Omer Levy and Yoav Goldberg. 2014b. Linguistic regularities in sparse and explicit word representations. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 171–180, Ann Arbor, Michigan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+2014b%2E+Linguistic+regularities+in+sparse+and+explicit+word+representations%2E+In+Proceedings+of+the+Eighteenth+Conference+on+Computational+Natural+Language+Learning%2C+pages+1) [our card](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/)
36. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211–225. [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Yoav+Goldberg%2C+and+Ido+Dagan%2E+2015%2E+Improving+distributional+similarity+with+lessons+learned+from+word+embeddings%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+3%3A211%E2%80%93225%2E)
37. Bofang Li, Tao Liu, Zhe Zhao, Buzhou Tang, Aleksandr Drozd, Anna Rogers, and Xiaoyong Du. 2017. Investigating different syntactic context types and context representations for learning word embeddings. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2… [find](https://scholar.google.com/scholar?q=Bofang+Li%2C+Tao+Liu%2C+Zhe+Zhao%2C+Buzhou+Tang%2C+Aleksandr+Drozd%2C+Anna+Rogers%2C+and+Xiaoyong+Du%2E+2017%2E+Investigating+different+syntactic+context+types+and+context+representations+for+learning+word+embeddings)
38. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec… [find](https://scholar.google.com/scholar?q=Nelson+F%2E+Liu%2C+Matt+Gardner%2C+Yonatan+Belinkov%2C+Matthew+E%2E+Peters%2C+and+Noah+A%2E+Smith%2E+2019a%2E+Linguistic+knowledge+and+transferability+of+contextual+representations%2E+In+Proceedings+of+the+2019+Conferenc) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/)
39. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019b%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+)
40. Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics - Volume 1, ETMTNLP ’02, pages 63–70, Stroudsburg, PA, USA. Association for Comput… [find](https://scholar.google.com/scholar?q=Edward+Loper+and+Steven+Bird%2E+2002%2E+Nltk%3A+The+natural+language+toolkit%2E+In+Proceedings+of+the+ACL%2D02+Workshop+on+Effective+Tools+and+Methodologies+for+Teaching+Natural+Language+Processing+and+Computat)
41. Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Huma… [find](https://scholar.google.com/scholar?q=Thomas+Manzini%2C+Lim+Yao+Chong%2C+Alan+W+Black%2C+and+Yulia+Tsvetkov%2E+2019%2E+Black+is+to+criminal+as+caucasian+is+to+police%3A+Detecting+and+removing+multiclass+bias+in+word+embeddings%2E+In+Proceedings+of+the+) [our card](https://md.research.elamur.ai/free/en/papers/manzini-2019-black-is-criminal-as-caucasian/)
42. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… [find](https://scholar.google.com/scholar?q=Chandler+May%2C+Alex+Wang%2C+Shikha+Bordia%2C+Samuel+R%2E+Bowman%2C+and+Rachel+Rudinger%2E+2019%2E+On+measuring+social+biases+in+sentence+encoders%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapte) [our card](https://md.research.elamur.ai/free/en/papers/may-2019-measuring-social-biases-sentence-encoders/)
43. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 14014– 14024. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Paul+Michel%2C+Omer+Levy%2C+and+Graham+Neubig%2E+2019%2E+Are+sixteen+heads+really+better+than+one%3F+In+H%2E+Wallach%2C+H%2E+Larochelle%2C+A%2E+Beygelzimer%2C+F%2E+dAlche%CC%81%2DBuc%2C+E%2E+Fox%2C+and+R%2E+Garnett%2C+editors%2C+Advances+in+Ne) [our card](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/)
44. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+neural+information+processing+)
45. George A Miller and Walter G Charles. 1991. Contextual correlates of semantic similarity. Language and cognitive processes, 6(1):1–28. [find](https://scholar.google.com/scholar?q=George+A+Miller+and+Walter+G+Charles%2E+1991%2E+Contextual+correlates+of+semantic+similarity%2E+Language+and+cognitive+processes%2C+6%281%29%3A1%E2%80%9328%2E)
46. Jiaqi Mu and Pramod Viswanath. 2018. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jiaqi+Mu+and+Pramod+Viswanath%2E+2018%2E+All%2Dbut%2Dthe%2Dtop%3A+Simple+and+effective+postprocessing+for+word+representations%2E+In+International+Conference+on+Learning+Representations%2E)
47. Malvina Nissim, Rik van Noord, and Rob van der Goot. 2020. Fair is better than sensational: Man is to doctor as woman is to doctor. Computational Linguistics, pages 1–17. [find](https://scholar.google.com/scholar?q=Malvina+Nissim%2C+Rik+van+Noord%2C+and+Rob+van+der+Goot%2E+2020%2E+Fair+is+better+than+sensational%3A+Man+is+to+doctor+as+woman+is+to+doctor%2E+Computational+Linguistics%2C+pages+1%E2%80%9317%2E)
48. Davide Nunes and Luis Antunes. 2018. Neural random projections for language modelling. CoRR, abs/1807.00930. — [arXiv:1807.00930](https://arxiv.org/abs/1807.00930) [find](https://scholar.google.com/scholar?q=Davide+Nunes+and+Luis+Antunes%2E+2018%2E+Neural+random+projections+for+language+modelling%2E+CoRR%2C+abs%2F1807%2E00930%2E+%E2%80%94+%5BarXiv%3A1807%2E00930%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1807%2E00930%29)
49. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+t)
50. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
51. Vikas Raunak, Vivek Gupta, and Florian Metze. 2019. Effective dimensionality reduction for word embeddings. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 235–243, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vikas+Raunak%2C+Vivek+Gupta%2C+and+Florian+Metze%2E+2019%2E+Effective+dimensionality+reduction+for+word+embeddings%2E+In+Proceedings+of+the+4th+Workshop+on+Representation+Learning+for+NLP+%28RepL4NLP%2D2019%29%2C+pages)
52. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how bert works. ArXiv, abs/2002.12327. [find](https://scholar.google.com/scholar?q=Anna+Rogers%2C+Olga+Kovaleva%2C+and+Anna+Rumshisky%2E+2020%2E+A+primer+in+bertology%3A+What+we+know+about+how+bert+works%2E+ArXiv%2C+abs%2F2002%2E12327%2E) [our card](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/)
53. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Commun. ACM, 8(10):627–633. [find](https://scholar.google.com/scholar?q=Herbert+Rubenstein+and+John+B%2E+Goodenough%2E+1965%2E+Contextual+correlates+of+synonymy%2E+Commun%2E+ACM%2C+8%2810%29%3A627%E2%80%93633%2E)
54. Paula Rubio-Fernández. 2008. Concept narrowing: The role of context-independent information. Journal of semantics, 25(4):381–409. [find](https://scholar.google.com/scholar?q=Paula+Rubio%2DFerna%CC%81ndez%2E+2008%2E+Concept+narrowing%3A+The+role+of+context%2Dindependent+information%2E+Journal+of+semantics%2C+25%284%29%3A381%E2%80%93409%2E)
55. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+ArXiv%2C+abs%2F1910%2E01108%2E)
56. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2019. Green AI. CoRR, abs/1907.10597. [find](https://scholar.google.com/scholar?q=Roy+Schwartz%2C+Jesse+Dodge%2C+Noah+A%2E+Smith%2C+and+Oren+Etzioni%2E+2019%2E+Green+AI%2E+CoRR%2C+abs%2F1907%2E10597%2E) [our card](https://md.research.elamur.ai/free/en/papers/schwartz-2019-green-ai/)
57. Dinghan Shen, Pengyu Cheng, Dhanasekar Sundararaman, Xinyuan Zhang, Qian Yang, Meng Tang, Asli Celikyilmaz, and Lawrence Carin. 2019. Learning compressed sentence representations for on-device text processing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Dinghan+Shen%2C+Pengyu+Cheng%2C+Dhanasekar+Sundararaman%2C+Xinyuan+Zhang%2C+Qian+Yang%2C+Meng+Tang%2C+Asli+Celikyilmaz%2C+and+Lawrence+Carin%2E+2019%2E+Learning+compressed+sentence+representations+for+on%2Ddevice+text+pr)
58. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+pytorch%2E)
59. Weijia Shi, Muhao Chen, Pei Zhou, and Kai-Wei Chang. 2019. Retrofitting contextualized word embeddings with paraphrases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Muhao+Chen%2C+Pei+Zhou%2C+and+Kai%2DWei+Chang%2E+2019%2E+Retrofitting+contextualized+word+embeddings+with+paraphrases%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+)
60. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Process)
61. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gabriel+Stanovsky%2C+Noah+A%2E+Smith%2C+and+Luke+Zettlemoyer%2E+2019%2E+Evaluating+gender+bias+in+machine+translation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C) [our card](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/)
62. Karl Stratos. 2017. A sub-character architecture for Korean language processing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 721–726, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Karl+Stratos%2E+2017%2E+A+sub%2Dcharacter+architecture+for+Korean+language+processing%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+721%E2%80%93726%2C+Copenhagen%2C+D)
63. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emma+Strubell%2C+Ananya+Ganesh%2C+and+Andrew+McCallum%2E+2019%2E+Energy+and+policy+considerations+for+deep+learning+in+NLP%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Lingu) [our card](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/)
64. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computati… [find](https://scholar.google.com/scholar?q=Tony+Sun%2C+Andrew+Gaut%2C+Shirlyn+Tang%2C+Yuxin+Huang%2C+Mai+ElSherief%2C+Jieyu+Zhao%2C+Diba+Mirza%2C+Elizabeth+Belding%2C+Kai%2DWei+Chang%2C+and+William+Yang+Wang%2E+2019%2E+Mitigating+gender+bias+in+natural+language+proce) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-mitigating-gender-bias-natural-language/)
65. Yi Chern Tan and L. Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 13230–13241. Curran Ass… [find](https://scholar.google.com/scholar?q=Yi+Chern+Tan+and+L%2E+Elisa+Celis%2E+2019%2E+Assessing+social+and+intersectional+biases+in+contextualized+word+representations%2E+In+H%2E+Wallach%2C+H%2E+Larochelle%2C+A%2E+Beygelzimer%2C+F%2E+dAlche%CC%81%2DBuc%2C+E%2E+Fox%2C+and+R%2E+G)
66. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019a%2E+BERT+rediscovers+the+classical+NLP+pipeline%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4593%E2%80%93) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/)
67. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Lea… [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Patrick+Xia%2C+Berlin+Chen%2C+Alex+Wang%2C+Adam+Poliak%2C+R+Thomas+McCoy%2C+Najoung+Kim%2C+Benjamin+Van+Durme%2C+Sam+Bowman%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019b%2E+What+do+you+learn+from+context%3F+probi) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/)
68. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+I%2E+Guyon%2C+U%2E+V%2E+Luxburg%2C+S%2E+Bengio%2C+H%2E)
69. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Shi+Feng%2C+Nikhil+Kandpal%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+2019%2E+Universal+adversarial+triggers+for+attacking+and+analyzing+NLP%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods)
70. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=John+Wieting%2C+Mohit+Bansal%2C+Kevin+Gimpel%2C+and+Karen+Livescu%2E+2016%2E+Towards+universal+paraphrastic+sentence+embeddings%2E+In+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/wieting-2015-towards-universal-paraphrastic-sentence-embeddings/)
71. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 33. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Ruslan+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+Advances+in+Neural+Inform) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
72. Jinxing Yu, Xun Jian, Hao Xin, and Yangqiu Song. 2017. Joint embeddings of Chinese words, characters, and fine-grained subcharacter components. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 286–291, Copenhagen, Denmark. Association for Computational… [find](https://scholar.google.com/scholar?q=Jinxing+Yu%2C+Xun+Jian%2C+Hao+Xin%2C+and+Yangqiu+Song%2E+2017%2E+Joint+embeddings+of+Chinese+words%2C+characters%2C+and+fine%2Dgrained+subcharacter+components%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Metho)
73. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
74. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019a. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1… [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Tianlu+Wang%2C+Mark+Yatskar%2C+Ryan+Cotterell%2C+Vicente+Ordonez%2C+and+Kai%2DWei+Chang%2E+2019a%2E+Gender+bias+in+contextualized+word+embeddings%2E+In+Proceedings+of+the+2019+Conference+of+the+North+Amer) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2019-gender-bias-contextualized-word-embeddings/)
75. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979–2989, Copenhagen, Denmark.… [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Tianlu+Wang%2C+Mark+Yatskar%2C+Vicente+Ordonez%2C+and+Kai%2DWei+Chang%2E+2017%2E+Men+also+like+shopping%3A+Reducing+gender+bias+amplification+using+corpus%2Dlevel+constraints%2E+In+Proceedings+of+the+2017+C) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2017-men-also-like-shopping-reducing/)
76. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018a. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Tianlu+Wang%2C+Mark+Yatskar%2C+Vicente+Ordonez%2C+and+Kai%2DWei+Chang%2E+2018a%2E+Gender+bias+in+coreference+resolution%3A+Evaluation+and+debiasing+methods%2E+In+Proceedings+of+the+2018+Conference+of+the+) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/)
77. Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018b. Learning gender-neutral word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4847–4853, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Yichao+Zhou%2C+Zeyu+Li%2C+Wei+Wang%2C+and+Kai%2DWei+Chang%2E+2018b%2E+Learning+gender%2Dneutral+word+embeddings%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2018-learning-gender-neutral-word-embeddings/)
78. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019b. Mover-Score: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internation… [find](https://scholar.google.com/scholar?q=Wei+Zhao%2C+Maxime+Peyrard%2C+Fei+Liu%2C+Yang+Gao%2C+Christian+M%2E+Meyer%2C+and+Steffen+Eger%2E+2019b%2E+Mover%2DScore%3A+Text+generation+evaluating+with+contextualized+embeddings+and+earth+mover+distance%2E+In+Proceeding)