---
title: "BERT Knows Punta Cana is not just beautiful, it's gorgeous:"
description: "The paper investigates whether a pre-trained BERT model encodes how strongly scalar adjectives… 48 references, cited by 18 papers here."
authors: ["Aina Garí Soler", "Marianna Apidianaki"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.598/"
doi: "10.18653/v1/2020.emnlp-main.598"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/soler-2020-bert-knows-punta-cana-is/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/soler-2020-bert-knows-punta-cana-is/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / BERT Knows Punta Cana is not just beautiful, it's gorgeous: Ranking Scalar Adjectives with Contextualised Representations

# BERT Knows Punta Cana is not just beautiful, it's gorgeous: Ranking Scalar Adjectives with Contextualised Representations

**Authors:** [Aina Garí Soler](https://md.research.elamur.ai/free/en/authors/aina-gari-soler/) and [Marianna Apidianaki](https://md.research.elamur.ai/free/en/authors/marianna-apidianaki/) · **Year:** 2020 · **Venue:** [Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2020](https://md.research.elamur.ai/free/en/venues/emnlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.598/) · [PDF](https://aclanthology.org/2020.emnlp-main.598.pdf) · [DOI 10.18653/v1/2020.emnlp-main.598](https://doi.org/10.18653/v1/2020.emnlp-main.598) · [OpenAlex](https://openalex.org/W3092399518)

**Citations:** 12 citations (OpenAlex, 2026-10-14) · [18](https://md.research.elamur.ai/free/en/papers/soler-2020-bert-knows-punta-cana-is/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper investigates whether a pre-trained BERT model encodes how strongly scalar adjectives express a property, for example that *gorgeous* is more intense than *beautiful*, which in turn is more intense than *pretty*. The authors use three gold datasets of adjective half-scales: DEMELO, CROWD and WILKINSON. For each scale, they take sentences from ukWaC and Flickr 30K and substitute every adjective of the scale into the same context, so the representations differ only in the adjective. They filter out unsuitable contexts using stanza parses, Hearst patterns and context2vec fit scores. They then rank adjectives in several ways. The best approach borrows from gender-bias research: an "intensity direction" is computed as the difference between the vectors of a mild and a strong adjective, possibly from just one pair, and new adjectives are ordered by their projection onto it. Evaluation covers intrinsic ranking and an Indirect Question Answering task. BERT-based rankings beat static embeddings and earlier systems that depend on patterns or lexicons. The approach is resource-light and could transfer to languages that lack scalar adjective resources.

## Key points
- Datasets: DEMELO has 87 half-scales and 548 adjective pairs, taken from WordNet "dumbbell" structures. CROWD has 79 half-scales and 330 pairs, built by crowdsourcing over PPDB vocabulary. WILKINSON has 21 half-scales and 61 pairs.
- For each adjective, 1,000 sentences are collected from ukWaC and from Flickr 30K. Flickr 30K covers 96.56% of DEMELO scales and 86.08% of CROWD scales.
- Context selection works in two steps:
  - Sentences matching Hearst-pattern is-a configurations, detected on stanza dependency parses, are removed.
  - Among the remaining substituted variants, the ten whose 600-dim context2vec fit scores have the lowest standard deviation are kept. context2vec judged substitutability better than BERT perplexity or masked-token probability.
- DIFFVEC-1 (+), built from one positive pair, consistently beats DIFFVEC-1 (−), built from one negative pair. A partial explanation is word frequency: good occurs 276M times and awesome 10M, against 65M for bad and 4M for horrible.
- Ranking quality improves after layer 3 and peaks in one of BERT's last four layers. Carefully chosen sentences gave only minor gains over random ones.

## Interesting ideas
- An abstract property such as intensity can be treated as a direction in embedding space, obtained by subtracting the vector of a weak member from that of a strong one; any candidate is then placed by projecting it onto that axis, which is cheap and needs almost no labelled data.
- Placing every alternative word into the identical sentence gives minimal-pair representations, so any measured difference comes from the word itself rather than the context.
- Before substituting, the authors drop contexts where the swap would distort meaning: sentences with is-a or instantiation structure, found by running Hearst patterns over dependency parses.
- Contexts are kept only when all candidates fit about equally well, measured as low spread of fit scores, rather than when they fit well on average.
- Indirect answers to yes/no questions can be resolved by comparing the intensity of the adjective in the question with the one in the reply.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (world): This could seed graded value scales in compiled knowledge (warm \< hot \< scalding) for adjectives that WordNet lists without an order, with the derived order stored as an explicit, inspectable relation.
- **I2** (learn): Minimal-pair substitution sets would make good probes and negative controls when checking whether a learned component really reacts to lexical meaning.
- **I3** (tree): The same parse-plus-pattern filter could guard automatic data augmentation, so that word-swapped treebank sentences stay valid for parser training.
- **I4** (learn): For silver data, preferring examples where all alternatives score evenly would avoid comparisons biased toward one candidate.
- **I5** (frames): The implicature module could decide whether a reply like "it was gorgeous" to "was it pretty?" means yes, using the ordered adjective scales as an explainable reason.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Université Paris-Saclay](https://md.research.elamur.ai/free/en/universities/universite-paris-saclay/) — University, Gif-sur-Yvette, France · 34 works · 5,733 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Aina Garí Soler and Marianna Apidianaki. 2020. BERT Knows Punta Cana is not just beautiful, it's gorgeous: Ranking Scalar Adjectives with Contextualised Representations. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{soler2020bert,
  title = {{BERT Knows Punta Cana is not just beautiful, it's gorgeous: Ranking Scalar Adjectives with Contextualised Representations}},
  author = {Aina Garí Soler and Marianna Apidianaki},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.598},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.598}
}
```

- [← previous Assessing Discourse Relations in Language Generation from GPT-2](https://md.research.elamur.ai/free/en/papers/ko-2020-assessing-discourse-relations-language-generation/)
- [next → Classifying Syntactic Errors in Learner Language](https://md.research.elamur.ai/free/en/papers/choshen-2020-classifying-syntactic-errors-learner-language/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

48 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. Journal of Language Resources and Evaluation, 43(3):209–226. [↗](https://doi.org/10.1007/s10579-009-9081-4)
2. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/). In Advances in Neural Information Processing Systems 29, pages 4349–4357. Barcelona, Spain.
3. Thorsten Brants and Alex Franz. 2006. Web 1T 5-gram Version 1. In LDC2006T13, Philadelphia, Pennsylvania. Linguistic Data Consortium. [find](https://scholar.google.com/scholar?q=Thorsten+Brants+and+Alex+Franz%2E+2006%2E+Web+1T+5%2Dgram+Version+1%2E+In+LDC2006T13%2C+Philadelphia%2C+Pennsylvania%2E+Linguistic+Data+Consortium%2E)
4. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w19-4828)
5. Anne Cocos, Skyler Wharton, Ellie Pavlick, Marianna Apidianaki, and Chris Callison-Burch. 2018. [Learning Scalar Adjective Intensity from Paraphrases](https://md.research.elamur.ai/free/en/papers/cocos-2018-learning-scalar-adjective-intensity-paraphrases/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1752–1762, Brussels, Belgium. Association for Computational Linguistics.
6. Sunipa Dev and Jeff M Phillips. 2019. Attenuating Bias in Word Vectors. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), Naha, Okinawa, Japan. [↗](https://doi.org/10.48550/arxiv.1901.07656)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
8. Vasileios Hatzivassiloglou and Kathleen R. McKeown. 1993. Towards the Automatic Identification of Adjectival Scales: Clustering Adjectives According to Meaning. In 31st Annual Meeting of the Association for Computational Linguistics, pages 172–182, Columbus, Ohio, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vasileios+Hatzivassiloglou+and+Kathleen+R%2E+McKeown%2E+1993%2E+Towards+the+Automatic+Identification+of+Adjectival+Scales%3A+Clustering+Adjectives+According+to+Meaning%2E+In+31st+Annual+Meeting+of+the+Associati)
9. Marti A. Hearst. 1992. Automatic Acquisition of Hyponyms from Large Text Corpora. In COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marti+A%2E+Hearst%2E+1992%2E+Automatic+Acquisition+of+Hyponyms+from+Large+Text+Corpora%2E+In+COLING+1992+Volume+2%3A+The+15th+International+Conference+on+Computational+Linguistics%2E)
10. John Hewitt and Percy Liang. 2019. Designing and Interpreting Probes with Control Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, China. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d19-1275)
11. John Hewitt and Christopher D. Manning. 2019. [A Structural Probe for Finding Syntax in Word Representations](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1419)
12. Christopher Kennedy and Louise McNally. 2005. Scale Structure and the Semantic Typology of Gradable Predicates. Language, 81:345–381. [find](https://scholar.google.com/scholar?q=Christopher+Kennedy+and+Louise+McNally%2E+2005%2E+Scale+Structure+and+the+Semantic+Typology+of+Gradable+Predicates%2E+Language%2C+81%3A345%E2%80%93381%2E)
13. Adam Kilgarriff. 2004. How Dominant Is the Commonest Sense of a Word? Lecture Notes in Computer Science (vol. 3206), Text, Speech and Dialogue, Sojka Petr, Kopeček Ivan, Pala Karel (eds.), pages 103–112. Springer, Berlin, Heidelberg. [find](https://scholar.google.com/scholar?q=Adam+Kilgarriff%2E+2004%2E+How+Dominant+Is+the+Commonest+Sense+of+a+Word%3F+Lecture+Notes+in+Computer+Science+%28vol%2E+3206%29%2C+Text%2C+Speech+and+Dialogue%2C+Sojka+Petr%2C+Kopec%CC%8Cek+Ivan%2C+Pala+Karel+%28eds%2E%29%2C+pages+103%E2%80%93)
14. Joo-Kyung Kim and Marie-Catherine de Marneffe. 2013. Deriving Adjectival Scales from Continuous Space Word Representations. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1625–1630, Seattle, Washington, USA. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d13-1169)
15. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. [Revealing the Dark Secrets of BERT](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365–4374, Hong Kong, China. Association for Computational Linguistics.
16. Gerhard Kremer, Katrin Erk, Sebastian Padó, and Stefan Thater. 2014. What Substitutes Tell Us - Analysis of an “All-Words” Lexical Substitution Corpus. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 540–549, Gothenburg, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gerhard+Kremer%2C+Katrin+Erk%2C+Sebastian+Pado%CC%81%2C+and+Stefan+Thater%2E+2014%2E+What+Substitutes+Tell+Us+%2D+Analysis+of+an+%E2%80%9CAll%2DWords%E2%80%9D+Lexical+Substitution+Corpus%2E+In+Proceedings+of+the+14th+Conference+of+the+Eu)
17. Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2020. A General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spaces. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York City, NY, USA. [↗](https://doi.org/10.1609/aaai.v34i05.6325)
18. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). Transactions of the Association for Computational Linguistics, 4:521– 535. [↗](https://doi.org/10.1162/tacl_a_00115)
19. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2010. [“Was It Good?](https://md.research.elamur.ai/free/en/papers/marneffe-2010-was-it-good-it-was/) It Was Provocative.” Learning the Meaning of Scalar Adjectives”. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 167–176, Uppsala, Sweden. Association for Computational Linguistics.
20. Diana McCarthy, Rob Koeling, Julie Weeds, and John Carroll. 2004. [Finding Predominant Word Senses in Untagged Text](https://md.research.elamur.ai/free/en/papers/mccarthy-2004-finding-predominant-word-senses-untagged/). In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 279–286, Barcelona, Spain.
21. Louise McNally. 2016. Scalar alternatives and scalar inference involving adjectives: A comment on van Tiel, et al. 2016. In Ruth Kramer Jason Ostrove and Joseph Sabbagh, editors, Asking the Right Questions: Essays in Honor of Sandra Chung, pages 17–28. [find](https://scholar.google.com/scholar?q=Louise+McNally%2E+2016%2E+Scalar+alternatives+and+scalar+inference+involving+adjectives%3A+A+comment+on+van+Tiel%2C+et+al%2E+2016%2E+In+Ruth+Kramer+Jason+Ostrove+and+Joseph+Sabbagh%2C+editors%2C+Asking+the+Right+Ques)
22. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. [context2vec: Learning Generic Context Embedding with Bidirectional LSTM](https://md.research.elamur.ai/free/en/papers/melamud-2016-context2vec-learning-generic-context-embedding/). In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 51–61, Berlin, Germany. Association for Computational Linguistics.
23. Gerard de Melo and Mohit Bansal. 2013. Good, Great, Excellent: Global Inference of Semantic Intensities. Transactions of the Association for Computational Linguistics, 1:279–290. [↗](https://doi.org/10.1162/tacl_a_00227)
24. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient Estimation of Word Representations in Vector Space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint:1301.3781v3. [↗](https://arxiv.org/abs/1301.3781)
25. Finn Årup Nielsen. 2011. A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. In Proceedings of the ESWC 2011 Workshop on ’Making Sense of Microposts: Big things come in small packages’, volume 718 in CEUR Workshop Proceedings, pages 93–98. [↗](https://doi.org/10.48550/arxiv.1103.2903)
26. Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1–2):1–135. [↗](https://doi.org/10.1561/9781601981516)
27. Ajay Patel, Alexander Sands, Chris Callison-Burch, and Marianna Apidianaki. 2018. Magnitude: A Fast, Efficient Universal Vector Embedding Utility Package. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 120–126, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ajay+Patel%2C+Alexander+Sands%2C+Chris+Callison%2DBurch%2C+and+Marianna+Apidianaki%2E+2018%2E+Magnitude%3A+A+Fast%2C+Efficient+Universal+Vector+Embedding+Utility+Package%2E+In+Proceedings+of+the+2018+Conference+on+Empi)
28. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. [PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification](https://md.research.elamur.ai/free/en/papers/pavlick-2015-ppdb-2-0-better-paraphrase/). In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 425–430, Beijing, China. Association for Computational Linguistics.
29. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://md.research.elamur.ai/free/en/papers/qi-2020-stanza-python-natural-language-processing/). arXiv preprint arXiv:2003.07082. [↗](https://arxiv.org/abs/2003.07082)
30. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://md.research.elamur.ai/free/en/papers/ravfogel-2020-null-it-out-guarding-protected/). arXiv preprint arXiv:2004.07667. [↗](https://arxiv.org/abs/2004.07667)
31. Sven Rill, J. vom Scheidt, Johannes Drescher, Oliver Schütz, Dirk Reinel, and Florian Wogenstein. 2012. A generic approach to generate opinion lists of phrases for opinion mining applications. In Proceedings of the First International Workshop on Issues of Sentiment Discovery and Opinion Mining (WIS-DOM), pages 1–8, Beijing, China. [find](https://scholar.google.com/scholar?q=Sven+Rill%2C+J%2E+vom+Scheidt%2C+Johannes+Drescher%2C+Oliver+Schu%CC%88tz%2C+Dirk+Reinel%2C+and+Florian+Wogenstein%2E+2012%2E+A+generic+approach+to+generate+opinion+lists+of+phrases+for+opinion+mining+applications%2E+In+Pro)
32. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. [A Primer in BERTology: What we know about how BERT works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). arXiv preprint:2002.12327v1. [↗](https://arxiv.org/abs/2002.12327)
33. Josef Ruppenhofer, Michael Wiegand, and Jasper Brandes. 2014. Comparing methods for deriving intensity scores for adjectives. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 117–122, Gothenburg, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Josef+Ruppenhofer%2C+Michael+Wiegand%2C+and+Jasper+Brandes%2E+2014%2E+Comparing+methods+for+deriving+intensity+scores+for+adjectives%2E+In+Proceedings+of+the+14th+Conference+of+the+European+Chapter+of+the+Assoc)
34. Raksha Sharma, Mohit Gupta, Astha Agarwal, and Pushpak Bhattacharyya. 2015. Adjective Intensity and Sentiment Analysis. In Proceedings of the 2015 Conference on Empirical Methods for Natural Language Processing, pages 2520–2526, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Raksha+Sharma%2C+Mohit+Gupta%2C+Astha+Agarwal%2C+and+Pushpak+Bhattacharyya%2E+2015%2E+Adjective+Intensity+and+Sentiment+Analysis%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+for+Natural+Language+)
35. Vera Sheinman, Christiane Fellbaum, Isaac Julien, Peter Schulam, and Takenobu Tokunaga. 2013. Large, huge or gigantic? Identifying and encoding intensity relations among adjectives in WordNet. Language resources and evaluation, 47(3):797–816. [find](https://scholar.google.com/scholar?q=Vera+Sheinman%2C+Christiane+Fellbaum%2C+Isaac+Julien%2C+Peter+Schulam%2C+and+Takenobu+Tokunaga%2E+2013%2E+Large%2C+huge+or+gigantic%3F+Identifying+and+encoding+intensity+relations+among+adjectives+in+WordNet%2E+Languag)
36. Vera Sheinman and Takenobu Tokunaga. 2009. AdjS-cales: Visualizing Differences between Adjectives for Language Learners. IEICE Transactions on Information and Systems, 92-D:1542–1550. [find](https://scholar.google.com/scholar?q=Vera+Sheinman+and+Takenobu+Tokunaga%2E+2009%2E+AdjS%2Dcales%3A+Visualizing+Differences+between+Adjectives+for+Language+Learners%2E+IEICE+Transactions+on+Information+and+Systems%2C+92%2DD%3A1542%E2%80%931550%2E)
37. Chaitanya Shivade, Marie-Catherine de Marneffe, Eric Fosler-Lussier, and Albert M. Lai. 2015. Corpus-based discovery of semantic intensity scales. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–493, Denver, Colorado. Association for Computational Linguistics. [↗](https://doi.org/10.3115/v1/n15-1051)
38. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. [Lexicon-based methods for sentiment analysis](https://md.research.elamur.ai/free/en/papers/taboada-2011-lexicon-based-methods-sentiment-analysis/). Computational linguistics, 37(2):267–307. [↗](https://doi.org/10.1162/coli_a_00049)
39. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2019. [oLMpics – On what Language Model Pre-training Captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/). arXiv preprint arXiv:1912.13283v1. [↗](https://arxiv.org/abs/1912.13283)
40. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [BERT Rediscovers the Classical NLP Pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1905.05950)
41. Bryan Wilkinson. 2017. Identifying and Ordering Scalar Adjectives Using Lexical Substitution. Ph.D. thesis, University of Maryland, Baltimore County. [↗](https://doi.org/10.13016/m29eeu-ic2j)
42. Bryan Wilkinson and Tim Oates. 2016. [A Gold Standard for Scalar Adjectives](https://md.research.elamur.ai/free/en/papers/wilkinson-2016-gold-standard-scalar-adjectives/). In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2669– 2675, Portorož, Slovenia. European Language Resources Association (ELRA).
43. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint:1609.08144. [↗](https://arxiv.org/abs/1609.08144)
44. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. [↗](https://doi.org/10.1162/tacl_a_00166)
45. Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018. [Learning gender-neutral word embeddings](https://md.research.elamur.ai/free/en/papers/zhao-2018-learning-gender-neutral-word-embeddings/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4847–4853, Brussels, Belgium. Association for Computational Linguistics.
46. George Kingsley Zipf. 1945. The meaning-frequency relationship of words. Journal of General Psychology, 33(2):251–256. [↗](https://doi.org/10.1080/00221309.1945.10544509)
47. A Bob Van Tiel, Emiel Van Miltenburg, Natalia Zevakhina, and Bart Geurts. 2016. Scalar Diversity. Journal of semantics, 33(1):137–175. Hearst Patterns Figure 4 illustrates the dependency structure of the following Hearst patterns: Elena Voita, Rico Sennrich, and Ivan Titov. 2019a. [The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives](https://md.research.elamur.ai/free/en/papers/voita-2019-bottom-up-evolution-representations-transformer/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4396–4406, Hong Kong, China. Association for Computational Linguistics.
48. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019b. [Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/). In Proceedings of the • [NP] and other [NP] • [NP] or other [NP] • [NP] such as [NP] • Such [NP] as [NP] • [NP], including [NP] • [NP], especially [NP] • [NP] like [NP] *Figure 4: Dependency structure of Hearst patterns.* We use these patterns to detect sentences where adjective substitution should not take place, as described in Section 4.2 of the paper. We remove these sentences from our ukWaC and Flickr datasets. 17

*Updated 24 October 2026*