---
title: "A Corpus of Non-Native Written English Annotated for Metaphor"
description: "Supervised metaphor detection has mostly been trained and tested on the VUA corpus, which draws… 19 references, cited by 22 papers here."
authors: ["Beata Beigman Klebanov", "Chee Wee (Ben) Leong", "Michael Flor"]
year: 2018
venue: "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)"
source: "https://aclanthology.org/N18-2014/"
doi: "10.18653/v1/n18-2014"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/klebanov-2018-corpus-non-native-written-english/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/klebanov-2018-corpus-non-native-written-english/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/) / A Corpus of Non-Native Written English Annotated for Metaphor

# A Corpus of Non-Native Written English Annotated for Metaphor

**Authors:** [Beata Beigman Klebanov](https://md.research.elamur.ai/free/en/authors/beata-beigman-klebanov/), [Chee Wee (Ben) Leong](https://md.research.elamur.ai/free/en/authors/chee-wee-ben-leong/) and [Michael Flor](https://md.research.elamur.ai/free/en/authors/michael-flor/) · **Year:** 2018 · **Venue:** [Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2018](https://md.research.elamur.ai/free/en/venues/naacl/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N18-2014/) · [PDF](https://aclanthology.org/N18-2014.pdf) · [DOI 10.18653/v1/n18-2014](https://doi.org/10.18653/v1/n18-2014) · [OpenAlex](https://openalex.org/W2803972085)

**Citations:** 23 citations (OpenAlex, 2026-10-14) · [22](https://md.research.elamur.ai/free/en/papers/klebanov-2018-corpus-non-native-written-english/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Supervised metaphor detection has mostly been trained and tested on the VUA corpus, which draws on BNC text. Learner writing had no public metaphor-annotated data. The authors fill this gap with 240 argumentative essays by non-native English writers, taken from the ETS Corpus of Non-Native Written English. Two annotators marked every content word for metaphor. They used a protocol aimed at *argumentation-relevant* metaphor, meaning figurative language that helps the writer push a position, and relied on intuition rather than dictionary senses. Existing VUA-era feature sets with Logistic Regression were benchmarked on the corpus. They score somewhat lower than on VUA, with the best F-score at 0.56. The authors then trained a detector on these essays and applied it to large scored essay collections. After controlling for essay length, the amount of detected metaphor tracks holistic essay scores, and it does so more strongly than a VUA-trained detector does. This makes the corpus useful both for metaphor detection and for studying how proficiency relates to figurative language.

## Key points
- **Sampling design:** 8 prompts × 3 native languages (Japanese, Italian, Arabic) × 2 proficiency levels (medium, high), with 5 essays per cell.
  - Low-proficiency essays were left out because their intended meaning is often unclear.
  - The data split is 75% train and 25% test.
  - The annotations are released on the EducationalTestingService GitHub.
- **Annotation agreement and labels:**
  - Kappa was 0.56 after the first annotation pass and 0.62 after the targeted self-review rounds.
  - The final labels are the union of both annotators.
  - About 3% of words are metaphorical.
  - By instance count, 7% of all-POS training instances (26,737) and 14% of verb instances (7,016) are metaphors.
- **Benchmark results:** feature sets all-15, v-16 and all-16 were run with Logistic Regression in scikit-learn/SKLL.
  - With optimized class re-weighting, F-scores are 0.52, 0.56 (v-16) and 0.53.
  - Without re-weighting, every system reaches about 0.50 F, with higher precision (about 0.68) and lower recall.
  - For comparison, v-16 reaches 0.60 F on VUA verbs.
- **Proficiency measure:** metaphor use was quantified as the log of metaphors per 1,000 words.
  - Its relationship to essay scores was assessed with a partial correlation that held essay length constant.
  - Six datasets from three testing programs were used (MGrF/S, IColF/S, DTLF/S), with 10k–40k essays each.
- **Proficiency results:**
  - The essay-trained detector (Arg, trained on these 240 essays plus 141 proprietary ones) gives significant correlations on all six sets, for example 0.166 vs 0.020 for VUA on MGrF.
  - Arg beats VUA everywhere, including test programs unrelated to the annotation data.
  - Correlations are stronger for free-position tasks than for source-based tasks.

## Interesting ideas
- Agreement was checked essay by essay, and only essays below a kappa threshold went back for review, first by one annotator and then by the other if the set was still weak; this concentrates review effort where disagreement actually is.
- Metaphor was defined by the job it does for the author's argument rather than by distance from a dictionary "basic" sense, so incidental figurative words like "behind" or "found" are not marked.
- A concreteness gap between a verb and its direct object, or between an adjective and its head noun, serves as a signal of non-literal use, and it is computed along specific syntactic relations.
- A new text feature was judged by its partial correlation with the target after removing a strong confound (length), not by raw correlation.
- The authors noted that long source-based prompts force much of the writer's vocabulary, which weakens lexical-choice signals; task design changes what a text can reveal.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): when witness parsers disagree on silver data, send only the sentences with low agreement to an LLM-teacher or repair pass, escalating tiers only if overall agreement stays below a threshold.
- **I2** (frames): mark figurative predicates in prag when they carry the speaker's stance or evaluation (e.g., "drain resources" as negative framing), alongside irony and implicature, rather than flagging every non-literal sense.
- **I3** (frames): compute such gaps over UD obj/amod edges, using concreteness or ontology data from the world layer, to flag frames whose literal VerbNet role restrictions are violated, with an explainable reason.
- **I4** (learn): when evaluating any new feature or repair rule, regress out trivial predictors such as sentence length or token count as a built-in negative control, so gains are not just proxies.
- **I5** (answer): when grading or reading student answers in school-course tests, discount words copied from the question or source text before drawing conclusions from vocabulary use.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Educational Testing Service](https://md.research.elamur.ai/free/en/ngos/educational-testing-service/) — Nonprofit, Princeton, United States · 27 works · 2,212 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,649 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 930 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works · [🧷 Idioms & multiword expressions](https://md.research.elamur.ai/free/en/topics/multiword-expressions/) · 222 works

## Cite

Cite as: Beata Beigman Klebanov, Chee Wee (Ben) Leong and Michael Flor. 2018. A Corpus of Non-Native Written English Annotated for Metaphor. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{klebanov2018corpus,
  title = {{A Corpus of Non-Native Written English Annotated for Metaphor}},
  author = {Beata Beigman Klebanov, Chee Wee (Ben) Leong and Michael Flor},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2014},
  url = {https://doi.org/10.18653/v1/n18-2014}
}
```

- [← previous EFLLex: A Graded Lexical Resource for Learners of English as a Foreign Language](https://md.research.elamur.ai/free/en/papers/durlich-2018-efllex-graded-lexical-resource-learners/)
- [next → Zero Pronoun Resolution with Attention-based Neural Network](https://md.research.elamur.ai/free/en/papers/yin-2018-zero-pronoun-resolution-attention-based/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/)

## References

19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yigal Attali and Jill Burstein. 2006. Automated Essay Scoring With e-rater R V.2. Journal of Technology, Learning, and Assessment 4(3). [https://www.learntechlib.org/p/103244/](https://www.learntechlib.org/p/103244/). [↗](https://www.learntechlib.org/p/103244/)
2. Olga Gurevich and Paul Deane. 2007. Document similarity measures to distinguish native vs. non-native essay writers. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers. Association for Computational Linguistics, Rochester, New York, pages 49–52. http://www.aclweb.org/anthology/N/N07/N07-2013. [↗](https://aclanthology.org/N07-2013/)
3. Beata Beigman Klebanov, Chee Wee Leong, and Michael Flor. 2013. Argumentation-relevant metaphors in test-taker essays. In Proceedings of the First Workshop on Metaphor in NLP. Association for Computational Linguistics, Atlanta, Georgia, pages 11–20. http://www.aclweb.org/ anthology/W13-0902. [↗](https://aclanthology.org/W13-0902/)
4. Beata Beigman Klebanov, Chee Wee Leong, and Michael Flor. 2015. Supervised word-level metaphor detection: Experiments with concreteness and reweighting of examples. In Proceedings of the Third Workshop on Metaphor in NLP. Association for Computational Linguistics, Denver, Colorado, pages 11–20. http://www.aclweb.org/anthology/W15-1402. [↗](https://aclanthology.org/W15-1402/)
5. Hessel Haagsma and Johannes Bjerva. 2016. Detecting novel metaphor using selectional preference information. In Proceedings of the Fourth Workshop on Metaphor in NLP. Association for Computational Linguistics, San Diego, California, pages 10– 17. http://www.aclweb.org/anthology/ W16-1102. [↗](https://aclanthology.org/W16-1102/)
6. Beata Beigman Klebanov, Chee Wee Leong, E. Dario Gutierrez, Ekaterina Shutova, and Michael Flor. 2016. Semantic classifications for detection of verb metaphors. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Berlin, Germany, pages 101–106. http://anthology.aclweb.org/P16-2017. [↗](https://aclanthology.org/P16-2017/)
7. Hyeju Jang, Yohan Jo, Qinlan Shen, Michael Miller, Seungwhan Moon, and Carolyn Rose. 2016. Metaphor detection with topic transition, emotion and cognition in context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 216–225. http://www.aclweb.org/anthology/P16-1021. First Workshop on Argumentation Mining. Association for Computational Linguistics, Baltimore, Maryland, pages 69–78. http://www.aclweb. org/anthology/W14-2110. [↗](https://aclanthology.org/P16-1021/)
8. Hyeju Jang, Seungwhan Moon, Yohan Jo, and Carolyn Rose. 2015. Metaphor detection in discourse. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics, Prague, Czech Republic, pages 384–392. http://aclweb.org/anthology/W15-4650. [↗](https://aclanthology.org/W15-4650/)
9. Christian Stab and Iryna Gurevych. 2014. [Annotating argument components and relations in persuasive essays](https://md.research.elamur.ai/free/en/papers/stab-2014-annotating-argument-components-relations-persuasive/). In Junichi Tsujii and Jan Hajic, editors, Proceedings of the 25th International Conference on Computational Linguistics (COLING 2014). Dublin City University and Association for Computational Linguistics, pages 1501–1510. http://www.aclweb.org/anthology/C14-1142. [↗](https://aclanthology.org/C14-1142/)
10. Jeannette Littlemore, Tina Krennmayr, James Turner, and Sarah Turner. 2013. An investigation into metaphor use at different levels of second language writing. Applied Linguistics https://doi.org/10.1093/applin/amt004. [↗](https://doi.org/10.1093/applin/amt004)
11. Gerard Steen, Aletta Dorst, Berenike Herrmann, Anna Kaal, Tina Krennmayr, and Trijntje Pasma. 2010. A Method for Linguistic Metaphor Identification. Amsterdam: John Benjamins. [↗](https://doi.org/10.1075/celcr.14)
12. Gözde Özbal, Carlo Strapparava, Serra Sinem Tekiroglu, and Daniele Pighin. 2016. Learning to identify metaphors from a corpus of proverbs. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2060–2065. https://aclweb.org/anthology/D16-1220. [↗](https://aclanthology.org/D16-1220/)
13. Tony Veale, Ekaterina Shutova, and Beata Beigman Klebanov. 2016. Metaphor: A computational perspective. Synthesis Lectures on Human Language Technologies 9(1):1–160. https://www.morganclaypool.com/doi/abs/ 10.2200/S00694ED1V01Y201601HLT031. [↗](https://doi.org/10.2200/s00694ed1v01y201601hlt031)
14. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12:2825–2830. [↗](https://doi.org/10.48550/arxiv.1201.0490)
15. Isaac Persing and Vincent Ng. 2015. [Modeling argument strength in student essays](https://md.research.elamur.ai/free/en/papers/persing-2015-modeling-argument-strength-student-essays/). In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China, pages 543–552. http://www.aclweb.org/ anthology/P15-1053. [↗](https://aclanthology.org/P15-1053/)
16. Group Pragglejaz. 2007. MIP: A Method for Identifying Metaphorically Used Words in Discourse. Metaphor and Symbol 22(1):1–39. https://www.tandfonline.com/doi/ abs/10.1080/10926480709336752. [↗](https://doi.org/10.1080/10926480709336752)
17. Sunny Rai, Shampa Chakraverty, and Devendra K. Tayal. 2016. Supervised metaphor detection using conditional random fields. In Proceedings of the Fourth Workshop on Metaphor in NLP. Association for Computational Linguistics, San Diego, California, pages 18–27. http://www.aclweb.org/ anthology/W16-1103. [↗](https://aclanthology.org/W16-1103/)
18. Swapna Somasundaran, Jill Burstein, and Martin Chodorow. 2014. [Lexical chaining for measuring discourse coherence quality in test-taker essays](https://md.research.elamur.ai/free/en/papers/somasundaran-2014-lexical-chaining-measuring-discourse-coherence/). In Proceedings of COLING. pages 950– 961. http://aclweb.org/anthology/C/ C14/C14-1090.pdf. [↗](https://aclanthology.org/C14-1090/)
19. Yi Song, Michael Heilman, Beata Beigman Klebanov, and Paul Deane. 2014. Applying argumentation schemes for essay scoring. In Proceedings of the [find](https://scholar.google.com/scholar?q=Yi+Song%2C+Michael+Heilman%2C+Beata+Beigman+Klebanov%2C+and+Paul+Deane%2E+2014%2E+Applying+argumentation+schemes+for+essay+scoring%2E+In+Proceedings+of+the)

*Updated 1 November 2026*