---
title: "The Creation of a Corpus of English Metalanguage — Wilson 2012"
description: "The paper addresses metalanguage, meaning language used to talk about language itself, as in… 24 references, cited by 8 papers here."
authors: ["Shomir Wilson"]
year: 2012
venue: "Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P12-1067/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wilson-2012-creation-corpus-english-metalanguage/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wilson-2012-creation-corpus-english-metalanguage/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/page/2/) / The Creation of a Corpus of English Metalanguage

# The Creation of a Corpus of English Metalanguage

**Authors:** [Shomir Wilson](https://md.research.elamur.ai/free/en/authors/shomir-wilson/) · **Year:** 2012 · **Venue:** [Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2012](https://md.research.elamur.ai/free/en/venues/acl/2012/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P12-1067/) · [PDF](https://aclanthology.org/P12-1067.pdf) · [OpenAlex](https://openalex.org/W2105187440)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/wilson-2012-creation-corpus-english-metalanguage/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses metalanguage, meaning language used to talk about language itself, as in sentences that discuss a word rather than its referent. NLP tools largely ignore this. Parsers mis-tag mentioned words, sentiment systems misread quoted terms, and dialog systems fail when a user corrects a misrecognized word. Wilson focuses on the tractable subtype called *mentioned language*. He gives a formal definition: tokens whose purpose is to highlight some feature of the tokens themselves or of their type. Because annotators found that definition too abstract, he adds a practical substitution rubric. Using English Wikipedia, whose writers often mark mentioned words with quote marks, italics or bold, he builds three successively richer corpora. The last, the Enhanced Cues Corpus, is the first to mark the exact word span that is mentioned. Expert re-annotation shows good agreement on whether mention is present but weaker agreement on its subtype. The resource lays groundwork for automatically detecting and delineating metalanguage.

## Key points
- Mentioned language is defined as a token or token set produced to highlight a property of that token or its type, such as spelling, pronunciation, meaning, connotation or source.
- The annotation rubric replaces a candidate X with "that [item]" (word, name, phrase, letter…). X counts as mentioned if the sentence's truth conditions stay the same when the phrase points back to X.
- The source is English Wikipedia. Stylistic cues (quotes, italic, bold) were used to collect candidates, which were then labeled with five categories: WW, NN, SP, OM and XX (not mentioned).
- Reliability check: three extra expert annotators labeled 100 sampled instances. All four annotators agreed on 46 positives and 30 negatives, with an average pairwise Kappa of 0.74 for presence of mention.
- Context and outlook:
  - The Stanford Parser tags *go* in "The button labeled go" as VB.
  - An earlier BNC-based study (Anderson et al. 2004) found metalanguage in about 11% of spoken utterances, but it lacked span-level labels.
  - Preliminary detection with bag-of-words plus a decision tree catches about 70% of instances.

## Interesting ideas
- A deictic substitution test ("replace X with *that word* and check whether the meaning holds") turns a vague semantic notion into a repeatable judgment that annotators can apply.
- Mentioned spans often attach to a small, closed set of mention-significant nouns (word, name, term) through apposition, or fill specific roles of verbs like *call* and *spell*.
- Stylistic cues (quotes, italics, bold) act as a cheap high-recall filter for mining candidates, which humans then judge. Redacting those cues during annotation probes how much judgments depend on them.
- Agreement can be measured twice: once on a coarse binary question (is this mention at all?) and once on fine subtypes, which shows where annotator consensus breaks down.
- Words that appear inside mention or quotation do not express the writer's own stance. A user turn like "No, I said 'Arlington'" is a correction about a word, not new content.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): Run the "that [item]" check when filling frame roles, so a span like *Mississippi* in "fun to spell" becomes a linguistic-token argument rather than a geographic entity.
- **I2** (tree): Add repair rules that re-attach and re-tag the mentioned span as a single quoted unit after nouns like *verb* or *word*, fixing parses such as *go*/VB in "labeled go".
- **I3** (learn): Mine Wikipedia-style highlighted spans as silver candidates for mention detection, and use cue-stripped copies as a negative-control test of whether the model relies only on punctuation.
- **I4** (learn): Score witness-parser or LLM-teacher agreement on collapsed labels and fine labels separately, to see whether disagreement sits in detection or in classification.
- **I5** (world): Store mentioned strings as language objects linked to their referents, so sentiment, attribution and dialog-repair reasoning do not treat a quoted term as a fact or opinion of the speaker.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Shomir Wilson. 2012. The Creation of a Corpus of English Metalanguage. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wilson2012creation,
  title = {{The Creation of a Corpus of English Metalanguage}},
  author = {Shomir Wilson},
  booktitle = {Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2012},
  url = {https://aclanthology.org/P12-1067/}
}
```

- [← previous Active Learning for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/laws-2012-active-learning-coreference-resolution/)
- [next → Alternative Lexicalizations of Discourse Connectives in Czech](https://md.research.elamur.ai/free/en/papers/rysova-2012-alternative-lexicalizations-discourse-connectives-czech/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/page/2/)

## References

24 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adler, B. Thomas, Luca de Alfaro, Ian Pye & Vishwanath Raman. 2008. Measuring author contributions to the Wikipedia. In Proc. of WikiSym '08. New York, NY, USA: ACM. [find](https://scholar.google.com/scholar?q=Adler%2C+B%2E+Thomas%2C+Luca+de+Alfaro%2C+Ian+Pye+%26+Vishwanath+Raman%2E+2008%2E+Measuring+author+contributions+to+the+Wikipedia%2E+In+Proc%2E+of+WikiSym+%2708%2E+New+York%2C+NY%2C+USA%3A+ACM%2E)
2. American Psychological Association. 2001. Publication Manual of the American Psychological Association. 5th ed. Washington, DC: American Psychological Association. [find](https://scholar.google.com/scholar?q=American+Psychological+Association%2E+2001%2E+Publication+Manual+of+the+American+Psychological+Association%2E+5th+ed%2E+Washington%2C+DC%3A+American+Psychological+Association%2E)
3. Anderson, Michael L, Yoshi A Okamoto, Darsana Josyula & Donald Perlis. 2002. The use-mention distinction and its importance to HCI. In Proc. of EDILOG 2002. 21–28. [find](https://scholar.google.com/scholar?q=Anderson%2C+Michael+L%2C+Yoshi+A+Okamoto%2C+Darsana+Josyula+%26+Donald+Perlis%2E+2002%2E+The+use%2Dmention+distinction+and+its+importance+to+HCI%2E+In+Proc%2E+of+EDILOG+2002%2E+21%E2%80%9328%2E)
4. Anderson, Michael L., Andrew Fister, Bryant Lee & Danny Wang. 2004. On the frequency and types of meta-language in conversation: A preliminary report. In Proc. of the 14th Annual Conference of the Society for Text & Discourse. [find](https://scholar.google.com/scholar?q=Anderson%2C+Michael+L%2E%2C+Andrew+Fister%2C+Bryant+Lee+%26+Danny+Wang%2E+2004%2E+On+the+frequency+and+types+of+meta%2Dlanguage+in+conversation%3A+A+preliminary+report%2E+In+Proc%2E+of+the+14th+Annual+Conference+of+the+Soc)
5. Cappelen, H & E Lepore. 1997. Varieties of quotation. Mind 106(423). 429 –450. [↗](https://doi.org/10.1093/mind/106.423.429)
6. Chicago Editorial Staff. 2010. The Chicago Manual of Style. 16th ed. University of Chicago Press. [find](https://scholar.google.com/scholar?q=Chicago+Editorial+Staff%2E+2010%2E+The+Chicago+Manual+of+Style%2E+16th+ed%2E+University+of+Chicago+Press%2E)
7. Clark, Herbert H. & Edward F. Schaefer. 1989. Contributing to discourse. Cognitive Science 13(2). 259–294. [↗](https://doi.org/10.1016/0364-0213(89)90008-6)
8. Davidson, Donald. 1979. Quotation. Theory and Decision 11(1). 27–40. [find](https://scholar.google.com/scholar?q=Davidson%2C+Donald%2E+1979%2E+Quotation%2E+Theory+and+Decision+11%281%29%2E+27%E2%80%9340%2E)
9. Fellbaum, Christiane. 1998. WordNet: An Electronic Lexical Database. Cambridge: MIT Press. [find](https://scholar.google.com/scholar?q=Fellbaum%2C+Christiane%2E+1998%2E+WordNet%3A+An+Electronic+Lexical+Database%2E+Cambridge%3A+MIT+Press%2E)
10. García-Carpintero, Manuel. 2004. The deferred ostension theory of quotation. Noûs 38(4). 674–692. [↗](https://doi.org/10.1111/j.0029-4624.2004.00488.x)
11. Klein, Dan & Christopher D. Manning. 2003. Fast exact inference with a factored model for natural language parsing. Advances in Neural Information Processing Systems 15. [find](https://scholar.google.com/scholar?q=Klein%2C+Dan+%26+Christopher+D%2E+Manning%2E+2003%2E+Fast+exact+inference+with+a+factored+model+for+natural+language+parsing%2E+Advances+in+Neural+Information+Processing+Systems+15%2E)
12. Loper, Edward & Steven Bird. 2002. NLTK: The Natural Language Toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics 1. 63–70. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.cs/0205028)
13. Maier, Emar. 2007. Mixed quotation: Between use and mention. In Proc. of LENLS 2007. [find](https://scholar.google.com/scholar?q=Maier%2C+Emar%2E+2007%2E+Mixed+quotation%3A+Between+use+and+mention%2E+In+Proc%2E+of+LENLS+2007%2E)
14. Partee, Barbara. 1973. The syntax and semantics of quotation. In Stephen Anderson & Paul Kiparsky (eds.), A Festschrift for Morris Halle. New York: Holt, Rinehart, Winston. [find](https://scholar.google.com/scholar?q=Partee%2C+Barbara%2E+1973%2E+The+syntax+and+semantics+of+quotation%2E+In+Stephen+Anderson+%26+Paul+Kiparsky+%28eds%2E%29%2C+A+Festschrift+for+Morris+Halle%2E+New+York%3A+Holt%2C+Rinehart%2C+Winston%2E)
15. Perlis, Donald, Khemdut Purang & Carl Andersen. 1998. Conversational adequacy: Mistakes are the essence. International Journal of Human-Computer Studies 48(5). 553–575. [↗](https://doi.org/10.1006/ijhc.1997.0181)
16. Quine, W. V. O. 1940. Mathematical Logic. Cambridge, MA: Harvard University Press. [find](https://scholar.google.com/scholar?q=Quine%2C+W%2E+V%2E+O%2E+1940%2E+Mathematical+Logic%2E+Cambridge%2C+MA%3A+Harvard+University+Press%2E)
17. Raux, Antoine, Brian Langner, Dan Bohus, Alan W Black & Maxine Eskenazi. 2005. Let’s Go public! Taking a spoken dialog system to the real world. In Proc. of Interspeech 2005. [find](https://scholar.google.com/scholar?q=Raux%2C+Antoine%2C+Brian+Langner%2C+Dan+Bohus%2C+Alan+W+Black+%26+Maxine+Eskenazi%2E+2005%2E+Let%E2%80%99s+Go+public%21+Taking+a+spoken+dialog+system+to+the+real+world%2E+In+Proc%2E+of+Interspeech+2005%2E)
18. Saka, Paul. 1998. Quotation and the use-mention distinction. Mind 107(425). 113 –135. [↗](https://doi.org/10.1093/mind/107.425.113)
19. Saka, Paul. 2003. Quotational constructions. Belgian Journal of Linguistics 17(1). [find](https://scholar.google.com/scholar?q=Saka%2C+Paul%2E+2003%2E+Quotational+constructions%2E+Belgian+Journal+of+Linguistics+17%281%29%2E)
20. Strunk, Jr. & E. B. White. 1979. The Elements of Style, Third Edition. Macmillan. [find](https://scholar.google.com/scholar?q=Strunk%2C+Jr%2E+%26+E%2E+B%2E+White%2E+1979%2E+The+Elements+of+Style%2C+Third+Edition%2E+Macmillan%2E)
21. Tarski, Alfred. 1933. The concept of truth in formalized languages. In J. H. Woodger (ed.), Logic, Semantics, Mathematics. Oxford: Oxford University Press. [find](https://scholar.google.com/scholar?q=Tarski%2C+Alfred%2E+1933%2E+The+concept+of+truth+in+formalized+languages%2E+In+J%2E+H%2E+Woodger+%28ed%2E%29%2C+Logic%2C+Semantics%2C+Mathematics%2E+Oxford%3A+Oxford+University+Press%2E)
22. Wilson, Shomir. 2010. Distinguishing use and mention in natural language. In Proc. of the NAACL HLT 2010 Student Research Workshop, 29–33. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wilson%2C+Shomir%2E+2010%2E+Distinguishing+use+and+mention+in+natural+language%2E+In+Proc%2E+of+the+NAACL+HLT+2010+Student+Research+Workshop%2C+29%E2%80%9333%2E+Association+for+Computational+Linguistics%2E)
23. Wilson, Shomir. 2011a. A Computational Theory of the Use-Mention Distinction in Natural Language. Ph.D. dissertation, University of Maryland at College Park. [find](https://scholar.google.com/scholar?q=Wilson%2C+Shomir%2E+2011a%2E+A+Computational+Theory+of+the+Use%2DMention+Distinction+in+Natural+Language%2E+Ph%2ED%2E+dissertation%2C+University+of+Maryland+at+College+Park%2E)
24. Wilson, Shomir. 2011b. In search of the use-mention distinction and its impact on language processing tasks. International Journal of Computational Linguistics and Applications 2(1-2). 139–154. [find](https://scholar.google.com/scholar?q=Wilson%2C+Shomir%2E+2011b%2E+In+search+of+the+use%2Dmention+distinction+and+its+impact+on+language+processing+tasks%2E+International+Journal+of+Computational+Linguistics+and+Applications+2%281%2D2%29%2E+139%E2%80%93154%2E)

*Updated 1 November 2026*