---
title: "Toward Automatic Processing of English Metalanguage (2013)"
description: "Metalanguage is language about language: sentences that talk about a word, a name, a… 0.74. kappa. 28 references, cited by 6 papers here."
authors: ["Shomir Wilson"]
year: 2013
venue: "Proceedings of the Sixth International Joint Conference on Natural Language Processing"
source: "https://aclanthology.org/I13-1091/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wilson-2013-automatic-processing-english-metalanguage/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wilson-2013-automatic-processing-english-metalanguage/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/page/3/) / Toward Automatic Processing of English Metalanguage

# Toward Automatic Processing of English Metalanguage

**Authors:** [Shomir Wilson](https://md.research.elamur.ai/free/en/authors/shomir-wilson/) · **Year:** 2013 · **Venue:** [Proceedings of the Sixth International Joint Conference on Natural Language Processing](https://md.research.elamur.ai/free/en/venues/ijcnlp/) ([IJCNLP 2013](https://md.research.elamur.ai/free/en/venues/ijcnlp/2013/)) · **Publisher:** [Asian Federation of Natural Language Processing](https://md.research.elamur.ai/free/en/publishers/asian-federation-of-natural-language-processing/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/I13-1091/) · [PDF](https://aclanthology.org/I13-1091.pdf) · [OpenAlex](https://openalex.org/W2252264672)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/wilson-2013-automatic-processing-english-metalanguage/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Metalanguage is language about language: sentences that talk about a word, a name, a spelling or a quoted phrase. Such sentences carry direct information about language, yet they confuse parsers and fit poorly into word-sense frameworks. Wilson reports the first attempt to find one common type automatically in English, called *mentioned language*. The data come from an earlier corpus built from English Wikipedia: 629 mention sentences and 1,764 non-mention sentences.

The work has two parts:
- **Detection.** The task is to tell whether a sentence contains a mention. Bag-of-words baselines in Weka were compared with a feature set reduced to about ten high-information "metawords". Its performance roughly reaches the ceiling implied by inter-annotator agreement.
- **Delineation.** The task is to mark the exact mentioned span. This was explored through case studies of three metawords: tree patterns over constituency parses for *term* and *word*, and semantic role labels for *call*.

The span patterns were precise but reliable only as far as the underlying parser or SRL. The results show that automatic processing of metalanguage is feasible.

## Key points
- **Corpus.** It was built by filtering Wikipedia sentences that have a stylistic cue (bold, italics, quotes) near one of 8,735 "mention-significant" words and collocations taken from WordNet. A human then labeled the candidates.
- **Reliability.** Three extra annotators labeled 100 sentences, and Kappa between each pair of them averaged 0.74. Kappa between the primary annotator and their majority vote was 0.90.
- **Detection setup.** Four feature sets (SW, UW, SWSB, UWUB) were run with five classifiers (Naive Bayes, SMO, J48, IBk, Decision Table) under ten-fold cross validation. The MW set kept the top ten stemmed words by information gain, chosen separately inside each training fold.
- **Delineation results.**

  | Metaword | Method | Sentences | Exact spans | Errors |
  |---|---|---|---|---|
  | *term* (noun) | TRegex apposition patterns on Stanford Parser output | 91 | 57 of 59 (~97%) | 2 underlabeled |
  | *word* (noun) | TRegex apposition patterns on Stanford Parser output | 107 | 57 of 64 (~89%) | 3 overlabeled, 4 underlabeled |
  | *call* (verb) | Illinois SRL "attribute of arg1" | 158 | 68 of 85 (~80%) | 16 overlabeled, 1 underlabeled |

- **Detection via patterns.** Pattern firing had precision 1.0 for *term* and *word*. Most errors traced back to parser or SRL mistakes. The three case studies together covered only 33% of the corpus's mention sentences.

## Interesting ideas
- A tiny lexicon of about ten information-gain-selected "metawords" can stand in for a full vocabulary when flagging that a sentence talks about language itself; it is cheap and readable.
- A single structural pattern does two jobs at once: when it matches, it marks the exact span and also counts as evidence that the sentence contains a mention.
- Spans can be scored as exact, overlabeled or underlabeled, kept apart from whether the pattern fired at all, so that boundary errors and detection errors are not mixed together.
- Semantic role arguments tend to "spill" past the true mention boundary, as when a trailing *for*-clause gets absorbed into the naming role.
- Mentioned language is direct, extractable knowledge about language, for example "graupel refers to a kind of precipitation" or "the name is actually Rolla".

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): Tag tokens near metawords such as *term*, *word*, *called* or *spelled* with a "possibly mentioned" feature, so that later stages do not force ordinary senses or morphology on them.
- **I2** (tree): Write explainable UD patterns, such as *term* plus an appositive, or *call* plus an xcomp or object predicate, that both detect and delimit mentioned spans.
- **I3** (learn): Use this three-way scope breakdown when evaluating span-producing stages such as SRL arguments, quotes and named spans, to see whether failures come from boundaries or from recognition.
- **I4** (frames): When filling a naming frame from VerbNet roles, trim the predicate argument to its core noun phrase and drop adjunct clauses, so that *X is called Y* yields a clean name Y.
- **I5** (world): Store detected definitions, alternate names and spellings as lexical facts or alias links for entities in the text, rather than treating the mentioned word as a real-world referent.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Shomir Wilson. 2013. Toward Automatic Processing of English Metalanguage. Proceedings of the Sixth International Joint Conference on Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wilson2013toward,
  title = {{Toward Automatic Processing of English Metalanguage}},
  author = {Shomir Wilson},
  booktitle = {Proceedings of the Sixth International Joint Conference on Natural Language Processing},
  year = {2013},
  url = {https://aclanthology.org/I13-1091/}
}
```

- [← previous Supersense Tagging for Arabic: the MT-in-the-Middle Attack](https://md.research.elamur.ai/free/en/papers/schneider-2013-supersense-tagging-arabic-mt-middle/)
- [next → A Discriminative Analysis of Fine-Grained Semantic Relations including Presupposition: Annotation and Classification](https://md.research.elamur.ai/free/en/papers/tremper-2013-discriminative-analysis-fine-grained-semantic/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/page/3/)

## References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adler, B. T., de Alfaro, L., Pye, I., & Raman, V. (2008). Measuring author contributions to the Wikipedia. In Proceedings of the 4th International Symposium on Wikis (pp. 15:1–15:10). New York, NY, USA: ACM. doi:10.1145/1822258.1822279 [↗](https://doi.org/10.1145/1822258.1822279)
2. Aha, D. W., & Kibler, D. (1991). [Instance-based learning algorithms](https://md.research.elamur.ai/free/en/papers/aha-1991-instance-based-learning-algorithms/). In Machine Learning (pp. 37– 66). [↗](https://doi.org/10.1007/bf00153759)
3. Anderson, M. L., Fister, A., Lee, B., & Wang, D. (2004). On the frequency and types of meta-language in conversation: A preliminary report. In 14th Annual Conference of the Society for Text and Discourse. [find](https://scholar.google.com/scholar?q=Anderson%2C+M%2E+L%2E%2C+Fister%2C+A%2E%2C+Lee%2C+B%2E%2C+%26+Wang%2C+D%2E+%282004%29%2E+On+the+frequency+and+types+of+meta%2Dlanguage+in+conversation%3A+A+preliminary+report%2E+In+14th+Annual+Conference+of+the+Society+for+Text+and+Discou)
4. Anderson, M. L., Okamoto, Y. A., Josyula, D., & Perlis, D. (2002). The Use-Mention Distinction and Its Importance to HCI. In Proceedings of the Sixth Workshop on the Semantics and Pragmatics of Dialog, 21–28. [find](https://scholar.google.com/scholar?q=Anderson%2C+M%2E+L%2E%2C+Okamoto%2C+Y%2E+A%2E%2C+Josyula%2C+D%2E%2C+%26+Perlis%2C+D%2E+%282002%29%2E+The+Use%2DMention+Distinction+and+Its+Importance+to+HCI%2E+In+Proceedings+of+the+Sixth+Workshop+on+the+Semantics+and+Pragmatics+of+Dialog)
5. Audi, R. (1995). The Cambridge Dictionary of Philosophy. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Audi%2C+R%2E+%281995%29%2E+The+Cambridge+Dictionary+of+Philosophy%2E+Cambridge+University+Press%2E)
6. Bird, S. (2006). [NLTK: The natural language toolkit](https://md.research.elamur.ai/free/en/papers/bird-2006-nltk-natural-language-toolkit/). In Proceedings of the COLING/ACL on Interactive Presentation Sessions (pp. 69–72). Stroudsburg, PA, USA: Association for Computational Linguistics.
7. Cappelen, H., & Lepore, E. (1997). Varieties of quotation. Mind, 106(423), 429–450. doi:10.1093/mind/106.423.429 [↗](https://doi.org/10.1093/mind/106.423.429)
8. Davidson, D. (1979). Quotation. Theory and Decision, 11(1), 27–40. doi:10.1007/BF00126690 [↗](https://doi.org/10.1007/bf00126690)
9. Fellbaum, C. (1998). WordNet: An electronic lexical database. Cambridge: MIT Press. [find](https://scholar.google.com/scholar?q=Fellbaum%2C+C%2E+%281998%29%2E+WordNet%3A+An+electronic+lexical+database%2E+Cambridge%3A+MIT+Press%2E)
10. García-Carpintero, M. (2004). The deferred ostension theory of quotation. Noûs, 38(4), 674–692. [↗](https://doi.org/10.1111/j.0029-4624.2004.00488.x)
11. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA data mining software: An update. ACM SIGKDD Explorations Newsletter, 11, 10–18. doi:10.1145/ 1656274.1656278 [↗](https://doi.org/10.1145/1656274.1656278)
12. Hu, G. (2010). A place for metalanguage in the L2 classroom. ELT Journal. doi:10.1093/elt/ccq037 [↗](https://doi.org/10.1093/elt/ccq037)
13. Jaworski, A., Coupland, N., & Galasinski, D. (Eds.). (2004). Metalanguage: Language, Power, and Social Process. De Gruyter. [find](https://scholar.google.com/scholar?q=Jaworski%2C+A%2E%2C+Coupland%2C+N%2E%2C+%26+Galasinski%2C+D%2E+%28Eds%2E%29%2E+%282004%29%2E+Metalanguage%3A+Language%2C+Power%2C+and+Social+Process%2E+De+Gruyter%2E)
14. John, G., & Langley, P. (1995). Estimating Continuous Distributions in Bayesian Classifiers. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence (pp. 338–345). Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=John%2C+G%2E%2C+%26+Langley%2C+P%2E+%281995%29%2E+Estimating+Continuous+Distributions+in+Bayesian+Classifiers%2E+In+Proceedings+of+the+Eleventh+Conference+on+Uncertainty+in+Artificial+Intelligence+%28pp%2E+338%E2%80%93345%29%2E+Morgan+K)
15. Keerthi, S. S., Shevade, S. K., Bhattacharyya, C., & Murthy, K. R. K. (2001). Improvements to Platt’s SMO Algorithm for SVM Classifier Design. Neural Computation, 13, 637–649. doi:10.1162/ 089976601300014493 [↗](https://doi.org/10.1162/089976601300014493)
16. Kohavi, R. (1995). The Power of Decision Tables. In Proceedings of the European Conference on Machine Learning (pp. 174–189). Springer Verlag. [find](https://scholar.google.com/scholar?q=Kohavi%2C+R%2E+%281995%29%2E+The+Power+of+Decision+Tables%2E+In+Proceedings+of+the+European+Conference+on+Machine+Learning+%28pp%2E+174%E2%80%93189%29%2E+Springer+Verlag%2E)
17. Levy, R., & Andrew, G. (2006). TRegex and TSurgeon: tools for querying and manipulating tree data structures. In Proceedings of the Fifth International Conference on Language Resources and Evaluation. [find](https://scholar.google.com/scholar?q=Levy%2C+R%2E%2C+%26+Andrew%2C+G%2E+%282006%29%2E+TRegex+and+TSurgeon%3A+tools+for+querying+and+manipulating+tree+data+structures%2E+In+Proceedings+of+the+Fifth+International+Conference+on+Language+Resources+and+Evaluation%2E)
18. Maier, E. (2007). Mixed quotation: between use and mention. In Logic and Engineering of Natural Language Semantics Workshop. Retrieved from [http://ncs.ruhosting.nl/emar/em_lenls_quot.pdf](http://ncs.ruhosting.nl/emar/em_lenls_quot.pdf) [↗](http://ncs.ruhosting.nl/emar/em_lenls_quot.pdf)
19. Marneffe, M. D., MacCartney, B., & Manning, C. D. (2006). [Generating typed dependency parses from phrase structure parses](https://md.research.elamur.ai/free/en/papers/marneffe-2006-generating-typed-dependency-parses-phrase/). In Proceedings of the International Conference on Language Resources and Evaluation, pp. 449–454.
20. Punyakanok, V., Roth, D., & Yih, W. (2008). [The importance of syntactic parsing and inference in semantic role labeling](https://md.research.elamur.ai/free/en/papers/punyakanok-2008-importance-syntactic-parsing-inference-semantic/). Computational Linguistics, 34(2), 257–287. doi:10.1162/coli.2008.34.2.257 [↗](https://doi.org/10.1162/coli.2008.34.2.257)
21. Quine, W. V. O. (1940). Mathematical logic. Cambridge, MA: Harvard University Press. [find](https://scholar.google.com/scholar?q=Quine%2C+W%2E+V%2E+O%2E+%281940%29%2E+Mathematical+logic%2E+Cambridge%2C+MA%3A+Harvard+University+Press%2E)
22. Quinlan, J. (1993). [C4.5: Programs for Machine Learning](https://md.research.elamur.ai/free/en/papers/quinlan-1993-c45-programs-machine-learning/). San Mateo, Calif.: Morgan Kaufmann Publishers.
23. Saka, P. (2006). The Demonstrative and Identity Theories of Quotation. Journal of Philosophy, 103(9), 452–471. [↗](https://doi.org/10.5840/jphil2006103920)
24. Sperber, D., & Wilson, D. (1981). Irony and the Use-Mention Distinction. In Radical Pragmatics (pp. 295– 318). New York. [find](https://scholar.google.com/scholar?q=Sperber%2C+D%2E%2C+%26+Wilson%2C+D%2E+%281981%29%2E+Irony+and+the+Use%2DMention+Distinction%2E+In+Radical+Pragmatics+%28pp%2E+295%E2%80%93+318%29%2E+New+York%2E)
25. Tarski, A. (1933). The concept of truth in formalized languages. In J. H. Woodger (Ed.), Logic, Semantics, Mathematics. Oxford: Oxford University Press. [find](https://scholar.google.com/scholar?q=Tarski%2C+A%2E+%281933%29%2E+The+concept+of+truth+in+formalized+languages%2E+In+J%2E+H%2E+Woodger+%28Ed%2E%29%2C+Logic%2C+Semantics%2C+Mathematics%2E+Oxford%3A+Oxford+University+Press%2E)
26. Wilson, S. (2011a). A computational theory of the use-mention distinction in natural language. University of Maryland at College Park. PhD Thesis, College Park, MD, USA. [find](https://scholar.google.com/scholar?q=Wilson%2C+S%2E+%282011a%29%2E+A+computational+theory+of+the+use%2Dmention+distinction+in+natural+language%2E+University+of+Maryland+at+College+Park%2E+PhD+Thesis%2C+College+Park%2C+MD%2C+USA%2E)
27. Wilson, S. (2011b). In search of the use-mention distinction and its impact on language processing tasks. International Journal of Computational Linguistics and Applications, 2(1-2), 139–154. [find](https://scholar.google.com/scholar?q=Wilson%2C+S%2E+%282011b%29%2E+In+search+of+the+use%2Dmention+distinction+and+its+impact+on+language+processing+tasks%2E+International+Journal+of+Computational+Linguistics+and+Applications%2C+2%281%2D2%29%2C+139%E2%80%93154%2E)
28. Wilson, S. (2012). [The Creation of a Corpus of English Metalanguage](https://md.research.elamur.ai/free/en/papers/wilson-2012-creation-corpus-english-metalanguage/). In Proceedings of the 50 th Annual Meeting of the Association for Computational Linguistics, (pp. 638–646).

*Updated 1 November 2026*