--- title: "MASC: the Manually Annotated Sub-Corpus of American English" description: "This paper describes the design and early work on MASC, a sub-corpus of public-domain American… 16 references, cited by 44 papers here." authors: ["Nancy Ide", "Collin Baker", "Christiane Fellbaum", "Charles Fillmore", "Rebecca Passonneau"] year: 2008 venue: "Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)" source: "https://aclanthology.org/L08-1105/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ide-2008-masc-manually-annotated-sub-corpus/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ide-2008-masc-manually-annotated-sub-corpus/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2008](https://md.research.elamur.ai/free/en/papers/2008/) / MASC: the Manually Annotated Sub-Corpus of American English # MASC: the Manually Annotated Sub-Corpus of American English **Authors:** [Nancy Ide](https://md.research.elamur.ai/free/en/authors/nancy-ide/), [Collin Baker](https://md.research.elamur.ai/free/en/authors/collin-f-baker/), [Christiane Fellbaum](https://md.research.elamur.ai/free/en/authors/christiane-fellbaum/), [Charles Fillmore](https://md.research.elamur.ai/free/en/authors/charles-j-fillmore/) and [Rebecca Passonneau](https://md.research.elamur.ai/free/en/authors/rebecca-j-passonneau/) · **Year:** 2008 · **Venue:** [Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2008](https://md.research.elamur.ai/free/en/venues/lrec/2008/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/L08-1105/) · [PDF](https://aclanthology.org/L08-1105.pdf) · [OpenAlex](https://openalex.org/W191693080) **Citations:** 75 citations (OpenAlex, 2026-10-14) · [44](https://md.research.elamur.ai/free/en/papers/ide-2008-masc-manually-annotated-sub-corpus/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Annotated corpora from different groups tend to be hard to reuse. Each one has its own formats and addressing schemes, and few cover more than a narrow range of genres or annotation layers. This paper describes the design and early work on MASC, a sub-corpus of public-domain American English drawn from the Open ANC, covering multiple genres. It will carry manual or human-checked annotations at several levels: WordNet senses, FrameNet frames and frame elements, shallow parses and named entities. Placing these layers on the same sentences lets lexical senses, frames and their entity fillers combine into richer semantic structures. The workflow alternates manual annotation of small, representative portions with retraining of automatic taggers, which are then applied to the full ANC. All layers are stored as separate stand-off files in a graph-based, feature-structure format, and a tool merges selected layers into the output format the user needs. Agreement statistics will be published for every layer. The authors expect MASC to become the largest semantically annotated English corpus, open to community contributions and linked to wordnets and framenets in other languages. ## Key points - The source material is the 15-million-word Open ANC (OANC), which is unrestricted, public-domain text. MASC and its annotations are to be freely downloadable from the ANC website. - Planned layers: - WordNet 3.0 sense tags on all content words, first assigned by existing WSD systems (e.g., SenseRelate and SenseLearner, the latter second in the SENSEVAL-3 English all-words task), then corrected by undergraduate annotators. - FrameNet full-text annotation on a smaller subset, plus automatic semantic role labeling over all of MASC. - Validated shallow parses and named entities. - Representation: - The format follows ISO TC37 SC4's Linguistic Annotation Framework, with each layer in its own stand-off document over read-only primary text. - The original annotation labels are kept, without linguistic harmonization. - ANCTool merges the chosen layers and exports to XML (for XAIRA), NLTK-friendly formats, GrAF, GraphViz input, and UIMA (planned). - Annotation quality is measured with Cohen's kappa (agreement beyond chance) and average F-measure (share of data all annotators agree on). These are combined with feedback from downstream users of the data. - A FrameNet example on a Dublin travel-guide sentence about the River Liffey shows three frames: Fluidic_motion, Part_inner_outer and Natural_feature. Composing them supports inferences such as a river named Liffey flowing through a city's inner part to a bay. ## Interesting ideas - Keep every annotation layer as its own stand-off file pointing at untouchable raw text, and merge only on demand; independent layers stay intact and new ones can be added without rewriting old ones. - Use an existing automatic system to pre-tag everything, have humans correct a smaller representative portion, retrain on the corrections, then reapply to the whole collection. This makes cheap automatic output progressively more trustworthy. - Check whether gold annotations on one layer improve automatic accuracy on another, as when validated senses and noun chunks help role labeling. This reveals which layers are worth correcting first. - Judge annotation quality with two complementary numbers, chance-corrected agreement and plain overlap F-measure, and also test whether different annotators' versions change downstream task results. - Compose frame annotations on one sentence (motion, part-whole, named natural feature) to derive explicit facts about entities and their relations. This shows how layered frames can become a small knowledge base. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tree): Store UD trees from different parsers and converted treebanks as separate layers over a shared tokenized text, so dialect conversions and parser comparisons never overwrite the source. - **I2** (learn): Fits the silver-data loop: witness parsers pre-annotate, a human or LLM teacher fixes a sampled slice, and retrained models relabel the full corpus. - **I3** (frames): Measure how much gold lemmas, senses or trees raise VerbNet role assignment in prag, to prioritize which upstream stage to repair. - **I4** (learn): Report both kappa-style and F-style agreement between witness parsers on silver data, and confirm usefulness by training on each witness's output separately. - **I5** (world): Turn co-occurring frames and their named fillers into world-of-text entities and relations, such as a river named Liffey flowing through a city, which the answer stage can then query. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 175 works · 227,440 citations - [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 241 works · 54,053 citations - [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 90 works · 39,109 citations - [International Computer Science Institute](https://md.research.elamur.ai/free/en/labs/international-computer-science-institute/) — Research institute, Berkeley, United States · 15 works · 11,471 citations - [Vassar College](https://md.research.elamur.ai/free/en/universities/vassar-college/) — University, Town of Poughkeepsie, United States · 6 works · 1,118 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works ## Cite Cite as: Nancy Ide, Collin Baker, Christiane Fellbaum, Charles Fillmore and Rebecca Passonneau. 2008. MASC: the Manually Annotated Sub-Corpus of American English. Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{ide2008masc, title = {{MASC: the Manually Annotated Sub-Corpus of American English}}, author = {Nancy Ide, Collin Baker, Christiane Fellbaum, Charles Fillmore and Rebecca Passonneau}, booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)}, year = {2008}, doi = {10.63317/2s5uigy45uwu}, url = {https://doi.org/10.63317/2s5uigy45uwu} } ``` - [← previous Shift-Reduce Dependency DAG Parsing](https://md.research.elamur.ai/free/en/papers/sagae-2008-shift-reduce-dependency-dag-parsing/) - [next → Constructing Corpora for the Development and Evaluation of Paraphrase Systems](https://md.research.elamur.ai/free/en/papers/cohn-2008-constructing-corpora-development-evaluation-paraphrase/) - [↑ back to papers of 2008](https://md.research.elamur.ai/free/en/papers/2008/) ## References 16 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Baker, C.; Ellsworth, M. & Erk, K. (2007). [SemEval-2007 Task 19: Frame Semantic Structure Extraction](https://md.research.elamur.ai/free/en/papers/baker-2007-semeval-2007-task-19-frame/) Proceedings of the SemEval 2007 Workshop, Association for Computational Linguistics, 99-104. 2. Cohen, J. 1960. A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20: 37–46. [find](https://scholar.google.com/scholar?q=Cohen%2C+J%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2C+Educational+and+Psychological+Measurement+20%3A+37%E2%80%9346%2E) 3. Erk, K. & Padó, S. (2006). Shalmaneser -- A Flexible Toolbox For Semantic Role Assignment. Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC-2006), 527-332. [find](https://scholar.google.com/scholar?q=Erk%2C+K%2E+%26+Pad%C3%B3%2C+S%2E+%282006%29%2E+Shalmaneser+%2D%2D+A+Flexible+Toolbox+For+Semantic+Role+Assignment%2E+Proceedings+of+the+Fifth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%2D2006%29%2C+527%2D332%2E) 4. Fillmore, C. J. (1976). Frame semantics and the nature of language Annals of the New York Academy of Sciences: Conference on the Origin and Development of Language and Speech, Vol. 280, 20-32. [find](https://scholar.google.com/scholar?q=Fillmore%2C+C%2E+J%2E+%281976%29%2E+Frame+semantics+and+the+nature+of+language+Annals+of+the+New+York+Academy+of+Sciences%3A+Conference+on+the+Origin+and+Development+of+Language+and+Speech%2C+Vol%2E+280%2C+20%2D32%2E) 5. Fillmore, C. J.; Baker, C. F. & Sato, H. (2004). Framenet as a "Net" Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC), 1091-1094. [find](https://scholar.google.com/scholar?q=Fillmore%2C+C%2E+J%2E%3B+Baker%2C+C%2E+F%2E+%26+Sato%2C+H%2E+%282004%29%2E+Framenet+as+a+%22Net%22+Proceedings+of+the+Fourth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%29%2C+1091%2D1094%2E) 6. Ide, N. and Suderman, K. (2007). [GrAF: A Graph-based Format for Linguistic Annotations](https://md.research.elamur.ai/free/en/papers/ide-2007-graf-graph-based-format-linguistic/). Proceedings of the Linguistic Annotation Workshop, Association for Computational Linguistics, 1-8. 7. Johansson, R. & Nugues, P. (2007). LTH: Semantic Structure Extraction using Nonprojective Dependency Trees Proceedings of the SemEval 2007 Workshop, Association for Computational Linguistics, 227-230. [find](https://scholar.google.com/scholar?q=Johansson%2C+R%2E+%26+Nugues%2C+P%2E+%282007%29%2E+LTH%3A+Semantic+Structure+Extraction+using+Nonprojective+Dependency+Trees+Proceedings+of+the+SemEval+2007+Workshop%2C+Association+for+Computational+Linguistics%2C+227%2D230%2E) 8. Krippendorff, K. (1980.) Content Analysis: An Introduction to its Methodology. Sage Publications, Beverly Hills, CA. [find](https://scholar.google.com/scholar?q=Krippendorff%2C+K%2E+%281980%2E%29+Content+Analysis%3A+An+Introduction+to+its+Methodology%2E+Sage+Publications%2C+Beverly+Hills%2C+CA%2E) 9. Litkowski, K. Mihalcea, R. & Edmonds, P. (ed.) (2004). Senseval-3 task: Automatic labeling of semantic roles. Proceedings of SENSEVAL-3: Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, Association for Computational Linguistics, 9-12. Lönneker-Rodman, B. & Baker, C. F. The FrameNet Model and its Applications. (ms.) [find](https://scholar.google.com/scholar?q=Litkowski%2C+K%2E+Mihalcea%2C+R%2E+%26+Edmonds%2C+P%2E+%28ed%2E%29+%282004%29%2E+Senseval%2D3+task%3A+Automatic+labeling+of+semantic+roles%2E+Proceedings+of+SENSEVAL%2D3%3A+Third+International+Workshop+on+the+Evaluation+of+Systems+for+t) 10. Mihalcea, R. & Edmonds, P. (ed.) (2004). Proceedings of SENSEVAL-3: Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mihalcea%2C+R%2E+%26+Edmonds%2C+P%2E+%28ed%2E%29+%282004%29%2E+Proceedings+of+SENSEVAL%2D3%3A+Third+International+Workshop+on+the+Evaluation+of+Systems+for+the+Semantic+Analysis+of+Text%2E+Association+for+Computational+Linguisti) 11. Passonneau, R., Habash, N., Rambow, O. 2006. Inter-annotator agreement on a multilingual semantic annotation task. Proceedings of the Fifith International Conference on Language Resources and Evaluation (LREC). Genoa, Italy. [find](https://scholar.google.com/scholar?q=Passonneau%2C+R%2E%2C+Habash%2C+N%2E%2C+Rambow%2C+O%2E+2006%2E+Inter%2Dannotator+agreement+on+a+multilingual+semantic+annotation+task%2E+Proceedings+of+the+Fifith+International+Conference+on+Language+Resources+and+Evaluati) 12. Passonneau, R., Nenkova, A., McKeown, K. and Sigelman, S. 2005. Applying the pyramid method in duc 2005. Proceedings of the 2005 Workshop of the Document Understanding Conference (DUC). [↗](https://doi.org/10.7916/d8tx3pvd) 13. Pradhan, S. S.; Ward, W. H.; Hacioglu, K.; Martin, J. H. & Jurafsky, D. Susan Dumais, D. M. & Roukos, S. (ed.) (2004). [Shallow Semantic Parsing using Support Vector Machines](https://md.research.elamur.ai/free/en/papers/pradhan-2004-shallow-semantic-parsing-support-vector/) HLT-NAACL 2004: Main Proceedings, Association for Computational Linguistics, 233-240. 14. Scheffczyk, J.; Baker, C. F. & Narayanan, S. Oltramari, A. (ed.) (2006). Ontology-based Reasoning about Lexical Resources. Proceedings of ONTOLEX 2006, 1-8 [find](https://scholar.google.com/scholar?q=Scheffczyk%2C+J%2E%3B+Baker%2C+C%2E+F%2E+%26+Narayanan%2C+S%2E+Oltramari%2C+A%2E+%28ed%2E%29+%282006%29%2E+Ontology%2Dbased+Reasoning+about+Lexical+Resources%2E+Proceedings+of+ONTOLEX+2006%2C+1%2D8) 15. Snyder, B. and Palmer, M. (2004). The English all-words task. In Mihalcea, R., Edmonds, P. (Eds.), Senseval-3: Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text. Association for Computational Linguistics, 41-43. [find](https://scholar.google.com/scholar?q=Snyder%2C+B%2E+and+Palmer%2C+M%2E+%282004%29%2E+The+English+all%2Dwords+task%2E+In+Mihalcea%2C+R%2E%2C+Edmonds%2C+P%2E+%28Eds%2E%29%2C+Senseval%2D3%3A+Third+International+Workshop+on+the+Evaluation+of+Systems+for+the+Semantic+Analysis+of+Te) 16. van Rijsbergen, C. J. 1979. Information Retrieval. 2 nd edition, London: Butterworths. [find](https://scholar.google.com/scholar?q=van+Rijsbergen%2C+C%2E+J%2E+1979%2E+Information+Retrieval%2E+2+nd+edition%2C+London%3A+Butterworths%2E) *Updated 1 November 2026*