---
title: "Towards Semi-Automated Annotation for Prepositional Phrase Attachment — Rosenthal 2010"
description: "The study used 1000 questions from Wall Street Journal sentences, each answered by three Amazon Mechanical Turk… 18 references, cited by 4 papers here."
authors: ["Sara Rosenthal", "William Lipovsky", "Kathleen McKeown", "Kapil Thadani", "Jacob Andreas"]
year: 2010
venue: "Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)"
source: "https://aclanthology.org/L10-1572/"
doi: "10.63317/4649a4tsyqwe"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rosenthal-2010-semi-automated-annotation-prepositional-phrase/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rosenthal-2010-semi-automated-annotation-prepositional-phrase/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/) / Towards Semi-Automated Annotation for Prepositional Phrase Attachment

# Towards Semi-Automated Annotation for Prepositional Phrase Attachment

**Authors:** [Sara Rosenthal](https://md.research.elamur.ai/free/en/authors/sara-rosenthal/), William J. Lipovsky, [Kathleen McKeown](https://md.research.elamur.ai/free/en/authors/kathleen-mckeown/), Kapil Thadani and [Jacob Andreas](https://md.research.elamur.ai/free/en/authors/jacob-andreas/) · **Year:** 2010 · **Venue:** Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10) **Link:** [aclanthology.org/L10-1572](https://aclanthology.org/L10-1572/) · **Licence:** copyrighted (ELRA, all rights reserved; listed as [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) by the ACL Anthology) — summary only

[**6** citations](https://openalex.org/works?filter=cites:W1905155149) OpenAlex, 2026-10-14 · **4** citations in our library of the 140,884 papers whose bibliographies we read · [Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)](https://aclanthology.org/volumes/L10-1/) · [licence](https://creativecommons.org/licenses/by-nc/4.0/)

- [Publisher's page](https://aclanthology.org/L10-1572/)
- [PDF](https://aclanthology.org/L10-1572.pdf)
- [DOI 10.63317/4649a4tsyqwe](https://doi.org/10.63317/4649a4tsyqwe)
- [OpenAlex](https://openalex.org/W1905155149)

**Authors (5)**

1. [Sara Rosenthal](https://openalex.org/A5103058383) [ORCID](https://orcid.org/0000-0002-9603-4699) · [Semantic Scholar](https://www.semanticscholar.org/author/144063596) · h 22 · 5,591 citations · 50 works
2. [William Lipovsky](https://openalex.org/A5048686130) [Semantic Scholar](https://www.semanticscholar.org/author/2809703) · h 1 · 6 citations · 1 works
3. [Kathleen McKeown](https://openalex.org/A5109565051) [ORCID](https://orcid.org/0000-0003-4455-2715) · [Semantic Scholar](https://www.semanticscholar.org/author/145590324) · h 67 · 18,940 citations · 450 works
4. [Kapil Thadani](https://openalex.org/A5005163929) [Semantic Scholar](https://www.semanticscholar.org/author/2695187) · h 13 · 626 citations · 37 works
5. [Jacob Andreas](https://openalex.org/A5070829652) [ORCID](https://orcid.org/0000-0002-3141-5845) · [Semantic Scholar](https://www.semanticscholar.org/author/2112400) · h 39 · 6,540 citations · 174 works

Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart.

## What it is about

The paper asks whether untrained crowd workers can label PP attachment well enough to make training data. The system turns Penn Treebank parses into multiple-choice questions: the PP and its alternative noun or verb heads. The study used 1000 questions from Wall Street Journal sentences, each answered by three Amazon Mechanical Turk workers, at a cost of $135 and 49 seconds per task. Workers picked the right attachment in 86.7% of the 3000 responses. The majority vote was right for 90.4% of questions, and 92.2% after the authors removed 18 of 96 wrong-majority cases caused by their own question generator. A unanimous answer was correct in 97% of cases, a bare majority in 82%.

## Ideas

- I3. Two-head check: the paper found items with two correct attachments, which the answer key treated as one. Mark PPs with more than one defensible head in the EWT gold set and score any of them as correct. Measure the ceiling before tuning any model.

## For Mova

- I1. `annot`: keep a silver PP head only when three independent sources agree (a large model, a `tools/treeagree` rule, and the `en` parse), and drop the rest. Measure the precision of the kept heads on a 300-item gold sample from EWT, and the share kept. Use EWT only, since it is CC BY-SA.
- I2. Error map: the workers failed mostly when the PP attached to a verb or an adjective, and rarely to a noun. Report PP head accuracy in `en parse-eval` split by the gold head's part of speech, on EWT test.
- I4. Only the agreement rule is taken from this work. Mova does not use a crowd, and the Penn Treebank data is test only.

The code is `en/src/parse.rs` for the PP scores and `annot` for the silver rule. The chapter is [languages/en/22-phrases.md](https://mova.elamur.ai/free/en/docs/languages/en/22-phrases/). The paper's questions come from WSJ text (LDC-licensed), so any measurement on that text is test only. Its 92.2% is a majority of three people on 1000 questions, so it is not a parser score and must not be read as one.

## Institutions

[🎓**Columbia University** New York · United States 118 works · 44,744 citations](https://md.research.elamur.ai/free/en/universities/columbia-university/)

## Cite

Cite as Sara Rosenthal, William Lipovsky, Kathleen McKeown, Kapil Thadani and Jacob Andreas. 2010. Towards Semi-Automated Annotation for Prepositional Phrase Attachment. Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10).

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.63317/4649a4tsyqwe](https://doi.org/10.63317/4649a4tsyqwe)

```bibtex
@inproceedings{rosenthal2010semi,
  title = {{Towards Semi-Automated Annotation for Prepositional Phrase Attachment}},
  author = {Sara Rosenthal, William Lipovsky, Kathleen McKeown, Kapil Thadani and Jacob Andreas},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  doi = {10.63317/4649a4tsyqwe},
  url = {https://doi.org/10.63317/4649a4tsyqwe}
}
```

- [← previous Cross-linguistic Attribute Selection for REG: Comparing Dutch and English](https://md.research.elamur.ai/free/en/papers/theune-2010-cross-linguistic-attribute-selection-reg/)
- [next → A Syntactic Lexicon for Arabic Verbs](https://md.research.elamur.ai/free/en/papers/loukil-2010-syntactic-lexicon-arabic-verbs/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/)

## References

18 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Timothy Baldwin, and David Martinez. 2008. Improving parsing and PP attachment performance with sense information. In Proceedings of ACL, pages 317–325, June. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Timothy+Baldwin%2C+and+David+Martinez%2E+2008%2E+Improving+parsing+and+PP+attachment+performance+with+sense+information%2E+In+Proceedings+of+ACL%2C+pages+317%E2%80%93325%2C+June%2E)
2. Michaela Atterer and Hinrich Schütze. 2007. Prepositional phrase attachment without oracles. Computational Linguistics, 33(4):469–476. [find](https://scholar.google.com/scholar?q=Michaela+Atterer+and+Hinrich+Schu%CC%88tze%2E+2007%2E+Prepositional+phrase+attachment+without+oracles%2E+Computational+Linguistics%2C+33%284%29%3A469%E2%80%93476%2E)
3. Eric Brill. 1995. Transformation-based error-driven learning and natural language processing: a case study in part-of-speech tagging. Computational Linguistics, 21(4):543–565. [our card](https://md.research.elamur.ai/free/en/papers/brill-1995-transformation-based-error-driven-learning/)
4. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of ACL, pages 132–139. [our card](https://md.research.elamur.ai/free/en/papers/charniak-2000-maximum-entropy-inspired-parser/)
5. Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational Linguistics, 29(4):589–637. [our card](https://md.research.elamur.ai/free/en/papers/collins-2003-head-driven-statistical-models-natural/)
6. Daniel Gruhl, R. Guha, David Liben-Nowell, and Andrew Tomkins. 2004. Information diffusion through blogspace. In WWW ’04: Proceedings of the 13th international conference on World Wide Web, pages 491–501. [find](https://scholar.google.com/scholar?q=Daniel+Gruhl%2C+R%2E+Guha%2C+David+Liben%2DNowell%2C+and+Andrew+Tomkins%2E+2004%2E+Information+diffusion+through+blogspace%2E+In+WWW+%E2%80%9904%3A+Proceedings+of+the+13th+international+conference+on+World+Wide+Web%2C+pages+491%E2%80%93)
7. Liangjie Hong and Brian D. Davison. 2009. A classification-based approach to question answering in discussion boards. In SIGIR ’09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 171– 178. [find](https://scholar.google.com/scholar?q=Liangjie+Hong+and+Brian+D%2E+Davison%2E+2009%2E+A+classification%2Dbased+approach+to+question+answering+in+discussion+boards%2E+In+SIGIR+%E2%80%9909%3A+Proceedings+of+the+32nd+international+ACM+SIGIR+conference+on+Resear)
8. Gilly Leshed and Joseph ’Jofish’ Kaye. 2006. Understanding how bloggers feel: recognizing affect in blog posts. In CHI ’06 extended abstracts on Human factors in computing systems, pages 1019–1024. [find](https://scholar.google.com/scholar?q=Gilly+Leshed+and+Joseph+%E2%80%99Jofish%E2%80%99+Kaye%2E+2006%2E+Understanding+how+bloggers+feel%3A+recognizing+affect+in+blog+posts%2E+In+CHI+%E2%80%9906+extended+abstracts+on+Human+factors+in+computing+systems%2C+pages+1019%E2%80%931024%2E)
9. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: the Penn Treebank. Computational Linguistics, 19(2):313–330. [our card](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/)
10. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The nombank project: An interim report. In Adam Meyers, editor, Proceedings of the HLT-NAACL Workshop on Frontiers in Corpus Annotation, pages 24– 31, Boston, Massachusetts, USA, May. [our card](https://md.research.elamur.ai/free/en/papers/meyers-2004-nombank-project-interim-report/)
11. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. [↗](https://doi.org/10.1162/0891201053630264) [our card](https://md.research.elamur.ai/free/en/papers/palmer-2005-proposition-bank-annotated-corpus-semantic/)
12. Sameer S. Pradhan and Nianwen Xue. 2009. Ontonotes: the 90% solution. In Proceedings of NAACL, pages 11– 12. [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2009-ontonotes-90-solution/)
13. Adwait Ratnaparkhi, Jeff Reynar, and Salim Roukos. 1994. A maximum entropy model for prepositional phrase attachment. In Proceedings of HLT, pages 250–255. [our card](https://md.research.elamur.ai/free/en/papers/ratnaparkhi-1994-maximum-entropy-model-prepositional-phrase/)
14. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP, pages 254–263. [our card](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/)
15. Jiri Stetina and Makoto Nagao. 1997. Corpus based pp attachment ambiguity resolution with a semantic dictionary. In Proceedings of the Workshop on Very Large Corpora, pages 66–80. [our card](https://md.research.elamur.ai/free/en/papers/stetina-1997-corpus-based-pp-attachment-ambiguity/)
16. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: chunking. In Proceedings of the CoNLL workshop on Learning Language in Logic, pages 127–132. [our card](https://md.research.elamur.ai/free/en/papers/sang-2000-introduction-conll-2000-shared-task/)
17. Alexander S. Yeh and Marc B. Vilain. 1998. Some properties of preposition and subordinate conjunction attachments. In Proceedings of COLING, pages 1436–1442. [find](https://scholar.google.com/scholar?q=Alexander+S%2E+Yeh+and+Marc+B%2E+Vilain%2E+1998%2E+Some+properties+of+preposition+and+subordinate+conjunction+attachments%2E+In+Proceedings+of+COLING%2C+pages+1436%E2%80%931442%2E)
18. Jakub Zavrel, Walter Daelemans, and Jorn Veenstra. 1997. Resolving PP attachment ambiguities with memory-based learning. In Proc. of the Workshop on Computational Language Learning (CoNLL’97), ACL, pages 136–144. [find](https://scholar.google.com/scholar?q=Jakub+Zavrel%2C+Walter+Daelemans%2C+and+Jorn+Veenstra%2E+1997%2E+Resolving+PP+attachment+ambiguities+with+memory%2Dbased+learning%2E+In+Proc%2E+of+the+Workshop+on+Computational+Language+Learning+%28CoNLL%E2%80%9997%29%2C+ACL%2C+pa)