--- title: "Evaluation of Automatically Generated Pronoun Reference" description: "The paper evaluates a system that automatically writes English pronoun reference questions: multiple-choice items in which a… 2 references." authors: ["Arief Yudha Satria", "Takenobu Tokunaga"] year: 2017 venue: "Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications" source: "https://aclanthology.org/W17-5008/" doi: "10.18653/v1/w17-5008" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/satria-2017-evaluation-automatically-generated-pronoun-reference/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/satria-2017-evaluation-automatically-generated-pronoun-reference/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/5/) / Evaluation of Automatically Generated Pronoun Reference Questions # Evaluation of Automatically Generated Pronoun Reference Questions **Authors:** Arief Yudha Satria and [Takenobu Tokunaga](https://md.research.elamur.ai/free/en/authors/takenobu-tokunaga/) · **Year:** 2017 · **Venue:** [Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications](https://md.research.elamur.ai/free/en/venues/bea/) ([BEA 2017](https://md.research.elamur.ai/free/en/venues/bea/2017/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W17-5008/) · [PDF](https://aclanthology.org/W17-5008.pdf) · [DOI 10.18653/v1/w17-5008](https://doi.org/10.18653/v1/w17-5008) · [OpenAlex](https://openalex.org/W2756495812) **Citations:** 8 citations (OpenAlex, 2026-10-14) ## Summary The paper evaluates a system that automatically writes English pronoun reference questions: multiple-choice items in which a reader picks the antecedent of a pronoun in a passage from four options. Off-the-shelf anaphora resolution is not reliable enough to supply correct answers, so the generator works the other way round. It takes human-written Project Gutenberg texts, finds sentences with nonrestrictive relative clauses, splits each into two sentences, and swaps the relative pronoun for a personal pronoun. The relative clause's antecedent then becomes the known correct answer. Centering theory filters out splits where the new pronoun would point to a different noun. Distractors come from nearby entities, excluding anything coreferent with the answer. The evaluation had two parts: - **Teachers** rated the generated items for quality. - **University students** (81 in total) answered machine-generated and human-made TOEFL-style items, and their responses were analysed with item statistics and correlations with TOEIC scores. The machine items came out roughly on par with human ones. This suggests cheap, scalable generation of reading-comprehension test items is feasible. ## Key points - **Generation pipeline:** the antecedent of the relative pronoun is accepted only if two methods agree on it. One is a constituency ("lexical") parser with hand-made rules using attachment and number. The other is a dependency parser's `acl:relcl` relation. Sentences where they disagree are dropped. - **Teacher judgement:** 39 of 60 generated questions (65%) were judged usable in a real test. - **Item discrimination:** 30 machine-generated questions (MGQs) and 30 human-made questions (HMQs) from TOEFL preparation books were compared. Average discrimination was 0.33 for MGQs and 0.37 for HMQs. Using the ≥0.2 acceptability threshold, 22 MGQs and 24 HMQs passed. High and low groups were the top and bottom 27% of test takers. - **Correlation with TOEIC:** MGQ scores correlated with TOEIC Reading at 0.65, Listening at 0.56 and the total at 0.74. The corresponding HMQ figures were 0.68, 0.57 and 0.77. MGQ and HMQ scores correlated with each other at 0.61. - **Difficulty:** passage difficulty, measured with the JACET8000 word list, correlated 0.56 with MGQ item difficulty. In a group of 25 students, the hardest item was solved by 5 and the easiest by 24. ## Interesting ideas - Turning nonrestrictive relative clauses into split sentences with a personal pronoun yields pronoun–antecedent pairs whose answer is known by construction, which is a cheap way to manufacture reliable coreference examples. - An example is kept only when two independent analyses (rule-based constituency traversal and a dependency parser) agree, and discordant cases are simply thrown away rather than adjudicated. - After rewriting text, the authors re-check with Centering theory that the newly introduced pronoun would still be read as pointing to the intended entity. - Distractors are drawn by recency, entities coreferent with the answer are excluded, and the error analysis showed that a distractor mentioned twice lured even strong readers. - Item quality is measured with a discrimination index between top and bottom 27% groups, and test scores are validated against an external proficiency measure. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (frames): we could auto-build a coreference test and training set from any parsed English corpus, using UD `acl:relcl` arcs as the source of truth for the pronoun's antecedent. - **I2** (learn): this matches our witness-parser silver data, and supports discarding disagreements outright instead of forcing a vote when precision matters more than coverage. - **I3** (back): when generating text from the meaning graph, we can run a salience-based check (subject outranks a PP object) on each emitted pronoun and fall back to a full noun phrase if the likely reading would be wrong. - **I4** (answer): when Mova builds or checks multiple-choice items for fables and school tests, it should exclude coreferent options and treat repeated mentions as a known trap to test its own resolver against. - **I5** (learn): we could score our evaluation questions by whether they separate stronger from weaker model versions, and drop items that every version passes or fails. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Tokyo Institute of Technology](https://md.research.elamur.ai/free/en/universities/tokyo-institute-of-technology/) — University, Tokyo, Japan · 22 works · 2,137 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 499 works 🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🔗 Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) · 574 works ## Cite Cite as: Arief Yudha Satria and Takenobu Tokunaga. 2017. Evaluation of Automatically Generated Pronoun Reference Questions. Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{satria2017evaluation, title = {{Evaluation of Automatically Generated Pronoun Reference Questions}}, author = {Arief Yudha Satria and Takenobu Tokunaga}, booktitle = {Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications}, year = {2017}, doi = {10.18653/v1/w17-5008}, url = {https://doi.org/10.18653/v1/w17-5008} } ``` - [← previous English partitives in Functional Discourse Grammar: types and constraints](https://md.research.elamur.ai/free/en/papers/keizer-2017-english-partitives-functional-discourse-grammar/) - [next → Splitting Complex English Sentences](https://md.research.elamur.ai/free/en/papers/lee-2017-splitting-complex-english-sentences/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/5/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Jun Araki, Dheeraj Rajagopal, Sreecharan Sankaranarayanan, Susan Holm, Yukari Yamakawa, and Teruko Mitamura. 2016. Generating questions and multiple-choice answers using semantic analysis of texts. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. [find](https://scholar.google.com/scholar?q=Jun+Araki%2C+Dheeraj+Rajagopal%2C+Sreecharan+Sankaranarayanan%2C+Susan+Holm%2C+Yukari+Yamakawa%2C+and+Teruko+Mitamura%2E+2016%2E+Generating+questions+and+multiple%2Dchoice+answers+using+semantic+analysis+of+texts%2E+In) 2. The COLING 2016 Organizing Committee, Osaka, Japan, pages 1125–1136. http://aclweb.org/anthology/C16-1107. [↗](https://aclanthology.org/C16-1107/) *Updated 31 October 2026*