--- title: "A Human-Computer Collaboration Approach to Improve Accuracy" description: "The authors define a redundant pair as two errors that overlap in position, come from different phases, and… 90.2% accuracy. 4 references." authors: ["Jee Eun Kim", "Kong Joo Lee"] year: 2010 venue: "Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications" source: "https://aclanthology.org/W10-1011/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/kim-2010-human-computer-collaboration-approach-improve/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kim-2010-human-computer-collaboration-approach-improve/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/page/3/) / A Human-Computer Collaboration Approach to Improve Accuracy of an Automated English Scoring System # A Human-Computer Collaboration Approach to Improve Accuracy of an Automated English Scoring System **Authors:** Jee Eun Kim and Kong Joo Lee · **Year:** 2010 · **Venue:** [Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications](https://md.research.elamur.ai/free/en/venues/naacl-hlt-workshop-on-innovative-use-of-nlp-for-building-educational/) ([NAACL HLT Workshop on Innovative Use of NLP for Building Educational Applications 2010](https://md.research.elamur.ai/free/en/venues/naacl-hlt-workshop-on-innovative-use-of-nlp-for-building-educational/2010/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W10-1011/) · [PDF](https://aclanthology.org/W10-1011.pdf) · [OpenAlex](https://openalex.org/W1814245531) **Citations:** 1 citations (OpenAlex, 2026-10-14) ## Summary The paper tackles duplicated feedback in an automated scorer for English sentences written by Korean learners. The scorer analyses each answer in three independent phases: word, syntax, and mapping, where mapping compares the answer with reference answers. Each phase gives 0–2 points. Because the phases do not share information, one mistake is often reported twice under different error IDs, which distorts the score. The authors define a redundant pair as two errors that overlap in position, come from different phases, and describe the same linguistic phenomenon. The first two conditions are checked automatically. Frequent co-occurring error-type pairs are then kept using a combined PMI and RFC score. English teachers sort the surviving types into three groups: always redundant, never redundant, and context-dependent. For the context-dependent group, a CART decision tree learns rules from features of the two errors, such as overlap, positions, and correction strings. When a pair is judged redundant, one error is dropped according to phase priority. Redundancy detection reached 90.2% accuracy, and exact agreement with human raters rose by 2.6%. This shows that a small amount of targeted expert judgement can clean up the output of loosely coupled modules. ## Key points - The scorer has 70 error types across the word, syntax and mapping phases. Each error report has three fields: Error_ID, Error_Position, and Error_Correction_Info. Missing-word errors are positioned on the reference answer instead of the student sentence. - Scoring 14,892 student answers produced 150,419 candidate pairs, grouped into 657 type pairs. Filtering with PMI·RFC ≥ k (k = 0.4) kept 111 types, covering 29,588 pairs. - Expert teachers, each with 10+ years of experience, sorted the 111 types into three classes: - Class A: 20 types, confirmed redundant. - Class B: 47 types, not redundant. - Class C: 44 types, undecidable without more context. - 12,178 Class C pairs were used to train CART. Its features were: - shared and non-shared length, normalised by the shorter sentence; - the correction info of each error; - the edit distance between the two correction strings; - each error's position and the difference between positions. - On 200 held-out sentences, accuracy was 94.1% for A, 98.0% for B and 82.3% for C, or 90.2% overall. Exact agreement with human raters improved by 2.6%. When removing an error, syntax-phase errors are kept first and mapping-phase errors are discarded first. ## Interesting ideas - Score co-occurring pairs of diagnostic types with PMI combined with relative frequency, so that reviewers see only the few type pairs that matter instead of every instance. - Have humans triage at the level of types into three outcomes: always yes, always no, and depends on context. Then train an interpretable learner only on the "depends" bucket. - When several independent modules report the same fact, merge the duplicates using an explicit priority that favours the module with the widest coverage (phrase-level over word-level). - Diagnose a learner answer by comparing it with a reference answer. Missing items are located on the reference side, and every other overlap test treats them as overlapping everything. - Report accuracy separately for classes decided by humans and by the model, and note that even human-labelled classes fall below 100% because upstream detection errors leak through. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Disagreements between witness parsers or repair-rule firings could be ranked this way before sending them to an LLM teacher tier, which cuts review cost. - **I2** (learn): Candidate self-induced repair rules could be split the same way, so that only context-dependent ones get a learned, explainable decision-tree condition. - **I3** (frames): When roles from VerbNet frames and from other cues overlap on one span, a declared priority order would resolve them while keeping the result explainable. - **I4** (answer): School-course test checkers could align student and reference trees and report omissions anchored to reference positions. - **I5** (learn): Evaluations should separate errors inherited from earlier stages from errors made by the component under test, much like a negative control. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Chungnam National University](https://md.research.elamur.ai/free/en/universities/chungnam-national-university/) — University, Daejeon, South Korea · 6 works · 14 citations - [Hankuk University of Foreign Studies](https://md.research.elamur.ai/free/en/universities/hankuk-university-of-foreign-studies/) — University, Seoul, South Korea · 3 works · 37 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works 🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works ## Cite Cite as: Jee Eun Kim and Kong Joo Lee. 2010. A Human-Computer Collaboration Approach to Improve Accuracy of an Automated English Scoring System. Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{kim2010human, title = {{A Human-Computer Collaboration Approach to Improve Accuracy of an Automated English Scoring System}}, author = {Jee Eun Kim and Kong Joo Lee}, booktitle = {Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications}, year = {2010}, url = {https://aclanthology.org/W10-1011/} } ``` - [← previous "Mind your p's and q's": or the peregrinations of an apostrophe in 17th Century English](https://md.research.elamur.ai/free/en/papers/piton-2010-mind-p-s-q-s/) - [next → Arabic Mention Detection: Toward Better Unit of Analysis](https://md.research.elamur.ai/free/en/papers/benajiba-2010-arabic-mention-detection-better-unit/) - [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/page/3/) ## References 4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Jee Eun Kim, K. J. Lee and K. A. Jin. 2007. Building an Automated Scoring System for a Single Sentence. Korea Information Processing Society Vol.4, No.3 (in Korean). [find](https://scholar.google.com/scholar?q=Jee+Eun+Kim%2C+K%2E+J%2E+Lee+and+K%2E+A%2E+Jin%2E+2007%2E+Building+an+Automated+Scoring+System+for+a+Single+Sentence%2E+Korea+Information+Processing+Society+Vol%2E4%2C+No%2E3+%28in+Korean%29%2E) 2. Keh-Yih Su, Ming-Wen We and Jing-Shin Chang. 1994. A Corpus-based Approach to Automatic Compound Extraction, In Proceedings of the ACL 94. [find](https://scholar.google.com/scholar?q=Keh%2DYih+Su%2C+Ming%2DWen+We+and+Jing%2DShin+Chang%2E+1994%2E+A+Corpus%2Dbased+Approach+to+Automatic+Compound+Extraction%2C+In+Proceedings+of+the+ACL+94%2E) 3. Leo Breiman, Jerome Friedman, Charles J. Stone, and R.A. Olshen. 1984. Classification and Regression Trees. Monterey, Calif., U.S.A.: Wadsworth, Inc. [find](https://scholar.google.com/scholar?q=Leo+Breiman%2C+Jerome+Friedman%2C+Charles+J%2E+Stone%2C+and+R%2EA%2E+Olshen%2E+1984%2E+Classification+and+Regression+Trees%2E+Monterey%2C+Calif%2E%2C+U%2ES%2EA%2E%3A+Wadsworth%2C+Inc%2E) 4. Yigal Attali and Jill Burstein. 2006. Automated Essay Scoring with e-raterÒ V.2. [find](https://scholar.google.com/scholar?q=Yigal+Attali+and+Jill+Burstein%2E+2006%2E+Automated+Essay+Scoring+with+e%2Drater%C3%92+V%2E2%2E) *Updated 2 November 2026*