--- title: "Human-AI Annotation Error Auditing for Hebrew Diacritization with Frontier LLMs — Gershuni 2026" description: "The authors audit the labels of the EACL 2023 Hebrew Homograph Challenge Set… 83.6% recall. 3 references." authors: ["Hillel Gershuni", "Avi Shmidman"] year: 2026 venue: "Proceedings of the 20th Linguistic Annotation Workshop (LAW XX)" source: "https://aclanthology.org/2026.law-main.4/" doi: "10.18653/v1/2026.law-main.4" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/gershuni-2026-human-ai-annotation-error-auditing/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gershuni-2026-human-ai-annotation-error-auditing/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / Human-AI Annotation Error Auditing for Hebrew Diacritization with Frontier LLMs # Human-AI Annotation Error Auditing for Hebrew Diacritization with Frontier LLMs **Authors:** [Hillel Gershuni](https://md.research.elamur.ai/free/en/authors/hillel-gershuni/); [Avi Shmidman](https://md.research.elamur.ai/free/en/authors/avi-shmidman/) · **Year:** 2026 · **Venue:** Proceedings of the 20th Linguistic Annotation Workshop (LAW XX) **Link:** [aclanthology.org/2026.law-main.4](https://aclanthology.org/2026.law-main.4/) · library `aclbulk-2026-gershuni-human-ai-annotation-error-auditing` · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology) [**0** citations](https://openalex.org/works?filter=cites:W7166872279) OpenAlex, 2026-10-14 · [Proceedings of the 20th Linguistic Annotation Workshop (LAW XX)](https://aclanthology.org/volumes/2026.law-main/) · [licence](https://creativecommons.org/licenses/by/4.0/) - [Publisher's page](https://aclanthology.org/2026.law-main.4/) - [PDF](https://aclanthology.org/2026.law-main.4.pdf) - [DOI 10.18653/v1/2026.law-main.4](https://doi.org/10.18653/v1/2026.law-main.4) - [OpenAlex](https://openalex.org/W7166872279) **Authors (2)** 1. [Hillel Gershuni](https://openalex.org/A5107544595) [Semantic Scholar](https://www.semanticscholar.org/author/2139215888) · h 1 · 1 citations · 8 works 2. [Avi Shmidman](https://openalex.org/A5013212409) [Semantic Scholar](https://www.semanticscholar.org/author/2408464206) · h 5 · 114 citations · 53 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about The authors audit the labels of the EACL 2023 Hebrew Homograph Challenge Set (about 150,000 sentences, niqqud on homographs in Modern Hebrew) for annotation errors. Frontier LLMs are asked to flag likely niqqud errors, with a recall-first prompt, and their flags are checked by a human adjudicator. On a focused set of 7,241 sentences, the gold standard has 271 errors. Gemini 3 Pro reaches 83.6% recall at batch size 50 and 99.1% precision, against 62.4% and 42.8% for the two human experts, and 73.4% for the union of their findings. Recall stays near 83% from batch 50 to 200 and falls to 71.4% at batch 500. Morphology and dictionary context did not help consistently. Errors missed by all models were construct-versus-adjective cases and dormant number mismatches that need collocational knowledge. The corrected set and difficulty labels (Easy, Medium, Hard, Undetected) are released. ## Ideas - I3. A checklist of the two error types the paper finds hard: the construct-versus-attributive ambiguity and the dormant singular-versus-plural cases. Each can be written as a feature check over the trees (noun in construct with an adjective-like neighbour; number mismatch with a collocate). ## For Mova - I1. An audit step for treebank labels: a teacher (a large model, offline) flags suspicious labels in batches of about 200 items, and a deterministic rule set decides which flags to keep. The large model never answers at test time; it only proposes what to check, which matches Mova's teacher role. - I2. Stratify every test by difficulty, using the cross-system detection rate of each error as a label. A Mova rule that finds only the easy errors shows its ceiling in the numbers. I1 and I2 fit `treebanks/hbo/ptnk` and `tools/macula compare`: an audit pass that flags disagreement between Mova's parse and the gold labels, with the batch and difficulty stratification in the report. I3 is a set of checks for [languages/hbo/04-noun.md](https://mova.elamur.ai/free/en/docs/languages/hbo/04-noun/) (construct state) and for [languages/hbo/05-pronouns-particles.md](https://mova.elamur.ai/free/en/docs/languages/hbo/05-pronouns-particles/) where number mismatch is discussed. The paper is on Modern Hebrew; its batch sizes and recall are not Mova's numbers and should be measured again on Biblical data. ## Institutions [🎓**Bar-Ilan University** Ramat Gan · Israel 102 works · 17,587 citations](https://md.research.elamur.ai/free/en/universities/bar-ilan-university/) Topics[Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Hebrew & Aramaic](https://md.research.elamur.ai/free/en/topics/hebrew-aramaic/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Cite Cite as Hillel Gershuni and Avi Shmidman. 2026. Human-AI Annotation Error Auditing for Hebrew Diacritization with Frontier LLMs. Proceedings of the 20th Linguistic Annotation Workshop (LAW XX). The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2026.law-main.4](https://doi.org/10.18653/v1/2026.law-main.4) ```bibtex @inproceedings{gershuni2026human, title = {{Human-AI Annotation Error Auditing for Hebrew Diacritization with Frontier LLMs}}, author = {Hillel Gershuni and Avi Shmidman}, booktitle = {Proceedings of the 20th Linguistic Annotation Workshop (LAW XX)}, year = {2026}, doi = {10.18653/v1/2026.law-main.4}, url = {https://doi.org/10.18653/v1/2026.law-main.4} } ``` - [← previous Human vs LLM in Conversational Repair Annotation: A New Resource and Comparative Study](https://md.research.elamur.ai/free/en/papers/ngo-2026-human-vs-llm-conversational-repair/) - [next → HumaniCA: A Benchmark Resource for the Detection of Users' Ascription of Humanness to Conversational Agents](https://md.research.elamur.ai/free/en/papers/villata-2026-humanica-benchmark-resource-detection-users/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Yaacov Choueka and Yoni Neeman. 1995. Nakdan-text,(an in-context text-vocalizer for modern hebrew). In BISFAI-95, The Fifth Bar Ilan Symposium for Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Yaacov+Choueka+and+Yoni+Neeman%2E+1995%2E+Nakdan%2Dtext%2C%28an+in%2Dcontext+text%2Dvocalizer+for+modern+hebrew%29%2E+In+BISFAI%2D95%2C+The+Fifth+Bar+Ilan+Symposium+for+Artificial+Intelligence%2E) 2. Domenic V. Cicchetti and Alvan R. Feinstein. 1990. High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6):551–558. [find](https://scholar.google.com/scholar?q=Domenic+V%2E+Cicchetti+and+Alvan+R%2E+Feinstein%2E+1990%2E+High+agreement+but+low+kappa%3A+II%2E+Resolving+the+paradoxes%2E+Journal+of+Clinical+Epidemiology%2C+43%286%29%3A551%E2%80%93558%2E) 3. Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549. [find](https://scholar.google.com/scholar?q=Alvan+R%2E+Feinstein+and+Domenic+V%2E+Cicchetti%2E+1990%2E+High+agreement+but+low+kappa%3A+I%2E+The+problems+of+two+paradoxes%2E+Journal+of+Clinical+Epidemiology%2C+43%286%29%3A543%E2%80%93549%2E)