---
title: "SemEval-2007 Task 18: Arabic Semantic Labeling (2007)"
description: "This SemEval-2007 task set out to give Arabic computational semantics a shared benchmark. 94% F1. 12 references, cited by 3 papers here."
authors: ["Mona Diab", "Musa Alkhalifa", "Sabry ElKateb", "Christiane Fellbaum", "Aous Mansouri", "Martha Palmer"]
year: 2007
venue: "Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)"
source: "https://aclanthology.org/S07-1017/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/diab-2007-semeval-2007-task-18-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/diab-2007-semeval-2007-task-18-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2007](https://md.research.elamur.ai/free/en/papers/2007/page/2/) / SemEval-2007 Task 18: Arabic Semantic Labeling

# SemEval-2007 Task 18: Arabic Semantic Labeling

**Authors:** [Mona Diab](https://md.research.elamur.ai/free/en/authors/mona-diab/), [Musa Alkhalifa](https://md.research.elamur.ai/free/en/authors/musa-alkhalifa/), Sabry ElKateb, [Christiane Fellbaum](https://md.research.elamur.ai/free/en/authors/christiane-fellbaum/), Aous Mansouri and [Martha Palmer](https://md.research.elamur.ai/free/en/authors/martha-palmer/) · **Year:** 2007 · **Venue:** [Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)](https://md.research.elamur.ai/free/en/venues/international-workshop-on-semantic-evaluations/) ([International Workshop on Semantic Evaluations 2007](https://md.research.elamur.ai/free/en/venues/international-workshop-on-semantic-evaluations/2007/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/S07-1017/) · [PDF](https://aclanthology.org/S07-1017.pdf) · [OpenAlex](https://openalex.org/W1988247094)

**Citations:** 22 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/diab-2007-semeval-2007-task-18-arabic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This SemEval-2007 task set out to give Arabic computational semantics a shared benchmark. At the time Arabic had almost no semantically annotated data, and English-oriented labelers were unlikely to transfer well. The organizers built four subtasks on Modern Standard Arabic newswire from the Arabic Treebank (ATB):
- Arabic word sense disambiguation (AWSD), using Arabic WordNet (AWN) synsets.
- An English-to-Arabic WSD variant.
- Argument boundary detection for semantic role labeling.
- Argument role classification, based on the Pilot Arabic Propbank.

The WSD and SRL texts were drawn from the same pool. This means one body of text ends up layered with morphology, syntax, senses, roles and English translations. The paper explains why Arabic is hard for these tasks: unwritten short vowels merge distinct words, subjects are often dropped, word order is flexible, and idafa possessive chains complicate noun phrase scope. Only one system, CUNIT (an SVM with polynomial kernels), took part, and it competed in SRL. It reached about 94% F1 on boundary detection and about 81% F1 on classification over test data. The main lasting contribution is the combined resource.

## Key points
- The data come from ATB III ver. 2: Annhar newspaper text from July–November 2002, released unvowelized in Buckwalter transliteration with the reduced Bies POS tags. In the ATB, roughly 30% of sentences drop the subject, and VSO and SVO each account for about 35% of sentences.
- AWN holds over 8,000 synsets and over 15,000 words, of which about 1,400 synsets are Named Entities. It links to PWN 2.0 through synset byte offsets.
- The WSD set had 1,176 candidate instances (256 verbs, 920 nouns). Gaps in AWN cut this to 888 annotated instances (677 nouns, 211 verbs), and annotators double-annotated an overlap of about 300 instances. The English-to-Arabic WSD subtask received no participants.
- The SRL data cover the 95 most frequent verbs, with gold ATB parses supplied:
  - train: 8,402 sentences, 21,194 arguments
  - dev: 886 sentences, 1,710 arguments
  - test: 902 sentences, 1,657 arguments
  
  There are 22 argument types, including STR (stranded) variants. Scoring used the CoNLL evaluator.
- CUNIT's best setting used a degree-2 kernel:
  - Boundary detection F1: 93.68% (dev), 94.06% (test).
  - Classification F1 on automatically detected boundaries: 77.84% (dev), 81.43% (test).
  - Precision clearly exceeded recall; for example, dev precision was 81.31%.

## Interesting ideas
- Annotating one shared text pool for both lexical senses and predicate-argument structure lets researchers study how the two layers interact, rather than evaluating each on unrelated corpora.
- The STR tag handles one semantic argument that is spread over several adjacent constituents: the first piece gets the main role and each later piece gets a continuation label, instead of forcing a single span.
- SRL is split into finding argument spans and then labeling them, and classification is scored on predicted rather than gold spans. This shows honestly how much the first-stage errors cost downstream.
- Before porting an English pipeline, the authors list concrete constructions that will break it: dropped subjects, ambiguous argument order, and the scope of possessive NP chains.
- The paper treats restoring missing diacritics as a partial form of sense disambiguation, because the spelling itself merges distinct words.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Build evaluation sets where the same English sentences carry gold senses, UD trees and frames, so a frame error can be traced to whether its cause was a sense error or a tree error.
- **I2** (frames): Let prag mark a role as split across sibling UD subtrees, for example appositive or stacked locatives like "in New York, in Brooklyn", so no piece of the argument is dropped or misassigned.
- **I3** (frames): Report frame-role accuracy both on gold UD trees and on parser output, so the cost of parse errors is visible separately from the cost of role-assignment errors.
- **I4** (tree): Keep a similar checklist for English trouble spots such as long noun compounds and of-chains, and test that the parser scopes the whole nominal before frames attach roles to it.
- **I5** (tokens): Treat English homographs whose lemma or POS depends on context, such as "lead" or "wound", as an early disambiguation step in crate en, with the chosen reading passed down and kept explainable.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 241 works · 54,053 citations
- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 90 works · 39,109 citations
- [University of Colorado Boulder](https://md.research.elamur.ai/free/en/universities/university-of-colorado-boulder/) — University, Boulder, United States · 69 works · 51,069 citations
- [University of Manchester](https://md.research.elamur.ai/free/en/universities/university-of-manchester/) — University, Manchester, United Kingdom · 87 works · 44,938 citations
- [Universitat de Barcelona](https://md.research.elamur.ai/free/en/universities/universitat-de-barcelona/) — University, Barcelona, Spain · 12 works · 1,338 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Mona Diab, Musa Alkhalifa, Sabry ElKateb, Christiane Fellbaum, Aous Mansouri and Martha Palmer. 2007. SemEval-2007 Task 18: Arabic Semantic Labeling. Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{diab2007semeval,
  title = {{SemEval-2007 Task 18: Arabic Semantic Labeling}},
  author = {Mona Diab, Musa Alkhalifa, Sabry ElKateb, Christiane Fellbaum, Aous Mansouri and Martha Palmer},
  booktitle = {Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)},
  year = {2007},
  doi = {10.3115/1621474.1621491},
  url = {https://doi.org/10.3115/1621474.1621491}
}
```

- [← previous Adapting WSJ-Trained Parsers to the British National Corpus Using In-Domain Self-Training](https://md.research.elamur.ai/free/en/papers/foster-2007-adapting-wsj-trained-parsers-british/)
- [next → Syllable-Based Speech Recognition for Amharic](https://md.research.elamur.ai/free/en/papers/abate-2007-syllable-based-speech-recognition-amharic/)
- [↑ back to papers of 2007](https://md.research.elamur.ai/free/en/papers/2007/page/2/)

## References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. [The berkeley FrameNet project](https://md.research.elamur.ai/free/en/papers/baker-1998-berkeley-framenet/). In COLING-ACL ’98: Proceedings of the Conference, held at the University of Montréal, pages 86–90.
2. Tim Buckwalter. 2002. Buckwalter Arabic Morphological Analyzer Version 1.0. Linguistic Data Consortium, University of Pennsylvania, 2002. LDC Catalog No.: LDC2002L49. [↗](https://doi.org/10.35111/7vzm-mb15)
3. Xavier Carreras and Lluís M`arquez. 2005. [Introduction to the CoNLL-2005 shared task: Semantic role labeling](https://md.research.elamur.ai/free/en/papers/carreras-2005-introduction-conll-2005-shared-task/). In Proceedings of CoNLL-2005, Ann Arbor, Michigan.
4. S. Elkateb, H. Rodriguez, M. Alkhalifa, P. Vossen, A. Pease, M. Bertran, W. Black, and C. Fellbaum. 2006. The arabic wordnet project. In Proceedings of the Conference on Lexical Resources in the European Community, Genoa, Italy, May. [find](https://scholar.google.com/scholar?q=S%2E+Elkateb%2C+H%2E+Rodriguez%2C+M%2E+Alkhalifa%2C+P%2E+Vossen%2C+A%2E+Pease%2C+M%2E+Bertran%2C+W%2E+Black%2C+and+C%2E+Fellbaum%2E+2006%2E+The+arabic+wordnet+project%2E+In+Proceedings+of+the+Conference+on+Lexical+Resources+in+the+Europ)
5. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press. [http://www.cogsci.princeton.edu/˜wn](http://www.cogsci.princeton.edu/˜wn) [2000, September 7]. [↗](http://www.cogsci.princeton.edu/˜wn)
6. Daniel Gildea and Daniel Jurafsky. 2002. [Automatic labeling of semantic roles](https://md.research.elamur.ai/free/en/papers/gildea-2002-automatic-labeling-semantic-roles/). Computational Linguistics, 28(3):245– 288. [↗](https://doi.org/10.1162/089120102760275983)
7. Nancy Ide and Jean Veronis. 1998. Word sense disambiguation: State of the art. In Computational Linguistics, number 24, pages 1–40. [find](https://scholar.google.com/scholar?q=Nancy+Ide+and+Jean+Veronis%2E+1998%2E+Word+sense+disambiguation%3A+State+of+the+art%2E+In+Computational+Linguistics%2C+number+24%2C+pages+1%E2%80%9340%2E)
8. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wig dan Mekki. 2004. The penn arabic treebank : Building a large-scale annota ted arabic corpus. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Ann+Bies%2C+Tim+Buckwalter%2C+and+Wig+dan+Mekki%2E+2004%2E+The+penn+arabic+treebank+%3A+Building+a+large%2Dscale+annota+ted+arabic+corpus%2E)
9. Martha Palmer, Dan Gildea, and Paul Kingsbury. 2005. The proposition bank: A corpus anotated with semantic roles. In Computational Linguistics Journal, number 31:1. [find](https://scholar.google.com/scholar?q=Martha+Palmer%2C+Dan+Gildea%2C+and+Paul+Kingsbury%2E+2005%2E+The+proposition+bank%3A+A+corpus+anotated+with+semantic+roles%2E+In+Computational+Linguistics+Journal%2C+number+31%3A1%2E)
10. Sameer Pradhan, Kadri Hacioglu, Wayne Ward, James H. Martin, and Daniel Jurafsky. 2003. Semantic role parsing: Adding semantic structure to unstructured text. In Proceedings of ICDM-2003, Melbourne, USA. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Kadri+Hacioglu%2C+Wayne+Ward%2C+James+H%2E+Martin%2C+and+Daniel+Jurafsky%2E+2003%2E+Semantic+role+parsing%3A+Adding+semantic+structure+to+unstructured+text%2E+In+Proceedings+of+ICDM%2D2003%2C+Melbourne%2C+U)
11. Dan Tufis. 2004. The balkanet project. In Special Issue of The Romanian Journal of Information Science and Technology, number 7, pages 1–248. [find](https://scholar.google.com/scholar?q=Dan+Tufis%2E+2004%2E+The+balkanet+project%2E+In+Special+Issue+of+The+Romanian+Journal+of+Information+Science+and+Technology%2C+number+7%2C+pages+1%E2%80%93248%2E)
12. Nianwen Xue and Martha Palmer. 2004. Calibrating features for semantic role labeling. In Dekang Lin and Dekai Wu, editors, Proceedings of EMNLP 2004, pages 88–94, Barcelona, Spain, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nianwen+Xue+and+Martha+Palmer%2E+2004%2E+Calibrating+features+for+semantic+role+labeling%2E+In+Dekang+Lin+and+Dekai+Wu%2C+editors%2C+Proceedings+of+EMNLP+2004%2C+pages+88%E2%80%9394%2C+Barcelona%2C+Spain%2C+July%2E+Association+f)

*Updated 1 November 2026*