--- title: "JHU System Description for the MADAR Arabic Dialect" description: "The paper describes the Johns Hopkins entry to the MADAR shared task on fine-grained Arabic dialect… 6 references, cited by 4 papers here." authors: ["Tom Lippincott", "Pamela Shapiro", "Kevin Duh", "Paul McNamee"] year: 2019 venue: "Proceedings of the Fourth Arabic Natural Language Processing Workshop" source: "https://aclanthology.org/W19-4634/" doi: "10.18653/v1/w19-4634" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lippincott-2019-jhu-system-description-madar-arabic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lippincott-2019-jhu-system-description-madar-arabic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/) / JHU System Description for the MADAR Arabic Dialect Identification Shared Task # JHU System Description for the MADAR Arabic Dialect Identification Shared Task **Authors:** [Tom Lippincott](https://md.research.elamur.ai/free/en/authors/tom-lippincott/), [Pamela Shapiro](https://md.research.elamur.ai/free/en/authors/pamela-shapiro/), [Kevin Duh](https://md.research.elamur.ai/free/en/authors/kevin-duh/) and [Paul McNamee](https://md.research.elamur.ai/free/en/authors/paul-mcnamee/) · **Year:** 2019 · **Venue:** [Proceedings of the Fourth Arabic Natural Language Processing Workshop](https://md.research.elamur.ai/free/en/venues/wanlp/) ([WANLP 2019](https://md.research.elamur.ai/free/en/venues/wanlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W19-4634/) · [PDF](https://aclanthology.org/W19-4634.pdf) · [DOI 10.18653/v1/w19-4634](https://doi.org/10.18653/v1/w19-4634) · [OpenAlex](https://openalex.org/W2971130934) **Citations:** 8 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/lippincott-2019-jhu-system-description-madar-arabic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper describes the Johns Hopkins entry to the MADAR shared task on fine-grained Arabic dialect identification. Subtask 1 assigns travel-domain sentences to one of 25 cities or MSA. Subtask 2 assigns Twitter users to their country. The main difficulties are very short inputs and the risk of overfitting to topic or proper names. The team's core tool was Prediction by Partial Matching (PPM). This is a compression-derived, variable-order character language model with automatic backoff. One PPM model is trained per class, and the class that fits the text best is chosen. On top of this they built an ensemble of CNN, RNN and MLP submodels and tried fastText embeddings trained on extra Arabic data (ADC, Twitter LID, Reddit, Twitter 1% sample). The ensemble won on Subtask 1 with a macro F1 of 63.4. A plain PPM model of order 3 was best on Subtask 2 with F1 54.3. Extra embedding data helped only marginally, and the ensemble barely outperformed its strongest member. The lesson is that a simple character LM is a very strong baseline. Diverse neural architectures may end up learning overlapping information. ## Key points - **PPM order matters per task:** on dev data, N=4 was best for Subtask 1 (0.591) and N=3 was best for Subtask 2 (0.543). Higher orders degraded noticeably on tweets (N=5 gave 0.287). - **Final scores:** the Subtask 1 ensemble reached P 63.7 / R 63.4 / F1 63.4. Subtask 2 PPM-3 reached P 74.9 / R 46.5 / F1 54.3. For PPM-3, the input was normalized to Arabic characters only, and the user name was prepended. - **Ensemble vs. its parts (Subtask 1):** CNN 0.545, RNN 0.554, MLP over PPM scores 0.591, full ensemble 0.634. On Subtask 2 the ensemble fell below PPM-3. - **Embeddings:** 300-dim fastText cbow vectors (random, MADAR, MADAR+) gave 0.632/0.626/0.634 on Subtask 1 and 0.399/0.397/0.411 on Subtask 2. The small gain on Subtask 2 is attributed to the extra data being in-domain Twitter text. - **Practical issues:** - 5–12% of Subtask 2 tweets were no longer retrievable. - The system was built to classify individual tweets, so user labels came from a majority vote over each user's tweets. - Errors mirrored known pairs such as TUN/SFX and BAS/BAG. - Subtask 2 confusions were notably one-directional (e.g., Iraq→Oman). ## Interesting ideas - Classification by "which class model would compress this text best" with a PPM character model needs no feature engineering. It is very robust on short strings, and the backoff chain shows exactly which contexts drove the decision. - Each submodel is first trained alone to emit a hidden vector of a common size. It is then frozen, and the vectors are concatenated under a step-down stack of dense layers. This lets heterogeneous signals, including non-neural probability scores and metadata, be fused in a uniform way. - The authors treat "the ensemble barely beats its best member" as a diagnostic of redundant, correlated components rather than a mere disappointment. They suggest training later members on data reweighted by earlier members' predictions. - Lower-order context worked better on noisy, sparse tweets than on clean travel phrases, so model capacity was tuned per domain rather than globally. - Inspecting the direction of confusions, not just their counts, revealed lopsided error patterns that aggregate F1 hides. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tokens): A per-class PPM could serve as a cheap, inspectable detector of input register or variety (e.g., British vs. American spelling, informal social-media text) before tagging, so the tagger can switch settings. - **I2** (learn): Witness parsers or LLM-teacher scores could be wrapped as frozen "distribution submodels" feeding a small arbiter, keeping each witness's contribution separable and auditable. - **I3** (learn): When silver data is agreed by several witness parsers, measure how often they err together and compare the agreement set against the single best parser. Doing this as a control checks that agreement is real evidence and not shared bias. - **I4** (tokens): Backoff or context-window settings for the morphology and lemma models could be chosen per text genre (school texts vs. user-generated text) on held-out data, instead of using one fixed value. - **I5** (answer): Evaluation of the readers could log asymmetric confusion between answer types or frame roles, exposing systematic one-way biases that call for targeted repair rules. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 248 works · 106,397 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works ## Cite Cite as: Tom Lippincott, Pamela Shapiro, Kevin Duh and Paul McNamee. 2019. JHU System Description for the MADAR Arabic Dialect Identification Shared Task. Proceedings of the Fourth Arabic Natural Language Processing Workshop. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{lippincott2019jhu, title = {{JHU System Description for the MADAR Arabic Dialect Identification Shared Task}}, author = {Tom Lippincott, Pamela Shapiro, Kevin Duh and Paul McNamee}, booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop}, year = {2019}, doi = {10.18653/v1/w19-4634}, url = {https://doi.org/10.18653/v1/w19-4634} } ``` - [← previous English Broadcast News Speech Recognition by Humans and Machines](https://md.research.elamur.ai/free/en/papers/thomas-2019-english-broadcast-news-speech-recognition/) - [next → Restoring Arabic vowels through omission-tolerant dictionary lookup](https://md.research.elamur.ai/free/en/papers/neme-2019-restoring-arabic-vowels-through-omission/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/) ## References 6 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the Association for Computational Linguistics, 5:135–146. [↗](https://doi.org/10.1162/tacl_a_00051) 2. John Cleary and Ian Witten. 1984. Data compression using adaptive coding and partial string matching. Transactions on Communications, 32:396–402. [↗](https://doi.org/10.1109/tcom.1984.1096090) 3. Eibe Frank, Chang Chui, and Ian Witten. 2000. Text categorization using compression models. In Proceedings of the IEEE Data Compression Conference, pages 200–209. [find](https://scholar.google.com/scholar?q=Eibe+Frank%2C+Chang+Chui%2C+and+Ian+Witten%2E+2000%2E+Text+categorization+using+compression+models%2E+In+Proceedings+of+the+IEEE+Data+Compression+Conference%2C+pages+200%E2%80%93209%2E) 4. Paul McNamee. 2016. Language and Dialect Discrimination Using Compression-Inspired Language Models. Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects. [find](https://scholar.google.com/scholar?q=Paul+McNamee%2E+2016%2E+Language+and+Dialect+Discrimination+Using+Compression%2DInspired+Language+Models%2E+Proceedings+of+the+Third+Workshop+on+NLP+for+Similar+Languages%2C+Varieties+and+Dialects%2E) 5. Mohammad Salameh and Houda Bouamor. 2018. [Fine-grained arabic dialect identification](https://md.research.elamur.ai/free/en/papers/salameh-2018-fine-grained-arabic-dialect-identification/). In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344. 6. Omar Zaidan and Chris Callison-Burch. 2011. [The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content](https://md.research.elamur.ai/free/en/papers/zaidan-2011-arabic-online-commentary-dataset-annotated/). In Proceedings of the Association for Computational Linguistics, pages 37–41. *Updated 1 November 2026*