---
title: "Exploring Pre-Trained Transformers and Bilingual — Min 2021"
description: "Arabic coreference resolution trails English by a wide margin, largely because annotated Arabic data is scarce: OntoNotes has roughly seven times more…"
authors: ["Bonan Min"]
year: 2021
venue: "Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference"
source: "https://aclanthology.org/2021.crac-1.10/"
doi: "10.18653/v1/2021.crac-1.10"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/min-2021-exploring-pre-trained-transformers-bilingual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/min-2021-exploring-pre-trained-transformers-bilingual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / Exploring Pre-Trained Transformers and Bilingual Transfer Learning for Arabic Coreference Resolution

# Exploring Pre-Trained Transformers and Bilingual Transfer Learning for Arabic Coreference Resolution

**Authors:** [Bonan Min](https://md.research.elamur.ai/free/en/authors/bonan-min/) · **Year:** 2021 · **Venue:** Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.crac-1.10/) · [PDF](https://aclanthology.org/2021.crac-1.10.pdf) · [DOI 10.18653/v1/2021.crac-1.10](https://doi.org/10.18653/v1/2021.crac-1.10) · [OpenAlex](https://openalex.org/W3214039522)

**Citations:** 7 citations (openalex-snapshot · 2026-10-14) · [4](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Arabic coreference resolution trails English by a wide margin, largely because annotated Arabic data is scarce: OntoNotes has roughly seven times more English documents than Arabic ones. This work adapts the end-to-end BERT-COREF model (Joshi et al., 2019b) to Arabic by swapping its English encoder for three pre-trained transformers: mBERT, AraBERT and GigaBERT. It then tests two ways to use the larger English portion of OntoNotes 5.0 for cross-lingual transfer. PIPELINE trains on English first and then fine-tunes on Arabic. JOINT trains on shuffled mini-batches drawn from both languages, with the Arabic data replicated so the two languages are roughly balanced. Adding English supervision lifts scores by about 2–4 average F1. The best system, GigaBERT-JOINT, reaches 64.55 F1 on the Arabic test set, a new state of the art. The comparison of encoders also shows that how much Arabic text an encoder saw in pre-training, and whether it was built specifically for Arabic, strongly affects results. The code is released on GitHub (bnmin/arabic_coref).

## Key points
- **Data:** OntoNotes 5.0 with the CoNLL 2012 splits. English has 2,802 training documents and Arabic only 359. In the JOINT setup, Arabic data was replicated k = 6 times, a value chosen on the dev set.
- **Arabic-only results (avg. F1):** GigaBERT scored 62.09, AraBERT 56.41 and mBERT 46.8. These rankings follow the amount of Arabic pre-training text: about 4.3B, 2.5B and 153M tokens respectively.
- **Bilingual results:** GigaBERT-JOINT reached 64.55 and GigaBERT-PIPELINE 64.06. For mBERT, PIPELINE scored 49.82 and JOINT 48.28. The best result beats the earlier 63.9 F1 of Aloraini et al. (2020).
- **Transfer behaviour:** PIPELINE and JOINT gave similar improvements, which suggests the extra data matters more than the training scheme. mBERT and GigaBERT benefited about equally from English data (≈3 vs. ≈2.5 F1). The author found this unexpected, since GigaBERT is an English-Arabic bilingual model.
- **Setup:** Encoder learning rate was 1e-5 and task learning rate 2e-4. Models trained for 30 epochs, except PIPELINE, which used 20 English plus 20 Arabic epochs. Training ran on a single NVIDIA Quadro RTX 8000 and took 3–12 hours per model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/)

## Cited by

Cited by

4 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 3 not yet.

- [Coreference Resolution through a seq2seq Transition-Based System](https://md.research.elamur.ai/free/en/papers/bohnet-2023-coreference-resolution-through-seq2seq-transition/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{min2021exploring,
  title = {{Exploring Pre-Trained Transformers and Bilingual Transfer Learning for Arabic Coreference Resolution}},
  author = {Min, Bonan},
  booktitle = {Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference},
  year = {2021},
  doi = {10.18653/v1/2021.crac-1.10},
  url = {https://doi.org/10.18653/v1/2021.crac-1.10}
}
```

- [← previous Exploiting Document Structures and Cluster Consistencies for Event Coreference Resolution](https://md.research.elamur.ai/free/en/papers/tran-2021-exploiting-document-structures-cluster-consistencies/)
- [next → Extractive Summarization Considering Discourse and Coreference Relations based on Heterogeneous Graph](https://md.research.elamur.ai/free/en/papers/huang-2021-extractive-summarization-considering-discourse-coreference/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/)

## References

16 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdulrahman Aloraini and Massimo Poesio. 2020. Cross-lingual Zero Pronoun Resolution. In Proceedings of LREC 2020. [find](https://scholar.google.com/scholar?q=Abdulrahman+Aloraini+and+Massimo+Poesio%2E+2020%2E+Cross%2Dlingual+Zero+Pronoun+Resolution%2E+In+Proceedings+of+LREC+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/aloraini-2020-cross-lingual-zero-pronoun-resolution/)
2. Abdulrahman Aloraini, Juntao Yu, and Massimo Poesio. 2020. Neural coreference resolution for Arabic. In Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference, pages 99–110, Barcelona, Spain (online). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2011.00286) [our card](https://md.research.elamur.ai/free/en/papers/aloraini-2020-neural-coreference-resolution-arabic/)
3. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019b. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Omer+Levy%2C+Luke+Zettlemoyer%2C+and+Daniel+Weld%2E+2019b%2E+BERT+for+coreference+resolution%3A+Baselines+and+analysis%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Langua) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-bert-coreference-resolution-baselines-analysis/)
4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. [↗](https://doi.org/10.48550/arxiv.2003.00104) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
5. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An Empirical Study of Pre-trained Transformers for Arabic Information Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), arXiv. [find](https://scholar.google.com/scholar?q=Wuwei+Lan%2C+Yang+Chen%2C+Wei+Xu%2C+and+Alan+Ritter%2E+2020%2E+An+Empirical+Study+of+Pre%2Dtrained+Transformers+for+Arabic+Information+Extraction%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Nat)
6. Anders Björkelund and Jonas Kuhn. 2014. Learning Structured Perceptrons for Coreference Resolution with Latent Antecedents and Non-local Features. pages 47–57. [find](https://scholar.google.com/scholar?q=Anders+Bj%C3%B6rkelund+and+Jonas+Kuhn%2E+2014%2E+Learning+Structured+Perceptrons+for+Coreference+Resolution+with+Latent+Antecedents+and+Non%2Dlocal+Features%2E+pages+47%E2%80%9357%2E) [our card](https://md.research.elamur.ai/free/en/papers/bjorkelund-2014-learning-structured-perceptrons-coreference-resolution/)
7. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Confer-Desislava Zhekova, Sandra Kübler, Joshua Bonner, Marwa Ragheb, and Yu-Yin Hsu. 2012. UBIU for Multilingual Coreference Resolution in OntoNotes. In Proceedings of the Joint Conference on EMNLP and CoNLL - Shared Task. ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Luheng+He%2C+and+Luke+Zettlemoyer%2E+2018%2E+Higher%2Dorder+coreference+resolution+with+coarse%2Dto%2Dfine+inference%2E+In+Proceedings+of+the+2018+Confer%2DDesislava+Zhekova%2C+Sandra+K%C3%BCbler%2C+Joshua+Bonner%2C) [our card](https://md.research.elamur.ai/free/en/papers/lee-2018-higher-order-coreference-resolution-coarse/)
8. Vincent Ng. 2010. Supervised noun phrase coreference research: The first fifteen years. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1396–1411, Uppsala, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vincent+Ng%2E+2010%2E+Supervised+noun+phrase+coreference+research%3A+The+first+fifteen+years%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1396%E2%80%931411%2C+Upp) [our card](https://md.research.elamur.ai/free/en/papers/ng-2010-supervised-noun-phrase-coreference-research/)
9. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Cardiff, United Kingdom. Leibniz-Institut für Deutsche Sprache. [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Beno%C3%AEt+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+Pipeline+for+Processing+Huge+Corpora+on+Medium+to+Low+Resource+Infrastructures%2E+In+7th+Workshop+on+the+Challenges+in+the)
10. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1493) [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
11. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+2012%2E+CoNLL%2D2012+shared+task%3A+Modeling+multilingual+unrestricted+coreference+in+OntoNotes%2E+In+Joint+Conference+on+EM) [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2012-conll-2012-shared-task-modeling/)
12. Marcus Stamborg, Dennis Medved, Peter Exner, and Pierre Nugues. 2012. Using Syntactic Dependencies to Solve Coreferences. In Proceedings of the Joint Conference on EMNLP and CoNLL - Shared Task. [find](https://scholar.google.com/scholar?q=Marcus+Stamborg%2C+Dennis+Medved%2C+Peter+Exner%2C+and+Pierre+Nugues%2E+2012%2E+Using+Syntactic+Dependencies+to+Solve+Coreferences%2E+In+Proceedings+of+the+Joint+Conference+on+EMNLP+and+CoNLL+%2D+Shared+Task%2E) [our card](https://md.research.elamur.ai/free/en/papers/stamborg-2012-syntactic-dependencies-solve-coreferences/)
13. Olga Uryupina and Alessandro Moschitti. 2013. Multilingual mention detection for coreference resolution. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 100–108, Nagoya, Japan. Asian Federation of Natural Language Processing. [find](https://scholar.google.com/scholar?q=Olga+Uryupina+and+Alessandro+Moschitti%2E+2013%2E+Multilingual+mention+detection+for+coreference+resolution%2E+In+Proceedings+of+the+Sixth+International+Joint+Conference+on+Natural+Language+Processing%2C+page)
14. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.622) [our card](https://md.research.elamur.ai/free/en/papers/wu-2020-corefqa-coreference-resolution-as-query/)
15. Hao Xiong and Qun Liu. 2012. ICT: System Description for CoNLL-2012. In Proceedings of the Joint Conference on EMNLP and CoNLL - Shared Task. [find](https://scholar.google.com/scholar?q=Hao+Xiong+and+Qun+Liu%2E+2012%2E+ICT%3A+System+Description+for+CoNLL%2D2012%2E+In+Proceedings+of+the+Joint+Conference+on+EMNLP+and+CoNLL+%2D+Shared+Task%2E)
16. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.60692/vm65h-ach57)