---
title: "Leveraging Domain Adaptation and Data Augmentation to"
description: "The work shows how to build workable domain retrieval when in-domain resources are thin. 46 references, cited by 4 papers here."
authors: ["Vera Pavlova"]
year: 2023
venue: "Proceedings of ArabicNLP 2023"
source: "https://aclanthology.org/2023.arabicnlp-1.7/"
doi: "10.18653/v1/2023.arabicnlp-1.7"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pavlova-2023-leveraging-domain-adaptation-data-augmentation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pavlova-2023-leveraging-domain-adaptation-data-augmentation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / Leveraging Domain Adaptation and Data Augmentation to Improve Qur'anic IR in English and Arabic

# Leveraging Domain Adaptation and Data Augmentation to Improve Qur'anic IR in English and Arabic

**Authors:** [Vera Pavlova](https://md.research.elamur.ai/free/en/authors/vera-pavlova/) · **Year:** 2023 · **Venue:** [Proceedings of ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.arabicnlp-1.7/) · [PDF](https://aclanthology.org/2023.arabicnlp-1.7.pdf) · [DOI 10.18653/v1/2023.arabicnlp-1.7](https://doi.org/10.18653/v1/2023.arabicnlp-1.7) · [OpenAlex](https://openalex.org/W4389518247)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/pavlova-2023-leveraging-domain-adaptation-data-augmentation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tackles retrieval of relevant Qur'anic verses for a user question, in both English and Arabic. Neural retrievers need far more training data than exists for this domain. The author's recipe is to train first on large general-domain data, then continue on in-domain data. In-domain data is enlarged with an augmentation scheme that exploits relations between Qur'anic verses.

English had no Islamic corpus or domain language model, so the author built a roughly 47M-word Islamic English corpus. On it she continued pre-training BERT with an extended vocabulary, producing BPIT. BPIT then served as the backbone for bi-encoder and cross-encoder retrievers.

Several Arabic LMs were compared. Arabic also got extra treatment to offset scarce general-domain data: teacher-student distillation and machine-translated training sets.

Evaluation uses QRCD recast as a retrieval benchmark. The augmentation markedly raised MRR@10 and NDCG@5 and gave new state-of-the-art Qur'anic IR in both languages. English systems still beat Arabic ones. The work shows how to build workable domain retrieval when in-domain resources are thin.

## Key points
- **Islamic English corpus (~47M words).** It combines:
  - scholarly books on Tafseer, Hadith, Seerah, Fiqh and Aqeedah (~28M words);
  - Islamic journals (~5.5M);
  - fatwas (~4.8M);
  - Wikipedia Islam-portal articles (~5.6M);
  - filtered Common Crawl pages (~2.5M).
- **BPIT model.** BERT was continued-pretrained from bert-base-uncased with a WordPiece tokenizer trained on the corpus.
  - 3992 new domain tokens were kept after frequency filtering; they replace [unused] slots, giving a 33511-token vocabulary.
  - New-token embeddings come from contextualized weight distillation.
  - Pre-training ran in two stages: subtoken masking first, then whole-word masking. Both stages used a 0.15 masking rate and the 80-10-10 rule.
- **Test set.** QRCD was converted to IR: train and dev splits combined, no-answer questions removed, 169 queries in total, with single verses as retrieval units.
  - Queries were translated into English and checked by Islamic scholars.
  - The Saheeh International translation supplies Qur'anic terminology and serves as the English collection.
- **Training and evaluation.** Retrievers (e.g., SBERT-BPIT, Cross-encoder-BPIT) were trained on MS MARCO (533k examples).
  - SBERT-BPIT uses Multiple Negative Ranking Loss with cross-encoder-mined hard negatives.
  - Metrics are MRR@10, NDCG@5 and Recall@100, computed in the BEIR framework.
- **Arabic and limitations.** Arabic training relied partly on machine-translated mMARCO and XNLI. The author lists their translation quality as the main limitation.

## Interesting ideas
- When adding new vocabulary items to a pretrained model, initialize each one from the averaged contextual vectors of its old subword pieces across a handful of sampled sentences, so new entries start near meaningful points instead of random ones
- Order masked-LM pre-training from easy (subtoken masking) to hard (whole-word masking) to squeeze more out of a small corpus
- Cap vocabulary growth with a frequency threshold, since every added item costs extra training that a small corpus cannot pay for
- Score a system with three complementary metric types (coverage via Recall, first-hit rank via MRR, graded ranking via NDCG) rather than a single number
- Build in-domain training pairs from known links between related passages when labeled queries are scarce, and validate translated test queries with domain experts using a fixed reference translation for key terms

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse could absorb domain or rare words into its transformer vocabulary this way, keeping parses stable when retraining on new text.
- **I2** (learn): a similar easy-to-hard curriculum could schedule training of the GPU parser on limited silver data, e.g. starting from short or high-agreement sentences.
- **I3** (tokens): when the en lexicon learns new lemmas or morphological entries from corpora, a count cutoff keeps noise and one-off forms out.
- **I4** (answer): QA on fables and school tests could report whether the right evidence was found at all separately from how highly it was ranked, which makes failures easier to explain.
- **I5** (learn): linked passages (e.g. paraphrased tale episodes) could serve as cheap positive pairs, and expert-checked, terminology-consistent test items would make evaluation more trustworthy.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (3): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

## Cite

Cite as: Vera Pavlova. 2023. Leveraging Domain Adaptation and Data Augmentation to Improve Qur'anic IR in English and Arabic. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pavlova2023leveraging,
  title = {{Leveraging Domain Adaptation and Data Augmentation to Improve Qur'anic IR in English and Arabic}},
  author = {Vera Pavlova},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.7},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.7}
}
```

- [← previous Investigating Multilingual Coreference Resolution by Universal Annotations](https://md.research.elamur.ai/free/en/papers/chai-2023-investigating-multilingual-coreference-resolution-universal/)
- [next → Multi-modal Sarcasm Generation: Dataset and Solution](https://md.research.elamur.ai/free/en/papers/zhao-2023-multi-modal-sarcasm-generation-dataset/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

46 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Khadige Abboud, Olga Golovneva, and Christopher DiPersio. 2022. [Cross-lingual transfer for low-resource Arabic language understanding](https://md.research.elamur.ai/free/en/papers/abboud-2022-cross-lingual-transfer-low-resource/). In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 225–237, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
2. Menwa Alshammeri, Eric Atwell, and Mhd ammar Alsalka. 2021. Detecting semantic-based similarity between verses of the quran with doc2vec. Procedia Computer Science, 189:351–358. [find](https://scholar.google.com/scholar?q=Menwa+Alshammeri%2C+Eric+Atwell%2C+and+Mhd+ammar+Alsalka%2E+2021%2E+Detecting+semantic%2Dbased+similarity+between+verses+of+the+quran+with+doc2vec%2E+Procedia+Computer+Science%2C+189%3A351%E2%80%93358%2E)
3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [AraBERT: Transformer-based model for Arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. [↗](https://doi.org/10.48550/arxiv.2003.00104)
4. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. [Ms marco: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/).
5. Muhammad Huzaifa Bashir, Aqil M Azmi, Haq Nawaz, Wajdi Zaghouani, Mona Diab, Ala Al-Fuqaha, and Junaid Qadir. 2023. Arabic natural language processing for qur’anic research: A systematic review. Artificial Intelligence Review, 56(7):6801–6854. [↗](https://doi.org/10.1007/s10462-022-10313-2)
6. Luiz Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzieh Fadaee, Roberto Lotufo, and Rodrigo Nogueira. 2022. mmarco: A multilingual version of the ms marco passage ranking dataset. [↗](https://doi.org/10.48550/arxiv.2108.13897)
7. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. [Curriculum learning](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/). In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, page 41–48, New York, NY, USA. Association for Computing Machinery.
8. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. Contact us at hello@rttl.ai Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993. Signature verification using a "siamese" time delay neural network. In Proceedings of the 6th International Conference on Neural Information Processing Systems, NIPS’93, page 737–744, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
9. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.740)
10. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.747)
11. Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec_eval. In The 41st International ACM SIGIR Conference on Research &amp Development in Information Retrieval. ACM. [find](https://scholar.google.com/scholar?q=Christophe+Van+Gysel+and+Maarten+de+Rijke%2E+2018%2E+Pytrec%5Feval%2E+In+The+41st+International+ACM+SIGIR+Conference+on+Research+%26amp+Development+in+Information+Retrieval%2E+ACM%2E)
12. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. [find](https://scholar.google.com/scholar?q=Matthew+Henderson%2C+Rami+Al%2DRfou%2C+Brian+Strope%2C+Yun+hsuan+Sung%2C+Laszlo+Lukacs%2C+Ruiqi+Guo%2C+Sanjiv+Kumar%2C+Balint+Miklos%2C+and+Ray+Kurzweil%2E+2017%2E+Efficient+natural+language+response+suggestion+for+smart+r)
13. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d18-1269)
14. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. [↗](https://doi.org/10.48550/arxiv.1905.01969)
15. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. [The interplay of variant, size, and task type in Arabic pre-trained language models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
16. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. [Pre-training with whole word masking for chinese bert](https://md.research.elamur.ai/free/en/papers/cui-2019-pre-training-whole-word-masking/). IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3504–3514. [↗](https://doi.org/10.1109/taslp.2021.3124365)
17. Yong Dai, Linyang Li, Cong Zhou, Zhangyin Feng, Enbo Zhao, Xipeng Qiu, Piji Li, and Duyu Tang. 2022. “is whole word masking always better for Chinese BERT?”: Probing on Chinese grammatical error correction. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1–8, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yong+Dai%2C+Linyang+Li%2C+Cong+Zhou%2C+Zhangyin+Feng%2C+Enbo+Zhao%2C+Xipeng+Qiu%2C+Piji+Li%2C+and+Duyu+Tang%2E+2022%2E+%E2%80%9Cis+whole+word+masking+always+better+for+Chinese+BERT%3F%E2%80%9D%3A+Probing+on+Chinese+grammatical+error+corre)
18. Ehsan Kamalloo, Nandan Thakur, Carlos Lassance, Xueguang Ma, Jheng-Hong Yang, and Jimmy Lin. 2023. Resources for brewing beir: Reproducible reference models and an official leaderboard. [↗](https://doi.org/10.48550/arxiv.2306.07471)
19. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-main.550)
20. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
21. Omar Khattab and Matei Zaharia. 2020. [Colbert: Efficient and effective passage search via contextualized late interaction over bert](https://md.research.elamur.ai/free/en/papers/khattab-2020-colbert-efficient-effective-passage/). [↗](https://doi.org/10.48550/arxiv.2004.12832)
22. Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2021. [Zero-shot neural passage retrieval via domain-targeted synthetic question generation](https://md.research.elamur.ai/free/en/papers/ma-2021-zero-shot-neural-passage-retrieval/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1075–1088, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2004.14503)
23. Fabio Henrique Kiyoiti dos Santos Tanaka and Claus Aranha. 2019. Data augmentation using gans. [↗](https://doi.org/10.48550/arxiv.1904.09135)
24. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. [Domain-specific language model pretraining for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/). ACM Trans. Comput. Healthcare, 3(1).
25. Rana Malhas and Tamer Elsayed. 2020. Ayatec: building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(6):1–21. [find](https://scholar.google.com/scholar?q=Rana+Malhas+and+Tamer+Elsayed%2E+2020%2E+Ayatec%3A+building+a+reusable+verse%2Dbased+test+collection+for+arabic+question+answering+on+the+holy+qur%E2%80%99an%2E+ACM+Transactions+on+Asian+and+Low%2DResource+Language+Infor)
26. Xu Guo and Han Yu. 2022. On the domain adaptation and generalization of pretrained language models: A survey. [↗](https://doi.org/10.48550/arxiv.2211.03154)
27. Rana Malhas and Tamer Elsayed. 2022. Arabic machine reading comprehension on the holy qur’an using cl-arabert. Information Processing Management, 59(6):103068. Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics. [↗](https://doi.org/10.1016/j.ipm.2022.103068)
28. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. Col-BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3715–3734, Seattle, United States. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Keshav+Santhanam%2C+Omar+Khattab%2C+Jon+Saad%2DFalcon%2C+Christopher+Potts%2C+and+Matei+Zaharia%2E+2022%2E+Col%2DBERTv2%3A+Effective+and+efficient+retrieval+via+lightweight+late+interaction%2E+In+Proceedings+of+the+2022+)
29. Rana R Malhas. 2023. ARABIC QUESTION ANSWER-ING ON THE HOLY QUR’AN. Ph.D. thesis. [find](https://scholar.google.com/scholar?q=Rana+R+Malhas%2E+2023%2E+ARABIC+QUESTION+ANSWER%2DING+ON+THE+HOLY+QUR%E2%80%99AN%2E+Ph%2ED%2E+thesis%2E)
30. Tom M Mitchell. 1997. Machine learning, volume 1. McGraw-hill New York. [find](https://scholar.google.com/scholar?q=Tom+M+Mitchell%2E+1997%2E+Machine+learning%2C+volume+1%2E+McGraw%2Dhill+New+York%2E)
31. Ensaf Hussein Mohamed and Eyad Mohamed Shokry. 2022. Qsst: A quranic semantic search tool based on word embedding. Journal of King Saud University -Computer and Information Sciences, 34(3):934–945. [find](https://scholar.google.com/scholar?q=Ensaf+Hussein+Mohamed+and+Eyad+Mohamed+Shokry%2E+2022%2E+Qsst%3A+A+quranic+semantic+search+tool+based+on+word+embedding%2E+Journal+of+King+Saud+University+%2DComputer+and+Information+Sciences%2C+34%283%29%3A934%E2%80%93945%2E)
32. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast WordPiece tokenization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2089–2103, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xinying+Song%2C+Alex+Salcianu%2C+Yang+Song%2C+Dave+Dopson%2C+and+Denny+Zhou%2E+2021%2E+Fast+WordPiece+tokenization%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages)
33. Vera Pavlova and Mohammed Makhlouf. 2023. BIOp-timus: Pre-training an optimal biomedical language model with curriculum learning for named entity recognition. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 337–349, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vera+Pavlova+and+Mohammed+Makhlouf%2E+2023%2E+BIOp%2Dtimus%3A+Pre%2Dtraining+an+optimal+biomedical+language+model+with+curriculum+learning+for+named+entity+recognition%2E+In+The+22nd+Workshop+on+Biomedical+Natura)
34. Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. 2022. Curriculum learning: A survey. [↗](https://doi.org/10.1007/s11263-022-01611-x)
35. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. [RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835–5847, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2010.08191)
36. Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021a. [Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks](https://md.research.elamur.ai/free/en/papers/thakur-2021-augmented-sbert-data-augmentation-method/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 296–310, Online. Association for Computational Linguistics.
37. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d19-1410)
38. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021b. [Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models](https://md.research.elamur.ai/free/en/papers/thakur-2021-beir-heterogenous-benchmark-zero/). In Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021) - Datasets and Benchmarks Track (Round 2). [↗](https://doi.org/10.48550/arxiv.2104.08663)
39. Nils Reimers and Iryna Gurevych. 2020. [Making monolingual sentence embeddings multilingual using knowledge distillation](https://md.research.elamur.ai/free/en/papers/reimers-2020-making-monolingual-sentence-embeddings-multilingual/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics.
40. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+OPUS%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+pages+2214%E2%80%932218%2C+Istanbul%2C+Turk)
41. Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. [RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking](https://md.research.elamur.ai/free/en/papers/ren-2021-rocketqav2-joint-training-method-dense/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2825–2835, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [↗](https://doi.org/10.60692/knanm-sbc86)
42. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. [Representation learning with contrastive predictive coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/). [↗](https://doi.org/10.48550/arxiv.1807.03748)
43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/).
44. Maxim Romanov and Masoumeh Seydi. 2019. Openiti: a machine-readable corpus of islamicate texts. Zenodo, URL: https://doi.org/10.5281/zenodo, 3082464. [↗](https://doi.org/10.5281/zenodo)
45. Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. TSDAE: Using transformer-based sequential denoising auto-encoderfor unsupervised sentence embedding learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 671–688, Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kexin+Wang%2C+Nils+Reimers%2C+and+Iryna+Gurevych%2E+2021%2E+TSDAE%3A+Using+transformer%2Dbased+sequential+denoising+auto%2Dencoderfor+unsupervised+sentence+embedding+learning%2E+In+Findings+of+the+Association+for+Com)
46. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. [KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/). In Proceedings of the Fourteenth Workshop on Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2345–2360, Seattle, United States. Association for Computational Linguistics.

*Updated 1 November 2026*