--- title: "NLP-LISAC at SemEval-2024 Task 1: Transformer-based" description: "This paper presents NLP-LISAC's submission to the supervised setting (Track A) of SemEval-2024 Task… 9 references, cited by 2 papers here." authors: ["Abdessamad Benlahbib", "Anass Fahfouh", "Hamza Alami", "Achraf Boumhidi"] year: 2024 venue: "Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)" source: "https://aclanthology.org/2024.semeval-1.33/" doi: "10.18653/v1/2024.semeval-1.33" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/benlahbib-2024-nlp-lisac-semeval-2024-task/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/benlahbib-2024-nlp-lisac-semeval-2024-task/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) / NLP-LISAC at SemEval-2024 Task 1: Transformer-based approaches for Determining Semantic Textual Relatedness # NLP-LISAC at SemEval-2024 Task 1: Transformer-based approaches for Determining Semantic Textual Relatedness **Authors:** [Abdessamad Benlahbib](https://md.research.elamur.ai/free/en/authors/abdessamad-benlahbib/), Anass Fahfouh, [Hamza Alami](https://md.research.elamur.ai/free/en/authors/hamza-alami/) and Achraf Boumhidi · **Year:** 2024 · **Venue:** [Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2024](https://md.research.elamur.ai/free/en/venues/semeval/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.semeval-1.33/) · [PDF](https://aclanthology.org/2024.semeval-1.33.pdf) · [DOI 10.18653/v1/2024.semeval-1.33](https://doi.org/10.18653/v1/2024.semeval-1.33) · [OpenAlex](https://openalex.org/W4401042623) **Citations:** 0 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 149,439 papers whose bibliographies we read ## Summary This paper presents NLP-LISAC's submission to the supervised setting (Track A) of SemEval-2024 Task 1, a shared task on Semantic Textual Relatedness. The task asks for a score from 0 to 1 that says how related two sentences are. Relatedness is broader than similarity. It also covers links such as hypernymy, part–whole relations, entailment and topical association. The team took part in four languages: Algerian Arabic, Moroccan Arabic, English and Spanish. For each one they fine-tuned a single transformer regressor: - MARBERTv2 for Algerian Arabic, - DarijaBERT for Moroccan Arabic, - DeBERTa-V3-Large for English, - mDeBERTa-V3-Base for Spanish. Their training data was the official train and dev splits merged together. They doubled it by adding each pair again with the two sentences swapped, keeping the same score. The newline between the sentences was replaced by a [SEP] token. The loss function (MAE, MSE or Huber) and the number of epochs were tuned separately for each language. Results were mid-to-upper table: 6th of 24 in Algerian Arabic and 13th of 36 in English. The paper shows that a simple, carefully tuned setup with a language-specific encoder is a competitive baseline. Ablations are thin, though. The swap augmentation is said to help, but no numbers are given for it. ## Key points - Official test scores (Spearman), with rankings: - Algerian Arabic: 0.6036 (6/24) - Spanish: 0.7171 (7/25) - Moroccan Arabic: 0.7894 (12/23) - English: 0.8346 (13/36) - Best configuration found for each language: | Language | Model | Epochs | Max length | Loss | |---|---|---|---|---| | Algerian Arabic | MARBERTv2 | 12 | 200 | MAE | | Moroccan Arabic | DarijaBERT | 5 | 200 | Huber | | English | DeBERTa-V3-Large | 5 | 150 | Huber | | Spanish | mDeBERTa-V3-Base | 12 | 200 | MSE | - Settings shared by all models: learning rate 1e-5, batch size 4, Adam optimizer. Training used Hugging Face Transformers and Keras on Kaggle. - The augmentation merges train and dev, then adds an order-swapped duplicate of every pair with an unchanged label. The authors report that this swap clearly improved dev-set results, but they give no ablation figures. - The data comes from the SemEval-2024 STR dataset, which covers 14 languages and is released on GitHub (semantic-textual-relatedness/Semantic_Relatedness_SemEval2024). ## Interesting ideas - **Symmetry as free data.** Since relatedness does not depend on which sentence comes first, swapping the pair yields a new training example with the same label at no cost. This also nudges the model away from encoding order effects it should not have. - **The regression loss is a tunable choice per dataset.** The authors did not fix one loss. They picked MAE, MSE or Huber by dev performance for each language, because robust losses limit the pull of noisy or extreme labels. - **Match the encoder to the language variety.** Dialect-specific pretrained models (DarijaBERT, MARBERTv2) were chosen over one generic multilingual model. Fit to the language variety often beats raw model size. - **Relatedness is a bundle of relation types.** The task framing lists synonymy, hyponymy/hypernymy, meronymy/holonymy, entailment and association as different routes to relatedness. A score can therefore be broken down into which relation actually links two texts. - **Judge graded outputs by ranking, not absolute values.** Spearman correlation only asks whether the system orders items the way humans do. This is the right test when the absolute scale is arbitrary. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): when building silver data for symmetric judgments (paraphrase, coreference-pair links), add swapped copies. Also use the swap as a negative control: an evaluation should fail if a supposedly symmetric scorer gives different scores for A–B and B–A. - **I2** (learn): when training confidence or score heads on silver labels from witness parsers or LLM teachers, try Huber or MAE so that a few wrong teacher labels do not dominate the gradients. - **I3** (tree): for gparse, it may pay to pick or adapt encoders per register or genre (fables, school texts, maths problems) rather than relying on one general backbone. - **I4** (world): an explainable relatedness judgment could trace these specific links between the frames' entities and predicates in the WordNet/FrameNet ontology and report the path, instead of giving one opaque number. - **I5** (answer): graded answer confidences or plausibility scores could be checked by rank correlation against human orderings, rather than by forcing them onto a calibrated scale. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works ## Cite Cite as: Abdessamad Benlahbib, Anass Fahfouh, Hamza Alami and Achraf Boumhidi. 2024. NLP-LISAC at SemEval-2024 Task 1: Transformer-based approaches for Determining Semantic Textual Relatedness. Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{benlahbib2024nlp, title = {{NLP-LISAC at SemEval-2024 Task 1: Transformer-based approaches for Determining Semantic Textual Relatedness}}, author = {Abdessamad Benlahbib, Anass Fahfouh, Hamza Alami and Achraf Boumhidi}, booktitle = {Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)}, year = {2024}, doi = {10.18653/v1/2024.semeval-1.33}, url = {https://doi.org/10.18653/v1/2024.semeval-1.33} } ``` - [← previous Methodology of Adapting Large English Language Models for Specific Cultural Contexts](https://md.research.elamur.ai/free/en/papers/zhang-2024-methodology-adapting-large-english-language/) - [next → Normalizing without Modernizing: Keeping Historical Wordforms of Middle French while Reducing Spelling Variants](https://md.research.elamur.ai/free/en/papers/rubino-2024-normalizing-without-modernizing-keeping-historical/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) ## References 9 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. [ARBERT & MARBERT: Deep bidirectional transformers for Arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.acl-long.551) 2. Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Abinew Ali Ayele, Pavan Baswani, Meriem Beloucif, Chris Biemann, Sofia Bourhim, Christine De Kock, Genet Shanko Dekebo, Oumaima Hourrane, Gopichand Kanumolu, Lokesh Madasu, Samuel Rutunda, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Hailegnaw Getaneh Tilaye, Krishnapriya Vishnubhotla, Genta Winata, Seid Muhie Yimam, and Saif M. Mohammad. 2024a. Semrel2024: A collection of semantic textual relatedness datasets for 14 languages. [find](https://scholar.google.com/scholar?q=Ahmad%2C+Sanchit+Ahuja%2C+Alham+Fikri+Aji%2C+Vladimir+Araujo%2C+Abinew+Ali+Ayele%2C+Pavan+Baswani%2C+Meriem+Beloucif%2C+Chris+Biemann%2C+Sofia+Bourhim%2C+Christine+De+Kock%2C+Genet+Shanko+Dekebo%2C+Oumaima+Hourrane%2C+Gopich) 3. Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Meriem Beloucif, Christine De Kock, Oumaima Hourrane, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Krishnapriya Vishnubhotla, Seid Muhie Yimam, and Saif M. Mohammad. 2024b. SemEval-2024 task 1: Semantic textual relatedness. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nedjma+Ousidhoum%2C+Shamsuddeen+Hassan+Muhammad%2C+Mohamed+Abdalla%2C+Idris+Abdulmumin%2C+Ibrahim+Said+Ahmad%2C+Sanchit+Ahuja%2C+Alham+Fikri+Aji%2C+Vladimir+Araujo%2C+Meriem+Beloucif%2C+Christine+De+Kock%2C+Oumaima+Hourr) 4. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. 5. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-demos.6) 6. Kamel Gaanoun, Abdou Mohamed Naira, Anass Allak, and Imade Benelallam. 2024. Darijabert: a step forward in nlp for the written moroccan dialect. International Journal of Data Science and Analytics. [↗](https://doi.org/10.1007/s41060-023-00498-2) 7. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021a. [Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/). [↗](https://doi.org/10.48550/arxiv.2111.09543) 8. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021b. [Deberta: Decoding-enhanced bert with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2006.03654) 9. Diederik P. Kingma and Jimmy Ba. 2015. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said [↗](https://doi.org/10.48550/arxiv.1412.6980) *Updated 1 November 2026*