---
title: "ASOS at Arabic LLMs Hallucinations 2024: Can LLMs detect their Hallucinations :) — Sibaee 2024"
description: "The paper describes the ASOS team's entry to Subtask A of the OSACT 2024 shared task. 44% F1. 23 references, cited by 2 papers here."
authors: ["Serry Taiseer Sibaee", "Abdullah I. Alharbi", "Samar Ahmed", "Omar Nacar", "Lahouri Ghouti", "Anis Koubaa"]
year: 2024
venue: "Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024"
source: "https://aclanthology.org/2024.osact-1.17/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sibaee-2024-asos-arabic-llms-hallucinations-2024/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sibaee-2024-asos-arabic-llms-hallucinations-2024/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / ASOS at Arabic LLMs Hallucinations 2024: Can LLMs detect their Hallucinations :)

# ASOS at Arabic LLMs Hallucinations 2024: Can LLMs detect their Hallucinations :)

**Authors:** [Serry Taiseer Sibaee](https://md.research.elamur.ai/free/en/authors/serry-sibaee/), [Abdullah I. Alharbi](https://md.research.elamur.ai/free/en/authors/abdullah-alharbi/), [Samar Ahmed](https://md.research.elamur.ai/free/en/authors/samar-ahmed/), Omar Nacar, [Lahouri Ghouti](https://md.research.elamur.ai/free/en/authors/lahouri-ghouti/) and [Anis Koubaa](https://md.research.elamur.ai/free/en/authors/anis-koubaa/) · **Year:** 2024 · **Venue:** Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024 · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.osact-1.17/) · [PDF](https://aclanthology.org/2024.osact-1.17.pdf) · [OpenAlex](https://openalex.org/W7131797684)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper describes the ASOS team's entry to Subtask A of the OSACT 2024 shared task. The task asks systems to label Arabic factual claims produced by ChatGPT and GPT-4 as FC (factually correct), FI (factually incorrect) or NF (non-factual, meaning not a declarative claim). The abstract expands these labels differently, but the data section uses this reading. Only the claim text and its label were allowed as inputs.

The team first tried supervised baselines: fine-tuning AraBERT, and a classifier on multilingual sentence embeddings. Both were weak. They then switched to zero-shot classification with three LLMs, each given one carefully iterated prompt that also fixes the output format:
- GPT-4 and Gemini read the Arabic directly.
- Mistral 7B received Google-translated English.

They also tried majority voting across the three models. GPT-4 was the strongest, with a macro F1 of 0.54 on dev and 0.516 on test.

Every model struggled most with factually incorrect claims, which differ from true ones only by subtle changes. The paper is mainly a useful, if modest, data point: general LLMs used as fact-checkers on a morphologically rich language perform weakly.

## Key points
- **Data and inputs:** the dataset is tab-separated, and each row stores the claim's ID, the model that generated it, its word position, its readability, the claim text itself and the label. Subtask A permits only the claim and label columns.
- **Failed baselines:**
  - Fine-tuned AraBERT peaked at 44% F1.
  - "distiluse base multilingual cased" sentence embeddings (512-dim) fed to several neural network architectures gave poor results.
- **Dev results (P / R / F1):**

  | Model | Precision | Recall | F1 |
  |---|---|---|---|
  | GPT-4 | 0.67 | 0.51 | 0.54 |
  | Mistral 7B | 0.67 | 0.43 | 0.42 |
  | Gemini | 0.58 | 0.38 | 0.34 |

- **Submitted run:** GPT-4 alone, scoring 0.663 precision, 0.495 recall and 0.516 F1 on test.
- **Per-class behaviour (from confusion matrices):**
  - FI was the weakest class for every model.
  - Gemini did well on NF but had low overall recall.
  - Majority voting raised recall on FC only, not on FI or NF.

## Interesting ideas
- **Format-locking prompt:** the prompt was refined over several rounds so that it both defines the three labels and forces a fixed answer format, which makes LLM output machine-parseable for a classification pipeline.
- **Per-class ensemble analysis:** instead of reporting only one ensemble score, confusion matrices showed that voting helped one class and left the others unchanged, so heterogeneous voters do not fix shared blind spots.
- **Non-declarative as a separate class:** the label set keeps "this is not a checkable claim" apart from true and false, so verification is only attempted on actual assertions.
- **Subtle alterations are hardest:** factually incorrect claims that differ only slightly from correct ones were the hardest case for all LLMs, suggesting surface fluency judgments miss small factual edits.
- **Translate-then-classify routing:** a translation step put text into the language a model handles best, at the risk of losing nuance (Mistral 7B, the model fed translated input, scored below GPT-4).

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): LLM teacher tiers could use format-locked prompts so that teacher labels on frames or trees can be parsed and compared against silver data without manual cleanup.
- **I2** (learn): when several witness parsers agree to produce silver data, agreement rates should be reported per label or relation, since witnesses may all fail on the same rare categories.
- **I3** (frames): the speech-act layer could first decide whether a clause asserts something before the world model tries to confirm or refute it.
- **I4** (world): checking claims against explicit entities, states and the knowledge graph targets exactly these small factual swaps, and minimally altered claims could be used as contrastive test items.
- **I5** (back): this is a cautionary baseline for comparing direct translation against translation through the meta-world, measuring whether factual details survive the round trip.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [🐪 Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/)

## Cite

Cite as: Serry Taiseer Sibaee, Abdullah I. Alharbi, Samar Ahmed, Omar Nacar, Lahouri Ghouti and Anis Koubaa. 2024. ASOS at Arabic LLMs Hallucinations 2024: Can LLMs detect their Hallucinations :). Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sibaee2024asos,
  title = {{ASOS at Arabic LLMs Hallucinations 2024: Can LLMs detect their Hallucinations :)}},
  author = {Serry Taiseer Sibaee, Abdullah I. Alharbi, Samar Ahmed, Omar Nacar, Lahouri Ghouti and Anis Koubaa},
  booktitle = {Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/2crdvsc554cw},
  url = {https://doi.org/10.63317/2crdvsc554cw}
}
```

- [← previous Arabic Text Diacritization In The Age Of Transfer Learning: Token Classification Is All You Need](https://md.research.elamur.ai/free/en/papers/skiredj-2024-arabic-text-diacritization-age-transfer/)
- [next → Can ChatGPT capture swearing nuances? Evidence from translating Arabic oaths](https://md.research.elamur.ai/free/en/papers/shormani-2024-chatgpt-capture-swearing-nuances-evidence/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. BSI. 1973a. Natural Fibre Twines, 3rd edition. British Standards Institution, London. BS 2570. [find](https://scholar.google.com/scholar?q=BSI%2E+1973a%2E+Natural+Fibre+Twines%2C+3rd+edition%2E+British+Standards+Institution%2C+London%2E+BS+2570%2E)
2. BSI. 1973b. Natural fibre twines. BS 2570, British Standards Institution, London. 3rd. edn. [find](https://scholar.google.com/scholar?q=BSI%2E+1973b%2E+Natural+fibre+twines%2E+BS+2570%2C+British+Standards+Institution%2C+London%2E+3rd%2E+edn%2E)
3. A. Castor and L. E. Pollux. 1992. The use of user modelling to guide inference and learning. Applied Intelligence, 2(1):37–53. [find](https://scholar.google.com/scholar?q=A%2E+Castor+and+L%2E+E%2E+Pollux%2E+1992%2E+The+use+of+user+modelling+to+guide+inference+and+learning%2E+Applied+Intelligence%2C+2%281%29%3A37%E2%80%9353%2E)
4. J.L. Chercheur. 1994. Case-Based Reasoning, 2nd edition. Morgan Kaufman Publishers, San Mateo, CA. [find](https://scholar.google.com/scholar?q=J%2EL%2E+Chercheur%2E+1994%2E+Case%2DBased+Reasoning%2C+2nd+edition%2E+Morgan+Kaufman+Publishers%2C+San+Mateo%2C+CA%2E)
5. N. Chomsky. 1973. Conditions on transformations. In A festschrift for Morris Halle, New York. Holt, Rinehart & Winston. [find](https://scholar.google.com/scholar?q=N%2E+Chomsky%2E+1973%2E+Conditions+on+transformations%2E+In+A+festschrift+for+Morris+Halle%2C+New+York%2E+Holt%2C+Rinehart+%26+Winston%2E)
6. Umberto Eco. 1990. The Limits of Interpretation. Indian University Press. [find](https://scholar.google.com/scholar?q=Umberto+Eco%2E+1990%2E+The+Limits+of+Interpretation%2E+Indian+University+Press%2E)
7. Paul Gerhard Hoel. 1971a. Elementary Statistics, 3rd edition. Wiley series in probability and mathematical statistics. Wiley, New York, Chichester. ISBN 0 471 40300. [find](https://scholar.google.com/scholar?q=Paul+Gerhard+Hoel%2E+1971a%2E+Elementary+Statistics%2C+3rd+edition%2E+Wiley+series+in+probability+and+mathematical+statistics%2E+Wiley%2C+New+York%2C+Chichester%2E+ISBN+0+471+40300%2E)
8. Paul Gerhard Hoel. 1971b. Elementary Statistics, 3rd edition, Wiley series in probability and mathematical statistics, pages 19–33. Wiley, New York, Chichester. ISBN 0 471 40300. [find](https://scholar.google.com/scholar?q=Paul+Gerhard+Hoel%2E+1971b%2E+Elementary+Statistics%2C+3rd+edition%2C+Wiley+series+in+probability+and+mathematical+statistics%2C+pages+19%E2%80%9333%2E+Wiley%2C+New+York%2C+Chichester%2E+ISBN+0+471+40300%2E)
9. Otto Jespersen. 1922. Language: Its Nature, Development, and Origin. Allen and Unwin. [find](https://scholar.google.com/scholar?q=Otto+Jespersen%2E+1922%2E+Language%3A+Its+Nature%2C+Development%2C+and+Origin%2E+Allen+and+Unwin%2E)
10. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. [our card](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
11. Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Halueval: A large-scale hallucination evaluation benchmark for large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing. [our card](https://md.research.elamur.ai/free/en/papers/li-2023-halueval-large-scale-hallucination-evaluation/)
12. Yuxin Liang, Zhuoyang Song, Hao Wang, and Jiaxing Zhang. 2024. Learning to trust your feelings: Leveraging self-awareness in llms for hallucination mitigation. arXiv preprint arXiv:2401.15449. [↗](https://arxiv.org/abs/2401.15449)
13. Junliang Luo, Tianyu Li, Di Wu, Michael Jenkin, Steve Liu, and Gregory Dudek. 2024. Hallucination detection and hallucination mitigation: An investigation. arXiv preprint arXiv:2401.08358. [↗](https://arxiv.org/abs/2401.08358)
14. OpenAI, :, and Josh Achiam etal. 2024. Gpt-4 technical report. [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
15. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. [our card](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/)
16. Charles Joseph Singer, E. J. Holmyard, and A. R. Hall, editors. 1954–58. A history of technology. Oxford University Press, London. 5 vol. [find](https://scholar.google.com/scholar?q=Charles+Joseph+Singer%2C+E%2E+J%2E+Holmyard%2C+and+A%2E+R%2E+Hall%2C+editors%2E+1954%E2%80%9358%2E+A+history+of+technology%2E+Oxford+University+Press%2C+London%2E+5+vol%2E)
17. Ben Snyder, Marius Moisescu, and Muhammad Bilal Zafar. 2023. On early detection of hallucinations in factual question answering. arXiv preprint arXiv:2312.14183. [↗](https://arxiv.org/abs/2312.14183)
18. Jannik Strötgen and Michael Gertz. 2012. Temporal tagging on different domains: Challenges, strategies, and gold standards. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 3746– 3753, Istanbul, Turkey. European Language Resource Association (ELRA). [our card](https://md.research.elamur.ai/free/en/papers/strotgen-2012-temporal-tagging-different-domains-challenges/)
19. S. Superman, B. Batman, C. Catwoman, and S. Spiderman. 2000. Superheroes experiences with books, 20th edition. The Phantom Editors Associates, Gotham City. [find](https://scholar.google.com/scholar?q=S%2E+Superman%2C+B%2E+Batman%2C+C%2E+Catwoman%2C+and+S%2E+Spiderman%2E+2000%2E+Superheroes+experiences+with+books%2C+20th+edition%2E+The+Phantom+Editors+Associates%2C+Gotham+City%2E)
20. Gemini Team and etal Rohan Anil. 2023. Gemini: A family of highly capable multimodal models. [our card](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/)
21. SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313. [↗](https://arxiv.org/abs/2401.01313) [our card](https://md.research.elamur.ai/free/en/papers/tonmoy-2024-comprehensive-survey-hallucination-mitigation-techniques/)
22. Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv preprint arXiv:2307.03987. [↗](https://arxiv.org/abs/2307.03987) [our card](https://md.research.elamur.ai/free/en/papers/varshney-2023-stitch-time-saves-nine-detecting/)
23. Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. 2023. Cognitive mirage: A review of hallucinations in large language models. arXiv preprint arXiv:2309.06794. [↗](https://arxiv.org/abs/2309.06794)