---
title: "Prediction of Arabic Legal Rulings using Large — Ammar 2023"
description: "The paper investigates whether current large language models can forecast the outcome of Arabic court cases, a legal-NLP area that has received little…"
authors: ["Adel Ammar", "Anis Koubaa", "Bilel Benjdira", "Omar Najar", "Serry Sibaee"]
year: 2023
venue: "arXiv:2310.10260"
source: "https://arxiv.org/abs/2310.10260"
doi: "10.48550/arxiv.2310.10260"
arxiv: "2310.10260"
licence: "CC BY-NC-ND 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ammar-2023-prediction-arabic-legal-rulings-using/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ammar-2023-prediction-arabic-legal-rulings-using/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Prediction of Arabic Legal Rulings using Large Language Models

# Prediction of Arabic Legal Rulings using Large Language Models

**Authors:** [Adel Ammar](https://md.research.elamur.ai/free/en/authors/adel-ammar/), [Anis Koubaa](https://md.research.elamur.ai/free/en/authors/anis-koubaa/), [Bilel Benjdira](https://md.research.elamur.ai/free/en/authors/bilel-benjdira/), [Omar Najar](https://md.research.elamur.ai/free/en/authors/omar-najar/), [Serry Sibaee](https://md.research.elamur.ai/free/en/authors/serry-sibaee/) · **Year:** 2023 · **Venue:** arXiv:2310.10260 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.10260) · [PDF](https://arxiv.org/pdf/2310.10260) · [DOI 10.48550/arxiv.2310.10260](https://doi.org/10.48550/arxiv.2310.10260) · [arXiv 2310.10260](https://arxiv.org/abs/2310.10260) · [OpenAlex](https://openalex.org/W4391843256)

**Citations:** 20 citations (openalex-snapshot · 2026-10-14) · [7](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper investigates whether current large language models can forecast the outcome of Arabic court cases, a legal-NLP area that has received little attention. The authors use 10,813 real commercial court cases and compare three base models: LLaMA-7b, JAIS-13b and GPT-3.5-turbo. Each model is run zero-shot, one-shot or after task-specific fine-tuning. Some variants first summarize and/or translate the Arabic input into English. This gives 14 configurations.

Outputs were scored by human judges, by GPT-3.5 acting as a grader, and with ROUGE and BLEU. GPT-3.5 configurations were clearly the strongest, with an average score roughly 50% above the Arabic-focused JAIS. LLaMA variants performed poorly. Even the best results remain weak, around 2.4 out of 5 from human raters.

A second finding concerns evaluation. The automatic metrics often disagreed with the human judgments, and this was worst for translated inputs. The study therefore functions both as an early Arabic legal-prediction baseline and as a warning against relying on automatic scores for open-ended legal generation.

## Key points
- **Dataset and models:** 10,813 real Arabic commercial court cases were tested with LLaMA-7b, JAIS-13b-chat and GPT-3.5-turbo.
- **Variants:** Combining training regimes (zero-shot, one-shot, fine-tuning) with input preprocessing (summarization, translation to English) produced 14 configurations.
- **Arabic share of training data:** Arabic made up about 29% of JAIS training data. For GPT-3 it was only about 0.03% by words. Despite this, GPT-3.5 variants won by a wide margin.
- **Agreement between human and GPT scores:** The correlation was 0.92 for Arabic-input models but only 0.60 for translated-input models.
- **ROUGE precision:** ROUGE-1 precision and ROUGE-L precision correlated negatively with human scores (−0.79 and −0.77). The best human score was about 2.4/5.

## Interesting ideas
- Before trusting an automatic metric, compute its correlation with human ratings separately for each pipeline condition; a metric can look acceptable overall yet collapse or even invert on one subgroup. — For Mova (learn): Evaluation should report metric-vs-human agreement per input route (e.g. native vs converted text) as a standing negative control, so a misleading score is caught before it drives training choices.
- An LLM grader gave its top rating to a configuration that humans judged only moderate, showing that a model-as-judge can reward plausible-sounding output. — For Mova (learn): Answers rated by LLM teacher tiers should be spot-audited against human or rule-checked gold, and tiers whose ratings drift from those audits should be down-weighted.
- Translating the input before processing seemed to drop or distort key details, which hurt both the predictions and the reliability of their grading. — For Mova (back): When translating through the meta-world, check that the entities and facts in the source graph survive the round trip, rather than judging fluency of the output alone.
- The strongest LLaMA variant borrowed GPT-3.5's summaries and translations, so its gains came partly from another system; the authors flag that it is not a pure result. — For Mova (learn): Ablations should attribute improvements to the component actually responsible, e.g. silver data from a stronger witness parser versus a change in our own model.
- The authors plotted input length against human score to see whether long cases are harder. — For Mova (answer): Breaking QA and test accuracy down by text length or depth would show whether failures come from long-range tracking in the world model rather than from local parsing.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Law](https://md.research.elamur.ai/free/en/shelves/law/)

## Cited by

Cited by

7 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 4 not yet.

- [How well can LLMs Grade Essays in Arabic?](https://md.research.elamur.ai/free/en/papers/ghazawi-2025-well-llms-grade-essays-arabic/)
- [Can Large Language Models Predict the Outcome of Judicial Decisions?](https://md.research.elamur.ai/free/en/papers/kmainasi-2025-large-language-models-predict-outcome/)
- [MURAD: A Large-Scale Multi-Domain Unified Reverse Arabic Dictionary Dataset](https://md.research.elamur.ai/free/en/papers/sibaee-2026-murad-large-scale-multi-domain/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ammar2023prediction,
  title = {{Prediction of Arabic Legal Rulings using Large Language Models}},
  author = {Ammar, Adel and Koubaa, Anis and Benjdira, Bilel and Najar, Omar and Sibaee, Serry},
  year = {2023},
  doi = {10.48550/arxiv.2310.10260},
  eprint = {2310.10260},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.10260}
}
```

- [← previous Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://md.research.elamur.ai/free/en/papers/fried-2023-pragmatics-language-grounding-phenomena-tasks/)
- [next → Preference Ranking Optimization for Human Alignment](https://md.research.elamur.ai/free/en/papers/song-2023-preference-ranking-optimization-human-alignment/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

47 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Habash, N.: Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers, ??? (2010) [find](https://scholar.google.com/scholar?q=Habash%2C+N%2E%3A+Introduction+to+Arabic+Natural+Language+Processing%2E+Morgan+%26+Claypool+Publishers%2C+%3F%3F%3F+%282010%29)
2. Attia, M.: Handling arabic morphological and syntactic ambiguity within the lfg framework with a view to machine translation. PhD thesis, University of Manchester (2008) [find](https://scholar.google.com/scholar?q=Attia%2C+M%2E%3A+Handling+arabic+morphological+and+syntactic+ambiguity+within+the+lfg+framework+with+a+view+to+machine+translation%2E+PhD+thesis%2C+University+of+Manchester+%282008%29)
3. Dai, A.M., Le, Q.V.: Semi-supervised sequence learning. Advances in neural information processing systems 28 (2015) [find](https://scholar.google.com/scholar?q=Dai%2C+A%2EM%2E%2C+Le%2C+Q%2EV%2E%3A+Semi%2Dsupervised+sequence+learning%2E+Advances+in+neural+information+processing+systems+28+%282015%29)
4. Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 (2018) [↗](https://arxiv.org/abs/1801.06146)
5. Katz, D.M., Bommarito, M.J., Blackman, J.: A general approach for predicting the behavior of the supreme court of the united states. PloS one 12(4), 0174698 (2017) [find](https://scholar.google.com/scholar?q=Katz%2C+D%2EM%2E%2C+Bommarito%2C+M%2EJ%2E%2C+Blackman%2C+J%2E%3A+A+general+approach+for+predicting+the+behavior+of+the+supreme+court+of+the+united+states%2E+PloS+one+12%284%29%2C+0174698+%282017%29)
6. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017) [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+et+al%2E%3A+Attention+is+all+you+need%2E+In%3A+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+30+%282017%29)
7. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, vol. 27 (2014) [find](https://scholar.google.com/scholar?q=Sutskever%2C+I%2E%2C+Vinyals%2C+O%2E%2C+Le%2C+Q%2EV%2E%3A+Sequence+to+sequence+learning+with+neural+networks%2E+In%3A+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+27+%282014%29)
8. Almusharraf, M.A.M.I.X.D.N.: Automated and human interaction in written discourse: A contrastive parallel corpus-based investigation of metadiscourse features in machine-human translations. Sage Open (2022) https://doi.org/10.1177/21582440221142210 [↗](https://doi.org/10.1177/21582440221142210)
9. Maas, A.L., et al.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (2011) [find](https://scholar.google.com/scholar?q=Maas%2C+A%2EL%2E%2C+et+al%2E%3A+Learning+word+vectors+for+sentiment+analysis%2E+In%3A+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technologies+%282011%29) [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)
10. Khan, A.: Improved multi-lingual sentiment analysis and recognition using deep learning. Journal Of Information Science (2023) https://doi.org/10.1177/01655515221137270 [↗](https://doi.org/10.1177/01655515221137270)
11. Chaudhry, H.N., Javed, Y., Kulsoom, F., Mehmood, Z., Khan, Z.I., Shoaib, U., Janjua, S.H.: Sentiment analysis of before and after elections: Twitter data of us election 2020. Electronics 10(17), 2082 (2021) [find](https://scholar.google.com/scholar?q=Chaudhry%2C+H%2EN%2E%2C+Javed%2C+Y%2E%2C+Kulsoom%2C+F%2E%2C+Mehmood%2C+Z%2E%2C+Khan%2C+Z%2EI%2E%2C+Shoaib%2C+U%2E%2C+Janjua%2C+S%2EH%2E%3A+Sentiment+analysis+of+before+and+after+elections%3A+Twitter+data+of+us+election+2020%2E+Electronics+10%2817%29%2C+2082+)
12. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al.: Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 (2022) [↗](https://arxiv.org/abs/2211.09110)
13. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A.A.M., Abid, A., Fisch, A., Brown, A.R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al.: Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 (2022) [↗](https://arxiv.org/abs/2206.04615) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
14. Elmadany, A., Nagoudi, E.M.B., Abdul-Mageed, M.: Orca: A challenging benchmark for arabic language understanding. arXiv preprint arXiv:2212.10758 (2022) [↗](https://arxiv.org/abs/2212.10758)
15. Abdelali, A., Mubarak, H., Chowdhury, S.A., Hasanain, M., Mousi, B., Boughorbel, S., Kheir, Y.E., Izham, D., Dalvi, F., Hawasly, M., et al.: Benchmarking arabic ai with large language models. arXiv preprint arXiv:2305.14982 (2023) [↗](https://arxiv.org/abs/2305.14982)
16. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356 (2022) [↗](https://arxiv.org/abs/2212.04356) [our card](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/)
17. Zhang, Y., Han, W., Qin, J., Wang, Y., Bapna, A., Chen, Z., Chen, N., Li, B., Axelrod, V., Wang, G., et al.: Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037 (2023) [↗](https://arxiv.org/abs/2303.01037)
18. White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., Schmidt, D.C.: A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023) [↗](https://arxiv.org/abs/2302.11382) [our card](https://md.research.elamur.ai/free/en/papers/white-2023-prompt-pattern-catalog-enhance-prompt/)
19. Zhou, Y., Muresanu, A.I., Han, Z., Paster, K., Pitis, S., Chan, H., Ba, J.: Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910 (2022) [↗](https://arxiv.org/abs/2211.01910) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/)
20. Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., Singh, S.: Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 (2020) [↗](https://arxiv.org/abs/2010.15980) [our card](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/)
21. Lauderdale, B.E., Clark, T.S.: Scaling politically meaningful dimensions using texts and votes. American Journal of Political Science 58(3), 754–771 (2014) [find](https://scholar.google.com/scholar?q=Lauderdale%2C+B%2EE%2E%2C+Clark%2C+T%2ES%2E%3A+Scaling+politically+meaningful+dimensions+using+texts+and+votes%2E+American+Journal+of+Political+Science+58%283%29%2C+754%E2%80%93771+%282014%29)
22. Medvedeva, M., Vols, M., Wieling, M.: Using machine learning to predict decisions of the european court of human rights. Artificial Intelligence and Law 27(3), 237–266 (2019) [find](https://scholar.google.com/scholar?q=Medvedeva%2C+M%2E%2C+Vols%2C+M%2E%2C+Wieling%2C+M%2E%3A+Using+machine+learning+to+predict+decisions+of+the+european+court+of+human+rights%2E+Artificial+Intelligence+and+Law+27%283%29%2C+237%E2%80%93266+%282019%29)
23. AL-Qurishi, M., AlQaseemi, S., Soussi, R.: AraLegal-BERT: A pretrained language model for Arabic Legal text (2022) [find](https://scholar.google.com/scholar?q=AL%2DQurishi%2C+M%2E%2C+AlQaseemi%2C+S%2E%2C+Soussi%2C+R%2E%3A+AraLegal%2DBERT%3A+A+pretrained+language+model+for+Arabic+Legal+text+%282022%29)
24. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Lee%2C+K%2E%2C+Toutanova%2C+K%2E%3A+BERT%3A+Pre%2Dtraining+of+Deep+Bidirectional+Transformers+for+Language+Understanding+%282019%29)
25. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023) [↗](https://arxiv.org/abs/2302.13971)
26. Sengupta, N., Sahu, S.K., Jia, B., Katipomu, S., Li, H., Koto, F., Afzal, O.M., Kamboj, S., Pandit, O., Pal, R., et al.: Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149 (2023) [↗](https://arxiv.org/abs/2308.16149) [our card](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/)
27. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020) [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2ED%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E%3A+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+informati)
28. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, 27730–27744 (2022) [find](https://scholar.google.com/scholar?q=Ouyang%2C+L%2E%2C+Wu%2C+J%2E%2C+Jiang%2C+X%2E%2C+Almeida%2C+D%2E%2C+Wainwright%2C+C%2E%2C+Mishkin%2C+P%2E%2C+Zhang%2C+C%2E%2C+Agarwal%2C+S%2E%2C+Slama%2C+K%2E%2C+Ray%2C+A%2E%2C+et+al%2E%3A+Training+language+models+to+follow+instructions+with+human+feedback%2E+Advanc)
29. Koubaa, A.: Gpt-4 vs. gpt-3.5: A concise showdown. Preprints (2023) https://doi.org/10.20944/preprints202303.0422.v1 [↗](https://doi.org/10.20944/preprints202303.0422.v1)
30. Floridi, L., Chiriatti, M.: Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines 30, 681–694 (2020) [find](https://scholar.google.com/scholar?q=Floridi%2C+L%2E%2C+Chiriatti%2C+M%2E%3A+Gpt%2D3%3A+Its+nature%2C+scope%2C+limits%2C+and+consequences%2E+Minds+and+Machines+30%2C+681%E2%80%93694+%282020%29)
31. OpenAI: GPT3 Dataset Language Statistics. [https://github.com/openai/gpt-3/tree/master/dataset](https://github.com/openai/gpt-3/tree/master/dataset) statistics. Accessed on 09 October 2023 [↗](https://github.com/openai/gpt-3/tree/master/dataset)
32. Shibata, Y., Kida, T., Fukamachi, S., Takeda, M., Shinohara, A., Shinohara, T., Arikawa, S.: Byte pair encoding: A text compression scheme that accelerates pattern matching (1999) [find](https://scholar.google.com/scholar?q=Shibata%2C+Y%2E%2C+Kida%2C+T%2E%2C+Fukamachi%2C+S%2E%2C+Takeda%2C+M%2E%2C+Shinohara%2C+A%2E%2C+Shinohara%2C+T%2E%2C+Arikawa%2C+S%2E%3A+Byte+pair+encoding%3A+A+text+compression+scheme+that+accelerates+pattern+matching+%281999%29)
33. Bostrom, K., Durrett, G.: Byte pair encoding is suboptimal for language model pretraining. arXiv preprint arXiv:2004.03720 (2020) [↗](https://arxiv.org/abs/2004.03720)
34. Kudo, T., Richardson, J.: Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226 (2018) [↗](https://arxiv.org/abs/1808.06226) [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
35. Pu, G., Jain, A., Yin, J., Kaplan, R.: Empirical analysis of the strengths and weaknesses of peft techniques for llms. arXiv preprint arXiv:2304.14999 (2023) [↗](https://arxiv.org/abs/2304.14999)
36. Hu, Z., Lan, Y., Wang, L., Xu, W., Lim, E.-P., Lee, R.K.-W., Bing, L., Poria, S.: Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933 (2023) [↗](https://arxiv.org/abs/2304.01933) [our card](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/)
37. AGI-Edgerunners: LLM-Adapters Github repository. [https://github.com/AGI-Edgerunners/LLM-Adapters](https://github.com/AGI-Edgerunners/LLM-Adapters). Accessed on 09 October 2023 [↗](https://github.com/AGI-Edgerunners/LLM-Adapters)
38. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021) [↗](https://arxiv.org/abs/2106.09685)
39. SJP: Saudi Justice Portal. [https://sjp.moj.gov.sa](https://sjp.moj.gov.sa). Accessed on 05 October 2023 [↗](https://sjp.moj.gov.sa)
40. PyPI: Selenium Python library. [https://pypi.org/project/selenium](https://pypi.org/project/selenium). Accessed on 05 October 2023 [↗](https://pypi.org/project/selenium)
41. PyPI: Beautiful Soup Python package. [https://pypi.org/project/bs4](https://pypi.org/project/bs4). Accessed on 04 October 2023 [↗](https://pypi.org/project/bs4)
42. PyPI: translators Python package. Accessed on 28 September 2023 [find](https://scholar.google.com/scholar?q=PyPI%3A+translators+Python+package%2E+Accessed+on+28+September+2023)
43. Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (2002) [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%2C+Roukos%2C+S%2E%2C+Ward%2C+T%2E%2C+Zhu%2C+W%2E%2DJ%2E%3A+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In%3A+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Computational+Linguist) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
44. Chen, B., Cherry, C.: A systematic comparison of smoothing techniques for sentence-level bleu. In: Proceedings of the Ninth Workshop on Statistical Machine [https://pypi.org/project/translators/](https://pypi.org/project/translators/). Translation, pp. 362–367 (2014) [↗](https://pypi.org/project/translators/) [our card](https://md.research.elamur.ai/free/en/papers/chen-2014-systematic-comparison-smoothing-techniques-sentence/)
45. NLTK: Bleu Python package. [https://www.nltk.org/api/nltk.translate.bleu](https://www.nltk.org/api/nltk.translate.bleu) score.html. Accessed on 04 October 2023 [↗](https://www.nltk.org/api/nltk.translate.bleu)
46. Lin, C.-Y.: Rouge: A package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004) [find](https://scholar.google.com/scholar?q=Lin%2C+C%2E%2DY%2E%3A+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In%3A+Text+Summarization+Branches+Out%2C+pp%2E+74%E2%80%9381+%282004%29)
47. PyPI: rouge Python package. [https://pypi.org/project/rouge](https://pypi.org/project/rouge). Accessed on 04 October 2023 [↗](https://pypi.org/project/rouge)