---
title: "Étude des facteurs de complexité des modèles — Antoine 2024"
description: "The authors propose a methodology that looks for complexity factors inherent to the NLP task itself rather than to model size."
authors: ["Elie Antoine", "Frederic Bechet", "Géraldine Damnati", "Philippe Langlais"]
year: 2024
venue: "Actes de la 31ème Conférence sur le Traitement Automatique des Langues Naturelles, volume 1 : articles longs et prises de position"
source: "https://aclanthology.org/2024.jeptalnrecital-taln.27/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/antoine-2024-etude-des-facteurs-de-complexite/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/antoine-2024-etude-des-facteurs-de-complexite/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2024](https://md.research.elamur.ai/free/papers/2024/page/4/) / Étude des facteurs de complexité des modèles de langage dans une tâche de compréhension de lecture à l'aide d'une expérience contrôlée sémantiquement

# Étude des facteurs de complexité des modèles de langage dans une tâche de compréhension de lecture à l'aide d'une expérience contrôlée sémantiquement

**Authors:** Elie Antoine, [Frederic Bechet](https://md.research.elamur.ai/free/authors/frederic-bechet/), Géraldine Damnati, Philippe Langlais · **Year:** 2024 · **Venue:** Actes de la 31ème Conférence sur le Traitement Automatique des Langues Naturelles, volume 1 : articles longs et prises de position · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.jeptalnrecital-taln.27/) · [PDF](https://aclanthology.org/2024.jeptalnrecital-taln.27.pdf) · [OpenAlex](https://openalex.org/W4399985342)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
The authors propose a methodology that looks for complexity factors inherent to the NLP task itself rather than to model size. They recommend moving away from broad benchmarks that evaluate a model as a "black box". Instead, they propose narrow evaluations: one task, one dataset and one language, described from a linguistic perspective. The approach was tested on a reading comprehension (question answering) task in French. For this, they used the Calor corpus, a collection of texts from Wikipedia and ClioTexte. Each question-answer pair in it is linked to Berkeley FrameNet semantic frames. Seven models of different architectures and sizes, from CamemBERT to GPT-3.5, were adapted on FQuAD. This was done through full fine-tuning, LoRA or few-shot prompting. The models were then evaluated on Calor using ROUGE-L and manual checking. It turned out that some examples are hard for all models regardless of their scale. These failures are linked to specific semantic factors. Based on these factors, the authors split Calor into subcorpora of gradually increasing complexity. The main conclusion: increasing the number of parameters alone is not enough to handle such examples. Hence, hard cases can also be studied on smaller models.

## Key points
- **The Calor corpus**: 1821 questions across 54 semantic frames. The texts cover mainly three topics: World War I, archaeology and antiquity. The answers are Frame Elements, and some of them require coreference resolution beyond a single sentence.
- Annotators wrote the questions without seeing the source sentence. In this way, the authors aimed for lexical diversity.
- **Models**:
  - CamemBERT (335M, classifier);
  - T5-LARGE, FLAN-T5-LARGE and MT5-LARGE (fine-tuned on FQuAD for 2 epochs);
  - LLAMA-2 7B (LoRA);
  - Mixtral-8x7B (47B) and GPT-3.5 (175B): both via prompts with one or two examples.
- **Results on the whole of Calor**:
  - ROUGE-L from 0.69 to 0.82, share of correct answers from 72 to 82 %;
  - the smallest model, CamemBERT (0.82 / 78 %), is on par with Mixtral (0.80 / 82 %) and GPT-3.5 (0.72 / 82 %);
  - the weakest result is LLAMA-2's (0.69 / 72 %).
- Calor is noticeably harder than FQuAD: its texts are specialized, and the reference answers are much longer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{antoine2024etude,
  title = {{Étude des facteurs de complexité des modèles de langage dans une tâche de compréhension de lecture à l'aide d'une expérience contrôlée sémantiquement}},
  author = {Antoine, Elie and Bechet, Frederic and Damnati, Géraldine and Langlais, Philippe},
  booktitle = {Actes de la 31ème Conférence sur le Traitement Automatique des Langues Naturelles, volume 1 : articles longs et prises de position},
  year = {2024},
  url = {https://aclanthology.org/2024.jeptalnrecital-taln.27/}
}
```

- [← previous Yi: Open Foundation Models by 01.AI](https://md.research.elamur.ai/free/papers/ai-2024-yi-open-foundation-models-01/)
- [next → τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://md.research.elamur.ai/free/papers/yao-2024-tau-bench-benchmark-tool/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/papers/2024/page/4/)

## References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. B AKER C. F., F ILLMORE C. J. & L OWE J. B. (1998). The berkeley framenet project. In COLING 1998 Volume 1 : The 17th International Conference on Computational Linguistics. [find](https://scholar.google.com/scholar?q=B+AKER+C%2E+F%2E%2C+F+ILLMORE+C%2E+J%2E+%26+L+OWE+J%2E+B%2E+%281998%29%2E+The+berkeley+framenet+project%2E+In+COLING+1998+Volume+1+%3A+The+17th+International+Conference+on+Computational+Linguistics%2E)
2. B ÉCHET F., A LOUI C., C HARLET D., D AMNATI G., H EINECKE J., N ASR A. & H ERLEDAN F. (2019). Calor-quest : generating a training corpus for machine reading comprehension models from shallow semantic annotations. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, p. 19–26… [find](https://scholar.google.com/scholar?q=B+%C3%89CHET+F%2E%2C+A+LOUI+C%2E%2C+C+HARLET+D%2E%2C+D+AMNATI+G%2E%2C+H+EINECKE+J%2E%2C+N+ASR+A%2E+%26+H+ERLEDAN+F%2E+%282019%29%2E+Calor%2Dquest+%3A+generating+a+training+corpus+for+machine+reading+comprehension+models+from+shallow+semantic)
3. B ROWN T., M ANN B., R YDER N., S UBBIAH M., K APLAN J. D., D HARIWAL P., N EELAKANTAN A., S HYAM P., S ASTRY G., A SKELL A. et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877–1901. [find](https://scholar.google.com/scholar?q=B+ROWN+T%2E%2C+M+ANN+B%2E%2C+R+YDER+N%2E%2C+S+UBBIAH+M%2E%2C+K+APLAN+J%2E+D%2E%2C+D+HARIWAL+P%2E%2C+N+EELAKANTAN+A%2E%2C+S+HYAM+P%2E%2C+S+ASTRY+G%2E%2C+A+SKELL+A%2E+et+al%2E+%282020%29%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+in)
4. D EVLIN J., C HANG M.-W., L EE K. & T OUTANOVA K. (2019). BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies, Volume 1 (L… [find](https://scholar.google.com/scholar?q=D+EVLIN+J%2E%2C+C+HANG+M%2E%2DW%2E%2C+L+EE+K%2E+%26+T+OUTANOVA+K%2E+%282019%29%2E+BERT+%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American)
5. G EHRMANN S., C LARK E. & S ELLAM T. (2023). Repairing the cracked foundation : A survey of obstacles in evaluation practices for generated text. J. Artif. Int. Res., 77. DOI : 10.1613/jair.1.13715. J IANG A. Q., S ABLAYROLLES A., R OUX A., M ENSCH A., S AVARY B., B AMFORD C., C HAPLOT D. S., C ASAS… [find](https://scholar.google.com/scholar?q=G+EHRMANN+S%2E%2C+C+LARK+E%2E+%26+S+ELLAM+T%2E+%282023%29%2E+Repairing+the+cracked+foundation+%3A+A+survey+of+obstacles+in+evaluation+practices+for+generated+text%2E+J%2E+Artif%2E+Int%2E+Res%2E%2C+77%2E+DOI+%3A+10%2E1613%2Fjair%2E1%2E13715%2E+J)
6. K HASHABI D., C HATURVEDI S., R OTH M., U PADHYAY S. & R OTH D. (2018). Looking beyond the surface : A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics : Human Languag… [find](https://scholar.google.com/scholar?q=K+HASHABI+D%2E%2C+C+HATURVEDI+S%2E%2C+R+OTH+M%2E%2C+U+PADHYAY+S%2E+%26+R+OTH+D%2E+%282018%29%2E+Looking+beyond+the+surface+%3A+A+challenge+set+for+reading+comprehension+over+multiple+sentences%2E+In+Proceedings+of+the+2018+Confe) [our card](https://md.research.elamur.ai/free/papers/khashabi-2018-looking-beyond-surface-challenge-set/)
7. L ASKAR M. T. R., B ARI M. S., R AHMAN M., B HUIYAN M. A. H., J OTY S. & H UANG J. (2023). A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. In A. R OGERS , J. B OYD -G RABER & N. O KAZAKI , Éds., Findings of the Association for Computational Linguistics : ACL 2023, p… [find](https://scholar.google.com/scholar?q=L+ASKAR+M%2E+T%2E+R%2E%2C+B+ARI+M%2E+S%2E%2C+R+AHMAN+M%2E%2C+B+HUIYAN+M%2E+A%2E+H%2E%2C+J+OTY+S%2E+%26+H+UANG+J%2E+%282023%29%2E+A+systematic+study+and+comprehensive+evaluation+of+ChatGPT+on+benchmark+datasets%2E+In+A%2E+R+OGERS+%2C+J%2E+B+OYD+%2DG)
8. L ECHELLE W., G OTTI F. & L ANGLAIS P. (2019). WiRe57 : A fine-grained benchmark for open information extraction. In A. F RIEDRICH , D. Z EYREK & J. H OEK , Éds., Proceedings of the 13th Linguistic Annotation Workshop, p. 6–15, Florence, Italy : Association for Computational Linguistics. DOI : 10.18… [find](https://scholar.google.com/scholar?q=L+ECHELLE+W%2E%2C+G+OTTI+F%2E+%26+L+ANGLAIS+P%2E+%282019%29%2E+WiRe57+%3A+A+fine%2Dgrained+benchmark+for+open+information+extraction%2E+In+A%2E+F+RIEDRICH+%2C+D%2E+Z+EYREK+%26+J%2E+H+OEK+%2C+%C3%89ds%2E%2C+Proceedings+of+the+13th+Linguistic+An)
9. L IU A., W U Z., M ICHAEL J., S UHR A., W EST P., K OLLER A., S WAYAMDIPTA S., S MITH N. & C HOI Y. (2023). We’re afraid language models aren’t modeling ambiguity. In H. B OUAMOR , J. P INO & K. B ALI , Éds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p.… [find](https://scholar.google.com/scholar?q=L+IU+A%2E%2C+W+U+Z%2E%2C+M+ICHAEL+J%2E%2C+S+UHR+A%2E%2C+W+EST+P%2E%2C+K+OLLER+A%2E%2C+S+WAYAMDIPTA+S%2E%2C+S+MITH+N%2E+%26+C+HOI+Y%2E+%282023%29%2E+We%E2%80%99re+afraid+language+models+aren%E2%80%99t+modeling+ambiguity%2E+In+H%2E+B+OUAMOR+%2C+J%2E+P+INO+%26+K%2E+B+ALI)
10. M ARTIN L., M ULLER B., S UÁREZ P. J. O., R OMARY L., DE L A C LERGERIE É. V., S EDDAH D., S AGOT B. et al. (2020). Camembert : a tasty french language model. In ACL 2020-58th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=M+ARTIN+L%2E%2C+M+ULLER+B%2E%2C+S+U%C3%81REZ+P%2E+J%2E+O%2E%2C+R+OMARY+L%2E%2C+DE+L+A+C+LERGERIE+%C3%89%2E+V%2E%2C+S+EDDAH+D%2E%2C+S+AGOT+B%2E+et+al%2E+%282020%29%2E+Camembert+%3A+a+tasty+french+language+model%2E+In+ACL+2020%2D58th+Annual+Meeting+of+the+As) [our card](https://md.research.elamur.ai/free/papers/martin-2019-camembert-tasty-french-language-model/)
11. M ARZINOTTO G., B ÉCHET F., D AMNATI G. & N ASR A. (2018). Sources of Complexity in Semantic Frame Parsing for Information Extraction. In International FrameNet Workshop 2018, Miyazaki, Japan. HAL : hal-01731385. [find](https://scholar.google.com/scholar?q=M+ARZINOTTO+G%2E%2C+B+%C3%89CHET+F%2E%2C+D+AMNATI+G%2E+%26+N+ASR+A%2E+%282018%29%2E+Sources+of+Complexity+in+Semantic+Frame+Parsing+for+Information+Extraction%2E+In+International+FrameNet+Workshop+2018%2C+Miyazaki%2C+Japan%2E+HAL+%3A+h)
12. R AJPURKAR P., Z HANG J., L OPYREV K. & L IANG P. (2016). Squad : 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, p. 2383–2392 : Association for Computational Linguistics. DOI : 10.18653/v1/D16-1264. [find](https://scholar.google.com/scholar?q=R+AJPURKAR+P%2E%2C+Z+HANG+J%2E%2C+L+OPYREV+K%2E+%26+L+IANG+P%2E+%282016%29%2E+Squad+%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+P) [our card](https://md.research.elamur.ai/free/papers/rajpurkar-2016-squad-100-000-questions-machine/)
13. R AVICHANDER A., G ARDNER M. & M ARASOVIC A. (2022). CONDAQA : A contrastive reading comprehension dataset for reasoning about negation. In Y. G OLDBERG , Z. K OZAREVA & Y. Z HANG , Éds., Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 8729–8755, Abu Dhabi,… [find](https://scholar.google.com/scholar?q=R+AVICHANDER+A%2E%2C+G+ARDNER+M%2E+%26+M+ARASOVIC+A%2E+%282022%29%2E+CONDAQA+%3A+A+contrastive+reading+comprehension+dataset+for+reasoning+about+negation%2E+In+Y%2E+G+OLDBERG+%2C+Z%2E+K+OZAREVA+%26+Y%2E+Z+HANG+%2C+%C3%89ds%2E%2C+Proceedings+)
14. R IBEIRO M. T., W U T., G UESTRIN C. & S INGH S. (2020). Beyond accuracy : Behavioral testing of NLP models with CheckList. In D. J URAFSKY , J. C HAI , N. S CHLUTER & J. T ETREAULT , Éds., Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 4902– 4912, Online… [find](https://scholar.google.com/scholar?q=R+IBEIRO+M%2E+T%2E%2C+W+U+T%2E%2C+G+UESTRIN+C%2E+%26+S+INGH+S%2E+%282020%29%2E+Beyond+accuracy+%3A+Behavioral+testing+of+NLP+models+with+CheckList%2E+In+D%2E+J+URAFSKY+%2C+J%2E+C+HAI+%2C+N%2E+S+CHLUTER+%26+J%2E+T+ETREAULT+%2C+%C3%89ds%2E%2C+Proceeding)
15. S RIVASTAVA A., R ASTOGI A., R AO A., S HOEB A. A. M., A BID A., F ISCH A., B ROWN A. R., S ANTORO A., G UPTA A. & A DRIÀ G ARRIGA -A LONSO E . A . (2023). Beyond the imitation game : Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=S+RIVASTAVA+A%2E%2C+R+ASTOGI+A%2E%2C+R+AO+A%2E%2C+S+HOEB+A%2E+A%2E+M%2E%2C+A+BID+A%2E%2C+F+ISCH+A%2E%2C+B+ROWN+A%2E+R%2E%2C+S+ANTORO+A%2E%2C+G+UPTA+A%2E+%26+A+DRI%C3%80+G+ARRIGA+%2DA+LONSO+E+%2E+A+%2E+%282023%29%2E+Beyond+the+imitation+game+%3A+Quantifying+and+) [our card](https://md.research.elamur.ai/free/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
16. T AN Y., M IN D., L I Y., L I W., H U N., C HEN Y. & Q I G. (2023). Can chatgpt replace traditional kbqa models ? an in-depth analysis of the question answering performance of the gpt llm family. In International Semantic Web Conference, p. 348–367 : Springer. [find](https://scholar.google.com/scholar?q=T+AN+Y%2E%2C+M+IN+D%2E%2C+L+I+Y%2E%2C+L+I+W%2E%2C+H+U+N%2E%2C+C+HEN+Y%2E+%26+Q+I+G%2E+%282023%29%2E+Can+chatgpt+replace+traditional+kbqa+models+%3F+an+in%2Ddepth+analysis+of+the+question+answering+performance+of+the+gpt+llm+family%2E+In+I)
17. T OUVRON H., M ARTIN L., S TONE K., A LBERT P., A LMAHAIRI A., B ABAEI Y., B ASHLYKOV N., B ATRA S., B HARGAVA P., B HOSALE S. et al. (2023). Llama 2 : Open foundation and fine-tuned chat models. arXiv preprint arXiv :2307.09288. [find](https://scholar.google.com/scholar?q=T+OUVRON+H%2E%2C+M+ARTIN+L%2E%2C+S+TONE+K%2E%2C+A+LBERT+P%2E%2C+A+LMAHAIRI+A%2E%2C+B+ABAEI+Y%2E%2C+B+ASHLYKOV+N%2E%2C+B+ATRA+S%2E%2C+B+HARGAVA+P%2E%2C+B+HOSALE+S%2E+et+al%2E+%282023%29%2E+Llama+2+%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXi)
18. T RUONG T. H., B ALDWIN T., V ERSPOOR K. & C OHN T. (2023). Language models are not naysayers : an analysis of language models on negation benchmarks. In A. P ALMER & J. C AMACHO -COLLADOS , Éds., Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), p. 101–114… [find](https://scholar.google.com/scholar?q=T+RUONG+T%2E+H%2E%2C+B+ALDWIN+T%2E%2C+V+ERSPOOR+K%2E+%26+C+OHN+T%2E+%282023%29%2E+Language+models+are+not+naysayers+%3A+an+analysis+of+language+models+on+negation+benchmarks%2E+In+A%2E+P+ALMER+%26+J%2E+C+AMACHO+%2DCOLLADOS+%2C+%C3%89ds%2E%2C+Pro)
19. T RUONG T. H., O TMAKHOVA Y., B ALDWIN T., C OHN T., L AU J. H. & V ERSPOOR K. (2022). Not another negation benchmark : The NaN-NLI test suite for sub-clausal negation. In Y. H E , H. J I , S. L I , Y. L IU & C.-H. C HANG , Éds., Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the A… [find](https://scholar.google.com/scholar?q=T+RUONG+T%2E+H%2E%2C+O+TMAKHOVA+Y%2E%2C+B+ALDWIN+T%2E%2C+C+OHN+T%2E%2C+L+AU+J%2E+H%2E+%26+V+ERSPOOR+K%2E+%282022%29%2E+Not+another+negation+benchmark+%3A+The+NaN%2DNLI+test+suite+for+sub%2Dclausal+negation%2E+In+Y%2E+H+E+%2C+H%2E+J+I+%2C+S%2E+L+I+%2C+Y)
20. W ANG A., P RUKSACHATKUN Y., N ANGIA N., S INGH A., M ICHAEL J., H ILL F., L EVY O. & B OWMAN S. (2019). Superglue : A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32. [find](https://scholar.google.com/scholar?q=W+ANG+A%2E%2C+P+RUKSACHATKUN+Y%2E%2C+N+ANGIA+N%2E%2C+S+INGH+A%2E%2C+M+ICHAEL+J%2E%2C+H+ILL+F%2E%2C+L+EVY+O%2E+%26+B+OWMAN+S%2E+%282019%29%2E+Superglue+%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understanding+systems%2E+Advances+i) [our card](https://md.research.elamur.ai/free/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
21. W EI J., B OSMA M., Z HAO V. Y., G UU K., Y U A. W., L ESTER B., D U N., D AI A. M. & L E Q. V. (2021). Finetuned language models are zero-shot learners. arXiv preprint arXiv :2109.01652. X UE L., C ONSTANT N., R OBERTS A., K ALE M., A L -R FOU R., S IDDHANT A., B ARUA A. & R AFFEL C. (2021). mt5 :… [find](https://scholar.google.com/scholar?q=W+EI+J%2E%2C+B+OSMA+M%2E%2C+Z+HAO+V%2E+Y%2E%2C+G+UU+K%2E%2C+Y+U+A%2E+W%2E%2C+L+ESTER+B%2E%2C+D+U+N%2E%2C+D+AI+A%2E+M%2E+%26+L+E+Q%2E+V%2E+%282021%29%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv+preprint+arXiv+%3A2109%2E01652%2E+X+UE+L%2E%2C+C+O)
22. Z HANG Y., Y ASUNAGA M., Z HOU Z., H AO C HEN J. Z., Z OU J., L IANG P. & Y EUNG S. (2023). Beyond positive scaling : How negation impacts scaling trends of language models. In A. R OGERS , J. B OYD -G RABER & N. O KAZAKI , Éds., Findings of the Association for Computational Linguistics : ACL 2023,… [find](https://scholar.google.com/scholar?q=Z+HANG+Y%2E%2C+Y+ASUNAGA+M%2E%2C+Z+HOU+Z%2E%2C+H+AO+C+HEN+J%2E+Z%2E%2C+Z+OU+J%2E%2C+L+IANG+P%2E+%26+Y+EUNG+S%2E+%282023%29%2E+Beyond+positive+scaling+%3A+How+negation+impacts+scaling+trends+of+language+models%2E+In+A%2E+R+OGERS+%2C+J%2E+B+OYD+%2D)