---
title: "Beyond Static Synthetic Noise — paper by Wu et al. (2026)"
description: "The authors build an automatic pipeline that mines Wikipedia revision histories. 58 references."
authors: ["Yulong Wu", "Viktor Schlegel", "Riza Batista-Navarro"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1796/"
doi: "10.18653/v1/2026.findings-acl.1796"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2026-beyond-static-synthetic-noise-assessing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2026-beyond-static-synthetic-noise-assessing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/) / Beyond Static Synthetic Noise: Assessing the Robustness of Large Language Models to Natural Context Variation in the Real World

# Beyond Static Synthetic Noise: Assessing the Robustness of Large Language Models to Natural Context Variation in the Real World

**Authors:** [Yulong Wu](https://md.research.elamur.ai/free/en/authors/yulong-wu/), [Viktor Schlegel](https://md.research.elamur.ai/free/en/authors/viktor-schlegel/) and [Riza Batista-Navarro](https://md.research.elamur.ai/free/en/authors/riza-batista-navarro/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1796/) · [PDF](https://aclanthology.org/2026.findings-acl.1796.pdf) · [DOI 10.18653/v1/2026.findings-acl.1796](https://doi.org/10.18653/v1/2026.findings-acl.1796) · [OpenAlex](https://openalex.org/W7166807178)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

Most QA robustness tests use artificial perturbations from fixed manipulation rules. These miss the meaning-preserving changes that real documents undergo as people edit them. This matters for LLMs in RAG and agent systems, which read sources that keep changing. The authors build an automatic pipeline that mines Wikipedia revision histories. It pairs a paragraph from an older revision with its edited version from the next revision, then swaps the edited text into existing QA benchmarks. Questions and gold answers stay unchanged, and an answer-preservation check is applied.

They apply this to six Wikipedia-based datasets and test 42 models: encoder-only BERT variants, Flan-T5 and many recent LLMs. On SQuAD, error analysis of the encoder models produced two challenge sets. Humans perform about the same on these sets, but state-of-the-art LLMs lose up to 28.28%. Similar drops appear on harder benchmarks such as DROP and HotpotQA. Adversarial retraining for encoders and perturbed in-context examples for LLMs help only partly. The authors argue that real-world text drift is a practical weakness that synthetic tests do not capture.

## Key points

- **Pipeline:** The method compares adjacent Wikipedia revisions and keeps only "modification" edits where both paragraphs exceed 500 characters. A perturbed passage is kept only if at least one question still has all its annotated answers in the new text.
- **Datasets:** SQuAD 1.1, SQuAD 2.0, BoolQ, DROP, HotpotQA (distractor) and TyDi QA (English gold passage). Development sets serve as test sets.
- **Models and metrics:** Encoders (BERT, DistilBERT, SpanBERT, RoBERTa, ALBERT, DeBERTa) and Flan-T5 are scored with Token-F1. Decoder LLMs (GPT-3.5 Turbo, GPT-4o, Llama, Gemma, Mistral, Qwen2.5, OLMo, Falcon, DeepSeek) are scored with Inclusion Match.
- **Results:** Drops on the SQuAD challenge sets range from 4.4% to 28.28%. Examples elsewhere include −5.25% for Mistral-7B-Instruct-v0.2 on BoolQ and −9.98% for Llama-3.2-1B-Instruct on DROP.
- **Defenses and resources:** Training on synthetic perturbations sometimes beats training on natural ones, but augmentation can lower accuracy on clean data. The approach only works for Wikipedia-sourced data. Code and challenge sets are available at github.com/Yulong-W/natural_perturbation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yulong Wu, Viktor Schlegel and Riza Batista-Navarro. 2026. Beyond Static Synthetic Noise: Assessing the Robustness of Large Language Models to Natural Context Variation in the Real World. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wu2026beyond,
  title = {{Beyond Static Synthetic Noise: Assessing the Robustness of Large Language Models to Natural Context Variation in the Real World}},
  author = {Yulong Wu, Viktor Schlegel and Riza Batista-Navarro},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1796},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1796}
}
```

- [← previous Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL](https://md.research.elamur.ai/free/en/papers/wang-2026-beyond-static-rules-automated-discovery/)
- [next → Beyond Static Toolsets: Self-Evolving LLM Tool Agents via Continual Documentation Adaptation](https://md.research.elamur.ai/free/en/papers/wu-2026-beyond-static-toolsets-self-evolving/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/)

## References

58 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. [The falcon series of open language mo…](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/)
2. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. [A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/). In Proceedings of…
3. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+Yonatan+Bisk%2E+2018%2E+Synthetic+and+natural+noise+both+break+neural+machine+translation%2E+In+International+Conference+on+Learning+Representations%2E)
4. Neeladri Bhuiya, Viktor Schlegel, and Stefan Winkler. 2024. Seemingly plausible distractors in multi-hop reasoning: Are large language models attentive readers? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2514–2528, Miami, Florida, USA. Associatio… [find](https://scholar.google.com/scholar?q=Neeladri+Bhuiya%2C+Viktor+Schlegel%2C+and+Stefan+Winkler%2E+2024%2E+Seemingly+plausible+distractors+in+multi%2Dhop+reasoning%3A+Are+large+language+models+attentive+readers%3F+In+Proceedings+of+the+2024+Conference+o)
5. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, and 16 others. 2022.… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+X)
6. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
7. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. [TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). Transactions of the Association for Computational Linguistics, 8:454–470…
8. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+Psychological+Measurement%2C+20%281%29%3A37%E2%80%9346%2E)
9. DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, and 69 others. 2024. Deepseek llm: Scaling open-source language models with longterm… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+%3A%2C+Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+Huazuo+Gao%2C+Kaige+Gao%2C+Wenjun+Gao%2C+Ruiqi+Ge%2C+Kang+Guan%2C+Daya+Guo%2C)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
11. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistic…
12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 516 others. 202… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
13. Wee Chung Gan and Hwee Tou Ng. 2019. Improving the robustness of question answering systems to question paraphrasing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6065–6075, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wee+Chung+Gan+and+Hwee+Tou+Ng%2E+2019%2E+Improving+the+robustness+of+question+answering+systems+to+question+paraphrasing%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Lin)
14. Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 24 others. 2024. OLMo: A… [find](https://scholar.google.com/scholar?q=Dirk+Groeneveld%2C+Iz+Beltagy%2C+Evan+Walsh%2C+Akshita+Bhagia%2C+Rodney+Kinney%2C+Oyvind+Tafjord%2C+Ananya+Jha%2C+Hamish+Ivison%2C+Ian+Magnusson%2C+Yizhong+Wang%2C+Shane+Arora%2C+David+Atkinson%2C+Russell+Authur%2C+Khyathi+Cha)
15. Ashim Gupta, Rishanth Rajendhran, Nathan Stringham, Vivek Srikumar, and Ana Marasovic. 2024. Whispers of doubt amidst echoes of triumph in NLP robustness. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… [find](https://scholar.google.com/scholar?q=Ashim+Gupta%2C+Rishanth+Rajendhran%2C+Nathan+Stringham%2C+Vivek+Srikumar%2C+and+Ana+Marasovic%2E+2024%2E+Whispers+of+doubt+amidst+echoes+of+triumph+in+NLP+robustness%2E+In+Proceedings+of+the+2024+Conference+of+the+)
16. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. [{DEBERTA}: {DECODING}-{enhanced} {bert} {with} {disentangled} {attention}](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In International Conference on Learning Representations.
17. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. 2021. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15262–15271. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Kevin+Zhao%2C+Steven+Basart%2C+Jacob+Steinhardt%2C+and+Dawn+Song%2E+2021%2E+Natural+adversarial+examples%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVP)
18. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.
19. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
20. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. [Span-BERT: Improving pre-training by representing and predicting spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/). Transactions of the Association for Computational Linguistics, 8:64–77. memory management for large language model serving with pagedat…
21. Yuxuan Lai, Chen Zhang, Yansong Feng, Quzhe Huang, and Dongyan Zhao. 2021. Why machine reading comprehension models learn shortcuts? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 989–1002, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yuxuan+Lai%2C+Chen+Zhang%2C+Yansong+Feng%2C+Quzhe+Huang%2C+and+Dongyan+Zhao%2E+2021%2E+Why+machine+reading+comprehension+models+learn+shortcuts%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DI)
22. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. [Albert: A lite bert for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). In International Conference on Learning Representations.
23. Thai Le, Jooyoung Lee, Kevin Yen, Yifan Hu, and Dongwon Lee. 2022. Perturbations in the wild: Leveraging human-written text perturbations for realistic adversarial attack and defense. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2953–2965, Dublin, Ireland. Associatio… [find](https://scholar.google.com/scholar?q=Thai+Le%2C+Jooyoung+Lee%2C+Kevin+Yen%2C+Yifan+Hu%2C+and+Dongwon+Lee%2E+2022%2E+Perturbations+in+the+wild%3A+Leveraging+human%2Dwritten+text+perturbations+for+realistic+adversarial+attack+and+defense%2E+In+Findings+of+t)
24. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In Proceedings of the 34th Internation…
25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized BERT pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). CoRR, abs/1907.11692.
26. Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang, and 7 others. 2025. [Large language model agent: A survey on methodo…](https://md.research.elamur.ai/free/en/papers/luo-2025-large-language-model-agent-survey/)
27. Diptesh Kanojia, Marina Fomicheva, Tharindu Ranasinghe, Frédéric Blain, Constantin Orăsan, and Lucia Specia. 2021. Pushing the right buttons: Adversarial evaluation of quality estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 625–638, Online. Association for Computati… [find](https://scholar.google.com/scholar?q=Diptesh+Kanojia%2C+Marina+Fomicheva%2C+Tharindu+Ranasinghe%2C+Fr%C3%A9d%C3%A9ric+Blain%2C+Constantin+Ora%CC%86san%2C+and+Lucia+Specia%2E+2021%2E+Pushing+the+right+buttons%3A+Adversarial+evaluation+of+quality+estimation%2E+In+Proceedi)
28. Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, L. Sifre, Morgane Riviere, Mihir Kale, J Christopher Love, Pouya Dehghani Tafti, L’eonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Am’elie H’elio… [find](https://scholar.google.com/scholar?q=Gemma+Team+Thomas+Mesnard%2C+Cassidy+Hardin%2C+Robert+Dadashi%2C+Surya+Bhupatiraju%2C+Shreya+Pathak%2C+L%2E+Sifre%2C+Morgane+Riviere%2C+Mihir+Kale%2C+J+Christopher+Love%2C+Pouya+Dehghani+Tafti%2C+L%E2%80%99eonard+Hussenot%2C+Aakanks)
29. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Zhuohan+Li%2C+Siyuan+Zhuang%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Cody+Hao+Yu%2C+Joseph+Gonzalez%2C+Hao+Zhang%2C+and+Ion+Stoica%2E+2023%2E)
30. Efficient Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. [find](https://scholar.google.com/scholar?q=Efficient+Aakanksha+Naik%2C+Abhilasha+Ravichander%2C+Norman+Sadeh%2C+Carolyn+Rose%2C+and+Graham+Neubig%2E+2018%2E+Stress+test+evaluation+for+natural+language+inference%2E)
31. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340[–2353, Santa Fe, New Mexico, USA](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). Association for Computational Linguistics. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Ha…
32. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. [PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification](https://md.research.elamur.ai/free/en/papers/pavlick-2015-ppdb-2-0-better-paraphrase/). In Proceedings of the 53rd Annual Meeting of the Association for Computational…
33. Anibal Pedraza, Oscar Deniz, and Gloria Bueno. 2022. Really natural adversarial examples. International Journal of Machine Learning and Cybernetics, 13(4):1065–1077. [find](https://scholar.google.com/scholar?q=Anibal+Pedraza%2C+Oscar+Deniz%2C+and+Gloria+Bueno%2E+2022%2E+Really+natural+adversarial+examples%2E+International+Journal+of+Machine+Learning+and+Cybernetics%2C+13%284%29%3A1065%E2%80%931077%2E)
34. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [GloVe: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
35. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
36. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don‘t know: Unanswerable questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling…
37. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
38. Gemma Team Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L’eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram’e, Johan Ferret, Peter Liu, Pouya Dehghani Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerom… [find](https://scholar.google.com/scholar?q=Gemma+Team+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%E2%80%99eonard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%E2%80%99e%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+D)
39. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/). In 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing @ NeurIPS 2019.
40. Viktor Schlegel, Marco Valentino, Andre Freitas, Goran Nenadic, and Riza Batista-Navarro. 2020. A framework for evaluation of machine reading comprehension gold standards. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5359–5369, Marseille, France. European Languag… [find](https://scholar.google.com/scholar?q=Viktor+Schlegel%2C+Marco+Valentino%2C+Andre+Freitas%2C+Goran+Nenadic%2C+and+Riza+Batista%2DNavarro%2E+2020%2E+A+framework+for+evaluation+of+machine+reading+comprehension+gold+standards%2E+In+Proceedings+of+the+Twelft)
41. Falcon-LLM Team. 2024. The falcon 3 family of open models. [find](https://scholar.google.com/scholar?q=Falcon%2DLLM+Team%2E+2024%2E+The+falcon+3+family+of+open+models%2E)
42. Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajič, Daniel Hershcovich, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, and Roberto Navigli. 2023. What‘s the meaning of superhuman performance in today‘s NLU? In Proceedings of the 61st Annual Meet… [find](https://scholar.google.com/scholar?q=Simone+Tedeschi%2C+Johan+Bos%2C+Thierry+Declerck%2C+Jan+Hajic%CC%8C%2C+Daniel+Hershcovich%2C+Eduard+Hovy%2C+Alexander+Koller%2C+Simon+Krek%2C+Steven+Schockaert%2C+Rico+Sennrich%2C+Ekaterina+Shutova%2C+and+Roberto+Navigli%2E+2023%2E)
43. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
44. Son Quoc Tran, Phong Nguyen-Thuan Do, Uyen Le, and Matt Kretchmar. 2023. The impacts of unanswerable questions on the robustness of machine reading comprehension models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1543– 1557,… [find](https://scholar.google.com/scholar?q=Son+Quoc+Tran%2C+Phong+Nguyen%2DThuan+Do%2C+Uyen+Le%2C+and+Matt+Kretchmar%2E+2023%2E+The+impacts+of+unanswerable+questions+on+the+robustness+of+machine+reading+comprehension+models%2E+In+Proceedings+of+the+17th+Con)
45. Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022. [Measure and improve robustness in NLP models: A survey](https://md.research.elamur.ai/free/en/papers/wang-2022-measure-improve-robustness-nlp-models/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4569–4586, Seattle, United States. Associati…
46. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In International Conference on Learning Representations.
47. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. [Transf…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
48. Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro. 2021. Is the understanding of explicit discourse relations required in machine reading comprehension? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3565–3579, On… [find](https://scholar.google.com/scholar?q=Yulong+Wu%2C+Viktor+Schlegel%2C+and+Riza+Batista%2DNavarro%2E+2021%2E+Is+the+understanding+of+explicit+discourse+relations+required+in+machine+reading+comprehension%3F+In+Proceedings+of+the+16th+Conference+of+the)
49. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+2369%E2%80%932380%2C+Brussels%2C+Belgium%2E+Association+for+Computational+Linguistics%2E)
50. Kun Zhang, Le Wu, Kui Yu, Guangyi Lv, and Dacao Zhang. 2025. Evaluating and improving robustness in large language models: A survey and future directions. Preprint, arXiv:2506.11111. — [arXiv:2506.11111](https://arxiv.org/abs/2506.11111) [find](https://scholar.google.com/scholar?q=Kun+Zhang%2C+Le+Wu%2C+Kui+Yu%2C+Guangyi+Lv%2C+and+Dacao+Zhang%2E+2025%2E+Evaluating+and+improving+robustness+in+large+language+models%3A+A+survey+and+future+directions%2E+Preprint%2C+arXiv%3A2506%2E11111%2E+%E2%80%94+%5BarXiv%3A2506%2E111)
51. A Table 7 shows the hyperparameters used to fine-tune the pre-trained encoder-only QA models in this work and their number of parameters contained. [find](https://scholar.google.com/scholar?q=A+Table+7+shows+the+hyperparameters+used+to+fine%2Dtune+the+pre%2Dtrained+encoder%2Donly+QA+models+in+this+work+and+their+number+of+parameters+contained%2E)
52. Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro. 2023a. Are machine reading comprehension systems robust to context paraphrasing? In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Co… [find](https://scholar.google.com/scholar?q=Yulong+Wu%2C+Viktor+Schlegel%2C+and+Riza+Batista%2DNavarro%2E+2023a%2E+Are+machine+reading+comprehension+systems+robust+to+context+paraphrasing%3F+In+Proceedings+of+the+13th+International+Joint+Conference+on+Natu)
53. Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro. 2025. Natural context drift undermines the natural language understanding of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 1248–1259, Suzhou, China. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Yulong+Wu%2C+Viktor+Schlegel%2C+and+Riza+Batista%2DNavarro%2E+2025%2E+Natural+context+drift+undermines+the+natural+language+understanding+of+large+language+models%2E+In+Findings+of+the+Association+for+Computation)
54. Yulong Wu, Viktor Schlegel, Daniel Beck, and Riza Batista-Navarro. 2023b. MMT‘s submission for the WMT 2023 quality estimation shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 856–862, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yulong+Wu%2C+Viktor+Schlegel%2C+Daniel+Beck%2C+and+Riza+Batista%2DNavarro%2E+2023b%2E+MMT%E2%80%98s+submission+for+the+WMT+2023+quality+estimation+shared+task%2E+In+Proceedings+of+the+Eighth+Conference+on+Machine+Translati)
55. Jun Yan, Yang Xiao, Sagnik Mukherjee, Bill Yuchen Lin, Robin Jia, and Xiang Ren. 2022. On the robustness of reading comprehension models to entity renaming. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi… [find](https://scholar.google.com/scholar?q=Jun+Yan%2C+Yang+Xiao%2C+Sagnik+Mukherjee%2C+Bill+Yuchen+Lin%2C+Robin+Jia%2C+and+Xiang+Ren%2E+2022%2E+On+the+robustness+of+reading+comprehension+models+to+entity+renaming%2E+In+Proceedings+of+the+2022+Conference+of+th)
56. Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2017. Identifying semantic edit intentions from revisions in Wikipedia. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2000–2010, Copenhagen, Denmark. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Diyi+Yang%2C+Aaron+Halfaker%2C+Robert+Kraut%2C+and+Eduard+Hovy%2E+2017%2E+Identifying+semantic+edit+intentions+from+revisions+in+Wikipedia%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+)
57. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/).
58. Encoder-only Model Parameters and Hyperparameters for Fine-tuning Model P arameters(M ) d b lr ep DistilBERT (66) BERT (110/340) SpanBERT (110/340) RoBERTa (125/355) ALBERT (11/17/223) DeBERTa (350) 384 384 512 384 384 384 8 8 4 8 4 4 3e − 5 3e − 5 2e − 5 3e − 5 3e − 5 3e − 6 3 2 4 2 2 3 [find](https://scholar.google.com/scholar?q=Encoder%2Donly+Model+Parameters+and+Hyperparameters+for+Fine%2Dtuning+Model+P+arameters%28M+%29+d+b+lr+ep+DistilBERT+%2866%29+BERT+%28110%2F340%29+SpanBERT+%28110%2F340%29+RoBERTa+%28125%2F355%29+ALBERT+%2811%2F17%2F223%29+DeBERTa+%28350%29+3)

*Updated 2 November 2026*