---
title: "DiaLLM — paper by Painter et al. (2026) — summary and ideas"
description: "The paper argues that today's LLMs can usually read regional Englishes but almost always reply in a standard, US-leaning… 46 references."
authors: ["Jordan Painter", "Dipankar Srirag", "Adarsh Kappiyath", "Diptesh Kanojia", "Aditya Joshi", "Lu Yin"]
year: 2026
venue: "arXiv:2607.07669"
source: "https://arxiv.org/abs/2607.07669"
doi: "10.48550/arxiv.2607.07669"
arxiv: "2607.07669"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/painter-2026-diallm-investigation-robustness-generation-gap/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/painter-2026-diallm-investigation-robustness-generation-gap/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) / DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

# DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

**Authors:** [Jordan Painter](https://md.research.elamur.ai/free/en/authors/jordan-painter/), [Dipankar Srirag](https://md.research.elamur.ai/free/en/authors/dipankar-srirag/), Adarsh Kappiyath, [Diptesh Kanojia](https://md.research.elamur.ai/free/en/authors/diptesh-kanojia/), [Aditya Joshi](https://md.research.elamur.ai/free/en/authors/aditya-joshi/) and [Lu Yin](https://md.research.elamur.ai/free/en/authors/lu-yin/) · **Year:** 2026 · **Venue:** arXiv:2607.07669 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2607.07669) · [PDF](https://arxiv.org/pdf/2607.07669) · [DOI 10.48550/arxiv.2607.07669](https://doi.org/10.48550/arxiv.2607.07669) · [arXiv 2607.07669](https://arxiv.org/abs/2607.07669) · [OpenAlex](https://openalex.org/W7167826577)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper argues that today's LLMs can usually read regional Englishes but almost always reply in a standard, US-leaning variety. The authors call this the generation half of the dialect problem and say it has been largely ignored. DiaLLM is a full adaptation pipeline. Three open-weight families are first continually pretrained on the International Corpus of English (ICE). Each then follows one of two post-training routes:
- **Implicit:** generic SFT plus broad alignment.
- **Explicit:** SFT on dialect-perturbed data plus alignment aimed at one variety.

Both routes are crossed with DPO, GRPO and GSPO, giving a controlled comparison for Australian, Indian and Northern British English.

The main finding is a robustness–generation gap. Benchmark scores depend mostly on CPT and SFT, while alignment changes generated text in ways those benchmarks do not register. Explicit targeting yields output that judges recognise as dialectal. However, the method that pushes hardest on the dialect-feature reward (GRPO) is not the one people find most dialectal. This shows that feature-count rewards and perceived authenticity can come apart. Code, checkpoints and preference data are released.

## Key points
- **Models:** Llama-3.1-8B, Qwen3-8B and Gemma-3-4B-IT. Gemma's CPT started from the instruction-tuned checkpoint because the base checkpoint did poorly in early runs.
- **CPT corpus:** 18 ICE varieties, about 20M tokens in total. Training used GaLore for memory-efficient full-parameter updates.
- **Dialect data:** preferred completions from the Cui et al. (2024) preference set were rewritten with Multi-VALUE, which applies eWAVE features, and code blocks were kept unchanged. This produced about 11.8k (en-AU), 15.4k (Northern British) and 18.4k (en-IN) preference pairs.
- **Evaluation:** benchmarks were VALUE and BESSTIE, generation was judged by Phi-4 and humans, and generation evaluation covered Llama-3.1-8B only. A linguistic analysis across all three families found that GRPO produced the fewest recognisable dialect markers.
- **Weaknesses the authors report:**
  - Human evaluation used six non-specialist annotators, and agreement was weakest for en-AU (mean AC1 0.02).
  - en-IN was the hardest variety.
  - The BESSTIE-trained en-UK classifier almost always predicted en-AU on model outputs.

## Interesting ideas
- Build dialect versions of existing supervision by applying typology-based rewrite rules to standard text, protecting spans such as code that must not change; this is a cheap way to get parallel variety data.
- Measure robustness on input separately from quality of output, since a stage can leave benchmark scores flat while clearly changing what the system produces.
- Do not reward and evaluate with the same feature detector; the authors demote feature-density gains to a training diagnostic because they are circular.
- Check whether an auxiliary classifier collapses onto one label when it sees out-of-distribution text, as the en-UK classifier did by predicting en-AU.
- Treat later stages as only reshaping signal that earlier exposure has already placed in the model, not as a source of new knowledge.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): Apply eWAVE-style rewrites to UD treebank sentences together with matching tree edits, giving dialect test and training sets for the perceptron parser and gparse.
- **I2** (back): Evaluate generation and the text → graph → text compressor on their own outputs, rather than inferring quality from parsing or QA scores.
- **I3** (learn): Self-induced repair rules and self-play should be scored by witnesses or controls independent of the signal used to train them, otherwise improvements may only reflect fitting the scorer.
- **I4** (learn): Add a collapse check, such as the label distribution on generated text, to negative controls for any tagger or classifier used to filter silver data.
- **I5** (tokens): Dialect lexemes and morphology (e.g. "arvo", "prepone", "summat") must exist in the en lexicon and lemmatiser first, because downstream frames and rules cannot recover forms the tokenizer never knew.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [UNSW Sydney](https://md.research.elamur.ai/free/en/universities/unsw-sydney/) — University, Sydney, Australia · 36 works · 1,509 citations
- [University of Surrey](https://md.research.elamur.ai/free/en/universities/university-of-surrey/) — University, Guildford, United Kingdom · 22 works · 677 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works

## Cite

Cite as: Jordan Painter, Dipankar Srirag, Adarsh Kappiyath, Diptesh Kanojia, Aditya Joshi and Lu Yin. 2026. DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation. arXiv:2607.07669.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{painter2026diallm,
  title = {{DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation}},
  author = {Jordan Painter, Dipankar Srirag, Adarsh Kappiyath, Diptesh Kanojia, Aditya Joshi and Lu Yin},
  year = {2026},
  doi = {10.48550/arxiv.2607.07669},
  eprint = {2607.07669},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.07669}
}
```

- [← previous DialectLLM: A Dialect-Aware Dialogue Generation Framework Beyond Standard American English](https://md.research.elamur.ai/free/en/papers/oh-2026-dialectllm-dialect-aware-dialog-ue/)
- [next → DialogGuard: Multi-Agent Psychosocial Safety Evaluation Interface of Sensitive LLM Responses](https://md.research.elamur.ai/free/en/papers/luo-2026-dialogguard-multi-agent-psychosocial-safety/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/)

## References

46 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 others. 2024. [Phi-4 technical report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/). Preprint, arXiv:2412.08905. [↗](https://arxiv.org/abs/2412.08905)
2. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery. [↗](https://doi.org/10.1145/3442188.3445922)
3. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. [Demographic dialectal variation in social media: A case study of African-American English](https://md.research.elamur.ai/free/en/papers/blodgett-2016-demographic-dialectal-variation-social-media/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130, Austin, Texas. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d16-1120)
4. Peter Collins. 2012. Australian English: Its evolution and current state. International Journal of Language, Translation and Intercultural Communication, 1:75– 86. [find](https://scholar.google.com/scholar?q=Peter+Collins%2E+2012%2E+Australian+English%3A+Its+evolution+and+current+state%2E+International+Journal+of+Language%2C+Translation+and+Intercultural+Communication%2C+1%3A75%E2%80%93+86%2E)
5. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. ULTRAFEEDBACK: Boosting language models with scaled AI feedback. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 9722–9744. PMLR. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Bingxiang+He%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Ruobing+Xie%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024%2E+ULTRAFEEDBACK%3A+Boosting+language+models+with+s)
6. Karin De Langis, Ryan Koo, and Dongyeop Kang. 2024. Dynamic multi-reward weighting for multi-style controllable generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6783–6800, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Karin+De+Langis%2C+Ryan+Koo%2C+and+Dongyeop+Kang%2E+2024%2E+Dynamic+multi%2Dreward+weighting+for+multi%2Dstyle+controllable+generation%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Langua)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
8. Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastasopoulos. 2024. DIALECTBENCH: An NLP benchmark for dialects, varieties, and closely-related languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14412–14454, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Fahim+Faisal%2C+Orevaoghene+Ahia%2C+Aarohi+Srivastava%2C+Kabir+Ahuja%2C+David+Chiang%2C+Yulia+Tsvetkov%2C+and+Antonios+Anastasopoulos%2E+2024%2E+DIALECTBENCH%3A+An+NLP+benchmark+for+dialects%2C+varieties%2C+and+closely%2Drel)
9. Eve Fleisig, Genevieve Smith, Madeline Bossi, Ishita Rustagi, Xavier Yin, and Dan Klein. 2024. [Linguistic bias in ChatGPT: Language models reinforce dialect discrimination](https://md.research.elamur.ai/free/en/papers/fleisig-2024-linguistic-bias-chatgpt-language-models/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13541–13564, Miami, Florida, USA. Association for Computational Linguistics.
10. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2023. A framework for few-shot language model evaluation. Zenodo, version v0.4.0. https://doi.org/10.5281/zenodo.10256836. [↗](https://doi.org/10.5281/zenodo.10256836)
11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. [The Llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783. [↗](https://arxiv.org/abs/2407.21783)
12. Sidney Greenbaum and Gerald Nelson. 1996. The International Corpus of English (ICE) project. World Englishes, 15(1):3–15. [find](https://scholar.google.com/scholar?q=Sidney+Greenbaum+and+Gerald+Nelson%2E+1996%2E+The+International+Corpus+of+English+%28ICE%29+project%2E+World+Englishes%2C+15%281%29%3A3%E2%80%9315%2E)
13. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.740)
14. Kilem L. Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48. [find](https://scholar.google.com/scholar?q=Kilem+L%2E+Gwet%2E+2008%2E+Computing+inter%2Drater+reliability+and+its+variance+in+the+presence+of+high+agreement%2E+British+Journal+of+Mathematical+and+Statistical+Psychology%2C+61%281%29%3A29%E2%80%9348%2E)
15. William Held, Caleb Ziems, and Diyi Yang. 2023. [TADA: Task agnostic dialect adapters for English](https://md.research.elamur.ai/free/en/papers/held-2023-tada-task-agnostic-dialect-adapters/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 813–824, Toronto, Canada. Association for Computational Linguistics.
16. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. 2024. AI generates covertly racist decisions about people based on their dialect. Nature, 633(8028):147–154. [find](https://scholar.google.com/scholar?q=Valentin+Hofmann%2C+Pratyusha+Ria+Kalluri%2C+Dan+Jurafsky%2C+and+Sharese+King%2E+2024%2E+AI+generates+covertly+racist+decisions+about+people+based+on+their+dialect%2E+Nature%2C+633%288028%29%3A147%E2%80%93154%2E)
17. Quzhe Huang, Mingxu Tao, Chen Zhang, Zhenwei An, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023. [Lawyer LLaMA technical report](https://md.research.elamur.ai/free/en/papers/huang-2023-lawyer-llama-technical-report/). Preprint, arXiv:2305.15062. [↗](https://arxiv.org/abs/2305.15062)
18. Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. 2025. Natural language processing for dialects of a language: A survey. ACM Computing Surveys, 57(6):1–37. [find](https://scholar.google.com/scholar?q=Aditya+Joshi%2C+Raj+Dabre%2C+Diptesh+Kanojia%2C+Zhuang+Li%2C+Haolan+Zhan%2C+Gholamreza+Haffari%2C+and+Doris+Dippold%2E+2025%2E+Natural+language+processing+for+dialects+of+a+language%3A+A+survey%2E+ACM+Computing+Surveys%2C+)
19. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. [The state and fate of linguistic diversity and inclusion in the NLP world](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.
20. Eeham Khan, Firas Saidani, Owen Van Esbroeck, Richard Khoury, and Leila Kosseim. 2025. Low-resource dialect adaptation of large language models: A French dialect case-study. Preprint, arXiv:2510.22747. [↗](https://arxiv.org/abs/2510.22747)
21. Bernd Kortmann and Kerstin Lunkenheimer, editors. 2013. The electronic world atlas of varieties of English (eWAVE). Max Planck Institute for Evolutionary Anthropology, Leipzig. [find](https://scholar.google.com/scholar?q=Bernd+Kortmann+and+Kerstin+Lunkenheimer%2C+editors%2E+2013%2E+The+electronic+world+atlas+of+varieties+of+English+%28eWAVE%29%2E+Max+Planck+Institute+for+Evolutionary+Anthropology%2C+Leipzig%2E)
22. Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors. 2020. The Electronic World Atlas of Varieties of English (eWAVE). Zenodo, Leipzig. [find](https://scholar.google.com/scholar?q=Bernd+Kortmann%2C+Kerstin+Lunkenheimer%2C+and+Katharina+Ehret%2C+editors%2E+2020%2E+The+Electronic+World+Atlas+of+Varieties+of+English+%28eWAVE%29%2E+Zenodo%2C+Leipzig%2E)
23. Klaus Krippendorff. 2004. Content Analysis: An Introduction to Its Methodology, 2nd edition. Sage, Thousand Oaks, CA. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2004%2E+Content+Analysis%3A+An+Introduction+to+Its+Methodology%2C+2nd+edition%2E+Sage%2C+Thousand+Oaks%2C+CA%2E)
24. Sanjh Maheshwari, Aniket Singh Rajpoot, Oana Cocarascu, and Mamta. 2026. [Improving dialect robustness in large language models via LoRA and mixture-of-experts](https://md.research.elamur.ai/free/en/papers/maheshwari-2026-improving-dialect-robustness-large-language/). In Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 293–303, Rabat, Morocco. Association for Computational Linguistics.
25. Joel Mire, Zubin Trivadi Aysola, Daniel Chechelnitsky, Nicholas Deas, Chrysoula Zerva, and Maarten Sap. 2025. Rejected dialects: Biases against African American language in reward models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7468–7487, Albuquerque, New Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Joel+Mire%2C+Zubin+Trivadi+Aysola%2C+Daniel+Chechelnitsky%2C+Nicholas+Deas%2C+Chrysoula+Zerva%2C+and+Maarten+Sap%2E+2025%2E+Rejected+dialects%3A+Biases+against+African+American+language+in+reward+models%2E+In+Findings+)
26. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Advances in Neural Information Processing Systems, volume 36.
27. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. [COMET-22: Unbabel-IST 2022 submission for the metrics shared task](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/). In Proceedings of the Seventh Conference on Machine Translation, pages 578–585.
28. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d19-1410)
29. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). Preprint, arXiv:2302.13971. [↗](https://arxiv.org/abs/2302.13971)
30. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. [GPQA: A graduate-level Google-proof Q&A benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling (COLM).
31. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w18-5446)
32. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. [The risk of racial bias in hate speech detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1163)
33. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. [DeepSeekMath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). Preprint, arXiv:2402.03300. [↗](https://arxiv.org/abs/2402.03300)
34. Dipankar Srirag, Aditya Joshi, and Jacob Eisenstein. 2025a. [Predicting the target word of game-playing conversations using a low-rank dialect adapter for decoder models](https://md.research.elamur.ai/free/en/papers/srirag-2025-predicting-target-word-game-playing/). In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 8–17, Albuquerque, New Mexico. Association for Computational Linguistics.
35. Dipankar Srirag, Aditya Joshi, Jordan Painter, and Diptesh Kanojia. 2025b. BESSTIE: A benchmark for sentiment and sarcasm classification for varieties of English. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8413–8429, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dipankar+Srirag%2C+Aditya+Joshi%2C+Jordan+Painter%2C+and+Diptesh+Kanojia%2E+2025b%2E+BESSTIE%3A+A+benchmark+for+sentiment+and+sarcasm+classification+for+varieties+of+English%2E+In+Findings+of+the+Association+for+Co)
36. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2023. [Challenging BIG-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada. Association for Computational Linguistics.
37. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. [MMLU-Pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). In Advances in Neural Information Processing Systems 37: Datasets and Benchmarks Track.
38. Nahathai Wongpakaran, Tinakon Wongpakaran, Danny Wedding, and Kilem L. Gwet. 2013. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Medical Research Methodology, 13:61. [find](https://scholar.google.com/scholar?q=Nahathai+Wongpakaran%2C+Tinakon+Wongpakaran%2C+Danny+Wedding%2C+and+Kilem+L%2E+Gwet%2E+2013%2E+A+comparison+of+Cohen%E2%80%99s+Kappa+and+Gwet%E2%80%99s+AC1+when+calculating+inter%2Drater+reliability+coefficients%3A+a+study+conducted)
39. Zedian Xiao, William Held, Yanchen Liu, and Diyi Yang. 2023. [Task-agnostic low-rank adapters for unseen English dialects](https://md.research.elamur.ai/free/en/papers/xiao-2023-task-agnostic-low-rank-adapters/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7857–7870, Singapore. Association for Computational Linguistics.
40. Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, Xinyu Zhou, Lingfei Qian, Huan He, Dennis Shung, Lucila Ohno-Machado, Yonghui Wu, Hua Xu, and Jiang Bian. 2025. Medical foundation large language models for comprehensive text analysis and beyond. npj Digital Medicine, 8(1):141. [find](https://scholar.google.com/scholar?q=Qianqian+Xie%2C+Qingyu+Chen%2C+Aokun+Chen%2C+Cheng+Peng%2C+Yan+Hu%2C+Fongci+Lin%2C+Xueqing+Peng%2C+Jimin+Huang%2C+Jeffrey+Zhang%2C+Vipina+Keloth%2C+Xinyu+Zhou%2C+Lingfei+Qian%2C+Huan+He%2C+Dennis+Shung%2C+Lucila+Ohno%2DMachado%2C+Yo)
41. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388. [↗](https://arxiv.org/abs/2505.09388)
42. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. [↗](https://arxiv.org/abs/2503.19786)
43. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. GaLore: Memory-efficient LLM training by gradient low-rank projection. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 61121–61143. PMLR. [find](https://scholar.google.com/scholar?q=Jiawei+Zhao%2C+Zhenyu+Zhang%2C+Beidi+Chen%2C+Zhangyang+Wang%2C+Anima+Anandkumar%2C+and+Yuandong+Tian%2E+2024%2E+GaLore%3A+Memory%2Defficient+LLM+training+by+gradient+low%2Drank+projection%2E+In+Proceedings+of+the+41st+Inte)
44. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. 2025. [Group sequence policy optimization](https://md.research.elamur.ai/free/en/papers/zheng-2025-group-sequence-policy-optimization/). Preprint, arXiv:2507.18071. [↗](https://arxiv.org/abs/2507.18071)
45. Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022. [VALUE: Understanding dialect disparity in NLU](https://md.research.elamur.ai/free/en/papers/ziems-2022-value-understanding-dialect-disparity-nlu/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3701–3720, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.acl-long.258)
46. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. [Multi-VALUE: A framework for cross-dialectal English NLP](https://md.research.elamur.ai/free/en/papers/ziems-2023-multi-value-framework-cross-dialectal/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 744–768, Toronto, Canada. Association for Computational Linguistics.

*Updated 2 November 2026*