---
title: "Evaluating Dialect Robustness of Language Models via"
description: "The paper asks whether LLMs that perform well on English understand conversations equally well… 26 references, cited by 10 papers here."
authors: ["Dipankar Srirag", "Nihar Ranjan Sahoo", "Aditya Joshi"]
year: 2025
venue: "Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation"
source: "https://aclanthology.org/2025.sumeval-2.3/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/srirag-2025-evaluating-dialect-robustness-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/srirag-2025-evaluating-dialect-robustness-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / Evaluating Dialect Robustness of Language Models via Conversation Understanding

# Evaluating Dialect Robustness of Language Models via Conversation Understanding

**Authors:** [Dipankar Srirag](https://md.research.elamur.ai/free/en/authors/dipankar-srirag/), Nihar Ranjan Sahoo and [Aditya Joshi](https://md.research.elamur.ai/free/en/authors/aditya-joshi/) · **Year:** 2025 · **Venue:** [Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation](https://md.research.elamur.ai/free/en/venues/sumeval/) ([Workshop on Scaling Up Multilingual Evaluation 2025](https://md.research.elamur.ai/free/en/venues/sumeval/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.sumeval-2.3/) · [PDF](https://aclanthology.org/2025.sumeval-2.3.pdf) · [OpenAlex](https://openalex.org/W4396822293)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/srirag-2025-evaluating-dialect-robustness-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether LLMs that perform well on English understand conversations equally well when the speakers use Indian English rather than US English. The authors reuse MD3, a corpus of transcribed dialogues from a Taboo-style guessing game. They keep only games the guesser won and cut each dialogue at the turn where the answer is spoken, then mask that answer. The resulting dataset, M-MD3, has four parts: the original en-US and en-IN dialogues, en-MV (US dialogues given Indian-English features by the rule-based Multi-VALUE tool) and en-TR (Indian dialogues neutralised by GPT-4). There are two tasks. In TWP, the model produces the hidden word freely. In TWS, it picks the word from the corpus-wide list of targets. Llama 3 70B Chat, GPT-4 and GPT-3.5 are tested with prompting and, where possible, after fine-tuning. Every model does worse on Indian English in every setting. GPT models score highest, but fine-tuned smaller models show a narrower gap. The method is a cheap, repeatable way to probe dialect bias using existing dialogue data.

## Key points
- M-MD3 subset sizes as single-word + multi-word targets:
  - en-US: 308 + 106
  - en-IN: 153 + 59
  - en-MV: 245 + 87
  - en-TR: 121 + 50
- Indian-English games are longer (6.8 turns and 57.4 words on average) than US games (4.1 turns and 42.1 words). The authors explain this by US players being more familiar with the target words.
- Scoring uses two metrics: strict exact-match accuracy and cosine similarity of Sentence-BERT embeddings, which gives credit for near-misses and partial multi-word answers.
- Training setup:
  - GPT-3.5 was fine-tuned per subset for 5 epochs, with top_p set to 0.2.
  - Llama-3 was trained with QLoRA on all linear layers for 20 epochs (batch size 16, Paged 8-bit AdamW, learning rate 2e-4).
- Fine-tuning on en-MV, where dialect features are added, helped more than fine-tuning on en-TR, where they are removed.
- Offering candidate words (TWS) fixed most errors. Models still failed when the answer depended on cultural knowledge shared by the speakers.

## Interesting ideas
- Hiding the final answer of a goal-directed dialogue turns any game transcript into a comprehension test, with no annotation needed, because the dialogue itself defines the gold label.
- Transforming data in both directions (adding dialect features to standard text and stripping them from dialect text) helps separate the effect of dialect from differences in content and source.
- Running the same item as open generation and as a closed choice shows whether a failure lies in understanding or in finding the right word.
- Pairing a strict exact-match score with a lenient relatedness score avoids punishing near-correct or partially matching multi-word answers.
- Rewrites by an LLM sometimes came back as summaries instead of transformed dialogues and had to be filtered out. Generated data needs a structural sanity check before use.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): Mask the entity a fable or tale names in its closing line and ask the reader to recover it. This checks whether the world model actually tracked the described referent.
- **I2** (tree): When testing UD dialect conversion, parse original/converted pairs in both directions and compare LAS. This shows whether parser drops come from dialect morphosyntax or from topic shift.
- **I3** (answer): Pose school-test and fable QA items both open-ended and multiple-choice. If only the open form fails, the gap is in lexical retrieval or generation rather than in the world of the text.
- **I4** (learn): Report exact match next to a soft score, ideally computed from WordNet relations instead of embeddings so it stays explainable.
- **I5** (learn): Before LLM-teacher rewrites enter silver data, verify that turn structure and speaker roles survived, and drop items that fail.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Indian Institute of Technology Bombay](https://md.research.elamur.ai/free/en/universities/indian-institute-of-technology-bombay/) — University, Mumbai, India · 29 works · 2,104 citations
- [UNSW Sydney](https://md.research.elamur.ai/free/en/universities/unsw-sydney/) — University, Sydney, Australia · 34 works · 1,509 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Dipankar Srirag, Nihar Ranjan Sahoo and Aditya Joshi. 2025. Evaluating Dialect Robustness of Language Models via Conversation Understanding. Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{srirag2025evaluating,
  title = {{Evaluating Dialect Robustness of Language Models via Conversation Understanding}},
  author = {Dipankar Srirag, Nihar Ranjan Sahoo and Aditya Joshi},
  booktitle = {Proceedings of the Second Workshop on Scaling Up Multilingual \& Multi-Cultural Evaluation},
  year = {2025},
  url = {https://aclanthology.org/2025.sumeval-2.3/}
}
```

- [← previous EMO: Embedding Model Distillation via Intra-Model Relation and Optimal Transport Alignments](https://md.research.elamur.ai/free/en/papers/truong-2025-emo-embedding-model-distillation-intra/)
- [next → Factual Dialogue Summarization via Learning from Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2025-factual-dialogue-summarization-learning-large/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/)

## References

26 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. [Language (technology) is power: A critical survey of “bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.485)
2. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. [SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization](https://md.research.elamur.ai/free/en/papers/gliwa-2019-samsum-corpus-human-annotated-dialogue/). In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.
3. Kingsley Bolton. 2012. World englishes and linguistic landscapes. World Englishes, 31(1):30–33. [↗](https://doi.org/10.1111/j.1467-971x.2011.01748.x)
4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, and David Schlangen. 2023. clembench: Using game play to evaluate chat-optimized language models as conversational agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11174–11219, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aaron+Grattafiori%2C+Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Alex+Vaughan%2C+Amy+Yang%2C+Angela+Fan%2C+Anirudh+Goyal%2C+Anth)
5. Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. [DialogSum: A real-life scenario dialogue summarization dataset](https://md.research.elamur.ai/free/en/papers/chen-2021-dialogsum-real-life-scenario-dialogue/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5062–5074, Online. Association for Computational Linguistics.
6. Zhi Chen, Lu Chen, Bei Chen, Libo Qin, Yuncong Liu, Su Zhu, Jian-Guang Lou, and Kai Yu. 2022. UniDU: Towards a unified generative dialogue understanding framework. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 442–455, Edinburgh, UK. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhi+Chen%2C+Lu+Chen%2C+Bei+Chen%2C+Libo+Qin%2C+Yuncong+Liu%2C+Su+Zhu%2C+Jian%2DGuang+Lou%2C+and+Kai+Yu%2E+2022%2E+UniDU%3A+Towards+a+unified+generative+dialogue+understanding+framework%2E+In+Proceedings+of+the+23rd+Annual+Me)
7. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. 8-bit optimizers via block-wise Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan Mc-Phie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783. Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, William Held, Caleb Ziems, and Diyi Yang. 2023. TADA : Task agnostic dialect adapters for English. In Findings of the Association for Computational Linguistics: ACL 2023, pages 813–824, Toronto, Canada. Association for Computational Linguistics. [↗](https://arxiv.org/abs/2407.21783)
8. Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. 2024. Natural language processing for dialects of a language: A survey. Preprint, arXiv:2401.05632. [↗](https://arxiv.org/abs/2401.05632)
9. Khyati Khandelwal, Manuel Tonneau, Andrew M. Bean, Hannah Rose Kirk, and Scott A. Hale. 2024. Indian-bhed: A dataset for measuring india-centric biases in large language models. In Proceedings of the 2024 International Conference on Information Technology for Social Good, GoodIT ’24, page 231–239, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Khyati+Khandelwal%2C+Manuel+Tonneau%2C+Andrew+M%2E+Bean%2C+Hannah+Rose+Kirk%2C+and+Scott+A%2E+Hale%2E+2024%2E+Indian%2Dbhed%3A+A+dataset+for+measuring+india%2Dcentric+biases+in+large+language+models%2E+In+Proceedings+of+the+)
10. Yanchen Liu, William Held, and Diyi Yang. 2023. [DADA: Dialect adaptation via dynamic aggregation of linguistic rules](https://md.research.elamur.ai/free/en/papers/liu-2023-dada-dialect-adaptation-dynamic-aggregation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13776–13793, Singapore. Association for Computational Linguistics.
11. Silvia Dal Negro and Alessandro Vietti. 2006. The interplay of dialect and the standard in anonymous street dialogues: Patterns of variation in northern italy. Language Variation and Change, 18(2):179–192. [↗](https://doi.org/10.1017/s0954394506060108)
12. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Preprint, arXiv:2303.08774. set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217–231. [↗](https://arxiv.org/abs/2303.08774)
13. Ayaka Ueyama and Yoshinobu Kano. 2023. Dialogue response generation using completion of omitted predicate arguments based on zero anaphora resolution. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 282–296, Prague, Czechia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ayaka+Ueyama+and+Yoshinobu+Kano%2E+2023%2E+Dialogue+response+generation+using+completion+of+omitted+predicate+arguments+based+on+zero+anaphora+resolution%2E+In+Proceedings+of+the+24th+Annual+Meeting+of+the+)
14. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
15. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2024. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, New Orleans, LA, USA. Curran Associates Inc.
16. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2212.10560)
17. Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. 2021. TIME-DIAL: Temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7066–7076, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lianhui+Qin%2C+Aditya+Gupta%2C+Shyam+Upadhyay%2C+Luheng+He%2C+Yejin+Choi%2C+and+Manaal+Faruqui%2E+2021%2E+TIME%2DDIAL%3A+Temporal+commonsense+reasoning+in+dialog%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Associ)
18. Wikipedia. 2023. Taboo (game). [find](https://scholar.google.com/scholar?q=Wikipedia%2E+2023%2E+Taboo+%28game%29%2E)
19. Zedian Xiao, William Held, Yanchen Liu, and Diyi Yang. 2023. [Task-agnostic low-rank adapters for unseen English dialects](https://md.research.elamur.ai/free/en/papers/xiao-2023-task-agnostic-low-rank-adapters/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7857–7870, Singapore. Association for Computational Linguistics.
20. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d19-1410)
21. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). Preprint, arXiv:2303.18223. [↗](https://arxiv.org/abs/2303.18223)
22. Michael J Ryan, William Held, and Diyi Yang. 2024. [Unintended impacts of LLM alignment on global representation](https://md.research.elamur.ai/free/en/papers/ryan-2024-unintended-impacts-llm-alignment-global/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16121–16140, Bangkok, Thailand. Association for Computational Linguistics.
23. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. [Multi-value: A framework for cross-dialectal english nlp](https://md.research.elamur.ai/free/en/papers/ziems-2023-multi-value-framework-cross-dialectal/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2023.acl-long.44)
24. Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. Transactions of the Association for Computational Linguistics, 8:810– 827. [↗](https://doi.org/10.1162/tacl_a_00347)
25. David Schlangen. 2023. On general language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8818–8825, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Schlangen%2E+2023%2E+On+general+language+understanding%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2023%2C+pages+8818%E2%80%938825%2C+Singapore%2E+Association+for+Computational+Linguistic)
26. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge data [find](https://scholar.google.com/scholar?q=Kai+Sun%2C+Dian+Yu%2C+Jianshu+Chen%2C+Dong+Yu%2C+Yejin+Choi%2C+and+Claire+Cardie%2E+2019%2E+DREAM%3A+A+challenge+data)

*Updated 1 November 2026*