---
title: "Do Multimodal LLMs Understand Order? Measuring — Wei 2026"
description: "Large vision-language models (LVLMs) do well on multimodal benchmarks, but it is unclear whether their answers stay stable when the same input is…"
authors: ["Sheng-Lun Wei", "Yu-Ling Liao", "Hen-Hsen Huang", "Hsin-Hsi Chen"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.716/"
doi: "10.63317/4jtpgzks8pbr"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wei-2026-do-multimodal-llms-understand-order/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wei-2026-do-multimodal-llms-understand-order/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) / Do Multimodal LLMs Understand Order? Measuring the Fragility of Multimodal Reasoning under Input Order Perturbations

# Do Multimodal LLMs Understand Order? Measuring the Fragility of Multimodal Reasoning under Input Order Perturbations

**Authors:** Sheng-Lun Wei, Yu-Ling Liao, [Hen-Hsen Huang](https://md.research.elamur.ai/free/en/authors/hen-hsen-huang/), [Hsin-Hsi Chen](https://md.research.elamur.ai/free/en/authors/hsin-hsi-chen/) · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.716/) · [PDF](https://aclanthology.org/2026.lrec-1.716.pdf) · [DOI 10.63317/4jtpgzks8pbr](https://doi.org/10.63317/4jtpgzks8pbr) · [OpenAlex](https://openalex.org/W7160158573)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
Large vision-language models (LVLMs) do well on multimodal benchmarks, but it is unclear whether their answers stay stable when the same input is arranged differently. Earlier work studied this positional bias only in text-only settings. This paper presents what the authors describe as the first systematic study of the problem for multimodal multiple-choice QA. They test two perturbations:

- **Choice order:** the answer options are reversed while the symbols (A, B, …) stay in place.
- **Modality order:** the image is placed either before or after the question text.

Stability is measured with Fluctuation Rate (FR), the share of predictions that change, and Relative Standard Deviation (RSD) of accuracy across answer positions. Fourteen models from five families were evaluated on MMMU, CVQA and MMBench.

Both perturbations change predictions. Instability is inversely related to accuracy and decreases as model size grows. Question difficulty, domain and image type strongly affect how robust a model is. The authors also examine whether text-based mitigations carry over to visual QA, including caption enhancement, self-consistency and reasoning-based methods. The results suggest that benchmark scores can overstate how reliable LVLM reasoning is.

## Key points
- **Models:** 14 LVLMs, all run through APIs with temperature 0 and simple-evals prompting.
  - Gemini 2.0/2.5 Flash and Flash Lite
  - Mistral: Pixtral 12B, Pixtral 124B, Small 3.1, Medium 3
  - Llama 4 Scout (109B) and Maverick (400B)
  - Gemma 3 at 4B, 12B and 27B
  - Phi-4
- **Data:**
  - MMMU: validation split
  - CVQA: English-translated version, 50 items sampled per country
  - MMBench: English validation split, 50 items per category
- **Benchmark difficulty:** MMMU was the hardest benchmark and showed the strongest sensitivity to modality order. MMBench was the least sensitive.
- **Difficulty and model size:** on hard MMMU questions, small models fell below 60% accuracy and exceeded 50% FR. Robustness declined faster as difficulty increased.
- **Domain:** FR was lower for Art and Humanities questions and higher for Science, Business and Engineering, which have denser visual and textual information.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wei2026do,
  title = {{Do Multimodal LLMs Understand Order? Measuring the Fragility of Multimodal Reasoning under Input Order Perturbations}},
  author = {Wei, Sheng-Lun and Liao, Yu-Ling and Huang, Hen-Hsen and Chen, Hsin-Hsi},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4jtpgzks8pbr},
  url = {https://doi.org/10.63317/4jtpgzks8pbr}
}
```

- [← previous Do LLMs Use Relational Structure in KG-Grounded Dialogue? Evidence from Entity and Relation Perturbation](https://md.research.elamur.ai/free/en/papers/okonkwo-2026-do-llms-use-relational-structure/)
- [next → Document-Level Text Simplification in Estonian Using Large Language Models](https://md.research.elamur.ai/free/en/papers/muru-2026-document-level-text-simplification-estonian/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/)

## References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas,… [find](https://scholar.google.com/scholar?q=Pravesh+Agrawal%2C+Szymon+Antoniak%2C+Emma+Bou+Hanna%2C+Baptiste+Bout%2C+Devendra+Chaplot%2C+Jessica+Chudnovsky%2C+Diogo+Costa%2C+Baudouin+De+Monicault%2C+Saurabh+Garg%2C+Theophile+Gervet%2C+Soham+Ghosh%2C+Am%C3%A9lie+H%C3%A9liou%2C+P)
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millicah, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock… [find](https://scholar.google.com/scholar?q=Jean%2DBaptiste+Alayrac%2C+Jeff+Donahue%2C+Pauline+Luc%2C+Antoine+Miech%2C+Iain+Barr%2C+Yana+Hasson%2C+Karel+Lenc%2C+Arthur+Mensch%2C+Katie+Millicah%2C+Malcolm+Reynolds%2C+Roman+Ring%2C+Eliza+Rutherford%2C+Serkan+Cabi%2C+Tengda+) [our card](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
3. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025. Do not think that much for 2+3=? on the overthinking of o1-like llms. [find](https://scholar.google.com/scholar?q=Xingyu+Chen%2C+Jiahao+Xu%2C+Tian+Liang%2C+Zhiwei+He%2C+Jianhui+Pang%2C+Dian+Yu%2C+Linfeng+Song%2C+Qiuzhi+Liu%2C+Mengfei+Zhou%2C+Zhuosheng+Zhang%2C+Rui+Wang%2C+Zhaopeng+Tu%2C+Haitao+Mi%2C+and+Dong+Yu%2E+2025%2E+Do+not+think+that+mu) [our card](https://md.research.elamur.ai/free/en/papers/chen-2024-think-that-much-2-3/)
4. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. page 104–120, Berlin, Heidelberg. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen%2C+Linjie+Li%2C+Licheng+Yu%2C+Ahmed+El+Kholy%2C+Faisal+Ahmed%2C+Zhe+Gan%2C+Yu+Cheng%2C+and+Jingjing+Liu%2E+2020%2E+Uniter%3A+Universal+image%2Dtext+representation+learning%2E+page+104%E2%80%93120%2C+Berlin%2C+Heidelberg%2E+S) [our card](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/)
5. Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, and Jiajun Chen. 2025. CapArena: Benchmarking and analyzing detailed image captioning in the LLM era. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14077–14… [find](https://scholar.google.com/scholar?q=Kanzhi+Cheng%2C+Wenpo+Song%2C+Jiaxin+Fan%2C+Zheng+Ma%2C+Qiushi+Sun%2C+Fangzhi+Xu%2C+Chenyang+Yan%2C+Nuo+Chen%2C+Jianbing+Zhang%2C+and+Jiajun+Chen%2E+2025%2E+CapArena%3A+Benchmarking+and+analyzing+detailed+image+captioning+in)
6. Hyeong Kyu Choi, Weijie Xu, Chi Xue, Stephanie Eckman, and Chandan K. Reddy. 2025. Mitigating selection bias with node pruning and auxiliary options. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5190–5215, Vienna, Austria.… [find](https://scholar.google.com/scholar?q=Hyeong+Kyu+Choi%2C+Weijie+Xu%2C+Chi+Xue%2C+Stephanie+Eckman%2C+and+Chandan+K%2E+Reddy%2E+2025%2E+Mitigating+selection+bias+with+node+pruning+and+auxiliary+options%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+A)
7. Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. 2021. Robustbench: a standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchma… [find](https://scholar.google.com/scholar?q=Francesco+Croce%2C+Maksym+Andriushchenko%2C+Vikash+Sehwag%2C+Edoardo+Debenedetti%2C+Nicolas+Flammarion%2C+Mung+Chiang%2C+Prateek+Mittal%2C+and+Matthias+Hein%2E+2021%2E+Robustbench%3A+a+standardized+adversarial+robustness)
8. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+2025%2E+Deepseek%2Dr1%3A+Incentivizing+reasoning+capability+in+llms+via+reinforcement+learning%2E) [our card](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/)
9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Amy+Yang%2C+Angela+Fan%2C+et+al%2E+2024%2E+The+llama+3+herd+of+models%2E+arXiv+prepr)
10. Google Gemini Team. 2023. Gemini: A family of highly capable multimodal models. ArXiv, abs/2312.11805. [find](https://scholar.google.com/scholar?q=Google+Gemini+Team%2E+2023%2E+Gemini%3A+A+family+of+highly+capable+multimodal+models%2E+ArXiv%2C+abs%2F2312%2E11805%2E) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/)
11. Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A. Smith, and Jiebo Luo. 2023. Promptcap: Prompt-guided image captioning for vqa with gpt-3. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2951–2963. [find](https://scholar.google.com/scholar?q=Yushi+Hu%2C+Hang+Hua%2C+Zhengyuan+Yang%2C+Weijia+Shi%2C+Noah+A%2E+Smith%2C+and+Jiebo+Luo%2E+2023%2E+Promptcap%3A+Prompt%2Dguided+image+captioning+for+vqa+with+gpt%2D3%2E+In+2023+IEEE%2FCVF+International+Conference+on+Computer+)
12. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Dongxu+Li%2C+Silvio+Savarese%2C+and+Steven+Hoi%2E+2023%2E+Blip%2D2%3A+bootstrapping+language%2Dimage+pre%2Dtraining+with+frozen+image+encoders+and+large+language+models%2E+In+Proceedings+of+the+40th+Internat) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/)
13. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. [find](https://scholar.google.com/scholar?q=Liunian+Harold+Li%2C+Mark+Yatskar%2C+Da+Yin%2C+Cho%2DJui+Hsieh%2C+and+Kai%2DWei+Chang%2E+2019%2E+Visualbert%3A+A+simple+and+performant+baseline+for+vision+and+language%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2019-visualbert-simple-performant-baseline-vision/)
14. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Qingyang+Wu%2C+and+Yong+Jae+Lee%2E+2023%2E+Visual+instruction+tuning%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+36%2C+pages+34892%E2%80%9334916%2E+Curran+Associates%2C+Inc%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)
15. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2025. Mmbench: Is your multi-modal model an all-around player? In Computer Vision – ECCV 2024, pages 216–233, Cham. Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=Yuan+Liu%2C+Haodong+Duan%2C+Yuanhan+Zhang%2C+Bo+Li%2C+Songyang+Zhang%2C+Wangbo+Zhao%2C+Yike+Yuan%2C+Jiaqi+Wang%2C+Conghui+He%2C+Ziwei+Liu%2C+Kai+Chen%2C+and+Dahua+Lin%2E+2025%2E+Mmbench%3A+Is+your+multi%2Dmodal+model+an+all%2Daround) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-mmbench-multi-modal-model-all/)
16. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Curran Associates Inc., Red Hook, NY, USA. [find](https://scholar.google.com/scholar?q=Jiasen+Lu%2C+Dhruv+Batra%2C+Devi+Parikh%2C+and+Stefan+Lee%2E+2019%2E+ViLBERT%3A+pretraining+task%2Dagnostic+visiolinguistic+representations+for+vision%2Dand%2Dlanguage+tasks%2E+Curran+Associates+Inc%2E%2C+Red+Hook%2C+NY%2C+USA%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/)
17. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tony+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+2022%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+q)
18. Meta AI. 2025. The llama 4 herd: The beginning of a new era of natively multimodal intelligence. [find](https://scholar.google.com/scholar?q=Meta+AI%2E+2025%2E+The+llama+4+herd%3A+The+beginning+of+a+new+era+of+natively+multimodal+intelligence%2E)
19. David Orlando Romero Mogrovejo, Chenyang Lyu, Haryo Akbarianto Wibowo, Santiago Góngora, Aishik Mandal, Sukannya Purkayastha, Jesus-German Ortiz-Barajas, Emilio Villa Cueva, Jinheon Baek, Soyeong Jeong, Injy Hamed, Zheng Xin Yong, Zheng Wei Lim, Paula Mónica Silva, Jocelyn Dunstan, Mélanie Jouitteau… [find](https://scholar.google.com/scholar?q=David+Orlando+Romero+Mogrovejo%2C+Chenyang+Lyu%2C+Haryo+Akbarianto+Wibowo%2C+Santiago+G%C3%B3ngora%2C+Aishik+Mandal%2C+Sukannya+Purkayastha%2C+Jesus%2DGerman+Ortiz%2DBarajas%2C+Emilio+Villa+Cueva%2C+Jinheon+Baek%2C+Soyeong+Jeon)
20. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, Mexico City, Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pouya+Pezeshkpour+and+Estevam+Hruschka%2E+2024%2E+Large+language+models+sensitivity+to+the+order+of+options+in+multiple%2Dchoice+questions%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+NAAC) [our card](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-large-language-models-sensitivity-order/)
21. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jong+Wook+Kim%2C+Chris+Hallacy%2C+Aditya+Ramesh%2C+Gabriel+Goh%2C+Sandhini+Agarwal%2C+Girish+Sastry%2C+Amanda+Askell%2C+Pamela+Mishkin%2C+Jack+Clark%2C+Gretchen+Krueger%2C+and+Ilya+Sutskever%2E+2021%2E+Learning) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
22. Noam Rotstein, David Bensaïd, Shaked Brody, Roy Ganz, and Ron Kimmel. 2024. Fusecap: Leveraging large language models for enriched fused image captions. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5677–5688. [find](https://scholar.google.com/scholar?q=Noam+Rotstein%2C+David+Bensa%C3%AFd%2C+Shaked+Brody%2C+Roy+Ganz%2C+and+Ron+Kimmel%2E+2024%2E+Fusecap%3A+Leveraging+large+language+models+for+enriched+fused+image+captions%2E+In+2024+IEEE%2FCVF+Winter+Conference+on+Applicati)
23. Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. Large language models are not yet human-level evaluators for abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4215–4233, Singapore. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Chenhui+Shen%2C+Liying+Cheng%2C+Xuan%2DPhi+Nguyen%2C+Yang+You%2C+and+Lidong+Bing%2E+2023%2E+Large+language+models+are+not+yet+human%2Dlevel+evaluators+for+abstractive+summarization%2E+In+Findings+of+the+Association+for)
24. Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. 2025. Stop overthinking: A survey on efficient reasoning for large language models. [find](https://scholar.google.com/scholar?q=Yang+Sui%2C+Yu%2DNeng+Chuang%2C+Guanchu+Wang%2C+Jiamu+Zhang%2C+Tianyi+Zhang%2C+Jiayi+Yuan%2C+Hongyi+Liu%2C+Andrew+Wen%2C+Shaochen+Zhong%2C+Hanjie+Chen%2C+and+Xia+Hu%2E+2025%2E+Stop+overthinking%3A+A+survey+on+efficient+reasoning) [our card](https://md.research.elamur.ai/free/en/papers/sui-2025-stop-overthinking-survey-efficient-reasoning/)
25. Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100–51… [find](https://scholar.google.com/scholar?q=Hao+Tan+and+Mohit+Bansal%2E+2019%2E+LXMERT%3A+Learning+cross%2Dmodality+encoder+representations+from+transformers%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and) [our card](https://md.research.elamur.ai/free/en/papers/tan-2019-lxmert-learning-cross-modality-encoder/)
26. Qwen Team. 2025. Qwen3 technical report. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2025%2E+Qwen3+technical+report%2E)
27. Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. 2025. Identifying and mitigating position bias of multi-image vision-language models. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10599– 10609. [find](https://scholar.google.com/scholar?q=Xinyu+Tian%2C+Shu+Zou%2C+Zhaoyuan+Yang%2C+and+Jing+Zhang%2E+2025%2E+Identifying+and+mitigating+position+bias+of+multi%2Dimage+vision%2Dlanguage+models%2E+2025+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recogn)
28. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xuezhi+Wang%2C+Jason+Wei%2C+Dale+Schuurmans%2C+Quoc+V+Le%2C+Ed+H%2E+Chi%2C+Sharan+Narang%2C+Aakanksha+Chowdhery%2C+and+Denny+Zhou%2E+2023%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+In+The) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
29. Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2024. Unveiling selection biases: Exploring order and token sensitivity in large language models. In Findings of the Association for Computational Linguistics: ACL 2024. [find](https://scholar.google.com/scholar?q=Sheng%2DLun+Wei%2C+Cheng%2DKuang+Wu%2C+Hen%2DHsen+Huang%2C+and+Hsin%2DHsi+Chen%2E+2024%2E+Unveiling+selection+biases%3A+Exploring+order+and+token+sensitivity+in+large+language+models%2E+In+Findings+of+the+Association+for+C)
30. Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian-Guang Lou, and Shuai Ma. 2024. Re-reading improves reasoning in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15549– 15575, Miami, Florida, USA. Associ… [find](https://scholar.google.com/scholar?q=Xiaohan+Xu%2C+Chongyang+Tao%2C+Tao+Shen%2C+Can+Xu%2C+Hongbo+Xu%2C+Guodong+Long%2C+Jian%2DGuang+Lou%2C+and+Shuai+Ma%2E+2024%2E+Re%2Dreading+improves+reasoning+in+large+language+models%2E+In+Proceedings+of+the+2024+Conference+)
31. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. Mmmu: A massive multi-disc… [find](https://scholar.google.com/scholar?q=Xiang+Yue%2C+Yuansheng+Ni%2C+Kai+Zhang%2C+Tianyu+Zheng%2C+Ruoqi+Liu%2C+Ge+Zhang%2C+Samuel+Stevens%2C+Dongfu+Jiang%2C+Weiming+Ren%2C+Yuxuan+Sun%2C+Cong+Wei%2C+Botao+Yu%2C+Ruibin+Yuan%2C+Renliang+Sun%2C+Ming+Yin%2C+Boyuan+Zheng%2C+Zhe) [our card](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
32. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+2024%2E+Large+language+models+are+not+robust+multiple+choice+selectors%2E+In+The+Twelfth+International+Conference+on+Learning+Representati)
33. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 3… [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+Hao+Zhang%2C+Joseph+E+Gonzalez%2C+and+Ion+Stoica%2E+2023%2E+Judging+llm%2Das%2Da%2Dj)