---
title: "Enhancing the Reasoning Ability of Multimodal Large Language"
description: "Open-source multimodal LLMs trained only with pre-training and supervised fine-tuning often score… 13 references, cited by 40 papers here."
authors: ["Weiyun Wang", "Zhe Chen", "Wenhai Wang", "Yue Cao", "Yangzhou Liu", "Zhangwei Gao", "Jinguo Zhu", "Xizhou Zhu", "Lewei Lu", "Yu Qiao", "Jifeng Dai"]
year: 2024
venue: "arXiv:2411.10442"
source: "https://arxiv.org/abs/2411.10442"
doi: "10.48550/arxiv.2411.10442"
arxiv: "2411.10442"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-enhancing-reasoning-ability-multimodal-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-enhancing-reasoning-ability-multimodal-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

# Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

**Authors:** [Weiyun Wang](https://md.research.elamur.ai/free/en/authors/weiyun-wang/), [Zhe Chen](https://md.research.elamur.ai/free/en/authors/zhe-chen/), [Wenhai Wang](https://md.research.elamur.ai/free/en/authors/wenhai-wang/), [Yue Cao](https://md.research.elamur.ai/free/en/authors/yue-cao/), [Yangzhou Liu](https://md.research.elamur.ai/free/en/authors/yangzhou-liu/), [Zhangwei Gao](https://md.research.elamur.ai/free/en/authors/zhangwei-gao/), [Jinguo Zhu](https://md.research.elamur.ai/free/en/authors/jinguo-zhu/), [Xizhou Zhu](https://md.research.elamur.ai/free/en/authors/xizhou-zhu/), [Lewei Lu](https://md.research.elamur.ai/free/en/authors/lewei-lu/), [Yu Qiao](https://md.research.elamur.ai/free/en/authors/yu-qiao/) and [Jifeng Dai](https://md.research.elamur.ai/free/en/authors/jifeng-dai/) · **Year:** 2024 · **Venue:** arXiv:2411.10442 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2411.10442) · [PDF](https://arxiv.org/pdf/2411.10442) · [DOI 10.48550/arxiv.2411.10442](https://doi.org/10.48550/arxiv.2411.10442) · [arXiv 2411.10442](https://arxiv.org/abs/2411.10442) · [OpenAlex](https://openalex.org/W4404569805)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [40](https://md.research.elamur.ai/free/en/papers/wang-2024-enhancing-reasoning-ability-multimodal-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Open-source multimodal LLMs trained only with pre-training and supervised fine-tuning often score lower when they reason step by step than when they answer directly. For example, InternVL2-8B drops from 58.3 to 56.8 on MathVista with Chain-of-Thought (CoT). The authors trace this to the gap between teacher-forced training and free generation at inference, which grows with longer outputs.

They respond with a preference-optimization stage built on two parts:
- **MMPR**, an automatically built preference dataset. For questions with known answers, sampled solutions are labeled chosen or rejected by correctness. For open-ended prompts, they use Dropout Next-Token Prediction: a response is cut in half and the model finishes it without seeing the image, and that completion becomes the rejected answer.
- **Mixed Preference Optimization (MPO)**, a training objective that blends a pairwise preference term, a per-response quality term and a generation (SFT) term. It needs no reward model.

The resulting InternVL2-8B-MPO reaches 67.0 on MathVista, roughly matching InternVL2-76B. This shows that preference learning can improve reasoning in MLLMs, not only reduce hallucinations.

## Key points
- MMPR holds about 3 million preference pairs, generated with InternVL2 models and no human annotation.
- InternVL2-8B-MPO gains 8.7 points on MathVista (58.3 → 67.0). The method also helps InternVL2-76B.
- MPO uses DPO as the preference loss and BCO as the quality loss, plus an SFT loss. In ablations on M3CoT, plain DPO variants gave little CoT benefit, adding the SFT loss was key, and a frozen reference model mattered.
- Dropout NTP data cuts hallucination about as well as RLAIF-V (Object HalBench 7.6 vs 7.3; MMHal-Bench 3.6 vs 3.5) at much lower cost. It also beats SeVa, POVID and STIC.
- Text-only benchmarks improve even though MMPR contains no text-only data. The average rises from 62.1 to 63.0, with TheoremQA reaching 20.8 (+5.2) and IFEval gaining +4.1.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 167 works · 20,987 citations
- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 126 works · 14,073 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 74 works · 7,814 citations
- [Sensetime](https://md.research.elamur.ai/free/en/companies/sensetime/) — Company, Beijing, China · 12 works · 2,065 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao and Jifeng Dai. 2024. Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization. arXiv:2411.10442.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2024enhancing,
  title = {{Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization}},
  author = {Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao and Jifeng Dai},
  year = {2024},
  doi = {10.48550/arxiv.2411.10442},
  eprint = {2411.10442},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.10442}
}
```

- [← previous Augmenting Math Word Problems via Iterative Question Composing](https://md.research.elamur.ai/free/en/papers/liu-2024-augmenting-math-word-problems-iterative/)
- [next → HyperCLOVA X Technical Report](https://md.research.elamur.ai/free/en/papers/yoo-2024-hyperclova-x-technical-report/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

13 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023. 1
2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). NIPS, 35:23716–23736, 2022. 3
3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021. 8
4. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024…
5. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609, 2023. 1, 3
6. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.[12966, 2023](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/). 1
7. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. Introducing our multimodal models, 2023. 3 [find](https://scholar.google.com/scholar?q=Rohan+Bavishi%2C+Erich+Elsen%2C+Curtis+Hawthorne%2C+Maxwell+Nye%2C+Augustus+Odena%2C+Arushi+Somani%2C+and+Sag%CC%86nak+Tas%CC%A7%C4%B1rlar%2E+Introducing+our+multimodal+models%2C+2023%2E+3)
8. Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, pages 4291–4301, 2019. 4 [find](https://scholar.google.com/scholar?q=Ali+Furkan+Biten%2C+Ruben+Tito%2C+Andres+Mafla%2C+Lluis+Gomez%2C+Marc%CC%A7al+Rusinol%2C+Ernest+Valveny%2C+CV+Jawahar%2C+and+Dimosthenis+Karatzas%2E+Scene+text+visual+question+answering%2E+In+ICCV%2C+pages+4291%E2%80%934301%2C+2019%2E+4)
9. Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. 3, 4 [find](https://scholar.google.com/scholar?q=Ralph+Allan+Bradley+and+Milton+E+Terry%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E+3%2C+4)
10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). NIPS, 2020. 1
11. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. [Internlm2 technical report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/). arXiv preprint arXiv:2403.17297, 2024. 1, 3
12. Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In COLING, pages 1511–1520, 2022. 4 [find](https://scholar.google.com/scholar?q=Jie+Cao+and+Jing+Xiao%2E+An+augmented+benchmark+dataset+for+geometric+question+answering+through+dual+parallel+text+encoding%2E+In+COLING%2C+pages+1511%E2%80%931520%2C+2022%2E+4)
13. Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211.[08545, 2022](https://md.research.elamur.ai/free/en/papers/lai-2024-step-dpo-step-wise-preference/). 4 evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241, 2023. 4 [40] Aniruddha Kembhavi…

*Updated 30 October 2026*