---
title: "Does the Judge Prefer English? Evaluating Language-Switching"
description: "This paper presents Judge-LS, a cheap, training-free protocol that tests whether a pairwise judge keeps its verdict when the… 12 references."
authors: ["Shaojie Yin"]
year: 2026
venue: "arXiv:2606.14278"
source: "https://arxiv.org/abs/2606.14278"
doi: "10.48550/arxiv.2606.14278"
arxiv: "2606.14278"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yin-2026-judge-prefer-english-evaluating-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yin-2026-judge-prefer-english-evaluating-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/30/) / Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge

# Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge

**Authors:** Shaojie Yin · **Year:** 2026 · **Venue:** arXiv:2606.14278 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2606.14278) · [PDF](https://arxiv.org/pdf/2606.14278) · [DOI 10.48550/arxiv.2606.14278](https://doi.org/10.48550/arxiv.2606.14278) · [arXiv 2606.14278](https://arxiv.org/abs/2606.14278) · [OpenAlex](https://openalex.org/W7164851912)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LLM judges are now a standard way to score open-ended outputs, but they may react to the language of the input rather than only to answer quality. This paper presents Judge-LS, a cheap, training-free protocol that tests whether a pairwise judge keeps its verdict when the same comparison is shown in English, Chinese, or a Chinese-English language-switched mix. All 419 LLMBar items were translated by gpt-4.1-mini, which was told to keep every original flaw so the gold labels still hold. Each item was then judged in both answer orders. Tie probes paired the English gold answer against its own Chinese translation. Four API judges produced 13,408 judgments. Every judge scored best in English, and the non-English versions flipped 10.7–14.4% of preferences. Yet the tie probes showed no consistent pro-English bias: most were called ties, and the decisive ones leaned toward Chinese. The real risk is instability across languages, which can shift correctness and interacts with position bias. That matters for any multilingual leaderboard.

## Key points
- Base data is the full LLMBar set of 419 items, split into one natural subset and four adversarial subsets: Natural 100, Adv-Neighbor 134, Adv-GPTInst 92, Adv-GPTOut 47 and Adv-Manual 46.
- Each item yields eight judgments per model: three language conditions (EN, ZH, LS) in two orders, plus two order-swapped tie probes. Across four judges this gives 13,408 successful judgments.
- The metrics are:
  - strict accuracy and tie-half accuracy, where a tie earns 0.5;
  - language-invariance flip rate and gold-correctness flip rate;
  - position inconsistency and language preference;
  - confidence intervals and paired significance tests.
- An automatic audit flagged 19 high-risk variants for empty fields, strong length shrinkage, or numeric-token count changes. After excluding them, the effect remains and strict-correctness changes stay significant.
- Practical notes:
  - One item with broken JSON was repaired by hand.
  - DeepSeek needed a larger output cap because of long reasoning before its answer.
  - The whole run used only API calls and stayed low-cost.

## Interesting ideas
- A tie probe compares an answer with its own translation, so any non-tie verdict reveals surface bias rather than quality judgment; using the strongest answer avoids confounding this with drift in how errors were rendered.
- When generating test variants, the transformation prompt explicitly asks to keep all errors, so a translation cannot quietly fix the weaker answer and spoil the label.
- A cheap mechanical audit catches likely meaning-breaking transformations before any results are interpreted; it checks empty fields, big length drops and changed counts of number tokens. Results are then rerun without the flagged items as a sensitivity check.
- Reporting how often a verdict flips under a label-preserving change, separately from accuracy, shows whether an evaluator is stable at all. Tie-half scoring keeps honest uncertainty from counting as a full error.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): when an LLM teacher tier grades silver data, include meaning-identical pairs that differ only in surface form as a negative control, and reject teachers that pick a side.
- **I2** (answer): when producing paraphrased school-test or word-problem variants, require that wrong distractors and planted mistakes survive, so the answer key stays valid.
- **I3** (back): apply the same checks to text → graph → text round-trips and translations, flagging outputs whose numbers or length diverge, and report scores with and without them.
- **I4** (learn): score witness parsers and repair rules by decision flips under order swaps or harmless rephrasings, not just accuracy, and give partial credit for abstentions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 152 works · 13,164 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 206 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Shaojie Yin. 2026. Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge. arXiv:2606.14278.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yin2026does,
  title = {{Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge}},
  author = {Shaojie Yin},
  year = {2026},
  doi = {10.48550/arxiv.2606.14278},
  eprint = {2606.14278},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.14278}
}
```

- [← previous Does RLVR Extend Reasoning Boundaries? Investigating Capability Expansion in Vision-Language Models](https://md.research.elamur.ai/free/en/papers/shen-2026-does-rlvr-extend-reasoning-boundaries/)
- [next → Does Translation Preserve Sentiment? An Analysis of Arabic-English Cross-Lingual Classification](https://md.research.elamur.ai/free/en/papers/mubarak-2026-does-translation-preserve-sentiment-analysis/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/30/)

## References

12 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [4] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan [15] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen-Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuoliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei han Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph Ji, Tiezheng Yu, Willy Chung, et al. 2023. A Multi-E. [find](https://scholar.google.com/scholar?q=%5B4%5D+Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+%5B15%5D+Yejin+Bang%2C+Samuel+Cahyawijaya%2C+Nayeon+Lee%2C+Wen%2DZhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuoliang+Dai%2C+Dan+Su%2C+Bryan+Wilie%2C+Holy+Lovenia)
2. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-atask, Multilingual, Multimodal Evaluation of ChatGPT Judge with MT-Bench and Chatbot Arena. In Advances on Reasoning, Hallucination, and Interactivity. In Pro-in Neural Information Processing Systems 36. ceedings of IJCNLP-AACL 2023, pages 675–718. [find](https://scholar.google.com/scholar?q=Gonzalez%2C+and+Ion+Stoica%2E+2023%2E+Judging+LLM%2Das%2Datask%2C+Multilingual%2C+Multimodal+Evaluation+of+ChatGPT+Judge+with+MT%2DBench+and+Chatbot+Arena%2E+In+Advances+on+Reasoning%2C+Hallucination%2C+and+Interactivity%2E+)
3. [5] Kishore Papineni, Salim Roukos, Todd Ward, and Wei- [16] Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Vey-Jing Zhu. 2002. BLEU: A Method for Automatic Eval-seh, Hieu Man, Franck Dernoncourt, Trung Bui, and uation of Machine Translation. In Proceedings of ACL Thien Huu Nguyen. 2023. ChatGPT Beyond English: 2002, pages 311–318. Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13171–13189. with selective aggregation for video rescaling. IEEE Transactions on Image Processing, 32:1300–1314. [find](https://scholar.google.com/scholar?q=%5B5%5D+Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2D+%5B16%5D+Viet+Dac+Lai%2C+Nghia+Trung+Ngo%2C+Amir+Pouran+Ben+Vey%2DJing+Zhu%2E+2002%2E+BLEU%3A+A+Method+for+Automatic+Eval%2Dseh%2C+Hieu+Man%2C+Franck+Dernoncourt%2C+Tru)
4. [25] Yuan Tian, Guo Lu, Xiongkuo Min, Z. Che, Guangtao Zhai, G. Guo, and Z. Gao. 2021. Self-conditioned prob-[17] Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent abilistic learning of video rescaling. In Proceedings of Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, the IEEE/CVF International Conference on Computer Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Vision, pages 4490–4499. [find](https://scholar.google.com/scholar?q=%5B25%5D+Yuan+Tian%2C+Guo+Lu%2C+Xiongkuo+Min%2C+Z%2E+Che%2C+Guangtao+Zhai%2C+G%2E+Guo%2C+and+Z%2E+Gao%2E+2021%2E+Self%2Dconditioned+prob%2D%5B17%5D+Kabir+Ahuja%2C+Harshita+Diddee%2C+Rishav+Hada%2C+Millicent+abilistic+learning+of+video+resca)
5. Bali, and Sunayana Sitaram. 2023. [MEGA: Multilingual Evaluation of Generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/). In Proceedings of EMNLP [26] Yuan Tian, Zhaohui Che, Wenbo Bao, Guangtao Zhai, 2023, pages 4232–4267. and Z. Gao. 2020. Self-supervised Motion Representation via Scattering Local Motion Cues. In European [18] Zicheng Zhang, Junying Wang, Yijin Guo, Farong Wen, Conference on Computer Vision, pages 71–89. Zijian Chen, Hanqing Wang, Wenzhe Li, Lu Sun, Yingjie Zhou, Jianbo Zhang, Bowen Yan, Ziheng Jia, Jiahao Xiao, Yuan Tian, Xiangyang Zhu, Kaiwei Zhang, Chunyi [27] Yuan Tian, Xiaoyue Ling, Cong Geng, Q. Hu, Guo Lu, and Guangtao Zhai. 2025. SMC++: Masked learning of Li, Xiaohong Liu, Xiongkuo Min, Qi Jia, and Guang-unsupervised video semantic compression.
6. IEEE Trans-tao Zhai. 2025. AIBench: Towards trustworthy evalu-actions on Pattern Analysis and Machine Intelligence. ◦ ation under the 45 law. Displays, page 103255. doi: 10.1016/j.displa.2025.103255. [↗](https://doi.org/10.1016/j.displa.2025.103255)
7. [19] Zicheng Zhang, Junying Wang, Farong Wen, Yijin Guo, Xiangyu Zhao, Xinyu Fang, Shengyuan Ding, Ziheng Jia, Jiahao Xiao, Ye Shen, Yushuo Zheng, Xiaorong Zhu, Yalun Wu, Ziheng Jiao, Wei Sun, Zijian Chen, Kaiwei Zhang, Kang Fu, Yuqin Cao, Ming Hu, Yue Zhou, Xuemei Zhou, Juntai Cao, Wei Zhou, Jinyu Cao, Ronghui Li, Donghao Zhou, Yuan Tian, Xiangyang Zhu, Chunyi Li, Haoning Wu, Xiaohong Liu, Junjun He, Yu Zhou, Hui Liu, Lin Zhang, Zesheng Wang, Huiyu Duan, Yingjie Zhou, Xiongkuo Min, Qi Jia, Dongzhan Zhou, Wenlong Zhang, Jiezhang Cao, Xue Yang, Junzhi Yu, Songyang Zhang, Haodong Duan, and Guangtao Zhai. 2025. Large Multimodal Models Evaluation: A Survey. SCIENCE CHINA Information Sciences, 68(12):221301–221369. doi: 10.1007/s11432-025-4676-4. [↗](https://doi.org/10.1007/s11432-025-4676-4)
8. [20] Yuan Tian, Xiongkuo Min, Guangtao Zhai, and Z. Gao. 2019. Video-based early ASD detection via temporal pyramid networks. In IEEE International Conference on Multimedia and Expo, pages 272–277. [find](https://scholar.google.com/scholar?q=%5B20%5D+Yuan+Tian%2C+Xiongkuo+Min%2C+Guangtao+Zhai%2C+and+Z%2E+Gao%2E+2019%2E+Video%2Dbased+early+ASD+detection+via+temporal+pyramid+networks%2E+In+IEEE+International+Conference+on+Multimedia+and+Expo%2C+pages+272%E2%80%93277%2E)
9. [21] Yuan Tian, Yichao Yan, Guangtao Zhai, G. Guo, and Z. Gao. 2022. EAN: Event adaptive network for enhanced action recognition. International Journal of Computer Vision, 130(10):2453–2471. [find](https://scholar.google.com/scholar?q=%5B21%5D+Yuan+Tian%2C+Yichao+Yan%2C+Guangtao+Zhai%2C+G%2E+Guo%2C+and+Z%2E+Gao%2E+2022%2E+EAN%3A+Event+adaptive+network+for+enhanced+action+recognition%2E+International+Journal+of+Computer+Vision%2C+130%2810%29%3A2453%E2%80%932471%2E)
10. [22] Yuan Tian, Guo Lu, Yichao Yan, Guangtao Zhai, L. Chen, and Z. Gao. 2024. A Coding Framework and Benchmark towards Low-Bitrate Video Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence. [find](https://scholar.google.com/scholar?q=%5B22%5D+Yuan+Tian%2C+Guo+Lu%2C+Yichao+Yan%2C+Guangtao+Zhai%2C+L%2E+Chen%2C+and+Z%2E+Gao%2E+2024%2E+A+Coding+Framework+and+Benchmark+towards+Low%2DBitrate+Video+Understanding%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machin)
11. [23] Yuan Tian, Guo Lu, Guangtao Zhai, and Z. Gao. 2023. Non-semantics suppressed mask learning for unsupervised video semantic compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13610–13622. [find](https://scholar.google.com/scholar?q=%5B23%5D+Yuan+Tian%2C+Guo+Lu%2C+Guangtao+Zhai%2C+and+Z%2E+Gao%2E+2023%2E+Non%2Dsemantics+suppressed+mask+learning+for+unsupervised+video+semantic+compression%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+)
12. [24] Yuan Tian, Yichao Yan, Guangtao Zhai, L. Chen, and Z. Gao. 2023. CLSA: A contrastive learning framework [find](https://scholar.google.com/scholar?q=%5B24%5D+Yuan+Tian%2C+Yichao+Yan%2C+Guangtao+Zhai%2C+L%2E+Chen%2C+and+Z%2E+Gao%2E+2023%2E+CLSA%3A+A+contrastive+learning+framework)

*Updated 31 October 2026*