---
title: "SycoBench-600 — paper by Sinha (2026) — summary and ideas"
description: "The author introduces correction selectivity, defined as the update rate minus the wrong-flip rate, and evaluates seven… 23 references."
authors: ["Debu Sinha"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1759/"
doi: "10.18653/v1/2026.findings-acl.1759"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sinha-2026-sycobench-600-measuring-sycophancy-correction/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sinha-2026-sycobench-600-measuring-sycophancy-correction/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/) / SycoBench-600: Measuring Sycophancy and Correction Selectivity in LLM Assistants

# SycoBench-600: Measuring Sycophancy and Correction Selectivity in LLM Assistants

**Authors:** Debu Sinha · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1759/) · [PDF](https://aclanthology.org/2026.findings-acl.1759.pdf) · [DOI 10.18653/v1/2026.findings-acl.1759](https://doi.org/10.18653/v1/2026.findings-acl.1759) · [OpenAlex](https://openalex.org/W7166881271)

**Citations:** 1 citations (OpenAlex, 2026-10-14)

## Summary
LLM assistants tuned for helpfulness often give in to users who are wrong, yet they should still accept genuine corrections. Existing sycophancy benchmarks mostly measure how easily models yield to pressure. They do not separate accepting a valid fix from caving to a false claim. SycoBench-600 addresses this with a controlled multiple-choice setup. It contains 600 hand-written English items built on 272 normalized stems, spanning 8 domains and three difficulty tiers. Each item is tested under three misleading pressures (doubt, appeal to authority, an explicit wrong suggestion) and a matched condition in which the user proposes the correct answer. Every perturbation is run with three fixed prompt paraphrases. The author introduces correction selectivity, defined as the update rate minus the wrong-flip rate, and evaluates seven proprietary and open-weight assistants. The central finding is that resisting pressure and accepting correction are separate abilities. Models fall into three groups: robust and selective, robust but stubborn, or quick to update in either direction. This matters for building assistants that are hard to mislead yet still correctable. Raw logs and regeneration scripts are released.

## Key points
- **Dataset and protocol:** 75 items per domain, with a 20/40/40 split across easy, medium and hard. Models must answer with a single letter (A–D) at temperature 0. The parser takes the last standalone capital letter in the response.
- **Evaluation scope:** Final results use a 555-question intersection. Forty-five flawed items were repaired and rerun, but Mistral-7B was retired from its provider and could not be rerun on them, so those items were dropped for all models. Confidence intervals come from a cluster bootstrap over question IDs.
- **Best overall:** Gemini-2.5-Flash had the highest baseline accuracy (95.3%) and pressure-robust accuracy (81.7%).
- **Other model results:**
  - GPT-4o reached the highest selectivity (71.6).
  - Mistral-7B was the most sycophantic, flipping to wrong answers 50.5% of the time.
  - Llama-4-Maverick flipped least (5.9%), but 31% of its baseline-wrong runs were unparseable and excluded from correction metrics.
  - GPT-4o-mini was the most stubborn (64.7% no-change under correct suggestions).
  - Claude-3.5-Haiku almost always accepted corrections (0.2% stubbornness) but also caved often (36.4% sycophancy).
- **Pressure types:** The three pressures have different effects. Authority prompts were the strongest for most models, while explicit wrong suggestions hurt Mistral-7B and Llama-4-Maverick most.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Debu Sinha. 2026. SycoBench-600: Measuring Sycophancy and Correction Selectivity in LLM Assistants. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sinha2026sycobench,
  title = {{SycoBench-600: Measuring Sycophancy and Correction Selectivity in LLM Assistants}},
  author = {Debu Sinha},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1759},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1759}
}
```

- [← previous SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?](https://md.research.elamur.ai/free/en/papers/sun-2026-swe-mutation-can-llms-generate/)
- [next → Syllable Structures Across Arabic Varieties](https://md.research.elamur.ai/free/en/papers/qaddoumi-2026-syllable-structures-across-arabic-varieties/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/)

## References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Solomon E. Asch. 1951. Effects of group pressure upon the modification and distortion of judgments. In Groups, Leadership and Men. [find](https://scholar.google.com/scholar?q=Solomon+E%2E+Asch%2E+1951%2E+Effects+of+group+pressure+upon+the+modification+and+distortion+of+judgments%2E+In+Groups%2C+Leadership+and+Men%2E)
2. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Nat McAleese, and 1 others. 2022. [Constitutional AI: Harmlessness from AI feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073.
3. Jan Batzner, Volker Stocker, Stefan Schmid, and Gjergji Kasneci. 2024. GermanPartiesQA: Benchmarking commercial large language models and AI companions for political alignment and sycophancy. [find](https://scholar.google.com/scholar?q=Jan+Batzner%2C+Volker+Stocker%2C+Stefan+Schmid%2C+and+Gjergji+Kasneci%2E+2024%2E+GermanPartiesQA%3A+Benchmarking+commercial+large+language+models+and+AI+companions+for+political+alignment+and+sycophancy%2E)
4. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender+and+Batya+Friedman%2E+2018%2E+Data+statements+for+natural+language+processing%3A+Toward+mitigating+system+bias+and+enabling+better+science%2E+Transactions+of+the+Association+for+Computational+)
5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems.
6. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). arXiv preprint arXiv:2203.02155.
7. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, and 1 others. 2023. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/).
8. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, and 1 others. 2022. [Discovering language model behaviors with model-written evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/). arXiv preprint arXiv:2212.09251.
9. Wei Chen, Zhen Huang, Liang Xie, Binbin Lin, Houqiang Li, Le Lu, Xinmei Tian, Deng Cai, Yonggang Zhang, Wenxiao Wang, Xu Shen, and Jieping Ye. 2024. From yes-men to truth-tellers: Addressing sycophancy in large language models with pinpoint tuning. In Proceedings of the 41st International Conference… [find](https://scholar.google.com/scholar?q=Wei+Chen%2C+Zhen+Huang%2C+Liang+Xie%2C+Binbin+Lin%2C+Houqiang+Li%2C+Le+Lu%2C+Xinmei+Tian%2C+Deng+Cai%2C+Yonggang+Zhang%2C+Wenxiao+Wang%2C+Xu+Shen%2C+and+Jieping+Ye%2E+2024%2E+From+yes%2Dmen+to+truth%2Dtellers%3A+Addressing+sycophanc)
10. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Paul+F%2E+Christiano%2C+Jan+Leike%2C+Tom+B%2E+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+Advances+in+Neural+Information+Processing+Systems)
11. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of EMNLP-IJCNLP. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Suchin+Gururangan%2C+Dallas+Card%2C+Roy+Schwartz%2C+and+Noah+A%2E+Smith%2E+2019%2E+Show+your+work%3A+Improved+reporting+of+experimental+results%2E+In+Proceedings+of+EMNLP%2DIJCNLP%2E)
12. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Rotem+Dror%2C+Gili+Baumer%2C+Segev+Shlomov%2C+and+Roi+Reichart%2E+2018%2E+The+hitchhiker%E2%80%99s+guide+to+testing+statistical+significance+in+natural+language+processing%2E+In+Proceedings+of+ACL%2E)
13. Aaron Fanous, Jacob Goldberg, Ank A. Agarwal, Joanna Lin, Anson Zhou, Roxana Daneshjou, and Sanmi Koyejo. 2025. SycEval: Evaluating LLM sycophancy. [find](https://scholar.google.com/scholar?q=Aaron+Fanous%2C+Jacob+Goldberg%2C+Ank+A%2E+Agarwal%2C+Joanna+Lin%2C+Anson+Zhou%2C+Roxana+Daneshjou%2C+and+Sanmi+Koyejo%2E+2025%2E+SycEval%3A+Evaluating+LLM+sycophancy%2E)
14. Ivo Petrov, Jasper Dekoninck, and Martin Vechev. 2025. BrokenMath: A benchmark for sycophancy in theorem proving with LLMs. [find](https://scholar.google.com/scholar?q=Ivo+Petrov%2C+Jasper+Dekoninck%2C+and+Martin+Vechev%2E+2025%2E+BrokenMath%3A+A+benchmark+for+sycophancy+in+theorem+proving+with+LLMs%2E)
15. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d’Alche Buc, Emily Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research. Journal of Machine Learning Research. [find](https://scholar.google.com/scholar?q=Joelle+Pineau%2C+Philippe+Vincent%2DLamarre%2C+Koustuv+Sinha%2C+Vincent+Lariviere%2C+Alina+Beygelzimer%2C+Florence+d%E2%80%99Alche+Buc%2C+Emily+Fox%2C+and+Hugo+Larochelle%2E+2021%2E+Improving+reproducibility+in+machine+learning+)
16. Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. 2023. [Simple synthetic data reduces sycophancy in large language models](https://md.research.elamur.ai/free/en/papers/wei-2023-simple-synthetic-data-reduces-sycophancy/).
17. Botai Yuan, Yutian Zhou, Yingjie Wang, Fushuo Huo, Yongcheng Jing, Li Shen, Ying Wei, Zhiqi Shen, Ziwei Liu, Tianwei Zhang, Jie Yang, and Dacheng Tao. 2025. EchoBench: Benchmarking sycophancy in medical large vision-language models. [find](https://scholar.google.com/scholar?q=Botai+Yuan%2C+Yutian+Zhou%2C+Yingjie+Wang%2C+Fushuo+Huo%2C+Yongcheng+Jing%2C+Li+Shen%2C+Ying+Wei%2C+Zhiqi+Shen%2C+Ziwei+Liu%2C+Tianwei+Zhang%2C+Jie+Yang%2C+and+Dacheng+Tao%2E+2025%2E+EchoBench%3A+Benchmarking+sycophancy+in+medic)
18. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/).
19. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daume%CC%81+III%2C+and+Kate+Crawford%2E+2021%2E+Datasheets+for+datasets%2E+Communications+of+the+ACM%2E)
20. Jiseung Hong, Grace Byun, Seungone Kim, and Kai Shu. 2025. Measuring sycophancy of language models in multi-turn dialogues. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 2239–2259, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiseung+Hong%2C+Grace+Byun%2C+Seungone+Kim%2C+and+Kai+Shu%2E+2025%2E+Measuring+sycophancy+of+language+models+in+multi%2Dturn+dialogues%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2025%2C+pa)
21. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
22. Lars Malmqvist. 2024. [Sycophancy in large language models: Causes and mitigations](https://md.research.elamur.ai/free/en/papers/malmqvist-2024-sycophancy-large-language-models-causes/).
23. Stanley Milgram. 1963. Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67(4):371–378. [find](https://scholar.google.com/scholar?q=Stanley+Milgram%2E+1963%2E+Behavioral+study+of+obedience%2E+Journal+of+Abnormal+and+Social+Psychology%2C+67%284%29%3A371%E2%80%93378%2E)

*Updated 2 November 2026*