---
title: "Cancer-Myth — paper by Zhu et al. (2025) — summary and ideas"
description: "Cancer patients increasingly ask LLMs for medical guidance. 54 references, cited by 5 papers here."
authors: ["Wang Bill Zhu", "Tianqi Chen", "Xinyan Velocity Yu", "Ching Ying Lin", "Jade Law", "Mazen Jizzini", "Jorge J. Nieva", "Ruishan Liu", "Robin Jia"]
year: 2025
venue: "arXiv:2504.11373"
source: "https://arxiv.org/abs/2504.11373"
doi: "10.48550/arxiv.2504.11373"
arxiv: "2504.11373"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhu-2025-cancer-myth-evaluating-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2025-cancer-myth-evaluating-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/) / Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions

# Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions

**Authors:** Wang Bill Zhu, [Tianqi Chen](https://md.research.elamur.ai/free/en/authors/tianqi-chen/), [Xinyan Velocity Yu](https://md.research.elamur.ai/free/en/authors/xinyan-yu/), Ching Ying Lin, Jade Law, Mazen Jizzini, Jorge J. Nieva, Ruishan Liu and [Robin Jia](https://md.research.elamur.ai/free/en/authors/robin-jia/) · **Year:** 2025 · **Venue:** arXiv:2504.11373 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2504.11373) · [PDF](https://arxiv.org/pdf/2504.11373) · [DOI 10.48550/arxiv.2504.11373](https://doi.org/10.48550/arxiv.2504.11373) · [arXiv 2504.11373](https://arxiv.org/abs/2504.11373) · [OpenAlex](https://openalex.org/W4415907988)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/zhu-2025-cancer-myth-evaluating-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Cancer patients increasingly ask LLMs for medical guidance. Their questions often carry personal details and hidden misconceptions, which existing medical benchmarks largely ignore. The authors first ran a pilot: three hematology-oncology physicians blindly rated LLM and social-worker answers to real CancerCare questions. The LLM answers scored well overall, but they often went along with mistaken beliefs embedded in the questions.

To study this failure systematically, the authors built Cancer-Myth. They gathered 994 myths with factual corrections and sources. An adversarial loop then generated patient-style questions that embed these myths. Each question was answered by a target model and checked by an LLM verifier. Questions the target model failed on were kept, and physicians vetted the result, producing 585 items in 7 categories. A companion set, Cancer-Myth-NFP, holds 150 questions that LLMs wrongly flagged as flawed.

No frontier model fully corrected more than 43% of the false premises. Prompt-based fixes raised that figure but caused frequent false alarms and hurt other benchmarks. The work matters because silently accepting a wrong premise can push patients away from effective care, and prompting alone does not reliably solve it.

## Key points
- Pilot study: 25 CancerCare questions, each with four blinded answers split into 648 advice paragraphs and rated 1–5. Mean scores were GPT-4-Turbo 4.13, Gemini-1.5-Pro 3.91, LLaMa-3.1-405B 3.57, and human social workers 3.20.
- Myth bank: 994 unique myths covering 127 cancer types listed by the National Cancer Institute. GPT-4o structured each myth into a cancer type, a correction and a source.
- Generation runs targeted GPT-4o, Gemini-1.5-Pro and Claude-3.5-Sonnet separately. The final set has 585 physician-checked questions in 7 categories, and models do worst on "Inevitable Side Effect".
- GPT-5, Gemini-2.5-Pro and Claude-4-Sonnet all stay at or below 43% full correction. Medical agent methods do not close the gap.
- GEPA-optimized precautionary prompts lift Gemini-2.5-Pro to 80% on Cancer-Myth. The costs are a 41% false-flag rate on Cancer-Myth-NFP and about a 10% relative drop on MedQA, PubMedQA, SymCat, Medbullets and Crafted-MD.

## Interesting ideas
- A dedicated set of clean questions that look suspicious (Cancer-Myth-NFP) shows whether a "fix" just makes the model flag everything; gains on the trap set mean little without it.
- The generator–responder–verifier loop keeps only questions the target fails on. It recycles successes as easy examples and failures as hard examples for later prompting rounds, so difficulty grows automatically.
- The paper judges an answer partly by whether it explicitly rejects a false premise before helping, not only by whether it is factually correct.
- Treating a premise as a separate claim to be verified, rather than as background, separates what the asker assumes from what they ask.
- Each myth is stored as a triple of myth, correction and source, giving every rebuttal a traceable justification.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Pair every new checker or repair rule with a look-alike negative-control set, and report its false-trigger rate next to its gains.
- **I2** (learn): A self-play loop could generate inputs, keep the ones Mova's readers get wrong, and feed them back as hard cases for repair-rule induction.
- **I3** (answer): Mova's readers could compare a question's presuppositions against the world of the text and say outright when the premise is false, instead of answering as if it were true.
- **I4** (frames): Prag could extract presuppositions as their own frames, marked as speaker assumptions, so later stages can check them independently.
- **I5** (world): Compiled knowledge used to refute claims could carry the correcting fact plus its provenance, keeping each refutation explainable.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 162 works · 35,836 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 241 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 184 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [😉 Implicature](https://md.research.elamur.ai/free/en/topics/implicature/) · 138 works

## Cite

Cite as: Wang Bill Zhu, Tianqi Chen, Xinyan Velocity Yu, Ching Ying Lin, Jade Law, Mazen Jizzini, Jorge J. Nieva, Ruishan Liu and Robin Jia. 2025. Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions. arXiv:2504.11373.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhu2025cancer,
  title = {{Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions}},
  author = {Wang Bill Zhu, Tianqi Chen, Xinyan Velocity Yu, Ching Ying Lin, Jade Law, Mazen Jizzini, Jorge J. Nieva, Ruishan Liu and Robin Jia},
  year = {2025},
  doi = {10.48550/arxiv.2504.11373},
  eprint = {2504.11373},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.11373}
}
```

- [← previous Can LLMs Help Create Grammar?: Automating Grammar Creation for Endangered Languages with In-Context Learning](https://md.research.elamur.ai/free/en/papers/spencer-2025-can-llms-help-create-grammar/)
- [next → Case2Code: Scalable Synthetic Data for Code Generation](https://md.research.elamur.ai/free/en/papers/shao-2025-case2code-scalable-synthetic-data-code/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/)

## References

54 references, 20 of them with a card of ours. ↗ leads to the work.

1. Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv, 2025. URL https://arxiv.org/abs/2507.19457. [↗](https://arxiv.org/abs/2507.19457)
2. Zaid Al-Ars, Obinna Agba, Zhuoran Guo, Christiaan Boerkamp, Ziyaad Jaber, and Tareq Jaber. Nlice: Synthetic medical record generation for effective primary healthcare differential diagnosis. 2023 IEEE 23rd International Conference on Bioinformatics and Bioengineering (BIBE), pp. 397–402, 2023. URL [https://api.semanticscholar.org/CorpusID:267211837](https://api.semanticscholar.org/CorpusID:267211837). [↗](https://api.semanticscholar.org/CorpusID:267211837)
3. Anthropic. Claude 3.5 sonnet. [https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet), 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet. [↗](https://www.anthropic.com/news/claude-3-5-sonnet)
4. John Ayers, Adam Poliak, Mark Dredze, Eric Leas, Zechariah Zhu, Jessica Kelley, Dennis Faix, Aaron Goodman, Christopher Longhurst, Michael Hogarth, and David Smith. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA internal medicine, 2023. URL https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2804309. [↗](https://doi.org/10.1001/jamainternmed.2023.1838)
5. Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. Improving question answering model robustness with synthetic adversarial data generation. In EMNLP, 2021. URL https://aclanthology.org/2021.emnlp-main.696/. [↗](https://aclanthology.org/2021.emnlp-main.696/)
6. Belong.Life. Dave, the world’s first real-time conversational ai oncology mentor., 2024. URL [https://belong.life/press/](https://belong.life/press/) belong-life-launches-dave-worlds-first-real-time-conversational-ai-oncology-mentor/. [↗](https://belong.life/press/)
7. Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman. Overview of the medical question answering task at trec 2017 liveqa. In TREC, 2017. URL [https://lhncbc.nlm.nih.gov/LHC-publications/pubs/](https://lhncbc.nlm.nih.gov/LHC-publications/pubs/) OverviewoftheMedicalQATaskTREC2017LiveQATrack.html. [↗](https://lhncbc.nlm.nih.gov/LHC-publications/pubs/)
8. Asma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis Goodwin, Sonya E. Shooshan, and Dina Demner-Fushman. Bridging the gap between consumers’ medication questions and trusted answers. In MEDINFO, 2019. URL https://pubmed.ncbi.nlm.nih.gov/ 31437878/. [↗](https://doi.org/10.3233/shti190176)
9. Amber Bourgeois, Tara Horrill, Ashley Mollison, Eleah Stringer, Leah K Lambert, and Kelli Stajduhar. Barriers to cancer treatment for people experiencing socioeconomic disadvantage in high-income countries: a scoping review. BMC Health Services Research, 2024. URL https://pubmed.ncbi.nlm.nih.gov/38807237/. [↗](https://doi.org/10.1186/s12913-024-11129-2)
10. Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal, and Yejin Choi. Adversarial filters of dataset biases. arXiv, 2020. URL https://arxiv.org/abs/2002.04108. [↗](https://arxiv.org/abs/2002.04108)
11. CancerCare. Cancercare, 2025. URL [https://www.cancercare.org](https://www.cancercare.org). [↗](https://www.cancercare.org)
12. Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. [Benchmarking large language models on answering and explaining challenging medical questions](https://md.research.elamur.ai/free/en/papers/chen-2025-benchmarking-large-language-models-answering/). In North American Chapter of the Association for Computational Linguistics, 2024. URL https://api.semanticscholar.org/CorpusID:268041175. [↗](https://doi.org/10.48550/arxiv.2402.18060)
13. Zeming Chen, Alejandro Hernández-Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. [Meditron-70b: Scaling medical pretraining for large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/). arXiv, 2023. URL https://arxiv.org/abs/2311.16079. [↗](https://arxiv.org/abs/2311.16079)
14. DeepSeek-AI. Deepseek-v3 technical report. arXiv, 2025. URL https://arxiv.org/abs/ 2412.19437. [↗](https://arxiv.org/abs/)
15. Deqing Fu, Ameya Godbole, and Robin Jia. SCENE: Self-labeled counterfactuals for extrapolating to negative examples. In EMNLP, 2023. URL https://aclanthology.org/2023. emnlp-main.485/. [↗](https://aclanthology.org/2023)
16. Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. Tldr: Token-level detective reward model for large vision language models. arXiv, 2025. URL https://arxiv.org/abs/2410.04734. [↗](https://arxiv.org/abs/2410.04734)
17. Google. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv, 2023. URL https://arxiv.org/abs/2312.11805. [↗](https://arxiv.org/abs/2312.11805)
18. Google. Gemma 2: Improving open language models at a practical size. 2024. URL https://arxiv.org/abs/2408.00118. [↗](https://arxiv.org/abs/2408.00118)
19. Google. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). 2025. URL https://arxiv.org/abs/ 2507.06261. [↗](https://arxiv.org/abs/)
20. Deepak Gupta, Kush Attal, and Dina Demner-Fushman. A dataset for medical instructional video classification and question answering. arXiv, 2022. URL https://arxiv.org/abs/ 2201.12888. [↗](https://arxiv.org/abs/)
21. Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv, 2020. URL https://arxiv.org/ abs/2003.10286. [↗](https://arxiv.org/abs/2003.10286)
22. Robin Jia and Percy Liang. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In EMNLP, 2017. URL https://aclanthology.org/D17-1215/. [↗](https://aclanthology.org/D17-1215/)
23. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). arXiv, 2020. URL https://arxiv.org/abs/2009.13081. [↗](https://arxiv.org/abs/2009.13081)
24. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. [Pubmedqa: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). arXiv, 2019. URL https://arxiv.org/abs/1909.06146. [↗](https://arxiv.org/abs/1909.06146)
25. S. Jerrold Kaplan. [Indirect responses to loaded questions](https://md.research.elamur.ai/free/en/papers/kaplan-1978-indirect-responses-loaded-questions/). American Journal of Computational Linguistics, 1978. URL https://aclanthology.org/J78-3037/. [↗](https://aclanthology.org/J78-3037/)
26. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in NLP. In NAACL, 2021. URL https://aclanthology.org/2021.naacl-main.324/. [↗](https://aclanthology.org/2021.naacl-main.324/)
27. Najoung Kim, Phu Mon Htut, Samuel R. Bowman, and Jackson Petty. (QA) 2 : Question answering with questionable assumptions. In ACL, 2023. URL https://aclanthology.org/2023.acl-long.472/. [↗](https://aclanthology.org/2023.acl-long.472/)
28. Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae Won Park. Mdagents: An adaptive collaboration of llms for medical decision-making. arXiv, 2024. URL https://arxiv.org/abs/2404.15155. [↗](https://arxiv.org/abs/2404.15155)
29. Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. [Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning](https://md.research.elamur.ai/free/en/papers/li-2024-mediq-question-asking-llms-benchmark/). arXiv, 2024. URL https://arxiv.org/abs/2406. 00922. [↗](https://arxiv.org/abs/2406)
30. Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 2022. URL https://doi.org/10.1093/bib/bbac409. [↗](https://doi.org/10.1093/bib/bbac409)
31. Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. [Augmented large language models with parametric knowledge guiding](https://md.research.elamur.ai/free/en/papers/luo-2023-augmented-large-language-models-parametric/). arXiv, 2023. URL https://arxiv.org/abs/2305.04757. [↗](https://arxiv.org/abs/2305.04757)
32. Lars Malmqvist. [Sycophancy in large language models: Causes and mitigations](https://md.research.elamur.ai/free/en/papers/malmqvist-2024-sycophancy-large-language-models-causes/). arXiv, 2024. URL https://arxiv.org/abs/2411.15287. [↗](https://arxiv.org/abs/2411.15287)
33. Tamir Mendel, Nina Singh, Devin M Mann, Batia Wiesenfeld, and Oded Nov. Laypeople’s use of and attitudes toward large language models and search engines for health queries: Survey study. Journal of Medical Internet Research, 2025. URL https://www.jmir.org/2025/ 1/e64290. [↗](https://doi.org/10.2196/64290)
34. Meta. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). ArXiv, abs/2407.21783, 2024a. URL https://api.semanticscholar.org/CorpusID:271571434. [↗](https://arxiv.org/abs/2407.21783)
35. Meta. Introducing llama 3.1: Our most capable models to date, 2024b. URL [https://ai.meta.com/blog/meta-llama-3-1/](https://ai.meta.com/blog/meta-llama-3-1/). [↗](https://ai.meta.com/blog/meta-llama-3-1/)
36. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, 2023. URL https://arxiv.org/abs/2303.08774. [↗](https://arxiv.org/abs/2303.08774)
37. OpenAI. Hello gpt-4o. [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/), 2024. URL https://openai.com/index/hello-gpt-4o/. [↗](https://openai.com/index/hello-gpt-4o/)
38. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. [Medmcqa : A large-scale multi-subject multi-choice dataset for medical domain question answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/). arXiv, 2022. URL https://arxiv.org/abs/2203.14371. [↗](https://arxiv.org/abs/2203.14371)
39. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. arXiv, 2024. URL https://arxiv.org/abs/2404.13076. [↗](https://arxiv.org/abs/2404.13076)
40. Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. [Discovering language model behaviors with model-written evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/). In Findings of ACL 2023, 2023. URL https://aclanthology.org/2023.findings-acl.847/. [↗](https://aclanthology.org/2023.findings-acl.847/)
41. Qwen. Qwen2.5 technical report. ArXiv, abs/2412.15115, 2024. semanticscholar.org/CorpusID:274859421. [↗](https://arxiv.org/abs/2412.15115)
42. URL https://api. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In ACL, 2018. [URL https://aclanthology.org/P18-2124/](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). [↗](https://aclanthology.org/P18-2124/)
43. Aswin Rrv, Nemika Tyagi, Md Nayem Uddin, Neeraj Varshney, and Chitta Baral. Chaos with keywords: Exposing large language models sycophancy to misleading keywords and evaluating defense strategies. In Findings of ACL, 2024. URL https://aclanthology.org/2024.findings-acl.755/. [↗](https://aclanthology.org/2024.findings-acl.755/)
44. Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. [Large language models encode clinical knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/). Nature, 2023. URL https://arxiv.org/abs/2212.13138. [↗](https://arxiv.org/abs/2212.13138)
45. Neha Srikanth, Rupak Sarkar, Heran Mane, Elizabeth Aparicio, Quynh Nguyen, Rachel Rudinger, and Jordan Boyd-Graber. [Pregnant questions: The importance of pragmatic awareness in maternal health question answering](https://md.research.elamur.ai/free/en/papers/srikanth-2024-pregnant-questions-importance-pragmatic-awareness/). In NAACL, 2024. URL https://aclanthology.org/2024.naacl-long.403/. [↗](https://aclanthology.org/2024.naacl-long.403/)
46. Yoo Yeon Sung, Maharshi Gor, Eve Fleisig, Ishani Mondal, and Jordan Lee Boyd-Graber. Is your benchmark truly adversarial? advscore: Evaluating human-grounded adversarialness. In North American Chapter of the Association for Computational Linguistics, 2024a. URL https://api.semanticscholar.org/CorpusID:270702965. [↗](https://doi.org/10.48550/arxiv.2406.16342)
47. Yoo Yeon Sung, Ishani Mondal, and Jordan Lee Boyd-Graber. How the advent of ubiquitous large language models both stymie and turbocharge dynamic adversarial question generation. ArXiv, abs/2401.11185, 2024b. URL https://api.semanticscholar.org/CorpusID: 267068488. [↗](https://arxiv.org/abs/2401.11185)
48. Neeraj Varshney, Pavel Dolin, Agastya Seth, and Chitta Baral. The art of defending: A systematic evaluation and analysis of LLM defense strategies on safety and over-defensiveness. In Findings of ACL, 2024. URL https://aclanthology.org/2024. findings-acl.776/. [↗](https://aclanthology.org/2024)
49. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. [FreshLLMs: Refreshing large language models with search engine augmentation](https://md.research.elamur.ai/free/en/papers/vu-2023-freshllms-refreshing-large-language-models/). In Findings of ACL, 2024. URL https://aclanthology.org/2024.findings-acl.813/. [↗](https://aclanthology.org/2024.findings-acl.813/)
50. Nahathai Wongpakaran, Tinakon Wongpakaran, and Kilem Gwet. A comparison of cohen’s kappa and gwet’s ac1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology, 13, 04 2013. doi: 10.1186/1471-2288-13-61. [↗](https://doi.org/10.1186/1471-2288-13-61)
51. Xinyan Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. [CREPE: Open-domain question answering with false presuppositions](https://md.research.elamur.ai/free/en/papers/yu-2023-crepe-open-domain-question-answering/). In ACL, 2023. URL https://aclanthology.org/2023.acl-long.583/. [↗](https://aclanthology.org/2023.acl-long.583/)
52. Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv, 2024. URL https://arxiv.org/abs/2305.10415. [↗](https://arxiv.org/abs/2305.10415)
53. Jiachen Zhao. In-context exemplars as clues to retrieving from large associative memory. arXiv, 2023. URL https://arxiv.org/abs/2311.03498. [↗](https://arxiv.org/abs/2311.03498)
54. Wang Zhu, Jesse Thomason, and Robin Jia. Generalization differences between end-to-end and neuro-symbolic vision-language reasoning systems. In Findings of EMNLP, 2022. URL https://aclanthology.org/2022.findings-emnlp.345/. [↗](https://aclanthology.org/2022.findings-emnlp.345/)

*Updated 1 November 2026*