--- title: "LLMs are Vulnerable to Malicious Prompts Disguised as Scientific" description: "The paper asks whether safety-aligned LLMs can be pushed into producing stereotyped or toxic output… 2 references, cited by 5 papers here." authors: ["Yubin Ge", "Neeraja Kirtane", "Hao Peng", "Dilek Hakkani-Tur"] year: 2025 venue: "arXiv:2501.14073" source: "https://arxiv.org/abs/2501.14073" doi: "10.48550/arxiv.2501.14073" arxiv: "2501.14073" licence: "CC0 1.0 (arXiv)" licence_url: "http://creativecommons.org/publicdomain/zero/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ge-2025-llms-vulnerable-malicious-prompts-disguised/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ge-2025-llms-vulnerable-malicious-prompts-disguised/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) / LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language # LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language **Authors:** Yubin Ge, Neeraja Kirtane, [Hao Peng](https://md.research.elamur.ai/free/en/authors/hao-peng/) and [Dilek Hakkani-Tur](https://md.research.elamur.ai/free/en/authors/dilek-hakkani-tur/) · **Year:** 2025 · **Venue:** arXiv:2501.14073 · **Licence:** [CC0 1.0 (arXiv)](http://creativecommons.org/publicdomain/zero/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2501.14073) · [PDF](https://arxiv.org/pdf/2501.14073) · [DOI 10.48550/arxiv.2501.14073](https://doi.org/10.48550/arxiv.2501.14073) · [arXiv 2501.14073](https://arxiv.org/abs/2501.14073) · [OpenAlex](https://openalex.org/W4406840539) **Citations:** 0 citations (OpenAlex, 2026-10-14) · 5 in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper asks whether safety-aligned LLMs can be pushed into producing stereotyped or toxic output when a harmful request is framed in academic terms. The idea draws on Cialdini's authority principle. The authors collected real social-science and psychology papers that discuss possible upsides of stereotyping and had GPT-4o condense them into a one-sided summary. That summary went into the system message, together with an instruction to continue a dialogue in a strongly biased way and to justify it. A second variant has an LLM invent fictitious paper titles and abstracts, which requires no expert knowledge from the attacker. Inputs came from StereoSet and GPT-4-generated opening utterances. Targets included GPT-4o, GPT-4o-mini, GPT-4, Gemini, Cohere and Llama3.1 models. GPT-4 graded bias and the Perspective API measured toxicity. Both real and fabricated "evidence" raised bias and toxicity compared with zero-shot, DAN and role-play baselines. Simple defenses did little to stop the attack. The authors argue this calls into question how scientific text is used to train LLMs. ## Key points - Two attack variants are compared: Sci-Paper Based Persuasion, built on real papers, and fabricated-paper persuasion, built on invented titles and abstracts. Both increase stereotypical bias and toxicity. - Bias is rated by GPT-4 as an LLM judge on a five-level rubric, running from no bias to extreme bias. Toxicity is measured with the Perspective API. - Ablations on GPT-4o show that stripping author names or venues from the summaries lowers bias scores. Rewriting the summaries in a simpler, K12-level style slightly raises them. GPT-4o-mini is largely unaffected by these changes. - In multi-turn dialogues (GPT-4o, GPT-4o-mini, GPT-3.5-turbo), average bias keeps rising with each turn. - Mutation defenses (Jain et al., 2023) mostly fail: - On GPT-4o, rephrasing cuts bias from 1.71 to 1.01, while retokenizing only reaches 1.63. - On GPT-4o-mini, bias goes from 2.59 to 2.74 with rephrasing and 2.65 with retokenizing. - On Llama3-405B, bias goes from 2.30 to 2.63 with rephrasing and 2.68 with retokenizing. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 184 works · 69,372 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works ## Cite Cite as: Yubin Ge, Neeraja Kirtane, Hao Peng and Dilek Hakkani-Tur. 2025. LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language. arXiv:2501.14073. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{ge2025llms, title = {{LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language}}, author = {Yubin Ge, Neeraja Kirtane, Hao Peng and Dilek Hakkani-Tur}, year = {2025}, doi = {10.48550/arxiv.2501.14073}, eprint = {2501.14073}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2501.14073} } ``` - [← previous LLM Dependency Parsing with In-Context Rules](https://md.research.elamur.ai/free/en/papers/ginn-2025-llm-dependency-parsing-context-rules/) - [next → LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing](https://md.research.elamur.ai/free/en/papers/wang-2025-llms-can-perform-multi-dimensional/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. concerns about the reliance on scientific data in the training of LLMs and the potential misuse of such models to reinforce societal stereotypes. [find](https://scholar.google.com/scholar?q=concerns+about+the+reliance+on+scientific+data+in+the+training+of+LLMs+and+the+potential+misuse+of+such+models+to+reinforce+societal+stereotypes%2E) 2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774. \#\# Ethics Statement *Updated 30 October 2026*