--- title: "TryggLLM: A Benchmark for Evaluating LLM — Touileb 2026" description: "Safety evaluation of generative language models has focused mostly on English, and Norwegian has had no dedicated resource for testing whether models…" authors: ["Samia Touileb", "Truls Pedersen", "Isabell Stinessen Haugen"] year: 2026 venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference" source: "https://aclanthology.org/2026.lrec-1.323/" doi: "10.63317/2rhfg2a92wim" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/touileb-2026-tryggllm-benchmark-evaluating-llm-safety/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/touileb-2026-tryggllm-benchmark-evaluating-llm-safety/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/5/) / TryggLLM: A Benchmark for Evaluating LLM Safety in Norwegian # TryggLLM: A Benchmark for Evaluating LLM Safety in Norwegian **Authors:** Samia Touileb, Truls Pedersen, Isabell Stinessen Haugen · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.323/) · [PDF](https://aclanthology.org/2026.lrec-1.323.pdf) · [DOI 10.63317/2rhfg2a92wim](https://doi.org/10.63317/2rhfg2a92wim) · [OpenAlex](https://openalex.org/W7160178528) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) ## Summary Safety evaluation of generative language models has focused mostly on English, and Norwegian has had no dedicated resource for testing whether models produce harmful output. Earlier Norwegian work studied mainly bias and stereotypes. The authors address this gap with TryggLLM, the first safety benchmark for Norwegian. It is built from two English sources, SimpleSafetyTests and SG-Bench. Four native-speaker research assistants translated the material by hand, two writing in Bokmål (BM) and two in Nynorsk (NN). They also localised it, for example by swapping American place names and emergency numbers for Norwegian ones. Machine translation was banned for prompts but allowed for long target responses, provided its use was flagged. Translations were deliberately not harmonised, so the benchmark captures natural variation both between and within the two written standards. The authors manually evaluated existing open Norwegian LLMs on TryggLLM. Depending on the model, 6.25% to 42.59% of generated answers were judged unsafe. This shows that locally developed models can carry serious safety risks and gives developers a tool for both testing and alignment. ## Key points - **Size and coverage:** TryggLLM contains 1,590 prompts and 1,188 target responses covering nine safety issues. - The five SimpleSafetyTests categories include self-harm, physical harm, fraud and child abuse. - The four SG-Bench categories include toxicity, bias, misinformation and dangerous information. - **SimpleSafetyTests portion:** each of the 100 prompts has four parallel versions (2 BM, 2 NN). This overlap lets the authors measure translation quality and diversity with text similarity metrics. - **SG-Bench portion:** only malicious-instruction prompts with predefined target responses were used. They were split into Set 1 and Set 2, and each item received one BM and one NN version. - The target responses can serve as references for evaluation or as data for instruction tuning toward Norwegian norms. - **Localisation example:** 911 was replaced with 113, and the Mental Helse helpline was added. - Annotators also chose natural everyday terms, such as *fremmedfrykt* rather than *xenofobi*. - Common English loanwords like *guide* and *dealer* were kept. - **Availability:** the benchmark is released on GitHub (SamiaTouileb/TryggLLM) and Hugging Face (SamiaT/TryggLLM). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{touileb2026tryggllm, title = {{TryggLLM: A Benchmark for Evaluating LLM Safety in Norwegian}}, author = {Touileb, Samia and Pedersen, Truls and Haugen, Isabell Stinessen}, booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference}, year = {2026}, doi = {10.63317/2rhfg2a92wim}, url = {https://doi.org/10.63317/2rhfg2a92wim} } ``` - [← previous Trustworthy NLP for Low-Resource Languages: Agent-Based Uncertainty Modeling for Hebrew Radiology Report Structuring](https://md.research.elamur.ai/free/papers/aclbulk-2026-atya-trustworthy-nlp-low-resource-languages/) - [next → TTSVowelViz: A Tool for Visualising Text-to-Speech Model Training via Vowel Spaces](https://md.research.elamur.ai/free/papers/udawatta-2026-ttsvowelviz-tool-visualising-text-speech/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/5/) ## References 22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Selma Bergstrand and Björn Gambäck. 2024. Detecting and mitigating LGBTQIA+ bias in large Norwegian language models. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 351– 364, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Selma+Bergstrand+and+Bj%C3%B6rn+Gamb%C3%A4ck%2E+2024%2E+Detecting+and+mitigating+LGBTQIA%2B+bias+in+large+Norwegian+language+models%2E+In+Proceedings+of+the+5th+Workshop+on+Gender+Bias+in+Natural+Language+Processing+%28G) 2. Zhibo Chu, Zichong Wang, and Wenbin Zhang. 2024. Fairness in large language models: A taxonomic survey. ACM SIGKDD explorations newsletter, 26(1):34–48. [↗](https://doi.org/10.1145/3682112.3682117) 3. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37 – 46. [↗](https://doi.org/10.1177/001316446002000104) 4. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097–1179. [↗](https://doi.org/10.1162/coli_a_00524) [our card](https://md.research.elamur.ai/free/papers/gallegos-2023-bias-fairness-large-language-models/) 5. Jon Atle Gulla, Peng Liu, and Lemei Zhang. 2026. Norwai’s large language models: Technical report. arXiv preprint arXiv:2601.03034. [↗](https://arxiv.org/abs/2601.03034) 6. Per E Kummervold, Javier De la Rosa, Freddy Wetjen, and Svein Arne Brygfjeld. 2021. Operationalizing a national digital library: The case for a norwegian transformer model. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 20–29. [↗](https://doi.org/10.48550/arxiv.2104.09617) 7. Jon Røyne Kyllingstad. 2017. The absence of race in norway. Journal of Anthropological Sciences, 95(1):1–9. [↗](https://doi.org/10.4436/jass.95012) 8. Peng Liu, Lemei Zhang, Terje Farup, Even Lauvrak, Jon Ingvaldsen, Simen Eide, Jon Atle Gulla, and Zhirong Yang. 2024. Nlebench+norglm: A comprehensive empirical analysis and benchmark dataset for generative language models in norwegian. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5543–5560. [↗](https://doi.org/10.48550/arxiv.2312.01314) 9. Yutao Mou, Shikun Zhang, and Wei Ye. 2024. Sg-bench: Evaluating llm safety generalization across diverse tasks and prompt types. In Advances in Neural Information Processing Systems, volume 37, pages 123032–123054. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Yutao+Mou%2C+Shikun+Zhang%2C+and+Wei+Ye%2E+2024%2E+Sg%2Dbench%3A+Evaluating+llm+safety+generalization+across+diverse+tasks+and+prompt+types%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+37%2C+pages+) 10. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) 11. David Samuel, Vladislav Mikhailov, Erik Velldal, Lilja Øvrelid, Lucas Georges Gabriel Charpentier, and Andrey Kutuzov. 2025. Small Languages, Big Models: A Study of Continual Training on Languages of Norway. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), Tallinn, Estonia. [↗](https://doi.org/10.48550/arxiv.2412.06484) 12. Martin Sjåvik and Samia Touileb. 2025. Ableism, ageism, gender, and nationality bias in Norwegian and multilingual language models. In Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 379–392, Vienna, Austria. Association for Computational Linguistics. [↗](https://doi.org/10.48448/6556-g231) 13. Samia Touileb. 2022. Exploring the effects of negation and grammatical tense on bias probes. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 423–429, Online only. Association for Computational Linguistics. [↗](https://doi.org/10.48448/ejc9-kq65) 14. Samia Touileb and Debora Nozza. 2022. Measuring harmful representations in Scandinavian language models. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS), pages 118–125, Abu Dhabi, UAE. Association for Computational Linguistics. [↗](https://doi.org/10.48448/8jx3-1s52) 15. Samia Touileb, Lilja Øvrelid, and Erik Velldal. 2022. Occupational biases in Norwegian and multilingual language models. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 200–211, Seattle, Washington. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samia+Touileb%2C+Lilja+%C3%98vrelid%2C+and+Erik+Velldal%2E+2022%2E+Occupational+biases+in+Norwegian+and+multilingual+language+models%2E+In+Proceedings+of+the+4th+Workshop+on+Gender+Bias+in+Natural+Language+Processin) 16. Samia Touileb, Lilja Øvrelid, and Erik Velldal. 2023. Measuring normative and descriptive biases in language models using census data. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2242–2248, Dubrovnik, Croatia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samia+Touileb%2C+Lilja+%C3%98vrelid%2C+and+Erik+Velldal%2E+2023%2E+Measuring+normative+and+descriptive+biases+in+language+models+using+census+data%2E+In+Proceedings+of+the+17th+Conference+of+the+European+Chapter+of+) 17. Bertie Vidgen, Nino Scherrer, Hannah Rose Kirk, Rebecca Qian, Anand Kannappan, Scott A Hale, and Paul Röttger. 2023. Simplesafetytests: a test suite for identifying critical safety risks in large language models. arXiv preprint arXiv:2311.08370. [↗](https://arxiv.org/abs/2311.08370) 18. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023. Do-notanswer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387. [↗](https://arxiv.org/abs/2308.13387) 19. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-demos.6) [our card](https://md.research.elamur.ai/free/papers/wolf-2020-transformers-state-art-natural-language/) 20. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [↗](https://arxiv.org/abs/1904.09675) 21. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309.07045, 34. [↗](https://arxiv.org/abs/2309.07045) [our card](https://md.research.elamur.ai/free/papers/zhang-2023-safetybench-evaluating-safety-large/) 22. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. [↗](https://arxiv.org/abs/2307.15043) [our card](https://md.research.elamur.ai/free/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/)