---
title: "ScandEval — paper by Nielsen (2023) — summary and ideas"
description: "As the number of Danish, Swedish, Norwegian and other Nordic language models grows… 63 references, cited by 68 papers here."
authors: ["Dan Nielsen"]
year: 2023
venue: "Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)"
source: "https://aclanthology.org/2023.nodalida-1.20/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nielsen-2023-scandeval-benchmark-scandinavian-natural-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nielsen-2023-scandeval-benchmark-scandinavian-natural-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / ScandEval: A Benchmark for Scandinavian Natural Language Processing

# ScandEval: A Benchmark for Scandinavian Natural Language Processing

**Authors:** Dan Nielsen · **Year:** 2023 · **Venue:** [Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)](https://md.research.elamur.ai/free/en/venues/nodalida/) ([NoDaLiDa 2023](https://md.research.elamur.ai/free/en/venues/nodalida/2023/)) · **Publisher:** [University of Tartu Library](https://md.research.elamur.ai/free/en/publishers/university-of-tartu-library/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.nodalida-1.20/) · [PDF](https://aclanthology.org/2023.nodalida-1.20.pdf) · [OpenAlex](https://openalex.org/W4362631726)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/nielsen-2023-scandeval-benchmark-scandinavian-natural-language/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
As the number of Danish, Swedish, Norwegian and other Nordic language models grows, practitioners find it hard to choose among them and researchers struggle to show real progress. No shared evaluation tool existed. The author built ScandEval to fill this gap. It is a benchmark with four tasks: NER, sentiment classification, linguistic acceptability and question answering. The last two use newly created datasets, ScaLA and ScandiQA. All task data were converted to one format and a fixed size, and the uniform versions are published on the Hugging Face Hub. ScandEval also ships as a Python package and CLI, `scandeval`, which can reproducibly finetune and score any Hub model. More than 100 models were evaluated, and the results appear in an online leaderboard. The best performers are national models such as NB-BERT-large, DFM-encoder-large-v1 and KB-BERT-large, which beat massively multilingual models. A clustering-based experiment shows strong transfer among the Mainland Scandinavian languages but little transfer between them and Icelandic or Faroese. This supports regional rather than global multilingual modelling.

## Key points
- The benchmark covers four tasks. ScaLA (linguistic acceptability, all five Scandinavian languages) and ScandiQA (question answering, Mainland languages) are new. NER and sentiment use existing datasets. Inference speed is measured as well.
- Finetuning is fixed for every model:
  - learning rate 2·10⁻⁵ with 100 warmup steps;
  - effective batch size 32;
  - AdamW optimiser;
  - linear schedule over up to 10,000 steps;
  - early stopping after 90 steps without validation-loss improvement;
  - fixed seeds.
- Training sets were cut to 1,024 samples, chosen as a trade-off between separating models and keeping runtime manageable. Evaluating all models consumed around 1,000 hours of GeForce RTX 2080 Ti GPU time, roughly 40 kg CO₂e.
- NB-BERT-large ranks first overall and in Norwegian, and it is in the top two for Danish and Swedish. DFM-encoder-large-v1 leads Danish and KB-BERT-large leads Swedish. RemBERT and mDeBERTaV3 are the strongest multilingual entries.
- Hypothesis 1 says transfer within Mainland Scandinavian (da, no, sv) is substantial. Hypothesis 2 says transfer to Insular Scandinavian (is, fo) is minimal. Both are supported. The leaderboard therefore shows only Mainland languages, while the package still supports Icelandic and Faroese.

## Interesting ideas
- **Two sources of noise in one confidence interval.** Each score comes from 10 runs. Every run uses a new seed for finetuning and a bootstrap resample of the test set, so the interval covers both training randomness and test-set sampling. This would make leaderboard differences for Ukrainian models easier to interpret.
- **Automatic batch-size fallback.** When GPU memory runs out, the tool halves the batch and doubles gradient accumulation, so the effective batch size stays the same. Large and small models can then be compared under the same hyperparameters on modest hardware.
- **Testing language grouping with clustering statistics.** One representative monolingual model per language, plus control languages (English, German, Dutch, Finnish, Russian, Arabic), is scored on candidate benchmark groups. Each split is ranked by its F-statistic, the ratio of between-group to within-group variance. The same procedure could check whether, for example, Ukrainian, Russian or Belarusian data should share a benchmark or a training pool.
- **Equal-size, uniform datasets for cross-language comparison.** Every task is cut to the same training size and format, so language-level scores are directly comparable. This is useful whenever source corpora differ greatly in size.
- **Calibration baselines.** A randomly initialised XLM-RoBERTa-base and an ELECTRA-small are included as reference points. They show how much of a pretrained model's score comes from knowledge gained in pretraining rather than from finetuning alone.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [Alexandra Institute](https://ror.org/054z83b16).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Dan Nielsen. 2023. ScandEval: A Benchmark for Scandinavian Natural Language Processing. Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nielsen2023scandeval,
  title = {{ScandEval: A Benchmark for Scandinavian Natural Language Processing}},
  author = {Dan Nielsen},
  booktitle = {Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)},
  year = {2023},
  url = {https://aclanthology.org/2023.nodalida-1.20/}
}
```

- [← previous Multi-VALUE: A Framework for Cross-Dialectal English NLP](https://md.research.elamur.ai/free/en/papers/ziems-2023-multi-value-framework-cross-dialectal/)
- [next → Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration](https://md.research.elamur.ai/free/en/papers/deutsch-2023-ties-matter-meta-evaluating-modern/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

63 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2021. Oxford English Dictionary. Oxford University Press. [https://www.lexico.com/](https://www.lexico.com/) definition/scandinavia. [↗](https://www.lexico.com/)
2. Amine Abdaoui, Camille Pradel, and Grégoire Sigel. 2020. Load What You Need: Smaller Versions of Mutililingual BERT. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 119–123, Online. Association for Computational Linguistics. Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). [find](https://scholar.google.com/scholar?q=Amine+Abdaoui%2C+Camille+Pradel%2C+and+Gr%C3%A9goire+Sigel%2E+2020%2E+Load+What+You+Need%3A+Smaller+Versions+of+Mutililingual+BERT%2E+In+Proceedings+of+SustaiNLP%3A+Workshop+on+Simple+and+Efficient+Natural+Language+Proc)
3. Kenneth Enevoldsen, Lasse Hansen, and Kristoffer L Nielbo. 2021. DaCy: A Unified Framework for Danish NLP. Proceedings http://ceur-ws.org ISSN, 1613:0073. [↗](https://doi.org/10.48550/arxiv.2107.05295)
4. Jan Terje Faarlund. 2019. The syntax of mainland Scandinavian. Oxford University Press. [↗](https://doi.org/10.1093/oso/9780198817918.001.0001)
5. Yvonne Adesam, Aleksandrs Berdicevskis, and Felix Morger. 2020. SwedishGLUE – Towards a Swedish Test Set For Evaluating Natural Language Understanding Models. Institutionen för svenska språket. [find](https://scholar.google.com/scholar?q=Yvonne+Adesam%2C+Aleksandrs+Berdicevskis%2C+and+Felix+Morger%2E+2020%2E+SwedishGLUE+%E2%80%93+Towards+a+Swedish+Test+Set+For+Evaluating+Natural+Language+Understanding+Models%2E+Institutionen+f%C3%B6r+svenska+spr%C3%A5ket%2E)
6. Sofia Gustafson-Capková and Britt Hartmann. 2006. Manual of the Stockholm Umeå corpus version 2.0. Stockholm University. [https://spraakbanken.gu.se/parole/](https://spraakbanken.gu.se/parole/) Docs/SUC2.0-manual.pdf. [↗](https://spraakbanken.gu.se/parole/)
7. Þórunn Arnardóttir, Hinrik Hafsteinsson, Einar Freyr Sigurðsson, Kristín Bjarnadóttir, Anton Karl Ingason, Hildur Jónsdóttir, and Steinþór Steingrímsson. 2020. [A Universal Dependencies conversion pipeline for a Penn-format constituency treebank](https://md.research.elamur.ai/free/en/papers/arnardottir-2020-universal-dependencies-conversion-pipeline-penn/). In Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020), pages 16–25.
8. Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020a. KorNLI and KorSTS: New benchmark datasets for Korean natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 422–430, Online. Association for Computational Linguistics. Lukasz Augustyniak, Kamil Tagowski, Albert Sawczyn, Denis Janiak, Roman Bartusiak, Adrian Dominik Szymczak, Arkadiusz Janz, Piotr Szymański, Marcin W atroba, Mikołaj Morzy, et al. This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [find](https://scholar.google.com/scholar?q=Jiyeon+Ham%2C+Yo+Joong+Choe%2C+Kyubyong+Park%2C+Ilji+Choi%2C+and+Hyungjoon+Soh%2E+2020a%2E+KorNLI+and+KorSTS%3A+New+benchmark+datasets+for+Korean+natural+language+understanding%2E+In+Findings+of+the+Association+for+C)
9. Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020b. Kornli and korsts: New benchmark datasets for korean natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 422– 430. [find](https://scholar.google.com/scholar?q=Jiyeon+Ham%2C+Yo+Joong+Choe%2C+Kyubyong+Park%2C+Ilji+Choi%2C+and+Hyungjoon+Soh%2E+2020b%2E+Kornli+and+korsts%3A+New+benchmark+datasets+for+korean+natural+language+understanding%2E+In+Findings+of+the+Association+for+C)
10. Davide Chicco and Giuseppe Jurman. 2020. The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC genomics, 21:1–13. [↗](https://doi.org/10.1186/s12864-019-6413-7)
11. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2020. [Rethinking Embedding Coupling in Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/). In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2010.12821)
12. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2019. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V+Le%2C+and+Christopher+D+Manning%2E+2019%2E+ELECTRA%3A+Pre%2Dtraining+Text+Encoders+as+Discriminators+Rather+Than+Generators%2E+In+International+Conference+on+Learning+Represe)
13. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+%C3%89douard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+Cross%2Dlingu)
14. Stefan Daniel Dumitrescu, Petru Rebeja, Beata Lorincz, Mihaela Gaman, Andrei Avram, Mihai Ilie, Andrei Pruteanu, Adriana Stan, Lorena Rosia, Cristina Ia-cobescu, et al. 2021. LiRo: Benchmark and leaderboard for romanian language tasks. In Thirty-fifth Einar Haugen. 1976. The Scandinavian languages: An introduction to their history. Cambridge, Mass., Harvard University Press. [find](https://scholar.google.com/scholar?q=Stefan+Daniel+Dumitrescu%2C+Petru+Rebeja%2C+Beata+Lorincz%2C+Mihaela+Gaman%2C+Andrei+Avram%2C+Mihai+Ilie%2C+Andrei+Pruteanu%2C+Adriana+Stan%2C+Lorena+Rosia%2C+Cristina+Ia%2Dcobescu%2C+et+al%2E+2021%2E+LiRo%3A+Benchmark+and+leade)
15. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. [DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/). arXiv preprint arXiv:2111.09543. [↗](https://arxiv.org/abs/2111.09543)
16. Matthew Honnibal, Ines Montani, Sofie Van Landeglem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal%2C+Ines+Montani%2C+Sofie+Van+Landeglem%2C+and+Adriane+Boyd%2E+2020%2E+spaCy%3A+Industrial%2Dstrength+Natural+Language+Processing+in+Python%2E)
17. Rasmus Hvingelby, Amalie Brogaard Pauli, Maria Barrett, Christina Rosted, Lasse Malm Lidegaard, and Anders Søgaard. 2020. DaNE: A named entity resource for Danish. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4597–4604, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Rasmus+Hvingelby%2C+Amalie+Brogaard+Pauli%2C+Maria+Barrett%2C+Christina+Rosted%2C+Lasse+Malm+Lidegaard%2C+and+Anders+S%C3%B8gaard%2E+2020%2E+DaNE%3A+A+named+entity+resource+for+Danish%2E+In+Proceedings+of+the+12th+Language+)
18. Malte Højmark-Bertelsen. 2021. Ælæctra - A Step Towards More Efficient Danish Natural Language Processing. [https://github.com/MalteHB/](https://github.com/MalteHB/) -l-ctra/. [↗](https://github.com/MalteHB/)
19. Svanhvít L Ingólfsdóttir, Ásmundur A Gudjónsson, and Hrafn Loftsson. 2020. Named Entity Recognition for Icelandic: Annotated Corpus and Models. In International Conference on Statistical Language and Speech Processing, pages 46–57. Springer. [find](https://scholar.google.com/scholar?q=Svanhv%C3%ADt+L+Ing%C3%B3lfsd%C3%B3ttir%2C+%C3%81smundur+A+Gudj%C3%B3nsson%2C+and+Hrafn+Loftsson%2E+2020%2E+Named+Entity+Recognition+for+Icelandic%3A+Annotated+Corpus+and+Models%2E+In+International+Conference+on+Statistical+Language+and+)
20. Tim Isbister and Magnus Sahlgren. 2020. Why Not Simply Translate? A First Swedish Evaluation Benchmark for Semantic Similarity. arXiv preprint arXiv:2009.03116. [↗](https://arxiv.org/abs/2009.03116)
21. Hildur Jónsdóttir and Anton Karl Ingason. 2020. Creating a parallel Icelandic dependency treebank from raw text to Universal Dependencies. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2924–2931. [find](https://scholar.google.com/scholar?q=Hildur+J%C3%B3nsd%C3%B3ttir+and+Anton+Karl+Ingason%2E+2020%2E+Creating+a+parallel+Icelandic+dependency+treebank+from+raw+text+to+Universal+Dependencies%2E+In+Proceedings+of+the+12th+Language+Resources+and+Evaluation+)
22. Fredrik Jørgensen, Tobias Aasmoe, Anne-Stine Ruud Husevåg, Lilja Øvrelid, and Erik Velldal. 2020. NorNE: Annotating named entities for Norwegian. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4547–4556, Marseille, France. European Language Resources Association. [↗](https://doi.org/10.48550/arxiv.1911.12146)
23. Simran Khanuja, Melvin Johnson, and Partha Talukdar. 2021. MergeDistill: Merging language models using pre-trained distillation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2874–2887, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Simran+Khanuja%2C+Melvin+Johnson%2C+and+Partha+Talukdar%2E+2021%2E+MergeDistill%3A+Merging+language+models+using+pre%2Dtrained+distillation%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DIJCNL)
24. Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabadi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh Rasooli, Sepideh Sadeghi, Erfan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saber Sheybani, Ali Tazarv, and Yadollah Yaghoobzadeh. 2020. ParsiNLU: A Suite of Language Understanding Challenges for Persian. Transactions of the Association for Computational Linguistics, 9:1147–1162. [↗](https://doi.org/10.1162/tacl_a_00419)
25. Fajri Koto, Afshin Rahimi, Jey Han Lau, and Timothy Baldwin. 2020. IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP. In Proceedings of the 28th International Conference on Computational Linguistics, pages 757–770, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Fajri+Koto%2C+Afshin+Rahimi%2C+Jey+Han+Lau%2C+and+Timothy+Baldwin%2E+2020%2E+IndoLEM+and+IndoBERT%3A+A+benchmark+dataset+and+pre%2Dtrained+language+model+for+Indonesian+NLP%2E+In+Proceedings+of+the+28th+International)
26. Matthias Trautner Kromann and Stine Kern Lynge. 2004. The Danish Dependency Treebank v. 1.0. [find](https://scholar.google.com/scholar?q=Matthias+Trautner+Kromann+and+Stine+Kern+Lynge%2E+2004%2E+The+Danish+Dependency+Treebank+v%2E+1%2E0%2E)
27. Per E Kummervold, Javier De la Rosa, Freddy Wetjen, and Svein Arne Brygfjeld. 2021. Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 20–29, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden. [↗](https://doi.org/10.48550/arxiv.2104.09617)
28. Andrey Kutuzov, Jeremy Barnes, Erik Velldal, Lilja Øvrelid, and Stephan Oepen. 2021. Large-scale contextualised language modelling for norwegian. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 30–40. [↗](https://doi.org/10.48550/arxiv.2104.06546)
29. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 7:453– 466. [↗](https://doi.org/10.1162/tacl_a_00276)
30. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. [From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers](https://md.research.elamur.ai/free/en/papers/lauscher-2020-zero-hero-limitations-zero-shot/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483–4499, Online. Association for Computational Linguistics.
31. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark datasetfor cross-lingual pre-training, understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008–6018, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yaobo+Liang%2C+Nan+Duan%2C+Yeyun+Gong%2C+Ning+Wu%2C+Fenfei+Guo%2C+Weizhen+Qi%2C+Ming+Gong%2C+Linjun+Shou%2C+Daxin+Jiang%2C+Guihong+Cao%2C+Xiaodong+Fan%2C+Ruofei+Zhang%2C+Rahul+Agrawal%2C+Edward+Cui%2C+Sining+Wei%2C+Taroon+Bharti%2C+)
32. Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. [MKQA: A linguistically diverse benchmark for multilingual open domain question answering](https://md.research.elamur.ai/free/en/papers/chen-2020-open-domain-question-answering/). Transactions of the Association for Computational Linguistics, 9:1389–1406. [↗](https://doi.org/10.1162/tacl_a_00433)
33. Ilya Loshchilov and Frank Hutter. 2018. Decoupled Weight Decay Regularization. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1711.05101)
34. Martin Malmsten, Love Börjeson, and Chris Haffenden. 2020. [Playing with words at the national library of sweden–making a swedish bert](https://md.research.elamur.ai/free/en/papers/malmsten-2020-playing-words-national-library-sweden/). arXiv preprint arXiv:2007.01658. [↗](https://arxiv.org/abs/2007.01658)
35. Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451. [↗](https://doi.org/10.1016/0005-2795(75)90109-9)
36. Jens Dahl Møllerhøj. 2020. Nordic BERT. [https://github.com/certainlyio/nordic_bert](https://github.com/certainlyio/nordic_bert). [↗](https://github.com/certainlyio/nordic_bert)
37. Joakim Nivre, Jens Nilsson, and Johan Hall. 2006. [Talbanken05: A Swedish treebank with phrase structure and dependency annotation](https://md.research.elamur.ai/free/en/papers/nivre-2006-talbanken05-swedish-treebank-phrase-structure/). In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA).
38. Lilja Øvrelid and Petter Hohle. 2016. Universal Dependencies for Norwegian. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1579–1585, Portorož, Slovenia. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Lilja+%C3%98vrelid+and+Petter+Hohle%2E+2016%2E+Universal+Dependencies+for+Norwegian%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9916%29%2C+pages+1579%E2%80%931585%2C+Portor)
39. Amalie Brogaard Pauli, Maria Barrett, Ophélie Lacroix, and Rasmus Hvingelby. 2021. DaNLP: An open-source toolkit for Danish natural language processing. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 460–466, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden. [find](https://scholar.google.com/scholar?q=Amalie+Brogaard+Pauli%2C+Maria+Barrett%2C+Oph%C3%A9lie+Lacroix%2C+and+Rasmus+Hvingelby%2E+2021%2E+DaNLP%3A+An+open%2Dsource+toolkit+for+Danish+natural+language+processing%2E+In+Proceedings+of+the+23rd+Nordic+Conference+on)
40. Charles Pierse. 2020. Pyinfer. [https://github.com/cdpierse/pyinfer](https://github.com/cdpierse/pyinfer). [↗](https://github.com/cdpierse/pyinfer)
41. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. [How multilingual is multilingual BERT?](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1493)
42. David C Plaut, Steven J Nowlan, and Geoffrey E Hinton. 1986. Experiments on learning by back propagation. Technical Report CMU-CS-86-126. [find](https://scholar.google.com/scholar?q=David+C+Plaut%2C+Steven+J+Nowlan%2C+and+Geoffrey+E+Hinton%2E+1986%2E+Experiments+on+learning+by+back+propagation%2E+Technical+Report+CMU%2DCS%2D86%2D126%2E)
43. Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. [Massively multilingual transfer for NER](https://md.research.elamur.ai/free/en/papers/rahimi-2019-massively-multilingual-transfer-ner/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164, Florence, Italy. Association for Computational Linguistics.
44. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992. [↗](https://doi.org/10.18653/v1/d19-1410)
45. Hannah Ritchie, Max Roser, and Pablo Rosado. 2022. Energy. Our World in Data. Https://ourworldindata.org/energy. [find](https://scholar.google.com/scholar?q=Hannah+Ritchie%2C+Max+Roser%2C+and+Pablo+Rosado%2E+2022%2E+Energy%2E+Our+World+in+Data%2E+Https%3A%2F%2Fourworldindata%2Eorg%2Fenergy%2E)
46. Eiríkur Rögnvaldsson, Anton Karl Ingason, Einar Freyr Sigurðsson, and Joel Wallenberg. 2012. [The Icelandic parsed historical corpus (IcePaHC)](https://md.research.elamur.ai/free/en/papers/rognvaldsson-2012-icelandic-parsed-historical-corpus-icepahc/). In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 1977–1984.
47. Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and Ireneusz Gawlik. 2020. KLEJ: Comprehensive benchmark for Polish language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1191– 1201, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Piotr+Rybak%2C+Robert+Mroczkowski%2C+Janusz+Tracz%2C+and+Ireneusz+Gawlik%2E+2020%2E+KLEJ%3A+Comprehensive+benchmark+for+Polish+language+understanding%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+)
48. Ali Safaya, Emirhan Kurtuluş, Arda Goktogan, and Deniz Yuret. 2022. Mukayese: Turkish nlp strikes back. In Findings of the Association for Computational Linguistics: ACL 2022, pages 846–863. [find](https://scholar.google.com/scholar?q=Ali+Safaya%2C+Emirhan+Kurtulus%CC%A7%2C+Arda+Goktogan%2C+and+Deniz+Yuret%2E+2022%2E+Mukayese%3A+Turkish+nlp+strikes+back%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2022%2C+pages+846%E2%80%93863%2E)
49. Magnus Sahlgren, Fredrik Carlsson, Fredrik Olsson, and Love Börjeson. 2021. It’s basically the same language anyway: the case for a nordic language model. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 367– 372, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden. [find](https://scholar.google.com/scholar?q=Magnus+Sahlgren%2C+Fredrik+Carlsson%2C+Fredrik+Olsson%2C+and+Love+B%C3%B6rjeson%2E+2021%2E+It%E2%80%99s+basically+the+same+language+anyway%3A+the+case+for+a+nordic+language+model%2E+In+Proceedings+of+the+23rd+Nordic+Conference+)
50. Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2020. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938–4947. [find](https://scholar.google.com/scholar?q=Paul%2DEdouard+Sarlin%2C+Daniel+DeTone%2C+Tomasz+Malisiewicz%2C+and+Andrew+Rabinovich%2E+2020%2E+Superglue%3A+Learning+feature+matching+with+graph+neural+networks%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+compu)
51. Philip Tamimi Sarnikowski. 2021. Danish Transformers. GitHub. [https://github.com/](https://github.com/) sarnikowski. [↗](https://github.com/)
52. Tatiana Shavrina, Alena Fenogenova, Emelyanov Anton, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020. RussianSuperGLUE: A Russian language understanding evaluation benchmark. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4717–4726, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-main.381)
53. Vésteinn Snæbjarnarson, Annika Simonsen, Goran Glavaš, and Ivan Vulić. 2023. Transfer to a low-resource language via close relatives: The case study on faroese. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), Tórshavn, Faroe Islands. Linköping University Electronic Press, Sweden. [↗](https://doi.org/10.48550/arxiv.2304.08823)
54. Kristoffer Svensson. 2017. Sentiment Analysis With Convolutional Neural Networks: Classifying sentiment in Swedish reviews. Bachelor’s thesis. [find](https://scholar.google.com/scholar?q=Kristoffer+Svensson%2E+2017%2E+Sentiment+Analysis+With+Convolutional+Neural+Networks%3A+Classifying+sentiment+in+Swedish+reviews%2E+Bachelor%E2%80%99s+thesis%2E)
55. Techpowerup.com. NVIDIA GeForce RTX 2080 Ti Specs. [https://www.techpowerup.com/gpu-specs/geforce-rtx-2080-ti](https://www.techpowerup.com/gpu-specs/geforce-rtx-2080-ti). c3305. Accessed: 2023-04-01. [↗](https://www.techpowerup.com/gpu-specs/geforce-rtx-2080-ti)
56. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. [Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/). In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147. [↗](https://doi.org/10.48550/arxiv.cs/0306050)
57. Erik Velldal, Lilja Øvrelid, Eivind Alexander Bergem, Cathrine Stadsnes, Samia Touileb, and Fredrik Jørgensen. 2018. NoReC: The Norwegian review corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Erik+Velldal%2C+Lilja+%C3%98vrelid%2C+Eivind+Alexander+Bergem%2C+Cathrine+Stadsnes%2C+Samia+Touileb%2C+and+Fredrik+J%C3%B8rgensen%2E+2018%2E+NoReC%3A+The+Norwegian+review+corpus%2E+In+Proceedings+of+the+Eleventh+International+Co)
58. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w18-5446)
59. Alex Warstadt, Amanpreet Singh, and Samuel Bowman. 2019. Neural Network Acceptability Judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [↗](https://doi.org/10.1162/tacl_a_00290)
60. Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. [IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding](https://md.research.elamur.ai/free/en/papers/wilie-2020-indonlu-benchmark-resources-evaluating-indonesian/). In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 843–857, Suzhou, China. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2009.05387)
61. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-demos.6)
62. Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. [CLiMP: A benchmark for Chinese language model evaluation](https://md.research.elamur.ai/free/en/papers/xiang-2021-climp-benchmark-chinese-language-model/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2784–2790, Online. Association for Computational Linguistics.
63. Liang Xu, Xuanwei Zhang, Lu Li, Hai Hu, Chenjie Cao, Weitang Liu, Junyi Li, Yudong Li, Kai Sun, Yechen Xu, Yiming Cui, Cong Yu, Qianqian Dong, Yin Tian, Dian Yu, Bo Shi, Jun jie Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhen-Yi Yang, Kyle Richardson, and Zhenzhong Lan. 2020. [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/). In International Conference on Computational Linguistics.

*Updated 24 October 2026*