Bibliographic Details
| Title: |
NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark |
| Authors: |
Mikhailov, Vladislav, Enstad, Tita, Samuel, David, Farsethås, Hans Christian, Kutuzov, Andrey, Velldal, Erik, Øvrelid, Lilja |
| Publication Year: |
2025 |
| Collection: |
Computer Science |
| Subject Terms: |
Computer Science - Computation and Language, Computer Science - Artificial Intelligence |
| More Details: |
This paper introduces NorEval, a new and comprehensive evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). NorEval consists of 24 high-quality human-created datasets -- of which five are created from scratch. In contrast to existing benchmarks for Norwegian, NorEval covers a broad spectrum of task categories targeting Norwegian language understanding and generation, establishes human baselines, and focuses on both of the official written standards of the Norwegian language: Bokm{\aa}l and Nynorsk. All our datasets and a collection of over 100 human-written prompts are integrated into LM Evaluation Harness, ensuring flexible and reproducible evaluation. We describe the NorEval design and present the results of benchmarking 19 open-source pre-trained and instruction-tuned LMs for Norwegian in various scenarios. Our benchmark, evaluation framework, and annotation materials are publicly available. |
| Document Type: |
Working Paper |
| Access URL: |
http://arxiv.org/abs/2504.07749 |
| Accession Number: |
edsarx.2504.07749 |
| Database: |
arXiv |
