Open Danish ASR Leaderboard 🇩🇰

Test sets

Methodology

Text normalisation

Applied identically to hypothesis and reference before scoring:

1. Unicode NFKC — folds compatibility characters (ligatures, full-width digits, ²→2) so visually equivalent forms match
2. Danish number canonicalisation — separators inside a numeral are removed so formatting differences don't inflate WER. Thousands and decimals are both stripped: 1.234 → 1234, and 3.14 / 3,14 → 314
3. Lowercase
4. Strip punctuation — apostrophes inside a word (e.g. det's) are preserved; all other punctuation and symbols are removed
5. Numerals → Danish words — standalone integers are spelled out with num2words (4→fire, 24→fireogtyve, 100→ethundrede), so a model that writes "4" is not penalised against a reference that spells out "fire". Only clean standalone integers are converted; digits embedded in larger tokens (decades like 1960'erne, ranges) are left untouched
6. Collapse whitespace

This is broadly consistent with the HF Open ASR Leaderboard (BasicTextNormalizer) and matches the de-facto Danish standard set by danish-speech-eval, whose "normalised WER" likewise spells numerals out as words.

Note: folding digit↔word formatting (step 5) is a fairness correction, not a leniency — it is applied symmetrically to reference and hypothesis, so genuine number-recognition errors are still counted. It lowers every model's WER by ~0.1–0.5 pp and does not reorder the board.

Note: Danish hesitation fillers (øh, hmm, …) are not removed in the published scores (the eval harness supports it as an opt-in, but it is off by default).

Note: Danish orthographic variants (aa↔å, oe↔ø, ae↔æ) are not normalised — the digraphs occur legitimately as letter sequences (ekstraarbejde, place names like Aarhus), so a blind substitution would introduce errors. Different Unicode encodings of the same letter are unified by NFC.

Metric

WER (Word Error Rate, %) and CER (Character Error Rate, %) — lower is better.

WER = (substitutions + deletions + insertions) / reference_words × 100

CER = (char substitutions + deletions + insertions) / reference_chars × 100

Mean WER and Mean CER are macro-averages across all five test sets (equally weighted).

WER can exceed 100 %. The denominator is the reference word count, but the numerator counts insertions without a ceiling — if a model hallucinates or loops, it can emit far more words than the reference contains, pushing WER well above 100 %. This is mathematically correct and a real signal: scores above 100 % on a given dataset indicate the model is not usable for that domain.

Speed

Speed (x) = total audio duration / total inference time. A value of 30x means 30 seconds of audio processed per wall-clock second. Only the transcription call is timed — model load time is excluded. Measured on an NVIDIA A100 (80 GB).

Speed reflects the serving stack, not just the model. Some models are run through high-throughput engines such as vLLM (continuous batching, paged KV cache), which can be several times faster than a plain transformers generation loop for the same weights. Treat Speed as "best-effort real-world throughput on this GPU" rather than an intrinsic model property — it is not comparable across different serving stacks or hardware, and API models are network-bound.

No warm-up run. Timing starts on the first batch, so one-off costs (CUDA lazy init, kernel autotuning, graph capture) are folded into the measurement. This slightly understates throughput, more so on smaller test sets where the fixed overhead is less amortised.

Code & data

Results dataset: RyeAI/danish-asr-leaderboard
Eval code: github.com/Rye-A1/danish-asr-leaderboard

Adding a model

There are two paths, depending on whether you've run the evaluation yourself:

Request a model — we run it

If you can't run the eval yourself, open a GitHub issue (or a Space discussion) with the model id, backend, and where to find it. We'll run it through the harness and add it.

Submit a score — you ran it

Run the harness on all five test sets, then open a pull request on the GitHub repo committing the two outputs the harness writes:

1. results/<model-slug>.json — scores and metadata.
2. outputs/<model-slug>/ — the raw per-dataset transcriptions (<dataset>.jsonl + meta.json), one {"id", "reference", "hypothesis"} per line. Lets reviewers inspect actual transcriptions and enables re-scoring under future normalisers.

Include in the PR description: the exact command you ran, the hardware used (for speed_x context), and whether the model is open or proprietary. On merge, CI automatically publishes both to the HF dataset and updates the leaderboard.

Do not modify the normalisation or metrics — run the harness as-is so results stay comparable.

Verification: whichever path a model arrives by, we re-evaluate it independently on our own hardware before publishing — to confirm the scores reproduce and catch any configuration differences.