🇩🇰 Open Danish ASR Leaderboard

Text normalisation

Applied identically to hypothesis and reference before scoring:

1. Unicode NFKC — folds compatibility characters (ligatures, full-width digits, ²→2) so visually equivalent forms match; optional acute accents such as dér/der are ignored
2. Exact word equivalents — selected historical and joined spellings (saa/så, afsted/af sted), abbreviations (fx/for eksempel, bl.a./blandt andet, m.m./med mere), units (km, cm, kg, kr, pct, numeric m, mm, m², km/h, km/t, m/s), and numeric mio./mia. match their spoken forms
3. Danish number canonicalisation — thousands separators are removed (1.234 → 1234), while decimals retain a marker (3.14 / 3,14 → 3komma14). Trailing decimal zeroes, frequency-unit formatting (including 2.400 GHz = 2.4 GHz), % / procent, clock times and ratios are normalized. Numeric slashes split numbers without assuming a fraction or date
4. Lowercase
5. Strip punctuation — apostrophes inside a word (e.g. det's) are preserved; all other punctuation and symbols are removed
6. Spelled-out numerals → digits — numbers written as words are folded to their digit form with text2num (otte og tredive and otteogtredive → 38; hundrede → 100), so spelling and spacing variants collapse onto one value before the expansion below. Bare en/et are left as indefinite articles
7. Numerals → Danish words — standalone integers are spelled out with num2words (4→fire, 24→fireogtyve, 100→ethundrede), so a model that writes "4" is not penalised against a reference that spells out "fire". Only clean standalone integers are converted, so decades such as 1960'erne keep their digits
8. Collapse whitespace

A dash between two digits becomes a space during the punctuation strip rather than being deleted, so 1-3 scores as two numbers. Deleting it glued the range into 13 → tretten, and the time range 10 00-11 00 into ti elleve nul — references no correct transcription could match, which also handed free matches to models that emit digits.

Known limitation: a bare 2.400 is read as two thousand four hundred; GHz context is needed for the decimal reading. A numeric slash separates its operands, but does not infer whether they form a fraction or date. Ordinals written with a full stop are likewise left alone (3. plads → tre plads): converting them to tredje was measured and rejected, since most N. here are sentence-final cardinals rather than true ordinals.

This is broadly consistent with the HF Open ASR Leaderboard (BasicTextNormalizer) and matches the de-facto Danish standard set by danish-speech-eval, whose "normalised WER" likewise spells numerals out as words.

Note: folding digit↔word formatting is a fairness correction, not a leniency — it is applied symmetrically to reference and hypothesis, so genuine number-recognition errors are still counted.

Note: Danish hesitation fillers (øh, hmm, …) are removed, from reference and hypothesis alike, using the same pattern as alexandrainst/coral. Scoring them verbatim measured transcription convention rather than accuracy. Verbatim scoring remains available offline via --no-filler-words.

Note: General Danish spelling substitutions (aa↔å, oe↔ø, ae↔æ) are not applied — these letter sequences occur legitimately in names such as Aarhus. Only a short, explicit list of historical words is folded.

Models and systems

The entry switch shows all results, single-checkpoint Models decoded greedily or with plain beam search, or Systems that add a language model or reranker, combine checkpoints, or use a hosted API. The small label by each name describes the submitted setup. A model may have one result of each type; the two scores come from separate evaluations. The Pareto chip compares open entries within their own category under the selected metric, ranking method, and filters.

Metric

WER (Word Error Rate, %) and CER (Character Error Rate, %) — lower is better.

WER = (substitutions + deletions + insertions) / reference_words × 100

CER = (char substitutions + deletions + insertions) / reference_chars × 100

Mean WER and Mean CER are macro-averages across all five test sets (equally weighted).

WER can exceed 100 %. The denominator is the reference word count, but the numerator counts insertions without a ceiling — if a model hallucinates or loops, it can emit far more words than the reference contains, pushing WER well above 100 %. This is mathematically correct and a real signal: scores above 100 % on a given dataset indicate the model is not usable for that domain.

Speed

Speed (x) = total audio duration / total inference time. A value of 30x means 30 seconds of audio processed per wall-clock second. Only the transcription call is timed — model load time is excluded. Measured on an NVIDIA A100 (80 GB).

Hosted API models show — instead. For those, the figure would measure network latency and our own request concurrency rather than anything about the provider, so it is not comparable with the locally-run models or between providers.

Speed reflects the serving stack, not just the model. Some models are run through high-throughput engines such as vLLM (continuous batching, paged KV cache), which can be several times faster than a plain transformers generation loop for the same weights. Treat Speed as "best-effort real-world throughput on this GPU" rather than an intrinsic model property — it is not comparable across different serving stacks or hardware, and API models are network-bound.

No warm-up run. Timing starts on the first batch, so one-off costs (CUDA lazy init, kernel autotuning, graph capture) are folded into the measurement. This slightly understates throughput, more so on smaller test sets where the fixed overhead is less amortised.

Code & data

Results dataset: RyeAI/danish-asr-leaderboard
Eval code: github.com/Rye-A1/danish-asr-leaderboard

Adding a model

There are two paths, depending on whether you've run the evaluation yourself:

Request a model — we run it

If you can't run the eval yourself, open a GitHub issue (or a Space discussion) with the model id, backend, and where to find it. We'll run it through the harness and add it.

Submit a score — you ran it

Run the harness on all five test sets, then open a pull request on the GitHub repo committing the two outputs the harness writes, plus a metadata entry:

1. results/<model-slug>.json — scores and metadata.
2. outputs/<model-slug>/ — the raw per-dataset transcriptions (<dataset>.jsonl + meta.json), one {"id", "reference", "hypothesis"} per line. Lets reviewers inspect actual transcriptions and enables re-scoring under future normalisers.
3. An entry in scripts/model_metadata.json: the Model/System type, release date, plus links to the training code, training data, model card and paper, or null where there is none (criteria in CONTRIBUTING.md).

Include in the PR description: the exact command you ran, the hardware used (for speed_x context), and whether the model is open or proprietary. On merge, CI automatically publishes both to the HF dataset and updates the leaderboard.

Do not modify the normalisation or metrics — run the harness as-is so results stay comparable.

Verification: whichever path a model arrives by, we re-evaluate it independently on our own hardware before publishing — to confirm the scores reproduce and catch any configuration differences.