Note: Many models do not disclose their training data. Some models may have been trained on one or more of the evaluation datasets, in which case their scores are not directly comparable to models that have not.
| Column | Dataset | Split | Domain |
|---|---|---|---|
| CoRal Conv. | CoRal-project/coral-v3 — conversation | test | Spontaneous conversation |
| CoRal Read | CoRal-project/coral-v3 — read_aloud | test | Read-aloud speech |
| CV17 (da) | mozilla-foundation/common_voice_17_0 — da | test | Crowd-sourced read speech |
| FLEURS (da) | google/fleurs — da_dk | test | Read speech |
| FTSpeech | alexandrainst/ftspeech | test_balanced | Parliamentary / broadcast |
Applied identically to hypothesis and reference before scoring:
²→2) so visually equivalent forms match1.234 → 1234,
and 3.14 / 3,14 → 314det's) are preserved; all other punctuation and
symbols are removedotte og tredive and otteogtredive →
38; hundrede → 100), so spelling and
spacing variants collapse onto one value before the expansion below. Bare
en/et are left as indefinite articles4→fire, 24→fireogtyve,
100→ethundrede), so a model that writes
"4" is not penalised against a reference that spells out
"fire". Only clean standalone integers are converted; digits
embedded in larger tokens (decades like 1960'erne, ranges) are
left untouchedThis is broadly consistent with the
HF Open ASR Leaderboard
(BasicTextNormalizer) and matches the de-facto Danish standard set
by
danish-speech-eval,
whose "normalised WER" likewise spells numerals out as words.
Note: folding digit↔word formatting is a fairness correction, not a leniency — it is applied symmetrically to reference and hypothesis, so genuine number-recognition errors are still counted.
Note: Danish hesitation fillers (øh,hmm, …) are not removed in the published scores (the eval harness supports it as an opt-in, but it is off by default).
Note: Danish orthographic variants (aa↔å,oe↔ø,ae↔æ) are not normalised — the digraphs occur legitimately as letter sequences (ekstraarbejde, place names likeAarhus), so a blind substitution would introduce errors. Different Unicode encodings of the same letter are unified by NFC.
WER (Word Error Rate, %) and CER (Character Error Rate, %) — lower is better.
WER = (substitutions + deletions + insertions) / reference_words × 100
CER = (char substitutions + deletions + insertions) / reference_chars × 100
Mean WER and Mean CER are macro-averages across all five test sets (equally weighted).
WER can exceed 100 %. The denominator is the reference word count, but the numerator counts insertions without a ceiling — if a model hallucinates or loops, it can emit far more words than the reference contains, pushing WER well above 100 %. This is mathematically correct and a real signal: scores above 100 % on a given dataset indicate the model is not usable for that domain.
Speed (x) = total audio duration / total inference time. A value of 30x means 30 seconds of audio processed per wall-clock second. Only the transcription call is timed — model load time is excluded. Measured on an NVIDIA A100 (80 GB).
Speed reflects the serving stack, not just the model.
Some models are run through high-throughput engines such as
vLLM (continuous batching, paged KV cache), which can be
several times faster than a plain transformers generation loop
for the same weights. Treat Speed as "best-effort real-world
throughput on this GPU" rather than an intrinsic model property — it is
not comparable across different serving stacks or hardware,
and API models are network-bound.
No warm-up run. Timing starts on the first batch, so one-off costs (CUDA lazy init, kernel autotuning, graph capture) are folded into the measurement. This slightly understates throughput, more so on smaller test sets where the fixed overhead is less amortised.
Results dataset:
RyeAI/danish-asr-leaderboard
Eval code:
github.com/Rye-A1/danish-asr-leaderboard
There are two paths, depending on whether you've run the evaluation yourself:
If you can't run the eval yourself, open a GitHub issue (or a Space discussion) with the model id, backend, and where to find it. We'll run it through the harness and add it.
Run the harness on all five test sets, then open a pull request on the GitHub repo committing the two outputs the harness writes:
results/<model-slug>.json — scores and metadata.outputs/<model-slug>/ — the raw per-dataset transcriptions
(<dataset>.jsonl + meta.json), one
{"id", "reference", "hypothesis"} per line. Lets reviewers inspect actual
transcriptions and enables re-scoring under future normalisers.Include in the PR description: the exact command you ran, the hardware used (for
speed_x context), and whether the model is open or
proprietary. On merge, CI automatically publishes both to the
HF dataset
and updates the leaderboard.
Do not modify the normalisation or metrics — run the harness as-is so results stay comparable.
Verification: whichever path a model arrives by, we re-evaluate it independently on our own hardware before publishing — to confirm the scores reproduce and catch any configuration differences.
38 and otteogtredive count as the same token.
That only worked when the model happened to spell a number the way
num2words does: a model writing otte og tredive
still mismatched, and because Danish compounds are written closed, three
tokens against one cost roughly three word errors rather than one.
100 was expanded to ethundrede, which never
matched the spoken hundrede. Numerals are now folded to
digits first and expanded once, so spelling and spacing variants land on
the same form. The scored text is unchanged in kind — still Danish
words — only the set of inputs reaching each form is wider.MediaCatch/xls-r-300m-danish-mc-v2, which spells numerals
out with spaces throughout and improves by 1.06, moving up two places.
Measured across all 29 published models before adoption: 28 improve or
hold, and the one regression is whisper-tiny at
+0.04 WER, already above 130 %.øh, hmm) was measured at the
same time and rejected. It changed no ranking across the 29 models, and it
is not monotone — whisper-tiny and whisper-base
get worse, because at their error rates removing reference tokens raises
the rate. It remains available as an opt-in for offline re-scoring.MediaCatch/xls-r-300m-danish-mc-v2 joins the board.speed (x). The models did not change:
beam search previously ran single-threaded and now runs in parallel, so the
old figures understated them.return_timestamps=True — a long-form setting — to every clip,
on a corpus that is ~99 % short-form. On short audio that sends the
Danish Whisper fine-tunes into repetition loops (a one-word reference
producing 100+ repeated words), badly inflating their WER. Decoding now
follows the reference implementations
(HF Open ASR Leaderboard,
CoRal):
plain short-form under 30 s, timestamped long-form above it. Every
transformers-backend model was re-run —
roest-v3-whisper-1.5b improves from 31.82 to 21.08 WER on
CoRal conversation. The wav2vec2 Røst models are unaffected: they use a
CTC path that cannot loop.whisper-large-v3 hallucinates on ~0.6 % of CoRal
conversation utterances — a 5-word reference typically yielding ~220 words —
emitting memorised training text such as Danish subtitle credits. This seems to be genuine
model behaviour rather than a scoring artifact, and is visible in the
published raw outputs. Its earlier score was flattered by the non-standard
setting above, which happened to suppress the effect for vanilla Whisper
while causing the opposite failure in the fine-tunes.num2words, Danish) before scoring, so digit-vs-word
formatting ("4" vs "fire") no longer counts as
an error. Lowers every model's WER by ~0.1–0.5 pp; does not reorder
the board.outputs/), so anyone can re-score under a different
normaliser without re-running inference.transformers 5.x fixed a silent truncation bug: the
previous transformers 4.x processor had no automatic chunking
and silently dropped audio beyond ~30 s, inflating WER on the longer
test sets. Affected models were re-measured.