Engineering

We benchmarked 13 speech-to-text APIs across 12 languages — and published everything

July 6, 2026 · by Masaki Kondo · 9 min read

If you try to pick a speech-to-text engine today, what you find is vendor self-reported numbers and English-centric leaderboards. "Best-in-class WER across 43 languages." "Parity with the leader at one-fifth the price." Almost nobody measures these claims independently, per language, under one methodology.

So we did: 13 hosted STT API variants × 12 languages × 200 utterances, scored under a preregistered, frozen methodology.

Leaderboard: koedesk.app/benchmark/
Repository (methodology, all raw data, scoring code): github.com/guide-inc-org/koedesk-stt-bench

This post covers how we tried to make a vendor-funded benchmark worth trusting, and the side findings — which turned out to be at least as interesting as the rankings.

Conflict of interest, stated first

This benchmark is built and funded by koedesk, a dictation app that uses ElevenLabs Scribe v2. We are not neutral, and we don't pretend to be.

Instead of claiming neutrality, we published everything you need to distrust us:

Preregistration: engines, normalization rules, scoring, and statistics were committed to a public repo before any headline measurement — the commit history is the proof there was no after-the-fact tuning
All raw data: every one of the 28,400 raw API responses is in the repo
All scoring code: the leaderboard is generated deterministically from scores.json; there are no hand-edited numbers
Losses published: the preregistration commits us to publishing every language where Scribe loses — and it does lose some

"Vendors publish benchmarks that favor themselves" is a structural problem. Engineering that structure away — rather than asking readers to trust our character — is the actual technical content of this project.

What we measured

Engines (13 variants): ElevenLabs Scribe v2, OpenAI gpt-4o-transcribe, whisper-1, Google Cloud STT v2 (Chirp 3), five Gemini variants (2.5 Flash, 2.5 Pro, 3.1 Flash-Lite, 3.1 Pro preview, 3.5 Flash), Deepgram Nova-3, Microsoft MAI-Transcribe-1.5, Mistral Voxtral Mini Transcribe V2, and AmiVoice (Japanese only).

Languages (12): Japanese, English, Mandarin Chinese (simplified), Korean, Vietnamese, Indonesian, Thai, German, French, Spanish, Portuguese, Russian.

Material: the first 200 utterances of the FLEURS test split per language — deterministic dataset order, so there is no sampling choice for us to fiddle. Every engine receives the identical 16-bit PCM file, peak-normalized to −6 dBFS.

In total: 28,400 transcriptions over ~95 hours of audio, for about $28 in API fees. Independent verification at this scale is cheap now; that fact deserves to be better known.

Exclusions are published too. AssemblyAI's terms prohibit using the service for benchmarking at all; xAI's agreement names "benchmark" in a prohibited-use clause. Both were excluded before measurement, with the clauses quoted in a public terms-of-service audit. That a vendor's terms forbid benchmarking is itself information you want before choosing an engine.

How we bound ourselves

The biggest lever in any STT benchmark isn't the model — it's text normalization. Punctuation, number formats ("twenty-one" vs "21"), width variants in CJK: whoever controls these choices after seeing the results can move the rankings. So:

The normalization pipeline is frozen in code with tests, committed before measurement, and applied identically to references and hypotheses
Any post-freeze change requires a dated, append-only amendment stating the change, the reason, and why it is not outcome-driven (we're at Amendment 7 — everything from normalizer bug fixes to a scope change is on the record)
For en/de/fr/es/pt we apply the pinned Whisper open-source normalizer, keeping our numbers methodologically comparable with the HF Open ASR Leaderboard

Statistics are frozen too: paired bootstrap (n=10,000, seed fixed at preregistration) puts engines into tie groups per language. Engines statistically indistinguishable from the leader share tier T1, and we never claim a win inside a tie group — including for our own engine. As you'll see, that clause has teeth.

Results

T1 finishes across the 12 languages:

| Engine | T1 count / 12 | |---|---| | Gemini 3.1 Pro (preview) | 10 | | MAI-Transcribe-1.5 | 9 | | ElevenLabs Scribe v2 | 7 | | gpt-4o-transcribe | 6 | | Google STT v2 (Chirp 3) | 6 |

There is no single champion. Vietnamese has a sole T1 (Gemini 3.1 Pro), Chinese has a different sole T1 (MAI), Korean is a two-engine tier (Scribe and MAI). Any ad that says "best across all languages" is compressing away exactly the structure that matters.

Japanese (headline metric: CER):

| Engine | CER | 95% CI | Tier | |---|---|---|---| | MAI-Transcribe-1.5 | 1.91% | [1.33, 2.60] | T1 | | ElevenLabs Scribe v2 | 2.26% | [1.75, 2.85] | T1 | | Gemini 3.1 Pro (preview) | 2.49% | [1.96, 3.06] | T1 | | Google STT v2 / gpt-4o and below | 2.92%– | — | T2+ |

The nominal leader is MAI — not our engine. The three T1 entries are statistically indistinguishable, and claiming a win on an overlapping confidence interval is precisely the practice this benchmark exists to criticize, so we won't.

One more point worth recording: Microsoft claims MAI beats Scribe, Whisper and Gemini across 43 languages. As far as we know, ours is the first independent multilingual test of that claim — and it largely holds up (T1 in 9 of our 12). We set out to audit our own engine choice and ended up independently corroborating a competitor's marketing. That's what preregistration does to you.

The side findings

1. LLM-based STT has a failure mode classic STT doesn't

Gemini 2.5 Pro's Chinese CER came out at an absurd 48.3%. The cause: on one utterance out of 200, the model returned 4,783 characters of English chain-of-thought as the "transcription". That single utterance accounts for 86.7% of the cell's error mass; without it, the CER is 6.45% — perfectly normal.

This is real engine behavior, not a pipeline bug, so per our preregistered rules it is published as-measured, with the breakdown. A 0.5% chance of receiving a wall of English reasoning instead of your transcription is invisible in averages and catastrophic in a dictation product. Means hide tails.

2. Public test sets have artifacts of their own

13 of the 200 Chinese FLEURS references contain English annotation text that was never spoken. It cancels out (every engine faces the same references), but "public benchmark references are clean" is not a safe assumption.

Separately, our pilot showed Gemini 3.1 Flash-Lite emits Japanese with token-separating spaces — which inflates character error rate nearly eightfold from whitespace alone. Whether you count spaces is a metric definition choice that can reorder a leaderboard. We published our resolution as Amendment 1, and recorded the behavior itself as a finding: token-spaced Japanese is a real usability defect for dictation, whatever the metric says.

3. The top of a leaderboard isn't always purchasable

The #1 engine on a well-known public English leaderboard is a private preview API that you cannot actually buy. "Best among things you can call today" is a different question from "best number on the chart" — it's why this benchmark covers hosted APIs only.

4. "Fastest" means two different things

A throughput figure ("276× real-time") measures server-side batch speed. What a dictation user feels is API round-trip time per second of audio (RTF) — and in our measurements, an engine that tops throughput charts took 1.4× longer than a competitor on round-trips. Both numbers are true. They measure different things, and quoting one as the other is how speed marketing works.

So, is koedesk switching engines?

No. In Japanese, only Scribe v2 simultaneously sits in the accuracy T1, near the top on round-trip speed (RTF 0.16), and cheapest within the T1 ($0.22/hour). The nominal accuracy gap to MAI is statistically indistinguishable — and switching engines over an indistinguishable difference would contradict the benchmark we just published.

The useful part is the counterfactual: if Scribe had clearly lost in Japanese, we were committed to publishing that and would have had to consider switching. Being able to say "our engine choice survives our own adversarially-designed benchmark" was the product-side payoff of doing this properly.

Limitations, honestly

Training-data contamination: FLEURS is public; every commercial engine has plausibly trained on it. This caveat applies to every number on the leaderboard, and we say so on the leaderboard. We originally planned a contamination-proof track on freshly recorded 2026 audio, and cancelled it before any measurement — its human-produced reference transcripts couldn't meet the deterministic-reproducibility bar the rest of the benchmark sets (the reasoning is public as Amendment 7). Everything is CC-BY / MIT, so anyone can run the full pipeline on their own recordings.
Latency is informational: the measurement machine is in Vietnam with egress through a VPN exit node in Tokyo — identical for every engine, so relative comparisons hold, but absolute RTF carries a fixed overhead (Amendment 6).
n=200 per language: every number ships with a bootstrap CI, but a larger sample would separate some engines that currently tie.

How this was built (the part that surprised us)

Nearly all of this — the methodology draft, 13 API adapters, the 28,400-transcription run with its quality gates, the frozen scoring implementation, the 36-language leaderboard site, and the amendments — was built and executed by Claude (Fable 5) over roughly three days, with humans in the loop only at decision and approval gates: scope calls, spending, amendment sign-offs, and anything that touched the public record.

We didn't set out to make an AI showcase, and this isn't a flashy demo. It's the opposite: patient, unglamorous survey work — reading terms of service, auditing reference texts by eye, re-running cells after finding our own process violations and recording them in public. That an AI agent can now carry that kind of work end-to-end, at a total API cost of about $28, changed our sense of what one small team can afford to verify independently.