Skip to content
Published on

Speech Recognition and Synthesis Technical Reports: What to Read, and What a Single WER Hides

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction

Part four is speech. This is the domain where a single number misleads hardest. Pick a model by WER alone and it falls apart in production on accents, noise and proper nouns.

The most honest statement of that problem happens to sit in one of the newest reports. The Qwen3-ASR abstract, covered below, notes that ASR models can differ very little on open benchmark scores while showing significant quality differences in real-world scenarios. The authors said this series' thesis for me.

Paper details were verified directly from the primary sources on 2026-08-12. This field moves fast, so check the current state yourself.

Why this article does not declare a winner

In ASR evaluation, a single text normalization rule moves WER. How you handle numerals, abbreviations, punctuation and casing goes straight into the score. TTS is worse: naturalness ultimately comes from human listening tests, so results shift with the rater pool and the instructions they were given.

So this article reads latency, language coverage, streaming capability and required data scale instead of scores. Those values hold up across harnesses.

What speech reports are actually contesting

[The axes speech reports contest]
 Coverage  : how many languages, from how little data
 Latency   : milliseconds until the first sound
 Streaming : start after the full sentence, or while receiving it
 Duplex    : can it listen while speaking
 Unification: one model for recognition, synthesis and understanding

Ten technical reports worth reading

Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)

arxiv.org/abs/2212.04356 — submitted 2022-12-06.

Trained on 680,000 hours of multilingual and multitask supervision, reporting that zero-shot transfer without any fine-tuning competes with prior fully supervised approaches. Nearly every ASR report since uses this as a baseline, so it is worth reading the original before the recent ones.

Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages

arxiv.org/abs/2511.09690 — submitted 2025-11-12.

Self-supervised pre-training scaled to 7B with an encoder-decoder architecture designed for zero-shot generalization. It covers more than 1,600 languages, over 500 of which the authors state were never before served by ASR. The design point is letting communities introduce an unserved language with only a handful of samples. And the report states its limit plainly: most of the world's 7,000-plus languages remain unsupported.

Qwen3-ASR Technical Report

arxiv.org/abs/2601.21337 — submitted 2026-01-29, v2 on 2026-01-30.

Two models, 1.7B and 0.6B, plus a timestamp alignment tool. Language identification and recognition for 52 languages and dialects, with latency down to 92 ms for the smaller model. The part worth quoting is not a score but the authors' own observation: models that look nearly identical on open benchmark scores can diverge substantially in real deployments. Released under Apache 2.0.

Open ASR Leaderboard

arxiv.org/abs/2510.06961 — submitted 2025-10-08, v4 on 2026-03-30.

Not a model, and the first thing anyone choosing an ASR system should read. It compares 86 open-source and proprietary systems across 12 datasets, standardizing WER and RTFx across differing architectures and toolkits. One reported trend is useful on its own: Conformer encoders with transformer decoders come out ahead on error rate, while CTC and TDT decoders lead on efficiency. Open-sourcing the code and dataset loaders so the comparison is reproducible is the substance of this work.

Seed-TTS and F5-TTS

arxiv.org/abs/2406.02430 (submitted 2024-06-04) and arxiv.org/abs/2410.06885 (submitted 2024-10-09, v3 on 2025-05-20).

The first is autoregressive, reporting speaker similarity and naturalness matching ground truth human speech, with self-distillation and reinforcement learning added for robustness. The second is non-autoregressive flow matching, foregrounding a 0.15 real-time factor, code switching and speed control. The F5-TTS abstract names slow convergence and low robustness in its predecessor as the motivation for its design.

CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

arxiv.org/abs/2412.10117 — submitted 2024-12-13, v3 on 2024-12-25.

Finite-scalar quantization to improve codebook utilization of speech tokens, plus a simplified architecture that lets a pre-trained LLM serve directly as the backbone. A chunk-aware causal flow matching model supports streaming and non-streaming synthesis together. The document describes itself as a tech report and work in progress, so its numbers should not be read as settled results.

Qwen3-TTS Technical Report

arxiv.org/abs/2601.15621 — submitted 2026-01-22.

Trained on over 5 million hours of speech across 10 languages, supporting 3-second voice cloning and description-based control. The distinguishing design is two speech tokenizers: one semantics-focused, the other for ultra-low-latency streaming, with a 97 ms first-packet emission time reported. Released under Apache 2.0.

Fish Audio S2 Technical Report

arxiv.org/abs/2603.08823 — submitted 2026-03-09, v2 on 2026-03-11.

Multi-speaker, multi-turn generation with instruction-following control through natural-language descriptions. The report describes a multi-stage training pipeline including captioning, quality assessment and reward modeling, and reports a 0.195 real-time factor with time-to-first-audio below 100 ms. Code, weights and fine-tuning resources are public.

Moshi: a speech-text foundation model for real-time dialogue

arxiv.org/abs/2410.00037 — submitted 2024-09-17, v2 on 2024-10-02.

A report with a different problem definition. It names three failures of component pipelines together: multi-second latency, information loss when text is the intermediate modality, and the inability to handle overlapping speech or interruptions. It models its own speech and the user's speech as parallel streams, and predicts text tokens before audio tokens in an Inner Monologue method. Reported latency is 160 ms theoretical, 200 ms in practice.

Qwen2.5-Omni and MOSS-Audio

arxiv.org/abs/2503.20215 (submitted 2025-03-26) and arxiv.org/abs/2606.01802 (submitted 2026-06-01, v3 on 2026-06-05).

The first takes text, images, audio and video and emits text and natural speech in a streaming manner, with TMRoPE aligning audio and video in time and a Thinker-Talker architecture separating text generation from speech generation to reduce interference. The second is an audio-language model covering speech, environmental sound and music understanding, using 12.5 Hz temporal representations, cross-layer feature injection that exposes the decoder to acoustic information from multiple encoder depths, and inserted time markers. The authors position it as an understanding foundation for future voice agents. Step-Audio 2 (arxiv.org/abs/2507.16632), which adds conversation, belongs to the same family.

Limitations the authors state themselves

  • Omnilingual ASR states directly in its abstract that most of the world's languages remain unsupported, and reflects on ethical considerations alongside that.
  • Qwen3-ASR writes that small differences in open benchmark scores can accompany large differences in real-world quality. Apply that sentence to every ASR selection you make.
  • CosyVoice 2 labels itself work in progress.
  • F5-TTS names its predecessor's convergence speed and robustness as the problems it set out to fix.
  • MOSS-Audio positions itself as an understanding foundation for future voice agents rather than a finished system.
  • Voice cloning carries impersonation risk. Reports rarely discuss that part when they discuss performance, which is worth remembering.

What a practitioner should read first

If you are choosing an ASR model, start with the Open ASR Leaderboard. A standardized comparison comes before any individual report's self-reported numbers.

If low-resource languages are your target, Omnilingual ASR. A design that attaches a new language from a handful of samples changes which use cases are feasible at all.

If you are building real-time conversation, read Moshi and Qwen3-TTS together. They address different bottlenecks: the structural problem of full-duplex dialogue, and first-packet latency.

If you want to fold several modalities into one model, Qwen2.5-Omni and MOSS-Audio are the design references.

How and when this was verified

Every report cited here was verified on 2026-08-12 by opening the arXiv abstract page directly. Candidates that could not be opened were not cited. All numbers, latency and real-time factors included, are as reported by the authors and vary with hardware and configuration.

Try it yourself

  • ML Training Data Explorer — see by example what training data recognition and synthesis models each consume.
  • Browser AI Lab — running a model in the browser with no server gives you a feel for what the latency figures in these reports actually mean.

Previous in the series: Video Generation and Understanding Technical Reports: What to Read

Next in the series: Image Generation and Understanding Technical Reports: What to Read

References

Discuss on Twitter • View on GitHub