- Introduction — why comparing TTS is hard
- What tts-bench actually measures
- The honest part — what the numbers can and cannot tell you
- Running it locally, and evaluating for your own case
- Closing
- References
Introduction — why comparing TTS is hard
Trending on GeekNews this week: tts-bench, a local benchmark by the developer 5uck1ess for comparing text-to-speech models on whatever hardware you put it on. Before the tool, the problem. Text-to-speech is unusually hard to benchmark because "good" is not one axis. A voice can be gorgeous but slur or drop words; a model can be perfectly intelligible but flat and robotic; another can be both and still too slow for a live agent. Naturalness, intelligibility, and speed pull in different directions, and the first two are subjective — there is no ground-truth scalar the way there is for, say, WER on a speech-recognition set. The author is blunt about it: "Quality is subjective, so the ground truth is your ears." What makes this bench worth reading is that it is built around that admission rather than papering over it.
The shape of the project, in one place:
- 55 models across three rigs (Windows + RTX 5090, Linux + RTX 3090, Apple M4)
- 16 predefined-voice engines plus 39 zero-shot cloning models
- Runs on CPU, CUDA, and Apple Silicon (MPS)
- Bench code is MIT-licensed; each model keeps its own license
What tts-bench actually measures
The bench is organized as three lenses.
Speed is the objective, publishable part: cold and warm TTFA (time to first audio), RTF (real-time factor; higher means faster than realtime), and memory, measured on CPU, CUDA, and Apple Silicon. The June 2026 leaders give the flavor:
- CPU (Ryzen 9 9950X3D): Piper — 107ms warm TTFA, 59x realtime
- CUDA (RTX 5090): Kokoro — 67ms warm TTFA, 104x realtime
- Apple M4 (16 GB): Piper — 208ms warm TTFA, 32x realtime
These are reproducible measurements, and they are the only part the bench is comfortable ranking outright.
Listen is the gallery: every model on every prompt, in both a default voice and a voice-cloning setting, each with an inline <audio> player so you can judge by ear. Samples are rig-independent — sourced once from the highest-fidelity rig — because timbre and artifacts should not depend on which box rendered them. As the README puts it, quality and artifacts are obvious in about five seconds, in a way a table cannot show.
Scores are objective quality proxies per model: UTMOS for naturalness, WER for intelligibility, and SIM for cloning fidelity, computed over the bench prompts with seed-tts-eval-style ASR and speaker-verification. The model list spans lightweight predefined-voice engines (Kokoro at 82M, Piper at ~15M) through zero-shot cloning systems (F5-TTS, Coqui XTTS-v2, ChatterBox, and many more), with a Default/Cloning toggle over the two groups.
The honest part — what the numbers can and cannot tell you
The most interesting thing about tts-bench is what it refuses to claim. There is no single "best sounding" ranking. An objective quality score (called NAQ) was prototyped and then pulled — in the author's words it "didn't track subjective ranking closely enough to publish," so it is being redesigned separately. The bench ranks speed; quality is left to the Listen lens and to a companion blind A/B voting arena that feeds a live human-preference Elo leaderboard. UTMOS, WER, and SIM are described as "objective backstops," not the verdict.
Why that caution is warranted shows up in the cloning results. Across 397 blind votes over 28 of 32 cloning models, the top three on voice-match were:
- OmniVoice — best voice match (24-1-3), but it "can garble or drop words"
- Echo-TTS — near-tied for first (21-1-6), clean 44.1 kHz
- IndexTTS-2 — third (16-2-5), accent held
The catch is in the first line. A timbre-focused A/B vote does not penalize garbled words, so the model that wins on sounding like the reference can simultaneously lose on saying the right words — precisely the gap the objective WER score exists to catch. Read one number and you would ship the wrong model.
One more caveat worth internalizing: the bench feeds every model the same plain prompts for fairness, so expressive features — emotion tags, voice descriptions, style knobs — are not exercised in any score. If your use case leans on that, the leaderboard is silent on the thing you care about most.
Running it locally, and evaluating for your own case
Setup needs uv and Python 3.11 and takes about 10-15 minutes. The full set is heavy:
- ~39 GB of per-model virtualenvs (in the repo)
- ~125 GB of model weights (in your Hugging Face cache, not the repo)
- ~165 GB all in
But you are not meant to install all 55. Pass the ones you want: ./install.sh kokoro piper miso (or the .ps1 on Windows), and bench.py only runs models whose venv exists. For quick feel-testing there is python speak.py kokoro, and python compare.py "your phrase" for a one-shot A/B.
That subset workflow is the point. The reason to run this locally rather than trust a global leaderboard is that TTS quality is use-case-specific: your prompts, your target voice, your language, your latency budget, your hardware. A model that tops the Elo on clean English sentences can fall apart on your domain jargon or your particular cloning reference. This is the familiar "evaluate on your own data" principle, turned into a tool — the bench hands you the harness and, honestly, tells you to listen for yourself.
Closing
What is genuinely good about tts-bench is its discipline about where objectivity ends. Speed it measures and ranks; quality it presents and lets you judge; cloning preference it collects through blind votes and labels as preference, not intelligibility. That separation is the honest way to benchmark something subjective.
The practical takeaway is small and concrete. Do not read a single score and pick a winner. Install the two or three models you are actually considering, feed them your text and your reference voice on the box you will deploy on, and decide by ear — with WER and RTF as backstops, not as the answer.
References
현재 단락 (1/33)
Trending on GeekNews this week: `tts-bench`, a local benchmark by the developer 5uck1ess for compari...