Choosing Speech Models: Practical Criteria for STT and TTS

• In Speech, Requirements Come Before Models
• STT: When You Need Multiple Languages
• STT: When English Needs Speed and Length
• Decomposing the Word Real-Time
• Diarization Is a Separate Model
• TTS: Where Does the Voice Come From
• The Warnings on Speech Cards Are Not Decoration
• Code Examples
• The Order to Decide In
• Try It Yourself
• Series Navigation
• References

Model details were read directly from the Hugging Face pages on 2026-08-12. Model cards and licenses change, so check the original again before you use anything.

In Speech, Requirements Come Before Models

With text models you can wire something up and compare quality afterwards. Speech does not work that way. Once you fix which languages come in, how long the audio runs, how many seconds you have before a result is needed, and whether speakers must be separated, most candidates fall away on their own.

So this post sorts out those four axes before listing models, because most of the values a card provides map directly onto them.

STT: When You Need Multiple Languages

It is easy to miss that the two Whisper variants carry different license tags. openai/whisper-large-v3 is marked apache-2.0 and openai/whisper-large-v3-turbo is marked mit. Same family does not mean same terms.

The openai/whisper-large-v3-turbo card states plainly that the model is way faster at the expense of a minor quality degradation. Which side you take depends on what a single misheard word costs. For a subtitle draft, speed; for a medical record or legal evidence, accuracy.

nvidia/canary-1b-flash states it supports translation between English and German, French, or Spanish in both directions. But the card also states it is designed for audio smaller than 40 seconds, that it does not handle special characters, and that it may need inverse text normalization.

STT: When English Needs Speed and Length

nvidia/parakeet-tdt-0.6b-v2 is cc-by-4.0 with 600 million parameters, described as a FastConformer encoder with a TDT decoder. Its practical strengths are the two things the card names: efficient transcription of audio segments up to 24 minutes in a single pass, and timestamp support at char, word, and segment level.

The trade is that the language is English only. And the card states that NVIDIA NeMo is required. If your pipeline is built purely on transformers, this is a decision to take on one more runtime. The card also states that transcripts may not be 100 percent accurate and that accuracy varies with language and the characteristics of the input audio.

Decomposing the Word Real-Time

A request for real-time is usually one of three things: partial results must appear before the utterance ends, a final result must arrive within a second after it ends, or the full result must be ready a few minutes after the meeting is over.

The Whisper family fits the third best. The openai/whisper-large-v3 card states a 30-second receptive field and that longer audio requires sequential or chunked algorithms, while the openai/whisper-large-v3-turbo card states outright that it is not designed for real-time transcription out of the box. If the first requirement is genuine, designing the streaming architecture comes before choosing a model.

Distribution format affects latency too. Systran/faster-whisper-large-v3 is a repository converting openai/whisper-large-v3 into the CTranslate2 format with ct2-transformers-converter, with weights saved in FP16 and the note that this can be changed with the CTranslate2 compute_type option. The same model under a different runtime has different latency behavior.

Diarization Is a Separate Model

If you are producing meeting minutes, transcription is only half of it. Splitting who spoke is a different model's job.

pyannote/speaker-diarization-3.1 is marked mit, assumes 16kHz mono, and states that stereo or multi-channel files are automatically downmixed to mono by averaging channels and that other sample rates are automatically resampled to 16kHz. Speaker counts can be steered with the num_speakers, min_speakers, and max_speakers options.

This repository requires accepting the access conditions and sharing contact information before files can be retrieved. That is something to confirm before you write an automated deployment pipeline.

TTS: Where Does the Voice Come From

The first question in picking a TTS model is where the voice comes from. If you choose from a prepared voice list, as with hexgrad/Kokoro-82M, the decision is simple. If you clone from a 6-second reference clip, as coqui/XTTS-v2 supports, then how you obtained those six seconds and whether that speaker consented becomes a blocker well before any technical question.

microsoft/speecht5_tts is assembled rather than dropped in. The card states you must supply speaker x-vector embeddings and load the microsoft/speecht5_hifigan vocoder alongside it. It is not a one-repository integration, and its supported languages along with bias, risks, limitations, and evaluation data remain More Information Needed on the card.

The SWivid/F5-TTS page does not state supported languages or a capability description. This post did not fill those blanks from memory. To actually use it you have to check the linked code repository and paper yourself.

The Warnings on Speech Cards Are Not Decoration

Speech is the modality where a card warning turns into a design constraint more directly than anywhere else.

The openai/whisper-large-v3 card states that predictions may include text not actually spoken in the audio, along with a tendency to generate repetitive text and uneven performance across languages. The same card says not to transcribe recordings taken without consent, and warns against deployment in high-risk decision-making contexts. The openai/whisper-large-v3-turbo card adds disparate performance across accents and dialects.

Those sentences become requirements verbatim. Output from a model with a documented hallucination risk must not become a finalized record without human review, and a service with no recording-consent procedure has a design problem that precedes model selection. The hexgrad/Kokoro-82M card stating that it was trained exclusively on permissive and non-copyrighted audio, and warning about fake sites impersonating it, belongs to the same category.

Code Examples

# Example: transcribe long audio using the chunked approach
from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="openai/whisper-large-v3-turbo",
    chunk_length_s=30,
    return_timestamps=True,
)

result = asr("meeting.wav", generate_kwargs={"language": "korean"})
for chunk in result["chunks"][:3]:
    print(chunk["timestamp"], chunk["text"])

Diarization is attached as a separate pipeline and merged on the time axis.

# Example: get speaker turns and align them with the transcript in time
from pyannote.audio import Pipeline

diarizer = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token="hf_...",
)

for turn, _, speaker in diarizer("meeting.wav").itertracks(yield_label=True):
    print(f"{turn.start:.1f}-{turn.end:.1f}", speaker)

The Order to Decide In

1. Fix your target languages. English-only changes the candidate set dramatically.
2. Look at the distribution of audio lengths and check it against the receptive field and recommended length on the card.
3. Rewrite the real-time requirement as one of the three concrete cases.
4. If diarization is needed, plan the separate model and its gating conditions together.
5. If you use voice cloning, put consent and rights verification ahead of the technical decision.
6. Copy the hallucination and bias warnings from the card directly into your human-review requirements.

Try It Yourself

• Browser AI Lab — run a small speech model in the browser and feel the latency.
• AI Benchmark Collection — check what speech recognition metrics actually measure.
• GPU VRAM Calculator for LLMs — size the memory when a speech model and a language model share one machine.

Series Navigation

• Previous: Open Models That Support Korean, and the Cost of Tokenization
• Next: Vision and Multimodal: Image Understanding, OCR, and VLMs

References

• Every value in the tables was read directly from that model page on Hugging Face on 2026-08-12. Anything absent from the page is written as not stated.
• The supported languages and capabilities of SWivid/F5-TTS are not stated on its page, so they are not written here.
• Read the full license text yourself and put commercial use through legal review.