Whisper Model Size vs. Accuracy Tradeoffs for Real-Time Transcription
Model size determines real-time feasibility more than accuracy alone.

- Written by
- The Voice Layer Editors
- Published
- September 30, 2026
- Reading time
- 8 min read
- Sources cited
- 6 sources ↓
What this covers
Whisper's biggest limitation for live transcription has nothing to do with model size and everything to do with how the model is built. Its encoder-decoder Transformer design forces the system to take in complete audio before it can do anything else, and that single fact is the root cause of every real-time compromise a practitioner will run into. The encoder processes audio in a fixed 30-second chunk, and the decoder then generates tokens autoregressively from that finished representation. There's no built-in path for feeding it audio incrementally as it arrives. NPUsper is a joint work of authors from Korea University and the University of Wisconsin–Madison, and is listed as a NeurIPS 2026 paper. Its authors include Sihyeon Lee, Hojeong Lee, Sungwon Woo, Chengpo Yan, Suman Banerjee, and Seyeon Kim<sup>1</sup><sup>2</sup>.
"Real-time" transcription with Whisper is always a workaround: chunking the input, padding shorter clips out to the expected window, buffering audio while waiting for enough of it to accumulate, or altering the encoder's attention mask to fake a smaller window. None of these approaches give you true causal streaming, where the model processes audio as it comes in and nothing more. WhisperKit, presented by Argmax at ICML 2025, puts a number on what this costs in practice: a naive low-latency streaming setup needs many encoder forward passes for every minute of audio, adding up to processing several times that minute's worth of audio, compared to a single pass for standard offline transcription. Argmax gave this talk, titled "WhisperKit: On-device Real-time ASR with Billion-Scale Transformers," at ICML 2025's On-device AI workshop.
The six official model families, tiny, base, small, medium, large, and turbo (809M parameters), each inherit this constraint; size determines how expensive each workaround becomes. OpenAI's official README lists the six size rows as tiny, base, small, medium, large, and turbo, with the large entry covering the whole large family and turbo weighing in at 809M parameters. Moving up in size doesn't remove the need for a workaround, it raises how expensive that workaround becomes. Picking a model size, in other words, is really picking how much architectural overhead a deployment can afford to pay for the privilege of calling itself real-time.
Accuracy, speed, and memory by size tier
Lay the six tiers side by side and the curve they form isn't a straight line. Accuracy improves fastest at the bottom of the range, while speed and memory costs climb steeply once you cross into the largest tiers, and turbo sits outside the pattern entirely: it holds a parameter count between medium and large, yet matches large-v3 for speed.
Tiny, at roughly 39 million parameters, is the fastest and lightest option available, and it pays for that with the weakest accuracy of any tier, particularly once audio conditions get difficult. In the whisper.cpp quantized GGML format most local transcription apps use, tiny's model file is about 75 MB and it uses roughly 273 MB of memory at runtime.
Base, at about 74 million parameters, buys a modest accuracy gain over tiny on clean speech, but the larger gain appears on harder audio, a pattern the next section covers in more detail.
Small, at 244 million parameters, is often the practical floor for anyone who needs dependable transcription outside of a lab-clean recording environment.
Medium, at 769 million parameters, delivers a meaningful further step up in accuracy but starts to demand real compute. Its resource cost rules it out on some of the hardware classes covered later in this piece.
Large, at 1.55 billion parameters, is Whisper's most accurate general-purpose tier and the one against which every other tier gets measured, at the cost of the heaviest memory footprint and slowest inference of the family.
Turbo, a distilled version of large-v3, is the tier that breaks the otherwise orderly progression from tiny through large.
Why accuracy gaps between sizes widen on difficult audio
Standard clean-speech benchmarks understate what larger Whisper models actually buy you, because robustness to noise and reverberation doesn't scale in a straight line with size. It grows much faster at the top of the range than the bottom. The clearest evidence is the reverberation penalty measured across tiers: tiny loses 15.50 percentage points of word error rate when reverberation is introduced, small loses 7.44 points, medium loses 5.94 points, and large-v3 loses only 2.31 points. The gap between tiny and large-v3 on reverberant audio dwarfs the gap between them on clean audio.
The same pattern appears with accented and noisy speech more broadly. Base.en's improvement over tiny.en on LibriSpeech's clean test set is modest, but on the harder test-other set, where the audio itself is more demanding, that improvement widens considerably. Whisper models trained mostly on adult speech show sharply higher error rates on children's voices, and even the largest models' extra robustness doesn't fully close that gap without fine-tuning aimed specifically at the domain, as studies from Ghent University and Radboud University have found. One of the relevant studies comes from Radboud University Nijmegen's Centre for Language Studies and evaluates Whisper specifically on Dutch native child speech.
For any deployment where audio conditions are inconsistent, a medical office with background noise, a legal deposition recorded in a room with bad acoustics, the case for a larger model is stronger than a clean-benchmark WER table would suggest, because the tiers separate most sharply where deployments actually live.
How real-time deployment constraints compress the effective choice set
Accuracy is only half the decision. A model also has to run fast enough, on the hardware actually available, to keep up with speech as it happens. For a real-time deployment, the real question is which is the largest model that can hold a real-time factor under 1.0 on the target device, not which model transcribes best in isolation, and that answer swings wildly depending on the platform.
Quantization redraws the line of what's feasible. INT4-quantized large-v3 achieves real-time speed from as few as 8 threads, with a measured real-time factor of 0.91. That single change collapses what looked like a hard tradeoff between small-FP16 and medium-INT8: a team can land medium-tier accuracy for something closer to small-tier compute, once quantization is on the table.
Mobile hardware narrows the field further, and less forgivingly. On a Snapdragon X Plus laptop, running just the encoder half of Whisper-medium takes more than 4 seconds, which blows past the short inference window that live transcription needs and effectively takes medium off the table for real-time use on that class of device. Whisper-large fares worse still: the NPUsper paper reports it triggered an out-of-memory error on a Samsung Galaxy S25. At the other end, tiny is rarely chosen as the primary model in current mobile streaming systems, because its transcription quality falls short of what most deployments need. The realistic mobile window ends up being base through small, or turbo if it's been modified architecturally for the platform.
What streaming workarounds cost at each size
Every method for making Whisper stream trades some amount of redundant computation for stability, and that redundancy scales with model size in ways that can make a tier impractical no matter how fast it runs offline. Established approaches like Whisper-Streaming, SimulStreaming, and SimulWhisper handle this by buffering previously processed audio alongside new audio and padding every inference round out to the full 30-second window Whisper expects, specifically to keep the model from hallucinating text it never heard. The computational cost of that padding scales directly with model size, so the same workaround gets proportionally more expensive as the tier gets bigger.
WhisperFlow takes a different approach, cutting most of the padding overhead by substituting a short, trained "hush word" segment in its place. Even so, consecutive inference rounds still overlap substantially, so the underlying redundancy problem shrinks rather than disappears.
NPUsper, a 2026 paper out of Korea University, addresses this differently: it detects hallucinated tokens online from temporal patterns in decoder cross-attention. It watches the temporal patterns in decoder cross-attention to catch hallucinated tokens as they're produced, which lets each inference round work from a short audio input with minimal carryover from the previous round. On commercial mobile devices from Samsung and Qualcomm, that design drops average power consumption by as much as 88.64% against baseline methods, alongside sizable cuts to per-word latency and time-to-first-token. WhisperKit approaches the same problem from the compression side, shrinking the model substantially while keeping word error rate close to what the uncompressed version achieves.
Hallucination itself gets worse, not better, at the top of the size range, specifically on silence. Across 18 adversarial test cases run by inferencebench.io, Whisper Large-v3-Turbo produced hallucinated transcriptions in 8 of them, every single one on silent audio and none on noise or music. Some practitioners report that Large-v2 hallucinates less often than newer large-tier models on quiet or silent stretches. The standard fix is voice activity detection: setting vad_filter=True in faster-whisper's transcribe() call, using Silero VAD, strips silence before it ever reaches the model, and that step alone eliminates most hallucinations in files with long pauses. In any streaming context where silence occurs often, conversational AI, live interview transcription, VAD filtering is a required part of the configuration for a large-tier model. It's a required part of the configuration.
Where the size-accuracy relationship inverts or requires domain fine-tuning
The assumption running through the rest of this piece, that more parameters mean better transcription, stops holding once language and domain enter the picture. A smaller model fine-tuned on the right data can beat a larger general-purpose one. On Pashto, Whisper-medium was the worst performer on one benchmark, at a word error rate of 461.2%, and the second-worst on another, despite carrying more parameters than both small and base. What the model was trained on matters more for accuracy there than parameter count.
Swedish tells a similar story: a version of Whisper-small fine-tuned for the language beat OpenAI's general-purpose Whisper-large on the relevant benchmarks. Dutch child speech follows the same logic. A fine-tuned Whisper-medium model posted the best overall results across the JASMIN and DART datasets, ahead of larger general-purpose systems including entries from the Parakeet and Wav2Vec2 families. That fine-tuned Whisper-medium model reached a word error rate of 5.54% on JASMIN and 70.37% on DART overall, though the detailed results show a fine-tuned Whisper-large model actually posted the lowest word error rate on JASMIN alone, at 5.46%.
Compression adds another layer to this. Research presented at LREC/SPEAKABLE found that dynamic INT8 quantization using Optimum-Quanto cuts model size by more than half while lowering word error rate below the uncompressed baseline on LibriSpeech. Pushing compression further into formats like nf4 or int3 raises the accuracy cost in acoustically difficult conditions. Compression choices end up interacting with domain difficulty the same way raw model size does: what works cleanly on benchmark audio can fall apart on the audio a deployment actually has to handle.
Turbo carries its own version of this risk. Its speed comes from pruning the decoder down from 32 layers to 4, and that pruning specifically weakens performance on low-resource languages, where the decoder had less redundancy to spare in the first place. None of this overturns the broader pattern set out earlier in this piece: larger models are, on average, more accurate and more robust. But for a deployment built around a specific language or a specific acoustic domain, that average matters less than the fine-tuned alternative, which deserves a real look before the largest available tier gets assumed to be the safest choice.
Methodology & sources
- Whisper Model Sizes: Complete Guide | OpenWhispr
Provided parameter counts, file sizes, and memory figures for each Whisper model tier.
- NPUsper: Eliminating Redundant Computation for Real-Time Whisper on Mobile NPUs
Provided data on mobile inference latency, out-of-memory errors, power consumption reductions, and the hallucination-detection approach used by NPUsper.
- WhisperKit: On-device Real-time ASR with Billion-Scale Transformers
Provided figures on the redundant encoder forward passes required for streaming transcription compared to offline transcription.
- Transcribing Children's Speech: ASR Performance and Obtaining Reliable Orthographic Transcriptions
Provided word error rate results for fine-tuned Whisper models on Dutch child speech datasets JASMIN and DART.
- GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision · GitHub
Cited as the official source listing all six Whisper model size tiers.
- Swedish Whispers; Leveraging a Massive Speech Corpus for Swedish Speech Recognition
Provided the finding that a fine-tuned Whisper-small for Swedish outperformed OpenAI's general-purpose Whisper-large.