What Is OpenAI Whisper and How Does It Work?

Whisper is a speech recognition model that OpenAI released in September 2022, with its weights and code open to anyone. This explainer covers what it was trained on, how it turns audio into text, how accurate its authors found it, where it goes wrong, and the two ways to run it.

What Is Whisper?

Whisper is an automatic speech recognition (ASR) model: it takes audio and returns text. OpenAI described it in a paper, Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., 2022), and released the code and the model weights under the MIT License (Whisper repository).

Its training data is what sets it apart. Rather than a small set of carefully transcribed recordings, Whisper was trained on 680,000 hours of audio and the transcripts that came with it, collected from the internet. About 65% of it, 438,000 hours, is English audio with English transcripts; the non-English data covers 98 other languages (Whisper model card). The paper calls this weak supervision: the transcripts are plentiful but were not checked by hand.

OpenAI has released several versions since: large-v2 in December 2022, large-v3 in November 2023, and large-v3-turbo in September 2024 (Whisper model card).

How Does Whisper Work?

Whisper is an encoder-decoder Transformer: one network reads the audio, and a second writes text from what the first one found (Radford et al., 2022, §2.2). The paper describes three steps.

The audio becomes a picture of sound. It is resampled to 16,000 Hz and turned into a log-Mel spectrogram: 80 frequency bands, measured every 10 milliseconds. Think of a heatmap, time across and pitch up the side. The model reads it 30 seconds at a time.
The encoder reads it. It turns each 30-second window into a representation of what is in it: speech sounds, pauses, noise.
The decoder writes. One token at a time, it predicts the most likely next piece of text, given the audio and the text so far. It writes the text as people write it, punctuation and capitals included, not a normalized form (§2.1).

One model does several jobs. Before the text, the decoder predicts a few special tokens: the language being spoken, whether there is any speech at all, the task (transcribe in the same language, or translate into English), and whether to mark timestamps (Radford et al., 2022, §2.3). The same model therefore identifies the language, transcribes, translates and times its output, with no separate system for each.

How Accurate Is Whisper?

Accuracy is measured as word error rate (WER): the share of words a transcript gets wrong. A WER of 5% means roughly one word in twenty is wrong. The paper tests Whisper on many public test sets without training it on any of them, and reports that, compared with human transcribers, the models approach their accuracy and robustness (Radford et al., 2022). The model card describes improved robustness to accents, background noise and technical language, and accuracy near the state of the art (Whisper model card).

That accuracy is not the same everywhere. The model card says the models perform unevenly across languages, worse on languages with less training data, and differently across the accents and dialects of one language (Whisper model card).

Where Whisper Goes Wrong

Whisper can write words nobody said. Its model card says the predictions may include text that is not actually spoken in the audio, and calls it hallucination (Whisper model card). The paper lists the failures it saw on long recordings: getting stuck repeating the same text, and, at worst, a transcript entirely unrelated to the audio (Radford et al., 2022, §6).

So the model card recommends against using the models in high-risk domains, such as contexts where decisions are made, and cautions against transcribing recordings of people taken without their consent (Whisper model card). Whatever a transcript is for, read it against the audio before relying on it.

What Languages Does Whisper Support?

English and the 98 other languages in its training data (Whisper model card). The amount of data differs greatly from one language to another, and accuracy follows it: highest where the data is plentiful, lower on languages with little. Whisper also identifies the language spoken, which helps when you do not know what language a recording is in.

Running Whisper

There are two ways to use it.

On your own computer. Download the code and the weights from the repository. It needs Python and the ffmpeg tool, and comes in six sizes, from tiny (39 million parameters, about 1 GB of graphics memory) to large (1,550 million, about 10 GB); the smaller ones run faster (Whisper repository). The audio stays on your machine.
Through OpenAI’s API. You send the audio and get text back, with no hardware of your own. Each file can be up to 25 MB, so a longer recording is compressed or split, ideally between sentences, and the parts’ text joined back together (OpenAI speech-to-text guide).

The API is changing. On August 26, 2026, OpenAI announced that whisper-1, the model the API offers under Whisper’s name, leaves the API on February 26, 2027, along with its other transcription models of that generation, and named their replacements (OpenAI deprecations). The open weights are not affected: they stay in the repository, under the same license.