What is a neural audio codec?

A neural audio codec is a codec whose encoder and decoder are neural networks trained on lots of audio, rather than hand-designed maths. Like any audio codec it shrinks sound into data and rebuilds it for playback. The difference is what the data looks like: a short sequence of discrete codes, one small set per slice of time. Those codes are what AI researchers call audio tokens.

How it works

1. Encoder. A network turns the waveform into a compact sequence of vectors, far fewer per second than audio samples.
2. Quantizer. Each vector is snapped to the nearest entry in a learned codebook. The best-known codecs use a residual vector quantizer: a first codebook gives a rough match, the next codes the leftover error, and so on. SoundStream trains this quantizer jointly with the encoder and decoder, end to end.
3. Decoder. A second network rebuilds audio from the codes. Training adds an adversarial "critic" so the result sounds natural rather than merely close on paper.

The three names you will see

• SoundStream (Google, 2021). Compresses speech, music and general audio at 3 to 18 kbps from one model. Its authors report that at 3 kbps it beat Opus at 12 kbps in listening tests on 24 kHz audio (paper).
• EnCodec (Meta, 2022). A real-time, streaming codec tested on 24 kHz mono and 48 kHz stereo, including music, with code released openly (paper).
• DAC, the Descript Audio Codec (2023). One universal model for speech, music and environmental sound, reporting about 90x compression of 44.1 kHz audio into tokens at 8 kbps. Its authors say plainly that the goal is generative modelling (paper).

What each paper added

• SoundStream uses structured dropout on the quantizer layers so that one model works across 3 to 18 kbps with a "negligible quality loss" compared with models trained at a fixed bitrate. It also supports low-latency, streamable inference that runs in real time on a smartphone CPU, and can denoise speech while compressing it (paper).
• EnCodec simplifies training with a single multiscale spectrogram adversary, and the authors show a lightweight Transformer can compress the codes by up to 40% further while staying faster than real time. They evaluated with MUSHRA listening tests on speech, noisy-reverberant speech and music (paper).
• DAC pairs better vector quantization techniques from the image domain with improved adversarial and reconstruction losses, and the authors release open-source code and trained weights (paper).

What a codec lets a music model do

MusicGen's authors say it is a single-stage transformer with "efficient token interleaving patterns", which removes the need to cascade several models, and that it can be conditioned on a text description or on melodic features (paper). Melody conditioning is the part that matters in a studio, because it lets you steer the output with a tune of your own rather than words alone.

Why music generators use tokens

Language models are good at one thing: predicting the next item in a sequence of discrete symbols. Raw audio is a terrible sequence for that, at 44,100 numbers per second per channel. A neural codec turns a song into a much shorter list of tokens a language model can predict, then the codec's decoder turns the predicted tokens back into sound. Meta's MusicGen is exactly this: a single transformer that writes several parallel streams of EnCodec-style tokens from a text prompt. Not every generator works this way; some use diffusion models instead. Our explainer on how AI music generators work covers both.

What we think

Read neural codecs as the alphabet of AI audio, not as an MP3 replacement. Nothing you deliver to a distributor uses one, and you should keep sending lossless masters. But the codec sets a ceiling on what a token-based generator can sound like: if the codec smears cymbals or stereo width, the model built on it will too. When an AI tool's output sounds slightly "phasey" or metallic on transients, the codec is a likely suspect (our inference, not a tested claim). It is also why sample rate claims from AI tools deserve a second look: the published codecs above work at 24 to 48 kHz, so ask what the tool's codec runs at.