What is a diffusion model in music?

A diffusion model is a kind of AI generator that learns to turn random noise into a sound by removing that noise a little at a time, guided by a prompt such as a text description.

How it works, in plain English

During training, the model sees real audio with more and more noise mixed in and learns to predict what to take away at each step. To generate something new, it runs that process in reverse: start from pure static, denoise repeatedly, and end with audio that matches the prompt. The approach most current models build on was set out in Denoising Diffusion Probabilistic Models by Jonathan Ho, Ajay Jain and Pieter Abbeel, a paper whose results were on images (the CIFAR10 and LSUN datasets). Music models adapted the idea later.

What it looks like in a real music model

Raw audio is long: tens of thousands of samples every second. So most music diffusion models denoise a compressed version instead. The Stable Audio Open 1.0 model card describes exactly this "latent diffusion" setup. An autoencoder squeezes the waveform into a shorter sequence, a T5-based text encoder turns your prompt into numbers, and a transformer diffusion model does the denoising in that compressed space. The card says it makes up to 47 seconds of 44.1 kHz stereo and was trained on about 486,000 Creative Commons-licensed recordings from Freesound and the Free Music Archive (checked 29 Oct 2026).

Not every music generator works this way. Some predict audio tokens one after another, like a language model. Our explainer on how AI music generators work covers both families.

What the model card says it can and cannot do

The same card sets out the limits, which matter more to a producer than the architecture. It lists the intended use as research and experimentation, including text-guided generation "to explore current abilities of generative AI models by machine learning practitioners and artists". It says the model "is not able to generate realistic vocals", was trained on English descriptions and will not perform as well in other languages, "does not perform equally well for all music styles and cultures", and is "better at generating sound effects and field recordings than music". It also says prompt engineering "may be required to obtain satisfying results". For commercial use it points to Stability AI's separate licence rather than the community licence the weights ship under (model card, checked 9 Nov 2026).

How the training data was filtered

The card says the dataset is 486,492 recordings, 472,618 from Freesound and 13,874 from the Free Music Archive, all under CC0, CC BY or CC Sampling+ licences. Before training, Stability AI says it ran Freesound music through Audible Magic's content-identification service to remove suspected copyrighted music (mostly field recordings with music playing in the background), and checked the Free Music Archive subset by metadata search against a large database of copyrighted tracks, with flagged items reviewed by humans. The accompanying paper describes the model as "trained with Creative Commons data" and says most current text-to-audio models "are private and not accessible for artists and researchers to build upon" (Stable Audio Open paper).

What we think

The useful thing about diffusion for musicians is the step-by-step process itself, because you can push on it partway through. Recent research makes an existing model follow a melody by steering it during sampling, with no retraining (see our news on melody guidance for audio diffusion). That kind of control fits a producer's workflow better than typing a prompt and hoping. The catch is the training data. A model card that says exactly what went in, as Stable Audio Open's does, is still rarer than it should be, and it is the first thing we check before recommending a tool. For how the commercial generators compare, see Suno vs Udio vs ElevenMusic.