How AI music generators work: language models, diffusion and the data behind them

Type "warm lo-fi beat with dusty piano" and thirty seconds later you have audio. It feels like magic, but the systems behind it come in two main families, and knowing which one you are using tells you a lot about what it can and cannot do for you.

This guide sticks to systems whose builders have published how they work. Commercial products often don't say, so treat any claim about a closed product's internals as a guess.

The shared problem: audio is huge

One second of CD-quality stereo is 88,200 numbers. No model predicts that directly. So both families first squash audio into a smaller representation, generate in that smaller space, and then decode back to sound. The difference is how they generate.

Family 1: language models over audio tokens

A language model predicts the next piece of a sequence, given everything before it. For text the pieces are word fragments. For music, a neural codec first turns audio into a stream of discrete tokens, and the model learns to predict the next token.

Meta's MusicGen paper is the clearest public example. It describes a single-stage transformer language model that works over several streams of compressed, discrete music tokens, using interleaving patterns so it doesn't need a cascade of separate models. The model card says the tokens come from an EnCodec model.

What this means for you: the model builds the track left to right, a little like writing a sentence. That makes continuation natural (give it a start, it carries on), and it makes conditioning on something time-shaped, like a melody, fit well.

Family 2: diffusion

A diffusion model learns to remove noise. In training, it sees real audio with noise added and learns to undo it; at generation time it starts from pure noise and cleans it up over many steps, steered by your prompt. It shapes the whole clip at once rather than left to right.

Google's Noise2Music chains diffusion models: one generates an intermediate representation (a spectrogram or lower-fidelity audio) from the text, and a second "cascader" turns that into high-fidelity audio, producing 30-second clips. Stability AI's Stable Audio instead runs diffusion in the compressed "latent" space of an autoencoder and adds timing embeddings, so you can ask for a length. The paper reports up to 95 seconds of 44.1 kHz stereo in about eight seconds on one A100 GPU.

What this means for you: duration and overall texture are easy to steer, and the model is good at sounds and beds. "Carry on from bar 16" is less natural than with a token model, though tools add tricks to approximate it.

Where the training data comes from

A model can only make what resembles what it has heard. The published cards show how different the inputs can be:

Noise2Music shows another common move: its paper says pretrained language models wrote the text descriptions paired with the training audio. So the "captions" a model learned from may themselves be machine-written, which is one reason prompt wording can feel hit-and-miss.

Both MusicGen's and Stable Audio Open's cards state that the model cannot generate realistic vocals, a plain consequence of their data. MusicGen's card adds that it works best with English prompts and does not perform equally across styles and cultures. Whether training on unlicensed recordings is lawful is a separate fight; see our guide to AI training data licensing.

What controls you actually get

• Text prompt. Every system here. It steers style and mood, not exact notes.
• Melody or audio conditioning. MusicGen accepts melodic features alongside text.
• Length. Explicit in Stable Audio via timing conditioning (Stable Audio Open's card lists up to 47 seconds).
• What you never get from audio models: separate, editable notes. The output is a recording. If you want pitches and velocities you can change, that is a symbolic model; our text-to-MIDI vs text-to-audio guide covers that route.

What we think

The "LM vs diffusion" split matters less to an artist than two plainer questions: what was it trained on, and what can I steer? A model trained on 20,000 hours of stock-library music will tend to sound like stock-library music, however clever its architecture. And a model whose only control is a text box gives you a lottery ticket, not an instrument.

So our advice is to prefer tools that publish a model card naming their data, and tools that take your material as input (a melody, a sketch, a length, a reference) rather than words alone. The more of the decision you hand in, the more of the result is yours, creatively and, as the copyright cases are showing, possibly legally too. A tool that won't say what it learned from is asking you to take a risk it won't describe.