How AI stem separation works, and a stem-splitting workflow that survives the artefacts

A stem splitter takes a finished stereo mix and hands back separate tracks, usually vocals, drums, bass and "other". That isn't undoing the mix. A splitter makes an educated guess at each part, and every guess has errors. The useful skill isn't finding a splitter with no artefacts. It's building a workflow where the artefacts don't matter.

How does a stem splitter work?

Modern splitters are neural networks trained on pairs: a full mix, plus the isolated parts that made it. From enough examples, they learn what a vocal or a kick drum tends to look like inside a mix.

Deezer's open-source Spleeter is a good place to start, because its paper (published 24 Jun 2020) explains the method plainly. The audio is turned into a spectrogram, a picture of which frequencies are sounding at each moment. For each stem, a 12-layer U-Net (a type of image-processing network) estimates a soft mask. Think of it as a stencil that decides what share of each point in the picture belongs to the vocal, the drums and so on. The models came in 2-stem (vocals and accompaniment), 4-stem and 5-stem (adding piano) versions. They were trained on Deezer's internal datasets, which the authors say they couldn't release "for copyright reasons".

What changed from Spleeter to Demucs to today?

Meta's Demucs took a different route. Version 4, described in a November 2022 paper, runs two branches at once: one works on the raw waveform, the other on the spectrogram. A Transformer in the middle lets each branch learn from the other. In the comparison table in the Demucs README, the fine-tuned v4 model scores 9.0 dB average signal-to-distortion ratio (SDR, a standard accuracy measure where higher is better) on the MUSDB test set. Spleeter scores 5.9, even though the same table lists it as trained on 25,000 extra songs against Demucs's 800. That's the Demucs authors' own table, on a 50-song test set, so read it as a direction, not a verdict.

The research has moved on since then. BS-RoFormer, the model behind the SAMI-ByteDance entry to a 2023 challenge, splits the spectrogram into frequency bands and models them with Transformers. Its paper says a version trained on MUSDB18HQ plus 500 extra songs ranked first in the music separation track of the 2023 Sound Demixing Challenge.

Why do artefacts happen?

Four reasons, and each one points to a fix.

Overlap. When a vocal and a guitar share the same frequencies at the same moment, a mask has to divide energy it can't truly attribute. Researchers measure the two resulting failures separately: interference, which you hear as bleed from other instruments, and artefacts, the watery or phasey sound left behind. The Spleeter paper reports both, as SIR and SAR.

Training data. The benchmark nearly everyone reports on is MUSDB18: 150 tracks, about 10 hours, split into just four stems (drums, bass, vocals and other), and licensed for academic use only. That's where models have had the most to learn from. Ask for more and quality drops: Demucs's own experimental six-stem model shows, in its authors' words, "okay quality for guitar, but a lot of bleeding and artifacts for the piano source".

Level. Separation errors can push a stem past full scale. The Demucs README says it rescales each stem by default to avoid clipping, which "can however break the relative volume between stems".

Source quality. The standard MUSDB18 files are AAC at 256 kbps, and the dataset's maintainers say that compression limits the bandwidth to 16 kHz, which is why they also publish an uncompressed HQ version. Our inference: a splitter can't recover detail an encoder has already thrown away, so a lossy file is a worse starting point.

A workflow that survives the artefacts

1. Use the best source you have. If it's your own track, export the real stems from the session; no splitter beats that. Otherwise, use a lossless file (WAV, AIFF or FLAC) rather than a stream rip or an MP3.
2. Ask for as few stems as the job needs. For an acapella or an instrumental, use a two-stem mode. Demucs has --two-stems=vocals, for example. Our reasoning: every extra stem is one more split the model can get wrong, and the four core stems are what the models were mostly trained on.
3. Pick the tool for the job. To learn a part, Moises says it separates 27 stem types and adds practice tools such as a speed changer and chord finder. To fix balance in a finished mix, Music Rebalance in iZotope's RX relevels vocals, bass, percussion and other instruments, offline or as a real-time plugin; it's in the Standard and Advanced editions of RX 12. If your DAW has a splitter built in, start there. Ableton Live Suite has four-stem separation from version 12.3, running locally and powered by Music AI, the company behind Moises. Logic Pro's Stem Splitter launched in May 2024 with four stems, processed on the device (it needs an M-series Mac or iPad). For a free, local, scriptable option there's Demucs.
4. Try two models and keep the best stem from each. Demucs alone ships several, and it says its fine-tuned htdemucs_ft takes four times as long but "might be a bit better". Different models fail on different stems.
5. Check what was lost. Our suggestion is a null test. Sum the stems, flip the polarity of the original mix and play both together. Whatever is left is what the splitter dropped or invented. With Demucs, pass --clip-mode clamp or lower the input level first, or the default rescaling will spoil the test.
6. Fix bleed in context, not in solo. A stem that sounds watery soloed can sit fine under new parts. EQ, gate or spectrally edit the worst bits, and arrange so something covers the gaps.
7. Write down what made each stem. Tool, model and settings, so you can redo a stem and answer questions when you clear a sample.

Should you split locally or in the cloud?

For anything unreleased, our view is that local should be the default. Demucs runs on your machine. It needs about 7 GB of GPU memory with default settings (3 GB at a smaller segment size), or it runs on the CPU at roughly 1.5 times the track's length. Ableton's and Logic's built-in splitters are local too. LALAL.AI's local model, Lyra, needs a Pro subscription, and LALAL.AI itself recommends its cloud models for final mixes and pre-release mastering. We covered that trade-off when LALAL.AI launched its plugin.

The cloud can offer more: LALAL.AI's cloud splitter lists synth, string and wind stems, which Lyra doesn't. But you're handing over the file. Their training terms differ in ways that matter, and we compare them line by line in our desk review of stem splitters.

Can you split someone else's record?

You can split it to learn from it. Releasing what you make from it is another matter. The U.S. Copyright Office explains that a song and a recording of it are two separate works for copyright purposes. U.S. law gives the owner of a sound recording the exclusive right to make versions in which its actual sounds are "rearranged, remixed, or otherwise altered in sequence or quality". Our reading, which isn't legal advice: an acapella you pulled from a record still belongs to both rights holders, and a flip that uses it needs both cleared. The tools put the risk on you. AudioShake's terms, for example, make you warrant that you have "all rights required" to upload and process the content. An easier tool doesn't change the clearance question.

What we think

Treat every stem as a draft. The field's main benchmark is four stems from 150 songs, so claims of dozens of stem types are claims about the long tail, where models have the least to learn from. Judge a splitter stem by stem, on your own material, and design your parts around its weak spots.

Separation hasn't made rights clearance any easier. It has made it matter more often. The technical barrier to lifting a vocal is gone. That means artists should assume anyone can extract the vocal from any release, and should decide on purpose whether to offer official stems or acapellas on terms they set. That's better than leaving the field to whoever has a splitter and no licence.