SMEBench: a new benchmark asks whether AI can edit music without wrecking the rest of it

Most AI music tools hand you a finished stereo file. If the bass line is wrong, you prompt again and get a different song. A new paper, SMEBench, posted to arXiv on 27 October 2026, tries to measure the opposite skill: changing one thing in a piece of audio and leaving everything else as it was.

What the researchers built

The authors, from Mila, Université Laval, Concordia, Maynooth University and French CNRS labs (paper), start from MIDI performances rather than recordings. They make a controlled change in the MIDI, then render both the original and the edited version to audio. That gives every example a source, an instruction, and a "correct answer" target to compare against (abstract).

The edits will sound familiar to anyone who has worked in a DAW. There are 12 operations (paper):

• Instruments: remove, replace, extract or double an instrument
• Drums: remove, replace or extract the drums
• Articulation: make a part staccato or legato
• Dynamics: crescendo or decrescendo
• Pitch: pitch correction

The balanced test set has 3,600 examples, plus a smaller set made from real multitrack recordings (abstract). The data is posted on Hugging Face.

What they found

Six editing systems were tested: SDEdit, DDIM inversion, ZETA, Virtual Consistency Inversion (VCI), SteerMusic and MusicGen. The headline is that the winner depends on the scorecard. Given an edit caption, a text-matching score (CLAP-Text) favoured MusicGen, while both scores that compare against the paired target favoured SteerMusic. Given an instruction instead, CLAP-Text favoured VCI and the target-based scores favoured ZETA (paper).

The paper also points out a trap: metrics that only check how close the output stays to the original can reward a system for doing nothing, because returning the input untouched scores perfectly on preservation (paper). In a 17-person listening test, the rendered targets themselves did not reliably read as following the instruction for legato, remove instrument and replace drums (paper), so even the "answers" are not perfect for every edit type.

What we think

This is the right question for producers. A flat-file generator is a slot machine: every pull is a new song. Real production is revision, where you fix a part and keep the rest. A benchmark with a known correct target is how researchers can tell editing from regenerating, and the finding that rankings flip by metric is a warning to read any "best AI editor" claim carefully.

Two caveats. The examples are synthesised from MIDI, which is clean in a way a dense mix is not, and the paper is a preprint, not yet peer-reviewed. Our view is that, for now, the most reliable way to "edit" music is still to keep it editable from the start: MIDI and stems you can open, not audio you have to ask a model to unpick. Tools that learn to pass SMEBench-style tests would narrow that gap, and that is worth watching.