MI-MIDI 2 paper tests whether sparse autoencoders can steer text-to-MIDI models

A paper posted to arXiv on 23 October, MI-MIDI 2: From Observation to Control with Sparse Autoencoders in Text-to-MIDI Generation by Jakub Poćwiardowski and Mateusz Modrzejewski, asks a practical question: can you find the parts of a text-to-MIDI model that stand for a musical trait, and then turn them up to get more of that trait?

What the researchers did

A sparse autoencoder (SAE) breaks a model's internal activity into many "features", each of which tends to fire for something specific. The authors trained SAEs on two text-to-MIDI models, text2midi and MIDI-LLM, and described each feature in three ways: how it correlates with musical metrics, which instrument programs the examples it fires on share, and automatically written descriptions, per the abstract.

Each of those descriptions came with its own control, such as adjusting for note density or comparing against matched excerpts where the feature is off. Then came the real test: intervening on a feature during generation and measuring the resulting MIDI against unmodified output and against a random direction of the same size.

What held up

• Instrumentation. Automated descriptions beat permutation controls for instruments. A feature described as a synth-lead program brought that program into the output in a pre-registered confirmation and again on a separate set of prompts, and similar features showed up in independently trained dictionaries.
• Other traits. Features chosen by density-adjusted correlations nudged their target metric up a little as a group. Features chosen by raw correlations did nothing.

The authors also published an online atlas with MIDI, piano-roll and audio examples for every feature in the screen, the abstract says.

Why producers should care

MIDI is the format you can actually work with: drag it into a DAW, swap the sound, fix the wrong note, change the voicing. That is the big advantage over generated audio, as our text-to-MIDI vs text-to-audio explainer sets out. But a MIDI generator is only useful if you can ask for what you want, more lead, sparser drums, busier bass, and get it reliably. Today that mostly means rewording the prompt and rolling again.

Feature steering points to a different kind of control: a knob tied to something the model has learned, rather than a hope that the prompt is read the way you meant.

Our call

This is a careful paper, and the care is the news. Its strongest finding is about one instrument feature, and its broader effect is described as small. The fact that raw correlations failed while density-adjusted ones worked is a useful warning: a lot of "this neuron means X" claims in music AI are probably just picking up how many notes there are.

We wouldn't expect steering sliders in a plugin soon. We do think this is the right direction for tools aimed at producers: tested, specific controls over generated MIDI, with a public atlas you can listen to, rather than another demo of a model making a whole track from one line of text.