AI music detectors are not proof: what a benchmark of eight of them found

Detector vendors sell certainty. Deezer says its detector is 99.8% accurate. IRCAM Amplify's home page advertises "99% accuracy" and fewer than 1% false positives. On 14 September, Intrect Research published something rarer: eight public detectors run on the same files, with the results table released for anyone to download.

One caveat before the numbers. This isn't an independent referee. Intrect makes ArtifactNet, the detector that came first, and the report page offers its own detector analysis, with the first one free. The data and method are public, which counts for a lot, but read it as a vendor's benchmark that can be checked, not a neutral audit.

What did the benchmark test?

According to Intrect's report, the test set (ArtifactBench v1.1) had 2,104 files: 1,388 AI-generated and 716 real. The run took place on 4 July 2026. Every detector was scored at its public fixed threshold of 0.5, meaning a file counts as "AI" once its score passes the halfway mark.

The results, from Intrect's table:

• ArtifactNet v9.4 (Intrect's own): found 97.3% of the AI tracks, and flagged 13.8% of real tracks as AI.
• Deezer's public research model (a "fakeprint" logistic regression): 64.6% recall and a 13.0% false-positive rate on real music.
• FST (Mippia): the fewest false alarms, 1.8%, but it caught only 58.7% of the AI tracks.
• CLAM and DeepFense: flagged 69.7% and 97.8% of the real tracks as AI.

By our arithmetic, a 13% false-positive rate on 716 real files is about 93 human-made tracks wrongly flagged. Even the winner flagged about 99. DeepFense flagged almost all of them.

Why do vendors say 99%?

They're measuring something different. Deezer's own research paper is the clearest example. Its ISMIR 2025 paper describes the method behind the public model and reports detection "surpassing 99% accuracy on several scenarios". On the Suno and Udio versions it was trained on, it scored 100%. On Udio 32, a version it hadn't seen in training, it detected 39.83%. The authors suggest Udio may have changed its architecture between versions.

That's the pattern. A detector is very good at the generators it has learned and much less reliable on anything new. A benchmark built from different sources, like Intrect's, finds that weak spot. A vendor quoting accuracy on its own test data mostly won't.

Two things are needed to be fair to Deezer. First, the model Intrect tested is the public research release from that paper (the paper links its code on GitHub), which the authors call a "by-product" of their analysis. It may not be the system Deezer runs on its service, and Intrect didn't test Deezer's production detector. So this result doesn't show that Deezer's platform flags 13% of human music. Second, Deezer has published its claimed error rates: it may miss about 2 in 1,000 AI tracks and wrongly flag fewer than 1 in 10,000 human ones. It also applies its AI tag per album to cut errors. But nobody outside Deezer has checked those numbers.

Does the threshold matter?

It matters a lot, and you will rarely be told what it is. Intrect makes this point itself: the right threshold depends on whether the detector is used as a triage signal, an investigative aid, or one input to a human appeal process. FST and ArtifactNet show the trade-off. Strict settings miss AI, loose ones accuse humans. A distributor or store picks a point on that curve based on its own costs, and the artist doesn't see the choice.

Intrect's conclusion is the line worth keeping: "Detection is not provenance." It says a score should not replace Content Credentials, platform records or a human review with an appeal route.

What to do if your release is flagged as AI

1. Get the decision in writing. Reply to the notice and ask three things: which tool flagged the track, whether a person reviewed it, and what the score or category was. A message like "flagged by our systems" tells you none of that.
2. Answer with files, not arguments. Send dated project files, stems, rough bounces, takes, and sample and plug-in licences. If you work in a browser DAW, the export menu is your friend: Veena, for example, can export every track as stems alongside the WAV and MIDI.
3. Point to the evidence on detectors. Cite this benchmark politely. Some public detectors flagged most real music, and Intrect, a detector maker itself, says scores are evidence to review, not a verdict to enforce.
4. Don't re-upload the same file and hope. The same audio will get the same score. Don't run it through an "artifact removal" tool either, which makes you look like you're hiding something.
5. Disclose honestly if AI is in the mix. If a generated element made the final master, declaring it is a stronger position than a detector finding it. Our guide to AI credits and DDEX disclosure covers the forms.
6. Escalate if you get no answer. Ask your distributor to raise it with the store. Keep every email.

Checklist before every release:

• Session files and stems archived, with dates
• Sample and plug-in licences saved
• AI use, if any, declared in the distributor's form
• Notes on which tools you used and where

This is general information, not legal advice.

What we think

A detector score is a smoke alarm, not a court. It's useful when a person checks what set it off, and harmful when it decides alone. On these numbers, the top-ranked public model wrongly flagged about one real track in seven at its default setting. That's a lot of wrongly accused singles at upload scale.

The bigger problem is the information gap. Generators already have their own detectors, and platforms have private ones. Artists have neither the scores nor the thresholds. Any platform that acts on a detector should publish three things: its false-positive rate on human-made music at the threshold it actually uses, the test set behind that number, and a human appeal with a deadline. Intrect deserves credit for making its benchmark data public. Now someone without a detector to sell needs to run it again.

Update, 23 October 2026. On 20 September Intrect's Heewon Oh posted the ArtifactBench paper, a v2 test of four detectors on 828 tracks. It states plainly that the public research version of Deezer's method "is not Deezer's production detector". At a threshold tuned on calibration data, the public Deezer model wrongly flagged 2.0% of one set of real tracks (FMA) but 21.1% of web-sourced "hard negatives". In other words, the false-positive rate depends on which real music you test. On 15 October Deezer repeated its 99.8% figure when its tool was named to TIME's Best Inventions of 2026. For how distributors use detectors and the appeal steps at DistroKid, see how distributors decide your song is "AI".