AI voice conversion explained: how RVC-style models swap one voice for another, and where consent and the law come in

AI voice conversion takes a performance in one voice and re-renders it so it sounds like another, keeping the words, melody and timing. It is the technology behind most "AI covers" and behind the voice-changer plug-ins now sold to producers. This guide explains how the most widely used open-source approach, RVC, works, why it needs so little audio, and what consent and the law ask of you. Everything is as we read the sources on 27 Oct 2026.

Conversion is not text-to-speech

Text-to-speech and singing synthesis generate a voice from text or notes. Voice conversion starts from a real recording. You (or a guide vocalist) sing the line, and the model replaces the timbre while the phrasing, pitch and emotion come from the original take. That is why converted vocals often sound more human than fully generated ones: a human performance is still underneath.

How RVC works, in plain terms

RVC stands for Retrieval-based Voice Conversion. Its project page is MIT-licensed and describes a pipeline built from a few parts:

1. Content features. A speech model (the project requires hubert_base for "inference and feature extraction") turns the input vocal into a representation of what is being sung, with as little of who is singing as possible.
2. Pitch. A pitch tracker follows the melody. RVC uses RMVPE, a vocal pitch-extraction method presented at InterSpeech 2023, which the project calls fast and light on resources.
3. Retrieval. This is the part in the name. The project says it uses top-1 retrieval to replace the input's features with the closest matching features from the training set, "to prevent timbre leakage", meaning less of the original singer bleeds through.
4. Re-synthesis. A decoder rebuilds audio from those features and the pitch line in the target voice. The project lists VITS and HiFi-GAN among the work it builds on.

It also ships a real-time voice-changer interface, a model-merging option for blending voices, and vocal/accompaniment separation so you can pull a vocal off a finished mix first.

Why ten minutes matters

The RVC page says you can "easily train a good VC model with voice data <= 10 mins", and recommends collecting at least 10 minutes of low-noise speech. The heavy lifting was done once: the base model was trained on nearly 50 hours of the open VCTK dataset, and a new voice is a fine-tune on top.

What we think: that ten-minute figure is the whole story for working artists. Ten clean minutes of a singer's voice is one isolated vocal stem, a few podcast interviews or a live session. If you have released music, enough of your voice to train a usable model is already public. Technical gatekeeping is not going to protect anyone; consent norms, contracts and law have to.

Consent: the line that matters

There are legitimate uses. Converting your own guide vocal into your own polished voice, demoing a song in the voice of a singer who has agreed to it, or licensing your voice to a tool on terms you set (see how to license your voice for AI) are all consent-first. Training on a famous singer's isolated stems and releasing the result is not, however good it sounds.

Our rule of thumb: if you would need permission to put a session singer's name on the track, you need permission to use a model of their voice.

What the law says

• United States, Tennessee. The ELVIS Act (Ensuring Likeness Voice and Image Security Act), signed on 21 Mar 2024, updated the state's Protection of Personal Rights law, which already covered name, image and likeness, to add "voice", aimed at generative AI clones.
• European Union. Article 50 of the AI Act requires deployers who create or alter deep fake audio to disclose that it "has been artificially generated or manipulated". A deep fake is content that "resembles existing persons" and would falsely appear authentic. For evidently artistic, creative or satirical work the duty is lighter: disclose in a way that "does not hamper the display or enjoyment of the work".

Laws differ by country and US state and are still moving. For the full patchwork, the federal NO FAKES Act, platform takedown routes and practical steps, read how to protect your voice from AI cloning. None of this is legal advice.

A short checklist before you convert a voice

• Is it your voice, or do you have written permission that covers this use and release?
• Will you label the track as using a converted voice? In the EU, deepfake audio needs disclosure.
• Did the training audio come from stems you have the rights to?
• If a distributor asks about AI use, answer honestly.