Abstract
Recent breakthroughs in text-to-speech (TTS) voice cloning have raised serious privacy concerns, allowing highly accurate vocal identity replication from just a few seconds of reference audio, while retaining the speaker's vocal authenticity. In this paper, we introduce CloneShield, a universal time-domain adversarial perturbation framework specifically designed to defend against zero-shot voice cloning. Our method provides protection that is robust across speakers and utterances, without requiring any prior knowledge of the synthesized text. We formulate perturbation generation as a multi-objective optimization problem, and propose Multi-Gradient Descent Algorithm (MGDA) to ensure the robust protection across diverse utterances. To preserve natural auditory perception for users, we decompose the adversarial perturbation via Mel-spectrogram representations and fine-tune it for each sample. This design ensures imperceptibility while maintaining strong degradation effects on zero-shot cloned outputs. Experiments on three state-of-the-art zero-shot TTS systems, five benchmark datasets and evaluations from 60 human listeners demonstrate that our method preserves near-original audio quality in protected inputs (PESQ = 3.90, SRS = 0.93) while substantially degrading both speaker similarity and speech quality in cloned samples (PESQ = 1.07, SRS = 0.08).
Abstract (translated)
最近在文本到语音（TTS）声音克隆领域的突破引发了严重的隐私担忧，仅通过几秒钟的参考音频即可实现高度准确的声音身份复制，并保留发言人的声音真实性。本文介绍了CloneShield，这是一个专门设计用于防御零样本声音克隆的通用时域对抗性扰动框架。我们的方法提供了一种无需任何关于合成文本的先验知识就能在不同说话人和话语中保持稳健性的保护方式。我们通过将扰动生成形式化为一个多目标优化问题，并提出多梯度下降算法（MGDA）来确保对各种话语具有鲁棒性的保护措施。 为了保留用户自然的听觉感知，我们将对抗性扰动分解为梅尔频谱图表示，并针对每个样本进行微调。这种设计在保证不可察觉的同时，还能保持对零样本克隆输出的强大破坏效果。我们在三个最先进的零样本TTS系统、五个基准数据集以及来自60名人类听众的评估中进行了实验，结果表明我们的方法可以保留受保护输入接近原始音频的质量（PESQ = 3.90, SRS = 0.93），同时大幅降低克隆样本中的说话人相似性和语音质量（PESQ = 1.07, SRS = 0.08）。
URL
https://arxiv.org/abs/2505.19119
