Abstract
All prior membership inference attacks for fine-tuned language models use hand-crafted heuristics (e.g., loss thresholding, Min-K\%, reference calibration), each bounded by the designer's intuition. We introduce the first transferable learned attack, enabled by the observation that fine-tuning any model on any corpus yields unlimited labeled data, since membership is known by construction. This removes the shadow model bottleneck and brings membership inference into the deep learning era: learning what matters rather than designing it, with generalization through training diversity and scale. We discover that fine-tuning language models produces an invariant signature of memorization detectable across architectural families and data domains. We train a membership inference classifier exclusively on transformer-based models. It transfers zero-shot to Mamba (state-space), RWKV-4 (linear attention), and RecurrentGemma (gated recurrence), achieving 0.963, 0.972, and 0.936 AUC respectively. Each evaluation combines an architecture and dataset never seen during training, yet all three exceed performance on held-out transformers (0.908 AUC). These four families share no computational mechanisms, their only commonality is gradient descent on cross-entropy loss. Even simple likelihood-based methods exhibit strong transfer, confirming the signature exists independently of the detection method. Our method, Learned Transfer MIA (LT-MIA), captures this signal most effectively by reframing membership inference as sequence classification over per-token distributional statistics. On transformers, LT-MIA achieves 2.8$\times$ higher TPR at 0.1\% FPR than the strongest baseline. The method also transfers to code (0.865 AUC) despite training only on natural language texts. Code and trained classifier available at this https URL.
Abstract (translated)
此前针对微调语言模型的所有成员推断攻击均依赖手工设计的启发式规则（例如损失阈值法、Min-K%、参考校准），其性能受设计者直觉的限制。我们提出了首个可迁移的习得攻击，其核心发现是：任何模型在任何语料库上微调时，由于成员身份在构造时已知，因此能产生无限标记数据。这消除了影子模型瓶颈，将成员推断带入深度学习时代——从“设计特征”转向“学习特征”，通过训练多样性和规模实现泛化。我们发现，微调语言模型会产生一个跨架构家族和数据域可检测的恒定记忆签名。我们仅在基于Transformer的模型上训练成员推断分类器，并实现零样本迁移至Mamba（状态空间模型）、RWKV-4（线性注意力）和RecurrentGemma（门控循环网络）， respective AUC分别达到0.963、0.972和0.936。每次评估均结合训练中未见的架构与数据集，但三者性能均超过在保留Transformer上的表现（0.908 AUC）。这四类架构共享无计算机制，其唯一共同点是基于交叉熵损失的梯度下降。即使简单基于似然的方法也展现出强迁移性，证实该签名独立于检测方法存在。我们的方法——习得迁移成员推断（LT-MIA）——通过将成员推断重构为基于每标记分布统计的序列分类任务，最有效地捕获该信号。在Transformer上，LT-MIA在0.1%假阳性率下实现比最强基线高2.8倍的真阳性率。该方法即使仅使用自然语言文本训练，也能迁移至代码领域（0.865 AUC）。代码与训练好的分类器可通过此链接获取。
URL
https://arxiv.org/abs/2604.03199
