Abstract
Speaker verification (SV) systems now achieve very low error rates on standard public protocols, but those results do not fully describe deployment generalization. VoxCeleb-style evaluations remain useful anchors for English in-the-wild media speech. Scenario-specific evaluations and challenges are also valuable. However, neither setting directly shows whether the same system remains reliable across diverse deployment scenarios. We introduce OpenSVBench, a public benchmark for cross-scenario SV evaluation. Its fixed public core contains 26 trial sets organized into 11 scenario groups, with 9.15M verification trials, frozen target/non-target keys, and metadata for scenario-wise reporting. Around this core, the online leaderboard acts as a diagnostic evaluation interface rather than only a ranked list. It accepts score packages from new systems, evaluates them with frozen keys and common scoring, and reports trial-set breakdowns, scenario-group metrics, macro EER/minDCF, model-wise profiles, and scenario-wise comparisons. We evaluate 18 public checkpoints spanning x-vector, ECAPA-TDNN, CAMPPlus, ResNet and ERes variants, ReDimNet2, SAM-ResNet, and pretrained Transformer speaker recipes. The results show clear cross-scenario gaps: systems with similar macro EER can have different scenario profiles, strong performance in one scenario does not guarantee strong performance in another, and same-scenario trial sets can still expose source and protocol gaps.
Abstract (translated)
URL
https://arxiv.org/abs/2610.13760
