Abstract
We present the first systematic study of Sparse Autoencoders (SAEs) on video representations. Standard SAEs decompose video into interpretable, monosemantic features but destroy temporal coherence: hard TopK selection produces unstable feature assignments across frames, reducing autocorrelation by 36%. We propose spatio-temporal contrastive objectives and Matryoshka hierarchical grouping that recover and even exceed raw temporal coherence. The contrastive loss weight controls a tunable trade-off between reconstruction and temporal coherence. A systematic ablation on two backbones and two datasets shows that different configurations excel at different goals: reconstruction fidelity, temporal coherence, action discrimination, or interpretability. Contrastive SAE features improve action classification by +3.9% over raw features and text-video retrieval by up to 2.8xR@1. A cross-backbone analysis reveals that standard monosemanticity metrics contain a backbone-alignment artifact: both DINOv2 and VideoMAE produce equally monosemantic features under neutral (CLIP) similarity. Causal ablation confirms that contrastive training concentrates predictive signal into a small number of identifiable features.
Abstract (translated)
我们首次对视频表示上的稀疏自编码器（SAEs）进行了系统性研究。标准SAEs能将视频分解为可解释的单语义特征，但破坏了时间连贯性：硬TopK选择导致帧间特征分配不稳定，使自相关性降低36%。我们提出了时空对比目标与马特罗什卡分层分组方法，恢复并甚至超越了原始的时间连贯性。对比损失权重可调节重建效果与时间连贯性之间的权衡。在两个主干网络和两个数据集上的系统性消融实验表明，不同配置在不同目标上各具优势：重建保真度、时间连贯性、动作判别或可解释性。对比SAE特征相比原始特征将动作分类性能提升3.9%，文本-视频检索性能最高提升至2.8倍R@1。跨主干分析揭示标准单语义性指标存在主干对齐伪影：在CLIP相似度下，DINOv2与VideoMAE产生的单语义特征程度相当。因果消融实验证实，对比训练将预测信号集中到少量可识别的特征中。
URL
https://arxiv.org/abs/2604.03919
