Abstract
We introduce the Convolutional Set Transformer (CST), a novel neural architecture designed to process image sets of arbitrary cardinality that are visually heterogeneous yet share high-level semantics - such as a common category, scene, or concept. Existing set-input networks, e.g., Deep Sets and Set Transformer, are limited to vector inputs and cannot directly handle 3D image tensors. As a result, they must be cascaded with a feature extractor, typically a CNN, which encodes images into embeddings before the set-input network can model inter-image relationships. In contrast, CST operates directly on 3D image tensors, performing feature extraction and contextual modeling simultaneously, thereby enabling synergies between the two processes. This design yields superior performance in tasks such as Set Classification and Set Anomaly Detection and further provides native compatibility with CNN explainability methods such as Grad-CAM, unlike competing approaches that remain opaque. Finally, we show that CSTs can be pre-trained on large-scale datasets and subsequently adapted to new domains and tasks through standard Transfer Learning schemes. To support further research, we release CST-15, a CST backbone pre-trained on ImageNet (this https URL).
Abstract (translated)
我们介绍了卷积集变压器（Convolutional Set Transformer，CST），这是一种新颖的神经网络架构，旨在处理任意大小、视觉上异质但具备高级语义关联（如共同类别、场景或概念）的图像集合。现有的集输入网络，例如Deep Sets和Set Transformer，仅限于向量输入，并且无法直接处理3D图像张量。因此，它们必须与特征提取器级联使用，通常是CNN，在集输入网络能够建模图间关系之前，将图像编码为嵌入表示。相比之下，CST可以直接操作3D图像张量，在执行特征提取的同时进行上下文建模，从而实现这两种过程的协同作用。这种设计在集合分类和集合异常检测等任务中表现出了更优的效果，并且与仅限于向量输入的方法不同，它原生兼容CNN可解释性方法（如Grad-CAM）。最后，我们展示了CST可以通过大规模数据集预训练，并通过标准的迁移学习方案适应新的领域和任务。为了支持进一步的研究，我们将发布CST-15，这是一个在ImageNet上进行过预训练的CST骨干网络（详见此链接: this https URL）。
URL
https://arxiv.org/abs/2509.22889
