Abstract
Conventional machine learning algorithms have traditionally been designed under the assumption that input data follows a vector-based format, with an emphasis on vector-centric paradigms. However, as the demand for tasks involving set-based inputs has grown, there has been a paradigm shift in the research community towards addressing these challenges. In recent years, the emergence of neural network architectures such as Deep Sets and Transformers has presented a significant advancement in the treatment of set-based data. These architectures are specifically engineered to naturally accommodate sets as input, enabling more effective representation and processing of set structures. Consequently, there has been a surge of research endeavors dedicated to exploring and harnessing the capabilities of these architectures for various tasks involving the approximation of set functions. This comprehensive survey aims to provide an overview of the diverse problem settings and ongoing research efforts pertaining to neural networks that approximate set functions. By delving into the intricacies of these approaches and elucidating the associated challenges, the survey aims to equip readers with a comprehensive understanding of the field. Through this comprehensive perspective, we hope that researchers can gain valuable insights into the potential applications, inherent limitations, and future directions of set-based neural networks. Indeed, from this survey we gain two insights: i) Deep Sets and its variants can be generalized by differences in the aggregation function, and ii) the behavior of Deep Sets is sensitive to the choice of the aggregation function. From these observations, we show that Deep Sets, one of the well-known permutation-invariant neural networks, can be generalized in the sense of a quasi-arithmetic mean.
Abstract (translated)
传统的机器学习算法通常假定输入数据遵循向量基格式，重点关注向量基范式。然而，随着涉及基于集的输入任务的需求不断增加，研究社区已经发生了范式转变，重点解决这些挑战。近年来，深度设置（如Deep Sets和Transformer）的出现为处理集数据提供了显著的进步。这些架构特意设计成能自然适应输入集，从而实现更有效的集结构表示和处理。因此，针对各种涉及近似集函数的任务，研究兴趣迅速增加。本全面调查旨在为神经网络 approximate set functions的问题提供概述，并探讨相关研究进展。通过深入研究这些方法并阐明相关挑战，调查旨在为读者提供对领域的全面理解。通过这种全面的视角，我们希望研究人员能够深入了解集-基神经网络的潜在应用、固有局限性和未来发展方向。事实上，从本次调查中我们获得了两个洞察：i）深度设置及其变体可以通过聚合函数的差异进行泛化；ii）深度设置的行为对聚合函数的选择敏感。从这些观察结果中，我们证明了深度设置，这是一种著名的不变序列神经网络，在泛化意义上是准算术平均。
URL
https://arxiv.org/abs/2403.17410
