Abstract
Binary concepts are empirically used by humans to generalize efficiently. And they are based on Bernoulli distribution which is the building block of information. These concepts span both low-level and high-level features such as "large vs small" and "a neuron is active or inactive". Binary concepts are ubiquitous features and can be used to transfer knowledge to improve model generalization. We propose a novel binarized regularization to facilitate learning of binary concepts to improve the quality of data generation in autoencoders. We introduce a binarizing hyperparameter $r$ in data generation process to disentangle the latent space symmetrically. We demonstrate that this method can be applied easily to existing variational autoencoder (VAE) variants to encourage symmetric disentanglement, improve reconstruction quality, and prevent posterior collapse without computation overhead. We also demonstrate that this method can boost existing models to learn more transferable representations and generate more representative samples for the input distribution which can alleviate catastrophic forgetting using generative replay under continual learning settings.
Abstract (translated)
二进制概念是人类以经验方式高效地泛化的基础。它们基于伯努利分布,是信息的基本构成单元。这些概念涵盖了低级和高级特征,如“大与小”和“神经元是否活跃或静止”。二进制概念是普遍存在的特征,可以用于传递知识以提高模型泛化能力。我们提出了一种新的二进制归一化方法,以促进二进制概念的学习,改善自动编码器中数据生成质量。我们在数据生成过程中引入了一个二进制归一化超参数$r$,以对称地分离隐层空间。我们证明,这种方法可以轻松地应用于现有的变体自编码器(VAE)中,以鼓励对称分离,提高重建质量,并防止后遗聚合。我们还证明,这种方法可以提高现有模型的学习可移植表示能力,生成更多的代表性样本,从而减轻使用生成回放在持续学习设置下引起的灾难性遗忘。
URL
https://arxiv.org/abs/2303.12255
