Abstract
Deep implicit functions have been found to be an effective tool for efficiently encoding all manner of natural signals. Their attractiveness stems from their ability to compactly represent signals with little to no off-line training data. Instead, they leverage the implicit bias of deep networks to decouple hidden redundancies within the signal. In this paper, we explore the hypothesis that additional compression can be achieved by leveraging the redundancies that exist between layers. We propose to use a novel run-time decoder-only hypernetwork - that uses no offline training data - to better model this cross-layer parameter redundancy. Previous applications of hyper-networks with deep implicit functions have applied feed-forward encoder/decoder frameworks that rely on large offline datasets that do not generalize beyond the signals they were trained on. We instead present a strategy for the initialization of run-time deep implicit functions for single-instance signals through a Decoder-Only randomly projected Hypernetwork (D'OH). By directly changing the dimension of a latent code to approximate a target implicit neural architecture, we provide a natural way to vary the memory footprint of neural representations without the costly need for neural architecture search on a space of alternative low-rate structures.
Abstract (translated)
深度隐含函数已被证明是一种有效的工具，用于高效地编码所有类型的自然信号。它们的吸引力源于它们能够以少量的离线训练数据来压缩信号。相反，它们利用深度网络的隐含偏见来解耦信号中的隐藏冗余。在本文中，我们探讨了通过利用层之间的冗余性来获得额外压缩的可能性。我们提出了一个新型的仅运行时解码器-仅网络 - 用于更好地建模跨层参数冗余。以前使用具有深度隐含函数的过网络的应用程序依赖于大型离线训练数据集，这些数据集不适用于它们所训练的信号。我们相反提出了一个通过Decoder-Only随机投影Hypernetwork（D'OH）初始化运行时深度隐含函数的策略。通过直接将潜在代码的维度变换为近似目标隐含神经架构，我们提供了在不需要进行神经架构搜索的情况下自然地变化神经表示的内存足迹的方法。
URL
https://arxiv.org/abs/2403.19163
