Abstract
Dimensionality reduction can distort vector space properties such as orthogonality and linear independence, which are critical for tasks including cross-modal retrieval, clustering, and classification. We propose a Relationship Preserving Loss (RPL), a loss function that preserves these properties by minimizing discrepancies between relationship matrices (e.g., Gram or cosine) of high-dimensional data and their low-dimensional embeddings. RPL trains neural networks for non-linear projections and is supported by error bounds derived from matrix perturbation theory. Initial experiments suggest that RPL reduces embedding dimensions while largely retaining performance on downstream tasks, likely due to its preservation of key vector space properties. While we describe here the use of RPL in dimensionality reduction, this loss can also be applied more broadly, for example to cross-domain alignment and transfer learning, knowledge distillation, fairness and invariance, dehubbing, graph and manifold learning, and federated learning, where distributed embeddings must remain geometrically consistent.
Abstract (translated)
维度减少可能会扭曲向量空间的性质，如正交性和线性独立性，这些性质对于跨模态检索、聚类和分类等任务至关重要。我们提出了一种关系保持损失（RPL），这是一种通过最小化高维数据的关系矩阵（例如Gram矩阵或余弦相似度矩阵）与其低维嵌入之间的差异来保留这些属性的损失函数。RPL 用于训练神经网络进行非线性投影，并且其误差界限是基于矩阵扰动理论得出的。初步实验表明，尽管减少了嵌入维度，但 RPL 能够在很大程度上保持下游任务的表现，这可能是由于它保留下了关键的向量空间属性。虽然这里描述了 RPL 在降维中的应用，但它也可以更广泛地使用，例如用于跨域对齐和迁移学习、知识蒸馏、公平性和不变性、去中心化处理、图与流形学习以及联邦学习等场景，在这些场景中分布式嵌入必须保持几何一致性。
URL
https://arxiv.org/abs/2509.01198
