Abstract
For natural language understanding and generation, embedding concepts using an order-based representation is an essential task. Unlike traditional point vector based representation, an order-based representation imposes geometric constraints on the representation vectors for explicitly capturing various semantic relationships that may exist between a pair of concepts. In existing literature, several approaches on order-based embedding have been proposed, mostly focusing on capturing hierarchical relationships; examples include vectors in Euclidean space, complex, Hyperbolic, order, and Box Embedding. Box embedding creates region-based rich representation of concepts, but along the process it sacrifices simplicity, requiring a custom-made optimization scheme for learning the representation. Hyperbolic embedding improves embedding quality by exploiting the ever-expanding property of Hyperbolic space, but it also suffers from the same fate as box embedding as gradient descent like optimization is not simple in the Hyperbolic space. In this work, we propose Binder, a novel approach for order-based representation. Binder uses binary vectors for embedding, so the embedding vectors are compact with an order of magnitude smaller footprint than other methods. Binder uses a simple and efficient optimization scheme for learning representation vectors with a linear time complexity. Our comprehensive experimental results show that Binder is very accurate, yielding competitive results on the representation task. But Binder stands out from its competitors on the transitive closure link prediction task as it can learn concept embeddings just from the direct edges, whereas all existing order-based approaches rely on the indirect edges.
Abstract (translated)
对于自然语言理解和生成，使用基于顺序表示的概念嵌入是非常重要的任务。与传统的基于点向量的表示方法不同，基于顺序表示的方法对表示向量施加了几何约束，从而明确地捕捉了概念之间可能存在的各种语义关系。在现有文献中，已经提出了几种基于顺序表示的方法，主要集中在捕捉层次关系；例如欧氏空间中的向量、复数、双曲、顺序和Box嵌入。Box嵌入创建了概念的空间丰富表示，但在过程中它牺牲了简单性，需要为学习表示定制一个优化方案。双曲嵌入通过利用双曲空间的无限扩张性质来提高嵌入质量，但与Box嵌入一样，在Hyperbolic空间中优化梯度分解不是简单的。在这篇论文中，我们提出了Binder，一种全新的基于顺序表示的方法。Binder使用二进制向量进行嵌入，因此嵌入向量具有大小较小且其他方法的足迹更小的 footprint。Binder 使用了一个简单而高效的优化方案来学习表示向量，具有线性时间复杂度。我们的全面实验结果表明，Binder非常准确，在表示任务上产生了竞争力的结果。但是，Binder在等价关系链接预测任务中从竞争对手脱颖而出，因为它可以从直接边缘学习概念嵌入，而所有现有的基于顺序的方法都需要从间接边缘学习。
URL
https://arxiv.org/abs/2404.10924
