Abstract
This study introduces De-DSI, a novel framework that fuses large language models (LLMs) with genuine decentralization for information retrieval, particularly employing the differentiable search index (DSI) concept in a decentralized setting. Focused on efficiently connecting novel user queries with document identifiers without direct document access, De-DSI operates solely on query-docid pairs. To enhance scalability, an ensemble of DSI models is introduced, where the dataset is partitioned into smaller shards for individual model training. This approach not only maintains accuracy by reducing the number of data each model needs to handle but also facilitates scalability by aggregating outcomes from multiple models. This aggregation uses a beam search to identify top docids and applies a softmax function for score normalization, selecting documents with the highest scores for retrieval. The decentralized implementation demonstrates that retrieval success is comparable to centralized methods, with the added benefit of the possibility of distributing computational complexity across the network. This setup also allows for the retrieval of multimedia items through magnet links, eliminating the need for platforms or intermediaries.
Abstract (translated)
这项研究引入了De-DSI，一种将大型语言模型（LLMs）与真正的去中心化信息检索相结合的新框架，特别是在去中心化环境中采用不同的可导搜索索引（DSI）概念。该研究将关注有效地将新颖的用户查询与文档标识符连接起来，而无需直接访问文档。De-DSI仅在查询-文档对上操作。为了提高可扩展性，引入了一个DSI模型的集成，其中数据集被划分为个人模型训练的较小的片段。这种方法不仅通过减少每个模型需要处理的数据量来保持准确性，而且通过将结果聚合起来，使可扩展性得到改善。这种聚合使用beam搜索来确定top docids，并应用softmax函数进行分数归一化，选择得分最高的文档进行检索。去中心化实现证明了检索成功与集中方法相当，同时还具有将计算复杂度分散在网络中的好处。这种设置还允许通过磁力链接检索多媒体项目，无需平台或中介机构。
URL
https://arxiv.org/abs/2404.12237
