Abstract
The Hyperspace Analogue to Language (HAL) model relies on global word co-occurrence matrices to construct distributional semantic representations. While these representations capture lexical relationships effectively, aggregating them into sentence-level embeddings via standard mean pooling often results in information loss. Mean pooling assigns equal weight to all tokens, thereby diluting the impact of contextually salient words with uninformative structural tokens. In this paper, we address this limitation by integrating a learnable, temperature-scaled additive attention mechanism into the HAL representation pipeline. To mitigate the sparsity and high dimensionality of the raw co-occurrence matrices, we apply Truncated Singular Value Decomposition (SVD) to project the vectors into a dense latent space prior to the attention layer. We evaluate the proposed architecture on the IMDB sentiment analysis dataset. Empirical results demonstrate that the attention-based pooling approach achieves a test accuracy of 82.38%, yielding an absolute improvement of 6.74 percentage points over the traditional mean pooling baseline (75.64%). Furthermore, qualitative analysis of the attention weights indicates that the mechanism successfully suppresses stop-words and selectively attends to sentiment-bearing tokens, improving both classification performance and model interpretability.
Abstract (translated)
超空间语言类比（HAL）模型依赖全局词共现矩阵构建分布语义表示。尽管这些表示能有效捕捉词汇关系，但通过标准均值池化聚合为句子级嵌入时往往导致信息损失。均值池化对所有词元赋予相同权重，从而稀释了语境显著词汇的影响，同时放大了无信息量的结构词元的作用。本文针对这一局限，在HAL表示流程中集成了可学习的温度缩放可加注意力机制。为缓解原始共现矩阵的稀疏性与高维问题，我们在注意力层前采用截断奇异值分解（SVD）将向量投影至稠密潜在空间。所提架构在IMDB情感分析数据集上进行了评估。实证结果表明，基于注意力池化的方法达到82.38%的测试准确率，相较于传统均值池化基线（75.64%）实现了6.74个百分点的绝对提升。此外，对注意力权重的定性分析显示，该机制成功抑制了停用词，并选择性地聚焦于携带情感信息的词元，从而提升了分类性能与模型可解释性。
URL
https://arxiv.org/abs/2603.20149
