Abstract
LSTM models used in current Machine Learning literature and applications, has a promising solution for permitting long term information using gating mechanisms that forget and reduce effect of current input information. However, even with this pipeline, they do not optimally focus on specific old index or long-term information. This paper elaborates upon input reordering approaches to prioritize certain input indices. Moreover, no LSTM based approach is found in the literature that examines weight normalization while choosing the right weight and exponent of Lp norms through main supervised loss function. In this paper, we find out which norm best finds relationship between weights to either smooth or sparsify them. Lastly, gates, as weighted representations of inputs and states, which control reduction-extent of current input versus previous inputs (~ state), are not nonlinearized enough (through a small FFNN). As analogous to attention mechanisms, gates easily filter current information to bold (emphasize on) past inputs. Nonlinearized gates can more easily tune up to peculiar nonlinearities of specific input in the past. This type of nonlinearization is not proposed in the literature, to the best of author's knowledge. The proposed approaches are implemented and compared with a simple LSTM to understand their performance in text classification tasks. The results show they improve accuracy of LSTM.
Abstract (translated)
当前的机器学习文献和应用中使用的LSTM模型通过使用门控机制来遗忘旧信息并减少当前输入的影响，从而允许可长期存储信息。然而，即使有了这些机制，它们也无法在处理特定旧索引或长时间信息时达到最佳聚焦效果。本文探讨了重新排序输入的方法以优先考虑某些输入索引。 此外，在现有的文献中尚未发现任何基于LSTM的方法会考察在选择合适的权重和$L_p$范数指数时进行权重归一化的做法，特别是在主要的监督损失函数范围内。本文通过研究哪种范数能最好地找到权重之间的关系（无论是平滑还是稀疏化），来寻找最优解决方案。 另外，门控机制作为输入和状态的加权表示，控制了当前输入与先前输入（~ 状态）之间的影响程度。但这些门控机制并未进行足够的非线性处理（通过小型前馈神经网络）。类似注意力机制，在过滤当前信息以突出显示过去的信息时，它们很容易做到这一点。经过非线性的门控可以更有效地调整个别过去的输入的特殊非线性。 据作者所知，这种类型的非线性化在文献中尚未被提出过。本文提出的几种方法与简单的LSTM进行对比测试，用以评估其在文本分类任务中的性能表现。实验结果表明，这些方法确实提高了LSTM模型的准确率。
URL
https://arxiv.org/abs/2509.00087
