Abstract
In this paper, a taxonomy for memory networks is proposed based on their memory organization. The taxonomy includes all the popular memory networks: vanilla recurrent neural network (RNN), long short term memory (LSTM ), neural stack and neural Turing machine and their variants. The taxonomy puts all these networks under a single umbrella and shows their relative expressive power , i.e. vanilla RNN <=LSTM<=neural stack<=neural RAM. The differences and commonality between these networks are analyzed. These differences are also connected to the requirements of different tasks which can give the user instructions of how to choose or design an appropriate memory network for a specific task. As a conceptual simplified class of problems, four tasks of synthetic symbol sequences: counting, counting with interference, reversing and repeat counting are developed and tested to verify our arguments. And we use two natural language processing problems to discuss how this taxonomy helps choosing the appropriate neural memory networks for real world problem.
Abstract (translated)
在本文中，提出了一种基于内存组织的内存网络分类。该分类包括所有流行的记忆网络：香草回归神经网络（RNN），长期短期记忆（LSTM），神经堆栈和神经图灵机及其变体。该分类将所有这些网络放在一个统一的伞下，并显示它们的相对表现力，即香草RNN <= LSTM <=神经堆栈<=神经RAM。分析了这些网络之间的差异和共性。这些差异还与不同任务的要求相关，这些任务可以向用户提供关于如何为特定任务选择或设计合适的存储器网络的说明。作为一个概念上简化的问题类别，合成符号序列的四个任务：计数，干扰计数，反转和重复计数被开发和测试以验证我们的论点。我们使用两种自然语言处理问题来讨论这种分类法如何帮助为真实世界问题选择合适的神经记忆网络。
URL
https://arxiv.org/abs/1805.00327
