Abstract
The article proposes a new architecture based on Multi-head attention to solve the problem of morphological tagging for the Russian language. The preprocessing of the word vectors includes splitting the words into subtokens, followed by a trained procedure for aggregating the vectors of the subtokens into vectors for tokens. This allows to support an open dictionary and analyze morphological features taking into account parts of words (prefixes, endings, etc.). The open dictionary allows in future to analyze words that are absent in the training dataset. The performed computational experiment on the SinTagRus and Taiga datasets shows that for some grammatical categories the proposed architecture gives accuracy 98-99% and above, which outperforms previously known results. For nine out of ten words, the architecture precisely predicts all grammatical categories and indicates when the categories must not be analyzed for the word. At the same time, the model based on the proposed architecture can be trained on consumer-level graphics accelerators, retains all the advantages of Multi-head attention over RNNs (RNNs are not used in the proposed approach), does not require pretraining on large collections of unlabeled texts (like BERT), and shows higher processing speed than previous results.
Abstract (translated)
本文提出了一种基于多头注意力机制的新架构，用于解决俄语形态标注问题。词向量的预处理包括将单词拆分为子词，随后通过训练好的程序将子词向量聚合为词向量。这支持使用开放词典，并能在分析形态特征时考虑单词的各个部分（如前缀、词尾等）。开放词典使得未来能够分析训练数据集中未出现的单词。在SinTagRus和Taiga数据集上进行的计算实验表明，对于某些语法范畴，该架构的准确率可达98%-99%以上，优于此前已知结果。每十个词中有九个，该架构能精确预测所有语法范畴，并能在无需分析某词的语法范畴时进行标注。同时，基于该架构的模型可在消费级显卡上训练，保留了多头注意力机制相对于循环神经网络（本方法未使用RNN）的所有优势，无需在大规模无标注文本上进行预训练（如BERT），且处理速度高于此前成果。
URL
https://arxiv.org/abs/2604.02926
