Abstract
Natural Language Processing enables computers to understand human language by analysing and classifying text efficiently with deep-level grammatical and semantic features. Existing models capture features by learning from large corpora with transformer models, which are computationally intensive and unsuitable for resource-constrained environments. Therefore, our proposed study incorporates comprehensive grammatical rules alongside semantic information to build a robust, lightweight classification model without resorting to full parameterised transformer models or heavy deep learning architectures. The novelty of our approach lies in its explicit encoding of sentence-level grammatical structure, including syntactic composition, phrase patterns, and complexity indicators, into a compact grammar vector, which is then fused with frozen contextual embeddings. These heterogeneous elements unified a single representation that captures both the structural and semantic characteristics of the text. Deep learning models such as Deep Belief Networks (DBNs), Long Short-Term Memory (LSTMs), BiLSTMs, and transformer-based BERT and XLNET were used to train and evaluate the model, with the number of epochs varied. Based on experimental results, the unified feature representation model captures both the semantic and structural properties of text, outperforming baseline models by 2%-15%, enabling more effective learning across heterogeneous domains. Unlike prior syntax-aware transformer models that inject grammatical structure through additional attention layers, tree encoders, or full fine-tuning, the proposed framework treats grammar as an explicit inductive bias rather than a learnable module, resulting in a very lightweight model that delivers better performance on edge devices
Abstract (translated)
自然语言处理（NLP）通过分析和分类文本中的深层次语法和语义特征，使计算机能够理解人类语言。现有模型利用大规模语料库训练的变压器模型来捕捉这些特征，但这种方法计算密集且不适合资源受限环境。因此，我们提出的研究结合了全面的语法规则与语义信息，构建了一个无需使用全参数化的变压器模型或重型深度学习架构的健壮、轻量级分类模型。 我们的方法新颖之处在于明确地将句子级别的语法结构（包括句法构成、短语模式和复杂度指标）编码为一个紧凑的语法规则向量，并将其与冻结的上下文嵌入相结合。这些异构元素统一在一个单一表示中，该表示同时捕捉文本的结构和语义特性。 我们使用深度信念网络（DBNs）、长短期记忆网络（LSTMs）、双向LSTMs以及基于变压器的BERT和XLNET等深度学习模型来训练和评估我们的模型，并通过改变训练轮数来进行实验。根据实验结果，统一特征表示模型能够捕捉文本的语义与结构属性，在各类基准模型上表现出2%-15%的优势，实现了异构领域中的更有效学习。 不同于之前的语法感知变换器模型（这些模型通过附加注意力层、树编码器或完全微调来注入语法结构），我们的框架将语法视为一种显式的归纳偏差而非可学习模块。这导致了一个非常轻量级的模型，在边缘设备上表现出更好的性能。
URL
https://arxiv.org/abs/2602.20749
