Abstract
Decoder-only language models, such as GPT and LLaMA, generally decode on the last layer. Motivated by human's hierarchical thinking capability, we propose that a hierarchical decoder architecture could be built with different layers decoding texts simultaneously. Due to limited time and computationally resources, we choose to adapt a pretrained language model into this form of hierarchical decoder. Language heads of the last layer are copied to different selected intermediate layers, and fine-tuned with different task inputs. By thorough experiments, we validate that these selective intermediate layers could be adapted to speak meaningful and reasonable contents, and this paradigm of hierarchical decoder can obtain state-of-the-art performances on multiple tasks such as hierarchical text classification, classification-guided generation, and hierarchical text generation. This study suggests the possibility of a generalized hierarchical reasoner, pretraining from scratch.
Abstract (translated)
类似于GPT和LLaMA这样的纯解码器语言模型一般在最后一层进行解码。受人类分层思维能力的启发，我们提出可以构建一个分层解码架构，在不同的层次上同时解码文本。由于时间和计算资源有限，我们将选择将预训练的语言模型改编成这种分层解码形式。将最后一层的语言头复制到选定的不同中间层，并使用不同任务输入进行微调。通过全面的实验，我们验证了这些选择性的中间层能够适应并生成有意义且合理的文本内容，并且这种分层解码架构在诸如分层文本分类、指导性生成和分层文本生成等多重任务上可以获得最先进的性能。这项研究表明从零开始预训练一个通用化分层推理模型的可能性。
URL
https://arxiv.org/abs/2507.12930
