Abstract
Pretrained transformer-based models have shown high performance in natural language generation task. However, a new wave of interest has surged: automatic programming language generation. This task consists of translating natural language instructions to a programming code. Despite the fact that well-known pretrained models on language generation have achieved good performance in learning programming languages, effort is still needed in automatic code generation. In this paper, we introduce JaCoText, a model based on Transformers neural network. It aims to generate java source code from natural language text. JaCoText leverages advantages of both natural language and code generation models. More specifically, we study some findings from the state of the art and use them to (1) initialize our model from powerful pretrained models, (2) explore additional pretraining on our java dataset, (3) carry out experiments combining the unimodal and bimodal data in the training, and (4) scale the input and output length during the fine-tuning of the model. Conducted experiments on CONCODE dataset show that JaCoText achieves new state-of-the-art results.
Abstract (translated)
预训练Transformer-based模型在自然语言生成任务中表现优异。然而,一股新的兴趣已经崛起:自动编程语言生成。这项任务是将自然语言指令转换为编程代码。尽管著名的语言生成预训练模型在学习编程语言方面取得了良好的表现,但在自动代码生成方面仍然需要努力。在本文中,我们介绍了JaCoText,一个基于Transformer神经网络模型的对象。它旨在从自然语言文本中提取Java源代码。JaCoText利用自然语言和代码生成模型的优势。具体来说,我们研究了最先进的研究 findings 并使用它们(1)从强大的预训练模型初始化我们的模型,(2)探索我们的Java数据集额外的预训练,(3)在训练期间结合单眼和双眼数据进行实验,(4)在模型微调期间调整输入和输出长度。在CONCODE数据集上开展的实验表明,JaCoText取得了新的最先进的结果。
URL
https://arxiv.org/abs/2303.12869