Abstract
Recent advancements in instruction-following models have made user interactions with models more user-friendly and efficient, broadening their applicability. In graphic design, non-professional users often struggle to create visually appealing layouts due to limited skills and resources. In this work, we introduce a novel multimodal instruction-following framework for layout planning, allowing users to easily arrange visual elements into tailored layouts by specifying canvas size and design purpose, such as for book covers, posters, brochures, or menus. We developed three layout reasoning tasks to train the model in understanding and executing layout instructions. Experiments on two benchmarks show that our method not only simplifies the design process for non-professionals but also surpasses the performance of few-shot GPT-4V models, with mIoU higher by 12% on Crello. This progress highlights the potential of multimodal instruction-following models to automate and simplify the design process, providing an approachable solution for a wide range of design tasks on visually-rich documents.
Abstract (translated)
近年来，指令跟随模型的进步使得用户与模型之间的交互更加友好和高效，拓宽了其应用范围。在图形设计中，非专业用户通常由于技能和资源有限，难以创建视觉上吸引人的布局。在这项工作中，我们引入了一个新颖的多模态指令跟随布局规划框架，允许用户通过指定画布大小和设计目的，轻松地将视觉元素排版到定制布局中，如书籍封面、海报、宣传册或菜单。我们开发了三个布局推理任务来训练模型理解并执行布局指令。在两个基准测试上的实验证明，我们的方法不仅简化了非专业用户的设计流程，而且超越了少样本GPT-4V模型的性能，在Crello上的mIoU值较高。这一进步突出了多模态指令跟随模型的潜力，可以自动化和简化设计过程，为各种视觉丰富的文档提供了一种易于设计的解决方案。
URL
https://arxiv.org/abs/2404.15271
