Abstract
For multimodal LLMs, the synergy of visual comprehension (textual output) and generation (visual output) presents an ongoing challenge. This is due to a conflicting objective: for comprehension, an MLLM needs to abstract the visuals; for generation, it needs to preserve the visuals as much as possible. Thus, the objective is a dilemma for visual-tokens. To resolve the conflict, we propose encoding images into morph-tokens to serve a dual purpose: for comprehension, they act as visual prompts instructing MLLM to generate texts; for generation, they take on a different, non-conflicting role as complete visual-tokens for image reconstruction, where the missing visual cues are recovered by the MLLM. Extensive experiments show that morph-tokens can achieve a new SOTA for multimodal comprehension and generation simultaneously. Our project is available at this https URL.
Abstract (translated)
对于多模态LLM，视觉理解（文本输出）和生成（视觉输出）的协同作用是一个持续的挑战。这是因为理解需要对视觉进行抽象，而生成需要尽可能地保留视觉。因此，对于视觉标记来说，目标是一个两难的困境。为了解决冲突，我们提出将图像编码成形位标记以实现双重目的：在理解方面，它们作为视觉提示指导MLLM生成文本；在生成方面，它们以一种非冲突的方式担任完整的视觉标记，以便MLLM通过修复缺失的视觉线索来重建图像。大量实验证明，形位标记可以同时达到多模态理解和生成的最优水平。我们的项目可以从该链接https://www.example.com/中获取。
URL
https://arxiv.org/abs/2405.01926
