Abstract
The rapid progress of Large Multimodal Models (LMMs) and cloud-based AI agents is transforming human-AI collaboration into bidirectional, multimodal interaction. However, existing codecs remain optimized for unimodal, one-way communication, resulting in repeated degradation under conventional compress-transmit-reconstruct pipelines. To address this limitation, we propose UniMIC, a Unified token-based Multimodal Interactive Coding framework that bridges edge devices and cloud AI agents. Instead of transmitting raw pixels or plain text, UniMIC employs compact tokenized representations as the communication medium, enabling efficient low-bitrate transmission while maintaining compatibility with LMMs. To further enhance compression, lightweight Transformer-based entropy models with scenario-specific designs-generic, masked, and text-conditioned-effectively minimize inter-token redundancy. Extensive experiments on text-to-image generation, text-guided inpainting, outpainting, and visual question answering show that UniMIC achieves substantial bitrate savings and remains robust even at ultra-low bitrates (<0.05bpp), without compromising downstream task performance. These results establish UniMIC as a practical and forward-looking paradigm for next-generation multimodal interactive communication.
Abstract (translated)
大型多模态模型（LMMs）和基于云的AI代理的快速进步正在将人机协作转变为双向、多模式互动。然而，现有的编解码器仍然针对单模式的一向通信进行优化，在传统的压缩-传输-重建管道中导致重复降质。为了克服这一限制，我们提出了UniMIC，这是一种统一的令牌基多模态交互编码框架，用于连接边缘设备和云端AI代理。与传输原始像素或纯文本不同，UniMIC采用紧凑的标记化表示作为通信媒介，能够在保持与LMMs兼容性的同时实现高效的低比特率传输。为了进一步提高压缩效果，我们设计了基于轻量级Transformer的熵模型，包括通用、掩码和文本条件三种特定场景的设计，有效减少了令牌间的冗余。 在文本到图像生成、文本引导的修补、扩展以及视觉问答等广泛的实验中显示，UniMIC实现了显著的比特率节省，并且即使在极低比特率（<0.05bpp）的情况下也保持了强大的性能稳定性，同时没有影响下游任务的表现。这些结果表明，UniMIC为下一代多模态交互通信建立了一种实用和前瞻性的范式。
URL
https://arxiv.org/abs/2509.22570
