Abstract
We present Lunima-OmniLV (abbreviated as OmniLV), a universal multimodal multi-task framework for low-level vision that addresses over 100 sub-tasks across four major categories: image restoration, image enhancement, weak-semantic dense prediction, and stylization. OmniLV leverages both textual and visual prompts to offer flexible and user-friendly interactions. Built on Diffusion Transformer (DiT)-based generative priors, our framework supports arbitrary resolutions -- achieving optimal performance at 1K resolution -- while preserving fine-grained details and high fidelity. Through extensive experiments, we demonstrate that separately encoding text and visual instructions, combined with co-training using shallow feature control, is essential to mitigate task ambiguity and enhance multi-task generalization. Our findings also reveal that integrating high-level generative tasks into low-level vision models can compromise detail-sensitive restoration. These insights pave the way for more robust and generalizable low-level vision systems.
Abstract (translated)
我们介绍了一种名为Lunima-OmniLV（简称OmniLV）的通用多模态多任务框架，该框架专门用于低级视觉处理，并涵盖了四个主要类别中的超过100个子任务：图像恢复、图像增强、弱语义密集预测以及风格化。OmniLV利用文本和视觉提示来提供灵活且用户友好的交互方式。基于扩散变换器（DiT）的生成先验知识构建，我们的框架支持任意分辨率，并在1K分辨率下实现最佳性能，同时保持了精细细节与高保真度。 通过广泛的实验研究，我们证明分别编码文本和视觉指令，并结合浅层特征控制协同训练方法，对于缓解任务模糊性以及增强多任务泛化能力至关重要。此外，我们的研究成果还表明，在低级视觉模型中整合高级生成任务可能会损害细节敏感型恢复的效果。这些见解为构建更为稳健且通用的低级视觉系统铺平了道路。
URL
https://arxiv.org/abs/2504.04903
