Abstract
This paper presents the first study on adapting the visual in-context learning (V-ICL) paradigm to optical character recognition tasks, specifically focusing on text removal and segmentation. Most existing V-ICL generalists employ a reasoning-as-reconstruction approach: they turn to using a straightforward image-label compositor as the prompt and query input, and then masking the query label to generate the desired output. This direct prompt confines the model to a challenging single-step reasoning process. To address this, we propose a task-chaining compositor in the form of image-removal-segmentation, providing an enhanced prompt that elicits reasoning with enriched intermediates. Additionally, we introduce context-aware aggregation, integrating the chained prompt pattern into the latent query representation, thereby strengthening the model's in-context reasoning. We also consider the issue of visual heterogeneity, which complicates the selection of homogeneous demonstrations in text recognition. Accordingly, this is effectively addressed through a simple self-prompting strategy, preventing the model's in-context learnability from devolving into specialist-like, context-free inference. Collectively, these insights culminate in our ConText model, which achieves new state-of-the-art across both in- and out-of-domain benchmarks. The code is available at this https URL.
Abstract (translated)
本文介绍了首个将视觉上下文学习（V-ICL）范式应用于光学字符识别任务的研究，具体聚焦于文本去除和分割。大多数现有的通用型V-ICL模型采用的是重构推理方法：它们使用简单的图像标签合成器作为提示和查询输入，然后通过屏蔽查询标签来生成所需输出。这种直接的提示将模型限制在一个具有挑战性的单步推理过程中。为了解决这一问题，我们提出了一种以“图像去除分割”形式的任务链式合成器，提供了增强的提示，使模型能够基于更加丰富的中间结果进行推理。 此外，我们还引入了上下文感知聚合方法，将链条式的提示模式整合到潜在查询表示中，从而增强了模型在上下文中进行推理的能力。同时，我们也考虑到了视觉异质性的问题，这使得选择同质化的演示示例在文本识别中变得复杂化。为此，通过一个简单的自我提示策略有效解决了这一问题，防止了模型的上下文学习能力退化为缺乏背景信息的专家式推断。 综上所述，这些见解汇聚成了我们的ConText模型，在域内和跨域基准测试中均达到了新的最先进水平。代码可在以下链接获取：[此URL]。
URL
https://arxiv.org/abs/2506.03799
