Abstract
Human vision combines low-resolution "gist" information from the visual periphery with sparse but high-resolution information from fixated locations to construct a coherent understanding of a visual scene. In this paper, we introduce MetamerGen, a tool for generating scenes that are aligned with latent human scene representations. MetamerGen is a latent diffusion model that combines peripherally obtained scene gist information with information obtained from scene-viewing fixations to generate image metamers for what humans understand after viewing a scene. Generating images from both high and low resolution (i.e. "foveated") inputs constitutes a novel image-to-image synthesis problem, which we tackle by introducing a dual-stream representation of the foveated scenes consisting of DINOv2 tokens that fuse detailed features from fixated areas with peripherally degraded features capturing scene context. To evaluate the perceptual alignment of MetamerGen generated images to latent human scene representations, we conducted a same-different behavioral experiment where participants were asked for a "same" or "different" response between the generated and the original image. With that, we identify scene generations that are indeed metamers for the latent scene representations formed by the viewers. MetamerGen is a powerful tool for understanding scene understanding. Our proof-of-concept analyses uncovered specific features at multiple levels of visual processing that contributed to human judgments. While it can generate metamers even conditioned on random fixations, we find that high-level semantic alignment most strongly predicts metamerism when the generated scenes are conditioned on viewers' own fixated regions.
Abstract (translated)
人类的视觉系统结合了来自视网膜周边区域的低分辨率“概览”信息与固定注视点处稀疏但高分辨率的信息，从而构建出对视觉场景的一致理解。本文介绍了MetamerGen这一工具，它用于生成符合潜在的人类场景表示的场景图像。MetamerGen是一种潜扩散模型，结合了从视网膜周边区域获取的概览信息和从固定注视点处获得的信息，以生成与人类在观看某一场景后所形成的理解一致的图像代用品（metamers）。同时从高分辨率（即“中央凹”）和低分辨率输入中生成图像构成了一个新的图到图合成问题。为了解决这一问题，我们引入了双重流表示形式来描述固定注视区域的场景：DINOv2令牌融合来自固定注视点详细特征与周边区域捕捉场景背景信息的降级特征。 为了评估MetamerGen生成的图像与其潜在的人类场景表征之间的感知一致性，我们进行了一个相同/不同行为实验，在该实验中参与者被要求在生成图像和原始图像之间做出“相同”或“不同”的判断。通过这种方式，我们可以识别出与观众形成的场景表示一致的场景代用品。MetamerGen是一个理解场景理解的强大工具。我们的概念验证分析揭示了视觉处理多个层级中的特定特征对人类判断的影响。尽管它可以基于随机注视点生成图像代用品，我们发现当产生的场景条件设定为观众自己的固定区域时，高层次语义一致性最能预测其是否为图像代用品。 该研究展示了通过结合不同层次的视觉信息来合成逼真的场景图像的新方法，并揭示了这些合成图像与人类感知之间的关联。MetamerGen不仅在技术上具有创新性，在理解人类如何处理复杂视觉场景方面也提供了重要的见解。
URL
https://arxiv.org/abs/2601.11675
