Abstract
While vision-language models (VLMs) have advanced into detailed image description, evaluation remains a challenge. Standard metrics (e.g. CIDEr, SPICE) were designed for short texts and tuned to recognize errors that are now uncommon, such as object misidentification. In contrast, long texts require sensitivity to attribute and relation attachments and scores that localize errors to particular text spans. In this work, we introduce PoSh, a metric for detailed image description that uses scene graphs as structured rubrics to guide LLMs-as-a-Judge, producing aggregate scores grounded in fine-grained errors (e.g. mistakes in compositional understanding). PoSh is replicable, interpretable and a better proxy for human raters than existing metrics (including GPT4o-as-a-Judge). To validate PoSh, we introduce a challenging new dataset, DOCENT. This novel benchmark contains artwork, paired with expert-written references, and model-generated descriptions, augmented with granular and coarse judgments of their quality from art history students. Thus, DOCENT enables evaluating both detailed image description metrics and detailed image description itself in a challenging new domain. We show that PoSh achieves stronger correlations (+0.05 Spearman $\rho$) with the human judgments in DOCENT than the best open-weight alternatives, is robust to image type (using CapArena, an existing dataset of web imagery) and is a capable reward function, outperforming standard supervised fine-tuning. Then, using PoSh, we characterize the performance of open and closed models in describing the paintings, sketches and statues in DOCENT and find that foundation models struggle to achieve full, error-free coverage of images with rich scene dynamics, establishing a demanding new task to gauge VLM progress. Through both PoSh and DOCENT, we hope to enable advances in important areas such as assistive text generation.
Abstract (translated)
尽管视觉-语言模型（VLMs）在详细图像描述方面取得了进展，但评估仍然面临挑战。标准度量（如CIDEr、SPICE）是为短文本设计的，并调整以识别现在少见的错误，例如对象误识。相比之下，长文本需要对属性和关系附加的敏感性以及将误差定位到特定文本片段的分数。 在这项工作中，我们引入了PoSh，这是一种用于详细图像描述的度量标准，使用场景图作为结构化的评分准则来指导LLMs（大型语言模型）作为裁判进行操作，生成基于细粒度错误（例如组合理解中的错误）的聚合得分。PoSh具有可重复性、解释性和比现有指标（包括GPT4o-as-a-Judge）更好的人类评分代理能力。 为了验证PoSh的有效性，我们引入了一个新的挑战数据集DOCENT。这个新颖的基准测试包含了艺术品及其专家撰写的参考描述，并附有模型生成的描述和艺术史学生对其质量进行细化与粗略判断的数据。因此，DOCENT能够在一个具有挑战性的新领域评估详细图像描述度量标准以及详尽的图像描述本身。 我们展示了PoSh在DOCENT的人类评分中实现了比最佳开放权重替代方案更强的相关性（+0.05 Spearman $\rho$），并且对图像类型具有鲁棒性（使用CapArena，一个现有的网络图像数据集）。此外，它是一个有效的奖励函数，在标准监督微调方面表现出色。然后，利用PoSh分析了在描述DOCENT中的绘画、素描和雕像时开放模型与封闭模型的表现，并发现基础模型难以实现包含丰富场景动态的图像的完全无误覆盖，从而建立了一个评估VLM进展的新任务。 通过PoSh和DOCENT这两项工作，我们希望能够在辅助文本生成等重要领域取得进展。
URL
https://arxiv.org/abs/2510.19060
