Abstract
Despite recent advances in text-conditioned 3D indoor scene generation, there remain gaps in the evaluation of these methods. Existing metrics primarily assess the realism of generated scenes by comparing them to a set of ground-truth scenes, often overlooking alignment with the input text - a critical factor in determining how effectively a method meets user requirements. We present SceneEval, an evaluation framework designed to address this limitation. SceneEval includes metrics for both explicit user requirements, such as the presence of specific objects and their attributes described in the input text, and implicit expectations, like the absence of object collisions, providing a comprehensive assessment of scene quality. To facilitate evaluation, we introduce SceneEval-100, a dataset of scene descriptions with annotated ground-truth scene properties. We evaluate recent scene generation methods using SceneEval and demonstrate its ability to provide detailed assessments of the generated scenes, highlighting strengths and areas for improvement across multiple dimensions. Our results show that current methods struggle at generating scenes that meet user requirements, underscoring the need for further research in this direction.
Abstract (translated)
尽管在文本条件下的3D室内场景生成方面取得了进展，但这些方法的评估仍然存在不足之处。现有的度量标准主要通过将生成的场景与一组真实的场景进行比较来衡量其逼真程度，往往忽略了与输入文本的一致性——这是决定一种方法能否有效满足用户需求的关键因素。我们提出了SceneEval这一评估框架，旨在解决这一局限性。SceneEval包括了针对明确的用户要求（如输入文本中描述的具体对象及其属性）和隐含期望（例如没有物体碰撞情况）的度量标准，从而对场景质量进行全面评估。为了便于评估，我们引入了包含注释的真实场景属性的100个场景描述的数据集SceneEval-100。我们使用SceneEval来评估最近的场景生成方法，并展示了它能够提供详细的生成场景评估结果，突出显示在多个维度上的优点和改进空间。我们的结果显示，当前的方法难以生成满足用户需求的场景，强调了在此方向上进行进一步研究的需求。
URL
https://arxiv.org/abs/2503.14756
