Abstract
We introduce Stylized Meta-Album (SMA), a new image classification meta-dataset comprising 24 datasets (12 content datasets, and 12 stylized datasets), designed to advance studies on out-of-distribution (OOD) generalization and related topics. Created using style transfer techniques from 12 subject classification datasets, SMA provides a diverse and extensive set of 4800 groups, combining various subjects (objects, plants, animals, human actions, textures) with multiple styles. SMA enables flexible control over groups and classes, allowing us to configure datasets to reflect diverse benchmark scenarios. While ideally, data collection would capture extensive group diversity, practical constraints often make this infeasible. SMA addresses this by enabling large and configurable group structures through flexible control over styles, subject classes, and domains-allowing datasets to reflect a wide range of real-world benchmark scenarios. This design not only expands group and class diversity, but also opens new methodological directions for evaluating model performance across diverse group and domain configurations-including scenarios with many minority groups, varying group imbalance, and complex domain shifts-and for studying fairness, robustness, and adaptation under a broader range of realistic conditions. To demonstrate SMA's effectiveness, we implemented two benchmarks: (1) a novel OOD generalization and group fairness benchmark leveraging SMA's domain, class, and group diversity to evaluate existing benchmarks. Our findings reveal that while simple balancing and algorithms utilizing group information remain competitive as claimed in previous benchmarks, increasing group diversity significantly impacts fairness, altering the superiority and relative rankings of algorithms. We also propose to use \textit{Top-M worst group accuracy} as a new hyperparameter tuning metric, demonstrating broader fairness during optimization and delivering better final worst-group accuracy for larger group diversity. (2) An unsupervised domain adaptation (UDA) benchmark utilizing SMA's group diversity to evaluate UDA algorithms across more scenarios, offering a more comprehensive benchmark with lower error bars (reduced by 73\% and 28\% in closed-set setting and UniDA setting, respectively) compared to existing efforts. These use cases highlight SMA's potential to significantly impact the outcomes of conventional benchmarks.
Abstract (translated)
我们介绍了Stylized Meta-Album（SMA），这是一个新的图像分类元数据集，包含24个数据集（12个内容数据集和12个风格化数据集），旨在推进出界分布（OOD）泛化的研究及相关话题。通过使用来自12个主题分类数据集的样式转换技术创建而成，SMA提供了一套多样且全面的4800组组合，将各种主题（对象、植物、动物、人类动作、纹理）与多种风格相结合。SMA允许对组和类进行灵活控制，使我们能够配置数据集以反映多样化基准测试场景。 虽然理想的数据收集会捕捉广泛的群体多样性，但实际限制通常使得这一目标难以实现。SMA通过对其风格、主题类别及领域进行灵活控制来解决此问题，从而建立大型且可配置的群体结构，让数据集能反映多种现实世界中的基准测试情景。这种设计不仅扩大了组和类别的多样性，还为评估模型在各种多样化的群组和领域设置下的表现打开了新的方法论方向，并可用于研究更多真实条件下公平性、鲁棒性和适应性的问题。 为了展示SMA的有效性，我们实施了两个基准测试：(1) 一个利用SMA的领域、类别及群体多样性进行OOD泛化和群体公平性评估的新基准；我们的发现表明，尽管简单的平衡以及使用群体信息的算法在现有基准中仍然具有竞争力，但增加群体多样性显著影响了公平性，改变了算法的优势及其相对排名。(2) 一项无监督域适应（UDA）基准测试，利用SMA的群组多样性来评价UDA算法跨更多场景的表现。与现有的努力相比，在封闭集设置和UniDA设置下，该基准测试提供了一个更全面的评估，误差范围降低了73% 和 28%，展示了SMA在扩大群体多样性的优化过程中实现更广泛公平性，并最终提高了最差群体准确性。 这些应用场景突显了SMA对传统基准结果具有显著影响的潜力。
URL
https://arxiv.org/abs/2512.09773
