Abstract
Estimating the nutritional content of food from images is a critical task with significant implications for health and dietary monitoring. This is challenging, especially when relying solely on 2D images, due to the variability in food presentation, lighting, and the inherent difficulty in inferring volume and mass without depth information. Furthermore, reproducibility in this domain is hampered by the reliance of state-of-the-art methods on proprietary datasets for large-scale pre-training. In this paper, we investigate the impact of large-scale pre-training datasets on the performance of deep learning models for nutritional estimation using only 2D images. We fine-tune and evaluate Vision Transformer (ViT) models pre-trained on two large public datasets, ImageNet and COYO, comparing their performance against baseline CNN models (InceptionV2 and ResNet-50) and a state-of-the-art method pre-trained on the proprietary JFT-300M dataset. We conduct extensive experiments on the Nutrition5k dataset, a large-scale collection of real-world food plates with high-precision nutritional annotations. Our evaluation using Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAE%) reveals that models pre-trained on JFT-300M significantly outperform those pre-trained on public datasets. Unexpectedly, the model pre-trained on the massive COYO dataset performs worse than the model pre-trained on ImageNet for this specific regression task, refuting our initial hypothesis. Our analysis provides quantitative evidence highlighting the critical role of pre-training dataset characteristics, including scale, domain relevance, and curation quality, for effective transfer learning in 2D nutritional estimation.
Abstract (translated)
从图像估计食物的营养成分是一项具有重要健康和饮食监测意义的关键任务。然而，仅依赖二维图像来完成这一任务极具挑战性，因为食物呈现方式、光线条件的变化以及缺乏深度信息使得推断体积和质量变得困难。此外，最先进的方法严重依赖专有数据集进行大规模预训练，这在很大程度上阻碍了该领域的可重复性。 在这篇论文中，我们研究了大规模预训练数据集对仅使用二维图像进行营养估计的深层学习模型性能的影响。我们将经过ImageNet和COYO两个大型公共数据集预训练的Vision Transformer（ViT）模型进行微调，并对其与基线CNN模型（InceptionV2和ResNet-50）以及在专有JFT-300M数据集上进行了大规模预训练的方法进行比较。我们在Nutrition5k数据集上进行了广泛的实验，这是一个包含大量现实世界食物照片的数据集，这些照片具有高精度的营养成分注释。我们的评估结果显示，在使用均方绝对误差（MAE）和平均绝对百分比误差（MAPE）作为评价指标时，预训练于JFT-300M数据集上的模型显著优于基于公共数据集预训练的模型。令人意外的是，大规模COYO数据集上预训练的模型在此特定回归任务中的表现不如在ImageNet上进行预训练的模型，这反驳了我们的初始假设。 我们分析提供了定量证据，强调了预训练数据集特征——包括规模、相关领域和整理质量对于二维营养估计中有效的迁移学习至关重要。
URL
https://arxiv.org/abs/2508.03996
