Abstract
We propose a unified food-domain QA framework that combines a large-scale multimodal knowledge graph (MMKG) with generative AI. Our MMKG links 13,000 recipes, 3,000 ingredients, 140,000 relations, and 14,000 images. We generate 40,000 QA pairs using 40 templates and LLaVA/DeepSeek augmentation. Joint fine-tuning of Meta LLaMA 3.1-8B and Stable Diffusion 3.5-Large improves BERTScore by 16.2\%, reduces FID by 37.8\%, and boosts CLIP alignment by 31.1\%. Diagnostic analyses-CLIP-based mismatch detection (35.2\% to 7.3\%) and LLaVA-driven hallucination checks-ensure factual and visual fidelity. A hybrid retrieval-generation strategy achieves 94.1\% accurate image reuse and 85\% adequacy in synthesis. Our results demonstrate that structured knowledge and multimodal generation together enhance reliability and diversity in food QA.
Abstract (translated)
我们提出了一种统一的食品领域问答框架，该框架结合了大规模多模态知识图谱（MMKG）和生成式人工智能。我们的MMKG连接了13,000个食谱、3,000种食材、140,000条关系以及14,000张图片。我们使用40个模板及LLaVA/DeepSeek增强技术生成了40,000对问答数据。通过将Meta LLaMA 3.1-8B与Stable Diffusion 3.5-Large联合微调，BERTScore提升了16.2%，FID降低了37.8%，CLIP一致性提高了31.1%。诊断分析——基于CLIP的不匹配检测（从35.2%降至7.3%）以及LLaVA驱动的幻觉检查——确保了事实和视觉的一致性。混合检索生成策略实现了94.1%的准确图像重用率及85%的合成充分性。我们的结果显示，结构化知识与多模态生成技术相结合能够提升食品问答领域的可靠性和多样性。
URL
https://arxiv.org/abs/2507.06571
