Abstract
The vast collection of machine learning records available on the web presents a significant opportunity for meta-learning, where past experiments are leveraged to improve performance. Two crucial meta-learning tasks are pipeline performance estimation (PPE), which predicts pipeline performance on target datasets, and dataset performance-based similarity estimation (DPSE), which identifies datasets with similar performance patterns. Existing approaches primarily rely on dataset meta-features (e.g., number of instances, class entropy, etc.) to represent datasets numerically and approximate these meta-learning tasks. However, these approaches often overlook the wealth of past experimental results and pipeline metadata available. This limits their ability to capture dataset - pipeline interactions that reveal performance similarity patterns. In this work, we propose KGmetaSP, a knowledge-graph-embeddings approach that leverages existing experiment data to capture these interactions and improve both PPE and DPSE. We represent datasets and pipelines within a unified knowledge graph (KG) and derive embeddings that support pipeline-agnostic meta-models for PPE and distance-based retrieval for DPSE. To validate our approach, we construct a large-scale benchmark comprising 144,177 OpenML experiments, enabling a rich cross-dataset evaluation. KGmetaSP enables accurate PPE using a single pipeline-agnostic meta-model and improves DPSE over baselines. The proposed KGmetaSP, KG, and benchmark are released, establishing a new reference point for meta-learning and demonstrating how consolidating open experiment data into a unified KG advances the field.
Abstract (translated)
网络上可获取的海量机器学习记录为元学习提供了重大机遇，即利用过往实验来提升性能。两个关键的元学习任务是：流水线性能预估（PPE），用于预测流水线在目标数据集上的表现；以及基于数据集性能的相似性预估（DPSE），用于识别具有相似性能模式的数据集。现有方法主要依赖数据集元特征（如实例数量、类别熵等）对数据集进行数值表征，并近似完成这些元学习任务。然而，这些方法往往忽视了海量的过往实验结果和流水线元数据，这限制了其捕捉能揭示性能相似性模式的数据集-流水线交互的能力。 本研究提出KGmetaSP，一种基于知识图谱嵌入的方法，利用现有实验数据捕捉这些交互，从而提升PPE和DPSE的性能。我们在统一的知识图谱（KG）中表征数据集和流水线，并导出支持与流水线无关的元模型（用于PPE）和基于距离的检索（用于DPSE）的嵌入。为验证方法，我们构建了包含144,177个OpenML实验的大规模基准，支持丰富的跨数据集评估。KGmetaSP能够通过单一与流水线无关的元模型实现准确的PPE，并在DPSE上优于基线方法。我们发布了提出的KGmetaSP、知识图谱及基准，为元学习领域建立了新的参考基准，并展示了将开放实验数据整合到统一知识图谱中如何推动该领域发展。
URL
https://arxiv.org/abs/2603.19888
