Abstract
This paper introduces a new technique to measure the feature dependency of neural network models. The motivation is to better understand a model by querying whether it is using information from human-understandable features, e.g., anatomical shape, volume, or image texture. Our method is based on the principle that if a model is dependent on a feature, then removal of that feature should significantly harm its performance. A targeted feature is "removed" by collapsing the dimension in the data distribution that corresponds to that feature. We perform this by moving data points along the feature dimension to a baseline feature value while staying on the data manifold, as estimated by a deep generative model. Then we observe how the model's performance changes on the modified test data set, with the target feature dimension removed. We test our method on deep neural network models trained on synthetic image data with known ground truth, an Alzheimer's disease prediction task using MRI and hippocampus segmentations from the OASIS-3 dataset, and a cell nuclei classification task using the Lizard dataset.
Abstract (translated)
本文提出了一种新的测量神经网络模型特征依赖性的技术。动机是更好地理解模型，通过查询它是否使用人类可理解特征（例如解剖形状、体积或图像纹理）来查询。我们的方法基于一个原理，即如果一个模型依赖于一个特征，那么移除该特征应显著损害其性能。我们通过在数据分布中沿着特征维度移动数据点，同时保持在数据流上，作为由深度生成模型估计的数据范式的“移除”来执行此操作。然后，我们在包含目标特征的修改测试数据集上观察模型性能的变化。我们用已知真实标准的深度神经网络模型进行了测试，该模型基于OASIS-3数据集的MRI和 hippocampus 分割预测阿尔茨海默病，以及使用Lizard数据集的细胞核分类任务。
URL
https://arxiv.org/abs/2404.12341
