Abstract
Large Deep Learning models are often compressed before being deployed in a resource-constrained environment. Can we trust the prediction of compressed models just as we trust the prediction of the original large model? Existing work has keenly studied the effect of compression on accuracy and related performance measures. However, performance parity does not guarantee trust-equivalence. We propose a two-dimensional framework for trust-equivalence evaluation. First, interpretability alignment measures whether the models base their predictions on the same input features. We use LIME and SHAP tests to measure the interpretability alignment. Second, calibration similarity measures whether the models exhibit comparable reliability in their predicted probabilities. It is assessed via ECE, MCE, Brier Score, and reliability diagrams. We conducted experiments using BERT-base as the large model and its multiple compressed variants. We focused on two text classification tasks: natural language inference and paraphrase identification. Our results reveal low interpretability alignment and significant mismatch in calibration similarity. It happens even when the accuracies are nearly identical between models. These findings show that compressed models are not trust-equivalent to their large counterparts. Deploying compressed models as a drop-in replacement for large models requires careful assessment, going beyond performance parity.
Abstract (translated)
大型深度学习模型在部署到资源受限的环境中之前通常会被压缩。我们可以像信任原始大规模模型那样完全信赖被压缩后的模型吗？现有研究已经深入探讨了压缩对准确性和相关性能指标的影响，但性能上的等价并不保证信任度的一致性。我们提出了一个用于评估信任度一致性的二维框架。 首先，可解释性一致性衡量的是这些模型是否基于相同的输入特征进行预测。我们使用LIME（局部可解释的模型无关方法）和SHAP（分值的重要性和属性贡献）测试来测量这一方面。 其次，校准相似性衡量的是这些模型在预测概率中的可靠性是否相当一致。通过预期校准误差（ECE）、最大校准误差（MCE）、布赖尔得分以及可靠性的图表来进行评估。 我们使用BERT-base作为大型原始模型，并对其多个压缩版本进行了实验。我们重点关注了两个文本分类任务：自然语言推理和同义句识别。我们的结果显示，即使在准确率几乎相同的情况下，可解释性一致性也很低且校准相似性存在显著差异。这表明被压缩的模型与它们的大规模原型并不具备等价的信任度。 因此，在将压缩模型作为大规模模型的直接替代品进行部署时，需要超越性能等价的考量来进行仔细评估。
URL
https://arxiv.org/abs/2508.13533
