Interpretable-by-design models vs. post-hoc explanation — MDI, permutation importance, ICE/PDP, LIME, and SHAP — and how to tell whether a model learned real chemistry or a shortcut.
The Clever Hans effect is not hypothetical. Lapuschkin et al. found published image classifiers winning a benchmark by keying on a source-tag watermark, not the object itself — a reminder that high performance alone never proves a model learned the right chemistry.
There is no single best method — pick by the question. MDI and permutation importance answer "which features matter globally," ICE/PDP answer "how does a feature act," and LIME/SHAP answer "why this specific prediction."
SHAP is grounded in cooperative game theory. Shapley values are the unique attribution satisfying efficiency (contributions sum exactly to the prediction), symmetry, and the dummy-feature axiom — a guarantee LIME's local surrogate does not carry.
The performance–interpretability trade-off is a common misconception, not a law. Recent large-scale benchmarking (Kruschel et al., 2026) finds interpretable and black-box models are frequently statistically indistinguishable in predictive accuracy.
Explanations can mislead as easily as they inform. Correlated descriptors split or hide each other's importance, a local explanation is not a global rule, and a feature attribution is not proof of causal mechanism.