Lecture 9

Model Interpretability

Interpretable-by-design models vs. post-hoc explanation — MDI, permutation importance, ICE/PDP, LIME, and SHAP — and how to tell whether a model learned real chemistry or a shortcut.

Michael Yusov
90 min
Back to schedule
Slides (PDF) Slides (PPTX) Recording (YouTube) Companion Tutorial



Lapuschkin et al. — "Unmasking Clever Hans Predictors and Assessing What Machines Really Learn" Paper
Nat. Commun., 2019, 10, 1096 · Models winning benchmarks via a copyright watermark, not the object
Molnar — "Interpretable Machine Learning: A Guide for Making Black Box Models Explainable" Website
3rd edn., 2025 · The taxonomy of interpretability methods used throughout this lecture
Van Noorden & Perkel — "AI and Science: What 1,600 Researchers Think" Paper
Nature, 2023, 621, 672 · Survey on trust, transparency, and adoption of AI in research
Kruschel et al. — "Challenging the Performance-Interpretability Trade-Off" Paper
Bus. Inf. Syst. Eng., 2026, 68, 159 · Interpretable models often match black-box accuracy
Lundberg & Lee — "A Unified Approach to Interpreting Model Predictions" Paper
NeurIPS, 2017 · Introduces SHAP and its axiomatic foundation
Ribeiro et al. — "Why Should I Trust You? Explaining the Predictions of Any Classifier" Paper
KDD, 2016 · Original LIME paper
Rodríguez-Pérez & Bajorath — "Interpretation of Machine Learning Models Using Shapley Values" Paper
J. Comput. Aided Mol. Des., 2020, 34, 1013 · SHAP applied to QSAR models with chemical context
Apley & Zhu — "Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models" Paper
J. R. Stat. Soc. B, 2020, 82, 1059 · Accumulated local effects (ALE) — feature-effect plots robust to correlated features
Rudin — "Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead" Paper
Nat. Mach. Intell., 2019, 1, 206 · A cautionary perspective on post-hoc explanation in high-stakes settings
Adebayo et al. — "Sanity Checks for Saliency Maps" Paper
NeurIPS, 2018 · Tests whether explanation maps actually depend on the model and data
Ying et al. — "GNNExplainer: Generating Explanations for Graph Neural Networks" Paper
arXiv:1903.03894 · Subgraph and node-feature explanations for GNNs
Captum Website
captum.ai · Integrated-gradients attribution for differentiable (neural-network) molecular models
Explainable Boosting Machines Website
interpret.ml/docs/ebm.html · Interpretable GAM-like models with learned nonlinear feature functions
Counterfactual Explanations (DiCE) Website
interpret.ml/DiCE · "What minimal molecular/property change would flip the prediction?"
Quantus Website
quantus.readthedocs.io · Benchmarking library for robustness and faithfulness of explanations
YouTube logo
Class-Aware Explainability for ML-Guided Catalyst Discovery Video
Parastoo Semnani · AiChemist-MSCA-DN lecture series
YouTube logo
The Role of Interpretability in Protein Language Models Video
Dr. Alex Vicente Sola · AiChemist-MSCA-DN lecture series
YouTube logo
Formal Reasoning for Explainable Neural Network Classification Video
Dr. Michael Wand · AiChemist-MSCA-DN lecture series