Train interpretable and black-box models to predict blood–brain barrier permeability, then explain them with permutation importance, ICE/PDP, SHAP, and LIME — and check whether the explanations make chemical sense.
What you will learn
Instructions
Open the Colab notebook using the button above. Run each cell in order; cells marked Exercise require you to fill in code.
Load and clean the BBBP dataset. Pull the MoleculeNet blood–brain-barrier-penetration dataset (~2,050 molecules), parse each SMILES with RDKit, and drop the ~11 rows that fail to parse — checking that this doesn't bias the class balance.
Compute interpretable descriptors. Build ~25 chemically meaningful descriptors per molecule (size, lipophilicity, polarity/H-bonding, flexibility, ring counts, shape indices), then remove one from each highly-correlated pair so importances aren't split arbitrarily.
Exercise 1 — Interpretable-by-design baseline. Train a standardized logistic regression and read its coefficients directly as a first, fully transparent model.
Train a gradient-boosted tree. Tune it via cross-validation, then compare its accuracy against the logistic regression before reaching for post-hoc explanations.
Exercise 2 — Permutation importance. Write a helper that shuffles one descriptor at a time and measures the drop in held-out ROC-AUC, for a model-agnostic global ranking.
Exercise 3 — ICE and PDP. Pick a handful of descriptors and plot individual conditional expectation curves plus their partial-dependence average to see how, not just whether, each one moves the prediction.
Exercise 4 — SHAP. Build a model-agnostic SHAP explainer, compute values for a subset of test molecules, and render a beeswarm plot.
Exercise 5 — SHAP vs. LIME. For one test molecule, compare its additive SHAP attribution to a local LIME surrogate fit around that same point.
Exercise 6 — Fingerprint bits. Retrain on Morgan fingerprints, extract MDI importances from the tree model, and draw the substructures behind the most important bits.
Capstone. Pick your own question about the BBBP data and design a question → features → model → evaluation → interpretation workflow to answer it.
Questions
Answer these questions as you work through the notebook. Discuss with your neighbour — some have no single right answer.
BBBP is class-imbalanced (BBB-permeable is the majority class). Why does that make raw accuracy a misleading metric here, and which metric should you check instead?
EasyCompare the logistic regression coefficients to the permutation importance ranking from the gradient-boosted tree. Do both models agree that polar surface area (TPSA) matters most? Where do they disagree?
MediumLook at the ICE/PDP curves for molecular weight and TPSA. Does the model's learned relationship match the chemical intuition that BBB permeability favors moderate size, lower polarity, and fewer H-bond donors — or has it learned something else?
MediumPick a test molecule and compare its SHAP attribution to its LIME explanation. Do they agree on the direction and rough magnitude of the top 2–3 descriptors? If they disagree, which explanation would you trust more, and why?
MediumA Morgan fingerprint bit can correspond to more than one substructure because of hash collisions. How would you check whether a bit's high MDI importance reflects one consistent chemical motif or several unrelated ones — and does its presence actually shift the BBB-positive fraction the way you'd expect?
ChallengeResources
References Cited in the Notebook