Tutorial 3

Model Training & Evaluation

A hands-on lab on the BBBP blood–brain barrier dataset: random vs. scaffold vs. temporal splitting, nested cross-validation, learning/validation curves, and a real data-leakage bug to find and fix.

Sabrina Wahler
105 min
Python · Google Colab
Back to schedule

Open in Google Colab

The notebook has most of the code pre-filled. Complete the exercises marked ### EXERCISE and # YOUR CODE HERE. End-to-end runtime once fully filled in is ~4–5 minutes on Colab's free CPU tier.

Open Notebook

Getting started

Open the Colab notebook using the button above. Run each cell in order; cells marked ### EXERCISE require you to fill in code — everything else is provided scaffolding.

  1. Set up the environment. Install rdkit, scikit-learn, pandas, and matplotlib. Confirm the setup cell prints Setup OK.

  2. Explore the dataset. Load the 2050-molecule BBBP (blood–brain barrier permeability) table. Exercise 2.1: count duplicated raw vs. canonical SMILES, flag duplicate groups with conflicting labels, and resolve them before splitting. Then plot the class balance.

  3. Implement splitting strategies. Featurize once with 1024-bit Morgan fingerprints, then complete three splits: 3a an 80/20 stratified random split; 3b a scaffold split that computes each molecule's Bemis–Murcko scaffold and greedily packs whole scaffold groups (largest first, shuffled for tie-breaking) into train/valid/test; 3c a temporal split on a provided simulated sim_year column (the public BBBP table has no real publication dates). Compare class distributions and visualize both splits in PCA space.

  4. Tune with nested cross-validation. Exercise 4: fill in a small hyperparameter grid for a RandomForestClassifier (n_estimators, max_depth, min_samples_leaf) and complete nested_cv_auc — an inner GridSearchCV fit on the training indices only, scored once on held-out test indices.

  5. Diagnose your model. On the scaffold split's training set, plot a learning curve (training size vs. AUC) in 5a and a validation curve (max_depth vs. AUC) in 5b. Identify the bias–variance regime your model is in.

  6. Find and fix the leakage bug. Exercise 6: implement a buggy branch that fits StandardScaler on train+test stacked together, and a fixed branch that fits it on the training data only. Compare AUC for a Random Forest, then 6b for a KNN and a Logistic Regression model.

  7. Challenge: balanced scaffold folds. Exercise 7: extend the greedy scaffold packer from step 3 into a k-fold splitter (balanced_scaffold_folds) that also balances each fold's positive rate, not just its size.


Answer these questions as you work through the notebook. Discuss with your neighbour — some have no single right answer.

    Warm-up

    What fraction of molecules are BBB-positive? Would you expect this class imbalance to affect AUC-ROC? What about accuracy?

    Easy
    Core

    Compare test AUC under random vs. scaffold splitting. Which is lower? By how much? What does this gap imply for model deployment?

    Medium
    Core

    Identify the data leakage bug in the provided preprocessing helper (hint: look at where StandardScaler is fit). Fix it and report how test AUC changes — for Random Forest, and for at least one other model type. Why does the size of the effect differ between models?

    Medium
    Challenge

    Extend the greedy scaffold packer to also balance class rate across folds. How does the AUC distribution across folds compare to an unstratified scaffold split? Which would you trust more to judge whether a new featurization is an improvement?

    Challenge

Notebook (Colab) Solutions notebook (Colab) GitHub repo Paired lecture notes Interactive: ROC-AUC curve


  1. Martins, I. F.; Teixeira, A. L.; Pinheiro, L.; Falcão, A. O. A Bayesian Approach to in Silico Blood-Brain Barrier Penetration Modeling. J. Chem. Inf. Model. 2012, 52 (6), 1686–1697. doi.org/10.1021/ci300124c
  2. Bemis, G. W.; Murcko, M. A. The Properties of Known Drugs. 1. Molecular Frameworks. J. Med. Chem. 1996, 39 (15), 2887–2893. doi.org/10.1021/jm9602928
  3. Ambroise, C.; McLachlan, G. J. Selection Bias in Gene Extraction on the Basis of Microarray Gene-Expression Data. Proc. Natl. Acad. Sci. 2002, 99 (10), 6562–6566. doi.org/10.1073/pnas.102102699