Tutorial 4

Neural Networks in Practice

Predict Tox21 toxicity with an MLP built from scratch in PyTorch — choose a fingerprint featurization, fight overfitting with regularization, question architecture against dataset size, run a fixed-split hyperparameter search, and extend to multi-task learning.

Jules Schleinitz
90 min
Python · Google Colab
Back to schedule

Open in Google Colab

The notebook has most of the code pre-filled. Complete the exercises marked Exercise or your code here as you go.

Open Notebook

Getting started

Open the Colab notebook using the button above. The notebook is structured as ten exercises, in order. Run each cell in order; cells marked Exercise require you to fill in code. Stuck on one? The solutions notebook (see Resources below) has a worked example for every exercise.

  1. Introduce the Tox21 dataset. Load dataset.csv and look at per-task label counts, positive rates, and the missing-label pattern across all 12 assays.

  2. Visualize and choose a featurization. Compute Morgan fingerprints from SMILES, project them to 2D with PCA, sanity-check them with a quick k-NN classifier, and use Tanimoto similarity to compare positively- vs. negatively-labeled molecules.

  3. Introduce the neural network. Build the simplest possible single-hidden-layer network, write one manual optimization cycle by hand, then generalize it into the configurable MolecularMLP class.

  4. Train a baseline model. Load the fixed train/val/test split for one Tox21 task and train a first MLP configuration end to end.

  5. Regularization. Recognize the overfitting in the baseline's learning curve, then fight it with dropout, weight decay, and tighter early stopping.

  6. Question architecture vs. dataset size. Sweep model capacity (tiny/medium/large) at fixed dataset size, see where overfitting kicks in, and quantify how much a single score varies by weight-init seed vs. by train/val/test split.

  7. Visualize learning curves vs. hyperparameters. Overlay validation curves across a swept hyperparameter (e.g. learning rate).

  8. Run a hyperparameter search. Sweep a small grid, select the best run by validation score only, and plot the efficiency/performance frontier.

  9. Draw conclusions. Plot an ROC curve for the best model and submit your run to the class leaderboard.

  10. Advanced: multi-task Tox21. Train one model across all 12 assays at once with a masked loss, and compare its score on your primary task against the single-task baseline.


Answer these questions as you work through the notebook. Discuss with your neighbour — some have no single right answer.

    Warm-up

    Not every Tox21 assay was tested on every molecule. For your chosen task, what fraction of molecules actually have a label? How does that compare to the total dataset size?

    Easy
    Core

    Does the k-NN sanity check in Exercise 2 suggest the Morgan fingerprint captures real signal for your task, or close to chance? Would you trust an MLP trained on this featurization?

    Easy
    Core

    In Exercise 5, which regularizer (dropout, weight decay, or tighter early stopping) shrinks the baseline's train/val loss gap the most? Does that same regularizer also give the best test ROC-AUC, or is there a cost/benefit tradeoff?

    Medium
    Core

    In the capacity sweep (Exercise 6), does the largest model actually win on test ROC-AUC, or does the train/val loss gap tell a different story? What does that imply about picking model size relative to labeled dataset size?

    Medium
    Core

    Still in Exercise 6: which is larger for these models — the variability from a different weight-init/shuffling seed, or from a different train/val/test split? What does that tell you about how much to trust a single reported test score?

    Medium
    Core

    Across the learning-rate sweep (Exercise 7), which value converges fastest, and which oscillates or fails to converge within the epoch budget? How do you tell the difference from the loss curve shape alone?

    Medium
    Challenge

    Does multi-task training (Exercise 10) improve or hurt your primary task's test ROC-AUC compared to the single-task baseline? Which of the 12 tasks benefit most from sharing a hidden trunk, and why might that be?

    Challenge

Notebook (Colab) Solutions notebook (Colab) GitHub repo Paired lecture notes