A personal-project hackathon: bring your own research question, and spend this afternoon turning it into a sharp, answerable plan — before writing a single line of model code. You'll write a one-page brief, trade it with a peer for review, and walk into Friday's flash talks with a measured baseline instead of a slide of good intentions.
This week's arc
The personal project runs across the rest of the week. This afternoon is framing only — no models yet.
Worksheet, 5-min TA conversation. Data and split only — no models.
Measure the baseline in writing before touching a model.
Score on the split you committed to. Build the 5 slides.
5 slides, 5 minutes, hard stop. Presentations & wrap-up.
Deliverables
Lastname_Firstname_takeaway.docx, and email it to jul@caltech.edu.Lastname_Firstname_plan.docx and email it to the same address so a peer can review it.A well-framed question with no results is a pass. A good number you cannot explain is not.
Start here
Fill in this sentence first. It is the first line of your plan template — if it needs the word "and," you have two projects; pick one.
I want to predict · rank · generate · explain ___________ for ___________, so that I can ___________, instead of ___________.
The plan, in order
A question worth answering. Name a result that would prove you wrong. If you can't find one, the question isn't sharp enough yet. Would you be interested in the answer if it came out the other way?
The data you actually have. Source, deduplicated sample count, the label and its error, whether negative results are recorded, and the cost of one new data point.
What the model is for. Not "predict the property well" — name the job: filter, ranker, interpolator, generator, or explainer. The job determines the metric.
Enough data, or not? If yes: design the split, and say why holding that out mimics how the model will really be used — the split is the scientific claim. If no: design the acquisition, and name the first 24 experiments specifically, not just a strategy.
What done looks like. The baseline you have to beat, what will exist by Friday, and where you're stuck — the most useful section for your reviewer.
Worked examples
What people say vs. the sharper question underneath it, across four chemistry backgrounds.
| Background | What people say | What you can be wrong about |
|---|---|---|
| Drug design | "Can ML help my series?" | Can a model trained on our 340 measured analogues rank the next 40 for hERG better than cLogP alone? |
| Materials | "I want to use ML for MOFs." | Can we predict CO₂ uptake at 1 bar within 0.4 mmol/g for MOFs built from linkers absent from the training set? |
| Organic | "ML for reaction optimisation." | Does a model on our 310 C–N couplings pick a better 24-well plate than our current screening panel? |
| Computational | "Can ML replace DFT?" | Can a Δ-model on 2,000 GFN2 geometries reproduce DFT barriers to 1 kcal/mol for our substrate class? |
Step 2
Be concrete — your reviewer cannot judge the plan without these numbers. Distinct compounds, materials, or reactions, not rows in a file.
| Source | N (deduplicated) | Label + its error | Negatives recorded? | Cost per new point |
|---|---|---|---|---|
| Our notebooks, 5 people | 310 C–N couplings | HPLC yield, ±5% | Partial — "no product," no conditions | 3 h, $40 |
| In-house assay | 340 analogues | hERG IC₅₀, ±0.4 log | Yes, inactives kept | 1 week, $600 |
| CoRE MOF + our 60 | 12,000 + 60 | GCMC uptake / measured | n/a — continuous | 4 h GPU |
| ____________ | ____________ | ____________ | ____________ | ____________ |
Also ask: do you believe the sample count, the stated uncertainty, and the answer about negative results? Is anything likely duplicated, correlated, or measured under different conditions?
Step 3
The job determines the right metric — and who actually uses the output.
| Function | Objective | Metric | Users |
|---|---|---|---|
| Filter | A shortlist, nothing discarded wrongly | Recall @ budget (not RMSE) | You, next week |
| Ranker | An ordering of the top few % | Enrichment · precision@k · top-k regret | The person picking the plate |
| Interpolator | The model itself, accurate everywhere | RMSE across the whole domain | Everyone, for years |
| Generator | Candidate structures | Validity · novelty · oracle score | A synthetic chemist |
| Explainer | A mechanism-shaped hypothesis | Does the follow-up experiment work? | Your next paper |
Step 4
Yes → the split is the scientific claim — choose it before you see a single score, write it in the brief, and do not change it afterward.
| Split | Held out | The claim you're making | Typical home |
|---|---|---|---|
| Random | Random 20% | Tomorrow's samples look like today's | Fixed library, screening triage |
| Scaffold / cluster | Whole chemotypes or clusters | I will be shown new chemotypes | Drug design, virtual libraries |
| Time / prospective | Everything after a date | I will predict the future of this project | Any live campaign |
| Leave-one-group-out | A linker, element, substrate class | I will generalise to a class I never measured | Materials, methodology |
| Leave-one-lab-out | A whole instrument or operator | Someone else will reproduce this | Assay-heavy work |
No → designing the acquisition is the project itself — a real project, not a consolation prize.
| Strategy | Use when | What you do | Where to look |
|---|---|---|---|
| Design of experiments | Cheap · parallel | Factorial, D-optimal, response surface | Tue — Schmidt |
| Active learning | Expensive · you want a model | Query where the model is uncertain | Wed — Schleinitz |
| Bayesian optimisation | Expensive · you want the best point | Surrogate + acquisition, EI / UCB | Tue — Schleinitz |
| Literature extraction | The data exists, just not in a table | RAG over PDFs and SI + hard validation | Thu W1 — Wahler |
...and the fifth option: borrow a pretrained model and fine-tune on the 200 points you have. Often correct. Takes an afternoon. Whichever you pick, the deliverable is the same: name the first 24 experiments — not a strategy, the list.
Before any result
| # | Baseline | Notes |
|---|---|---|
| 0 | Predict the mean | Free — if you cannot beat it, stop and say so |
| 1 | One descriptor + a line | cLogP · Hammett σ · tolerance factor — beat the number a chemist computes in Excel |
| 2 | The human | What your group does right now, quantified — the only baseline that decides adoption |
| 3 | Random, at the same budget | For any project that selects or designs — the control that exposes memorisation |
Calibration
Watch for
Duplicates across the split — same compound, two names, two salts, two tautomers. Deduplicate on InChIKey before splitting.
Tuning on the test set — every look costs you a little of it. Keep a validation set, touch the test set once.
Descriptors that leak — batch, date, plate position, sample ID. If MW is your top SHAP feature, ask why.
Extrapolation dressed as prediction — report where test points sit relative to training. Outside the hull, it's a guess with a decimal point.
Metric mismatch — optimising RMSE when the job needed enrichment.
No baseline — still, somehow, the most common one.
Artrith, N. et al. Best practices in machine learning for chemistry. Nat. Chem. 2021, 13, 505–508. doi.org/10.1038/s41557-021-00716-z — two pages, free, read it tonight.
Thursday
You'll trade briefs with a peer and review the framing, not the chemistry — assume the author knows their system far better than you do. Be specific and be kind, in that order, and say what's good first. Use the peer-review template, save as Lastname_Firstname_plan_review.docx, and send it to jul@caltech.edu.
Say it back. Without looking at their sentence, write their project in one sentence of your own. If you can't, say exactly where the plan stopped making sense.
Question, or topic? Name a result that would prove them wrong. Would you care about the answer if it came out the other way?
Is the data enough? What didn't they tell you that you'd need before believing a result? Anything likely duplicated, correlated, or measured under different conditions?
Does the evaluation match the intended use? Does their held-out set resemble how the model would actually be used? If they're acquiring data, are the first 24 experiments the ones that teach the most?
Is the baseline honest? Is it beatable, and hard enough that beating it means something?
The most useful next step. What could they do in the next three hours that would tell them the most — including that the project doesn't work?
What you would steal. One thing from their plan you'll take back to your own project.
Friday
| # | Slide | Content | Time |
|---|---|---|---|
| 1 | The sentence | Verbatim from your worksheet, in large type | 45 s |
| 2 | The data | N · label · error · negatives — one table | 45 s |
| 3 | The split, or the first 24 experiments | Your actual contribution | 90 s |
| 4 | Baseline vs. you | Two bars minimum | 90 s |
| 5 | What would change your mind | And what three more weeks would buy | 30 s |
"It did not work, and here is the diagnosis" is a strong talk. No outline slide. No history of ML.
Grading, loosely
A brief and a baseline — sharp question, honest data inventory, baseline measured.
+ one model on your own split, and a number you're willing to defend.
+ the diagnosis — why it worked, or why it didn't. Rare in three afternoons.
All three are a pass. You are not practising how to fit a model — you are practising how to decide what is worth fitting, and that is the only part of this week nobody can hand you.