🧑‍💻 Hackathon

Frame Your Own Project

A personal-project hackathon: bring your own research question, and spend this afternoon turning it into a sharp, answerable plan — before writing a single line of model code. You'll write a one-page brief, trade it with a peer for review, and walk into Friday's flash talks with a measured baseline instead of a slide of good intentions.

Bring your own data · Individual or pairs
Back to schedule

Four questions, three afternoons, then a 5-minute talk

The personal project runs across the rest of the week. This afternoon is framing only — no models yet.

Wed · now

Kickoff & framing

Worksheet, 5-min TA conversation. Data and split only — no models.

Thu · 13:30

Baseline, then a first model

Measure the baseline in writing before touching a model.

Thu · 15:15

Honest evaluation, then slides

Score on the split you committed to. Build the 5 slides.

Fri

Flash talks

5 slides, 5 minutes, hard stop. Presentations & wrap-up.


Two things leave this room today

A well-framed question with no results is a pass. A good number you cannot explain is not.


Before any code

Fill in this sentence first. It is the first line of your plan template — if it needs the word "and," you have two projects; pick one.

I want to predict · rank · generate · explain ___________ for ___________, so that I can ___________, instead of ___________.


Five headings, one page

  1. A question worth answering. Name a result that would prove you wrong. If you can't find one, the question isn't sharp enough yet. Would you be interested in the answer if it came out the other way?

  2. The data you actually have. Source, deduplicated sample count, the label and its error, whether negative results are recorded, and the cost of one new data point.

  3. What the model is for. Not "predict the property well" — name the job: filter, ranker, interpolator, generator, or explainer. The job determines the metric.

  4. Enough data, or not? If yes: design the split, and say why holding that out mimics how the model will really be used — the split is the scientific claim. If no: design the acquisition, and name the first 24 experiments specifically, not just a strategy.

  5. What done looks like. The baseline you have to beat, what will exist by Friday, and where you're stuck — the most useful section for your reviewer.


The same interest, twice

What people say vs. the sharper question underneath it, across four chemistry backgrounds.

BackgroundWhat people sayWhat you can be wrong about
Drug design"Can ML help my series?"Can a model trained on our 340 measured analogues rank the next 40 for hERG better than cLogP alone?
Materials"I want to use ML for MOFs."Can we predict CO₂ uptake at 1 bar within 0.4 mmol/g for MOFs built from linkers absent from the training set?
Organic"ML for reaction optimisation."Does a model on our 310 C–N couplings pick a better 24-well plate than our current screening panel?
Computational"Can ML replace DFT?"Can a Δ-model on 2,000 GFN2 geometries reproduce DFT barriers to 1 kcal/mol for our substrate class?

Inventory your data

Be concrete — your reviewer cannot judge the plan without these numbers. Distinct compounds, materials, or reactions, not rows in a file.

SourceN (deduplicated)Label + its errorNegatives recorded?Cost per new point
Our notebooks, 5 people310 C–N couplingsHPLC yield, ±5%Partial — "no product," no conditions3 h, $40
In-house assay340 analogueshERG IC₅₀, ±0.4 logYes, inactives kept1 week, $600
CoRE MOF + our 6012,000 + 60GCMC uptake / measuredn/a — continuous4 h GPU
____________________________________________________________

Also ask: do you believe the sample count, the stated uncertainty, and the answer about negative results? Is anything likely duplicated, correlated, or measured under different conditions?


Define the model's job

The job determines the right metric — and who actually uses the output.

FunctionObjectiveMetricUsers
FilterA shortlist, nothing discarded wronglyRecall @ budget (not RMSE)You, next week
RankerAn ordering of the top few %Enrichment · precision@k · top-k regretThe person picking the plate
InterpolatorThe model itself, accurate everywhereRMSE across the whole domainEveryone, for years
GeneratorCandidate structuresValidity · novelty · oracle scoreA synthetic chemist
ExplainerA mechanism-shaped hypothesisDoes the follow-up experiment work?Your next paper

Can existing data answer your question?

Yes → the split is the scientific claim — choose it before you see a single score, write it in the brief, and do not change it afterward.

SplitHeld outThe claim you're makingTypical home
RandomRandom 20%Tomorrow's samples look like today'sFixed library, screening triage
Scaffold / clusterWhole chemotypes or clustersI will be shown new chemotypesDrug design, virtual libraries
Time / prospectiveEverything after a dateI will predict the future of this projectAny live campaign
Leave-one-group-outA linker, element, substrate classI will generalise to a class I never measuredMaterials, methodology
Leave-one-lab-outA whole instrument or operatorSomeone else will reproduce thisAssay-heavy work

No → designing the acquisition is the project itself — a real project, not a consolation prize.

StrategyUse whenWhat you doWhere to look
Design of experimentsCheap · parallelFactorial, D-optimal, response surfaceTue — Schmidt
Active learningExpensive · you want a modelQuery where the model is uncertainWed — Schleinitz
Bayesian optimisationExpensive · you want the best pointSurrogate + acquisition, EI / UCBTue — Schleinitz
Literature extractionThe data exists, just not in a tableRAG over PDFs and SI + hard validationThu W1 — Wahler

...and the fifth option: borrow a pretrained model and fine-tune on the 200 points you have. Often correct. Takes an afternoon. Whichever you pick, the deliverable is the same: name the first 24 experiments — not a strategy, the list.


Define your baselines

#BaselineNotes
0Predict the meanFree — if you cannot beat it, stop and say so
1One descriptor + a linecLogP · Hammett σ · tolerance factor — beat the number a chemist computes in Excel
2The humanWhat your group does right now, quantified — the only baseline that decides adoption
3Random, at the same budgetFor any project that selects or designs — the control that exposes memorisation

Four briefs that would pass

Drug design

Question
Rank 40 new analogues for hERG
Data
340 in-house IC₅₀, ±0.4 log, inactives kept
Job
Ranker
Step 4
Yes → scaffold split, hold out 3 chemotypes
Baseline
Beat cLogP alone

Materials

Question
CO₂ uptake for MOFs with unseen linkers
Data
CoRE MOF GCMC (12k) + 60 measured in-house
Job
Interpolator
Step 4
Yes → leave-linker-out, then transfer to the 60
Baseline
Beat gravimetric surface area

Organic

Question
Pick a better 24-well plate for C–N coupling
Data
310 couplings, ±5% yield; failures being recovered
Job
Ranker
Step 4
Yes → leave-substrate-class-out
Baseline
Beat our standard panel

Computational

Question
Δ-learn DFT barriers from GFN2
Data
2,000 GFN2 geometries; 40 DFT barriers so far
Job
Interpolator
Step 4
No → active learning, 200 more DFT points
Baseline
Beat plain GFN2 barriers

Six ways this goes wrong

×

Duplicates across the split — same compound, two names, two salts, two tautomers. Deduplicate on InChIKey before splitting.

×

Tuning on the test set — every look costs you a little of it. Keep a validation set, touch the test set once.

×

Descriptors that leak — batch, date, plate position, sample ID. If MW is your top SHAP feature, ask why.

×

Extrapolation dressed as prediction — report where test points sit relative to training. Outside the hull, it's a guess with a decimal point.

×

Metric mismatch — optimising RMSE when the job needed enrichment.

×

No baseline — still, somehow, the most common one.

Artrith, N. et al. Best practices in machine learning for chemistry. Nat. Chem. 2021, 13, 505–508. doi.org/10.1038/s41557-021-00716-z — two pages, free, read it tonight.


Peer review, in seven questions

You'll trade briefs with a peer and review the framing, not the chemistry — assume the author knows their system far better than you do. Be specific and be kind, in that order, and say what's good first. Use the peer-review template, save as Lastname_Firstname_plan_review.docx, and send it to jul@caltech.edu.

  1. Say it back. Without looking at their sentence, write their project in one sentence of your own. If you can't, say exactly where the plan stopped making sense.

  2. Question, or topic? Name a result that would prove them wrong. Would you care about the answer if it came out the other way?

  3. Is the data enough? What didn't they tell you that you'd need before believing a result? Anything likely duplicated, correlated, or measured under different conditions?

  4. Does the evaluation match the intended use? Does their held-out set resemble how the model would actually be used? If they're acquiring data, are the first 24 experiments the ones that teach the most?

  5. Is the baseline honest? Is it beatable, and hard enough that beating it means something?

  6. The most useful next step. What could they do in the next three hours that would tell them the most — including that the project doesn't work?

  7. What you would steal. One thing from their plan you'll take back to your own project.


Five slides. Five minutes. In this order.

#SlideContentTime
1The sentenceVerbatim from your worksheet, in large type45 s
2The dataN · label · error · negatives — one table45 s
3The split, or the first 24 experimentsYour actual contribution90 s
4Baseline vs. youTwo bars minimum90 s
5What would change your mindAnd what three more weeks would buy30 s

"It did not work, and here is the diagnosis" is a strong talk. No outline slide. No history of ML.


What done looks like

Tier 1

A brief and a baseline — sharp question, honest data inventory, baseline measured.

Tier 2

+ one model on your own split, and a number you're willing to defend.

Tier 3

+ the diagnosis — why it worked, or why it didn't. Rare in three afternoons.

All three are a pass. You are not practising how to fit a model — you are practising how to decide what is worth fitting, and that is the only part of this week nobody can hand you.