Organise research data so it stays findable, reusable, and ready for downstream modelling and collaboration.
What you will learn
Instructions
Your labmate left the group for an industry job. Their reactions log — sprint-dataset-messy.xlsx — is now yours, and your PI wants an answer today.
Read the briefing card. sprint-briefing-card.docx sets the scene and the question: which catalyst/temperature combination gives the highest mean yield?
Set a 5-minute timer. Open sprint-dataset-messy.xlsx in Excel or Sheets and try to answer the PI's question before it runs out.
Come back with an answer or an excuse. Either a number you'd defend, or a clear explanation of why the file wouldn't let you get one.
Use tidy-cleaning-cheatsheet.docx as your guide and relational-diagram.png to see how the tables relate.
Apply the three tidy-data rules. One variable per column, one observation per row, one table per kind of entity.
Split the log into three linked tables. Produce reactions_log_tidy.csv, catalyst_reference_tidy.csv, and solvent_reference_tidy.csv, joined by ID columns.
Watch for the common traps. Trailing spaces, mixed units in one cell, averaged replicates, and metadata rows mixed in with measurements.
Start a data dictionary. Fill in data-dictionary-template.xlsx with the name, type, unit, and description of every column in your tidy reactions log.
Check your work against the answer key and the "payoff" notebook under Resources below — it answers the PI's question from the tidy CSVs in a few lines of pandas.
If you finish early, or as take-home: turn the manual cleaning you just did into a reusable pipeline.
Open pipeline_starter.py. Each function has a docstring and a TODO — fill them in following stretch-pipeline-handout.docx.
Run pytest test_pipeline.py. The first six tests check individual helpers; the last two run the whole pipeline end to end.
Generalise to a second dataset. The final test runs your pipeline on new-experiments-messy.xlsx — a log you haven't seen. If it passes, your pipeline is genuinely reusable.
Questions
These are the questions the facilitator will ask after the sprint. Discuss with your neighbour — some have no single right answer.
What answer did you land on for the PI's question, and how confident are you in it?
EasyWhat was the most annoying thing about the messy file — and which tidy-data rule would have prevented it?
MediumIf you handed your three tidy CSVs to a stranger, could they pandas.merge them and answer the PI's question in ~8 lines of code? What's still missing if not?
Resources