Tutorial 8

Data Stewardship

Organise research data so it stays findable, reusable, and ready for downstream modelling and collaboration.

Alix Schmidt
90 min
Python · Google Colab
Back to schedule

Start here

Open sprint-briefing-card.docx and sprint-dataset-messy.xlsx in Excel or Google Sheets — no Colab needed for this one.

Download dataset

Phase 1 — The sprint (5 min)

Your labmate left the group for an industry job. Their reactions log — sprint-dataset-messy.xlsx — is now yours, and your PI wants an answer today.

  1. Read the briefing card. sprint-briefing-card.docx sets the scene and the question: which catalyst/temperature combination gives the highest mean yield?

  2. Set a 5-minute timer. Open sprint-dataset-messy.xlsx in Excel or Sheets and try to answer the PI's question before it runs out.

  3. Come back with an answer or an excuse. Either a number you'd defend, or a clear explanation of why the file wouldn't let you get one.

Phase 2 — The cleaning exercise (~30 min)

Use tidy-cleaning-cheatsheet.docx as your guide and relational-diagram.png to see how the tables relate.

  1. Apply the three tidy-data rules. One variable per column, one observation per row, one table per kind of entity.

  2. Split the log into three linked tables. Produce reactions_log_tidy.csv, catalyst_reference_tidy.csv, and solvent_reference_tidy.csv, joined by ID columns.

  3. Watch for the common traps. Trailing spaces, mixed units in one cell, averaged replicates, and metadata rows mixed in with measurements.

  4. Start a data dictionary. Fill in data-dictionary-template.xlsx with the name, type, unit, and description of every column in your tidy reactions log.

Check your work against the answer key and the "payoff" notebook under Resources below — it answers the PI's question from the tidy CSVs in a few lines of pandas.

Phase 3 — Stretch / take-home

If you finish early, or as take-home: turn the manual cleaning you just did into a reusable pipeline.

  1. Open pipeline_starter.py. Each function has a docstring and a TODO — fill them in following stretch-pipeline-handout.docx.

  2. Run pytest test_pipeline.py. The first six tests check individual helpers; the last two run the whole pipeline end to end.

  3. Generalise to a second dataset. The final test runs your pipeline on new-experiments-messy.xlsx — a log you haven't seen. If it passes, your pipeline is genuinely reusable.


These are the questions the facilitator will ask after the sprint. Discuss with your neighbour — some have no single right answer.

    Warm-up

    What answer did you land on for the PI's question, and how confident are you in it?

    Easy
    Core

    What was the most annoying thing about the messy file — and which tidy-data rule would have prevented it?

    Medium
    Challenge

    If you handed your three tidy CSVs to a stranger, could they pandas.merge them and answer the PI's question in ~8 lines of code? What's still missing if not?

    Challenge

Sprint & cleaning

Sprint briefing card Messy dataset (xlsx) Tidy-data cleaning cheatsheet Relational diagram Data dictionary template

Answer key & payoff notebook

reactions_log_tidy.csv catalyst_reference_tidy.csv solvent_reference_tidy.csv Payoff notebook (Colab)

Stretch take-home

Stretch handout pipeline_starter.py test_pipeline.py Second dataset (new-experiments-messy.xlsx) Reference solution (Colab)

General

Paired lecture notes