Tutorial 14

Does This Chemistry Task Need an LLM?

Pit a manual parser, a direct LLM predictor, and a self-optimizing DSPy Flex program against each other on a chemical-record extraction task — and let a metric, not intuition, decide which one deserves a place in your pipeline.

Ben Reiner
James Wade
Braun 152
105 min
Python · Google Colab
Back to schedule

Open in Google Colab

The notebook is fully runnable end to end — no fill-in-the-blank cells. When opened in Colab, the first cell auto-installs dspy, pandas, python-dotenv, and deno (Flex's sandbox runtime). The next cell prompts for an API key (e.g. OPENAI_API_KEY) — add it as a Colab secret (key icon in the left sidebar) or paste it when prompted. Never hardcode a key into a notebook cell or share it in chat.

Open Notebook

Getting started

Open the Colab notebook using the button above. Run each cell in order; cells marked Exercise require you to fill in code.

  1. Configure the runtime. Set DSPY_STUDENT_MODEL and DSPY_REFLECTION_MODEL (any LiteLLM identifier, e.g. openai/gpt-5-mini) and the matching API key. The dataset — 40 train, 15 validation, 20 held-out synthetic boiling-point records adapted from the NIST Chemistry WebBook — is embedded in the notebook, so no external files are needed.

  2. Define the signature. ExtractBoilingPoint declares the contract: text in; compound name, boiling point in kelvin, supporting evidence, and a needs_review flag out. All three approaches below implement this same signature.

  3. Build the three competing programs. A hand-written regex/arithmetic parser (0 LM calls), dspy.Predict(ExtractBoilingPoint) as a direct one-call baseline, and dspy.Flex(ExtractBoilingPoint, max_predictor_calls=5), which starts as a predictor but can be rewritten entirely by an optimizer.

  4. Define the metric. chemistry_metric scores normalized value, compound name, exact evidence substring, and correct abstention — then subtracts a small penalty (capped at 0.10) per predictor call, so a correct zero-call answer never loses to a correct one-call answer.

  5. Run GEPA on Flex. With a 180-metric-call budget, GEPA runs the program, scores it, reflects on failures, and proposes new module source — not just new prompt text. Read the resulting optimized_flex.module_src and annotate which parts are deterministic parsing versus LM calls.

  6. Evaluate once, held out. Score the manual parser, direct predictor, and optimized Flex on the 20 held-out test records — the first and only time this split is used — and answer the reflection questions in the final section using the resulting table.


Answer these questions as you work through the notebook. Discuss with your neighbour — some have no single right answer.

    Warm-up

    On the held-out test set, what fraction of the optimized Flex program's predictions use zero, one, or more than one predictor call? Does zero-call execution coincide with correctness, or just with lower cost?

    Easy
    Core

    Read optimized_flex.module_src. Which branches are ordinary Python (parsing, unit conversion, validation) and which route to the student model? Is the split scientifically defensible, or did GEPA introduce brittle or unnecessary complexity?

    Medium
    Core

    Break down held-out accuracy by challenge category (celsius conversion, distractor numbers, ranges, conflicts, missing values). Which categories, if any, actually benefit from semantic interpretation rather than deterministic parsing?

    Medium
    Challenge

    Swap the metric's per-call penalty (currently capped at 0.10) for a larger or smaller value and rerun GEPA. Does the optimizer converge on a different implementation — e.g. one that never calls the LM, or one that calls it more freely?

    Challenge

Notebook (Colab) GitHub repo Paired lecture notes