Pit a manual parser, a direct LLM predictor, and a self-optimizing DSPy Flex program against each other on a chemical-record extraction task — and let a metric, not intuition, decide which one deserves a place in your pipeline.
What you will learn
dspy.Predict baseline, and a self-optimizing dspy.Flex program on the same boiling-point extraction taskInstructions
Open the Colab notebook using the button above. Run each cell in order; cells marked Exercise require you to fill in code.
Configure the runtime. Set DSPY_STUDENT_MODEL and DSPY_REFLECTION_MODEL (any LiteLLM identifier, e.g. openai/gpt-5-mini) and the matching API key. The dataset — 40 train, 15 validation, 20 held-out synthetic boiling-point records adapted from the NIST Chemistry WebBook — is embedded in the notebook, so no external files are needed.
Define the signature. ExtractBoilingPoint declares the contract: text in; compound name, boiling point in kelvin, supporting evidence, and a needs_review flag out. All three approaches below implement this same signature.
Build the three competing programs. A hand-written regex/arithmetic parser (0 LM calls), dspy.Predict(ExtractBoilingPoint) as a direct one-call baseline, and dspy.Flex(ExtractBoilingPoint, max_predictor_calls=5), which starts as a predictor but can be rewritten entirely by an optimizer.
Define the metric. chemistry_metric scores normalized value, compound name, exact evidence substring, and correct abstention — then subtracts a small penalty (capped at 0.10) per predictor call, so a correct zero-call answer never loses to a correct one-call answer.
Run GEPA on Flex. With a 180-metric-call budget, GEPA runs the program, scores it, reflects on failures, and proposes new module source — not just new prompt text. Read the resulting optimized_flex.module_src and annotate which parts are deterministic parsing versus LM calls.
Evaluate once, held out. Score the manual parser, direct predictor, and optimized Flex on the 20 held-out test records — the first and only time this split is used — and answer the reflection questions in the final section using the resulting table.
Questions
Answer these questions as you work through the notebook. Discuss with your neighbour — some have no single right answer.
On the held-out test set, what fraction of the optimized Flex program's predictions use zero, one, or more than one predictor call? Does zero-call execution coincide with correctness, or just with lower cost?
EasyRead optimized_flex.module_src. Which branches are ordinary Python (parsing, unit conversion, validation) and which route to the student model? Is the split scientifically defensible, or did GEPA introduce brittle or unnecessary complexity?
Break down held-out accuracy by challenge category (celsius conversion, distractor numbers, ranges, conflicts, missing values). Which categories, if any, actually benefit from semantic interpretation rather than deterministic parsing?
Swap the metric's per-call penalty (currently capped at 0.10) for a larger or smaller value and rerun GEPA. Does the optimizer converge on a different implementation — e.g. one that never calls the LM, or one that calls it more freely?
ChallengeResources