Build a complete RAG pipeline over chemistry paper abstracts — chunk, embed, retrieve, generate, and evaluate whether grounding actually reduces hallucination.
What you will learn
The notebook ships with a built-in 40-abstract demo corpus spanning ten chemistry subfields — including several on Suzuki-Miyaura coupling solvents, and one deliberately fictional paper (the "Kessler-Onyema coupling") used later to catch the bare LLM hallucinating. It's small on purpose: the whole notebook runs in a couple of minutes on a plain Colab CPU runtime, no external data download required. The notebook also outlines an optional path to scale up to a larger, real literature corpus.
Instructions
Open the Colab notebook using the button above. Run each cell in order; cells marked Exercise require you to fill in code.
Set up & load the corpus. Install sentence-transformers, faiss-cpu, bert-score, and transformers. The notebook loads the built-in 40-abstract demo corpus by default — everything runs in a couple of minutes on a plain CPU runtime.
Chunk and embed (Exercise 1). Implement chunk_text() — word-based chunking with overlap — then embed every chunk with all-MiniLM-L6-v2 and build a FAISS index.
Implement retrieval (Exercise 2). Complete retrieve(query, k=5): embed the query, search the FAISS index, and return the top-k chunks with their source metadata.
Build the RAG chain (Exercise 3). Complete build_prompt() so it assembles numbered context passages into an instruction to answer only from what's retrieved. Generation runs on a free local flan-t5-base fallback by default — no API key required.
RAG vs. bare LLM. Ask about the deliberately fictional "Kessler-Onyema coupling" — a reaction that only exists in the demo corpus — and see whether the bare LLM hallucinates a confident, detailed answer anyway.
Evaluate (Exercises 4a/4b). Score answers with BERTScore recall against the retrieved context (faithfulness) and embedding cosine similarity against the question (relevance), then sweep k = 3 → 10 to see whether more retrieved context keeps helping.
(Challenge) Re-rank with a cross-encoder (Exercise 5). Use cross-encoder/ms-marco-MiniLM-L-6-v2 to re-score the FAISS shortlist before picking the final top-3 context chunks.
Questions
Answer these questions as you work through the notebook. Discuss with your neighbour — some have no single right answer.
Run retrieve() on a query clearly inside the corpus and one clearly outside it (e.g. "What is the boiling point of ethanol?"). Does the FAISS index refuse the out-of-corpus query, or just return its least-bad chunks? What does that tell you about a retriever "knowing nothing" vs. an LLM "knowing nothing"?
For the query "What solvents are used in Suzuki coupling?", look at the top-3 retrieved chunks. Do they come from different papers, or does one dominate? Do they actually answer the question, or just mention "Suzuki coupling" without discussing solvents?
EasyAsk about the fictional Kessler-Onyema coupling. Does the bare LLM admit it doesn't know, or confidently invent plausible-sounding solvents, catalyst loadings, and yields? Does the RAG answer correctly ground itself in the retrieved abstract? Now try a real obscure reaction of your choice — does the bare LLM still hallucinate?
MediumSweep k = [3, 5, 7, 10] over the 10 evaluation questions. Does faithfulness keep improving as k grows, or does it plateau (or drop)? How does latency scale with k, and where's the point of diminishing returns for this corpus?
After cross-encoder re-ranking, does faithfulness improve on average, and is the improvement consistent across questions or concentrated in a few? Re-ranking costs an extra model call per candidate — when would that added latency be worth it in a production chemistry-literature assistant?
Challenge