Today we build:
A bigram-overlap reranker — a phrase-level signal neither BM25 nor vector similarity can see
A second pass over Day 4’s hybrid search candidates
A visual dashboard showing exactly which passages moved, and why
Why This Matters
Both of Day 4’s scorers — BM25 and vector similarity — treat a passage as a bag of independent words. Neither can tell the difference between “cancel at any time” and “at any point you may cancel,” even though only one of those actually contains the phrase a user asked about. Reranking exists because word order carries real information, and it’s cheap to check once the candidate pool is already small.
Core Concept
Bigram overlap scores a query against a passage by comparing their two-word sequences, not their individual words. “Cancel at any time” produces the bigrams (cancel, at), (at, any), (any, time) — a passage containing that exact phrase shares all three; a passage with the same words scrambled shares none.
This is a real, order-sensitive reranking signal, distinct from anything in the retrieval stage. It’s also deliberately cheap enough to run on only the top handful of candidates — reranking every passage in a large corpus this way wouldn’t scale, but reranking Day 4’s top 3 costs nothing.
Running today’s code on the query “cancel at any time” produces a genuine, verified rank change: hybrid search ranks doc_cancel_p0 first and doc_refund_p1 second, because doc_cancel_p0 shares more individual words overall. But doc_refund_p1 contains the exact phrase “cancel at any time” verbatim, while doc_cancel_p0 only has “cancel your subscription at any time” — the phrase is broken up. Reranking correctly promotes doc_refund_p1 to first place. That’s not a scripted outcome; it’s what the bigram math actually produces on this query.
Architecture
Reranker.rerank() takes Day 4’s HybridSearcher output directly — it doesn’t touch the passage store or recompute anything from scratch. It scores each candidate’s bigram overlap against the query, re-sorts by that score, and records both the original hybrid rank and the new final rank, so the dashboard can show the movement explicitly.
Implementation
def bigram_overlap_score(query: str, passage_text: str) -> float:
query_bigrams = bigrams(tokenize_ordered(query))
passage_bigrams = bigrams(tokenize_ordered(passage_text))
intersection = query_bigrams & passage_bigrams
union = query_bigrams | passage_bigrams
return len(intersection) / len(union)Note tokenize_ordered() deliberately keeps stopwords that Day 3 and Day 4’s tokenizer dropped — bigrams need real word adjacency to mean anything; filtering out “at” before forming pairs would destroy the exact phrase this lesson depends on.
Build, Test, Verify
chmod +x start.sh stop.sh
./start.shstart.sh installs dependencies, builds the Docker image if available, runs the lesson, runs the tests, and generates dashboard.html. Each passage renders with a rank badge — moved up, moved down, or unchanged — plus its bigram and hybrid scores side by side. Tests: 6 passed.
./stop.shImplementation Guide
GitHub Link
https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_05_package/lesson_05_package
Package contents
lesson_05_package/
├── lesson_code.py # bigram scoring, Reranker, generate_dashboard (+ Day 4's HybridSearcher, BM25, embeddings)
├── test_lesson.py # 6 tests verifying bigram scoring and rank changes
├── requirements.txt # pytest only
├── Dockerfile # minimal python:3.11-slim image
├── start.sh # install deps, build, run, test, verify — one command
├── stop.sh # cleanup — removes .venv, dashboard.html, caches, Docker image
└── README.md # quick-start reference
Run it
cd lesson_05_package
chmod +x start.sh stop.sh
./start.shYou should see dashboard.html generated, and 6 tests passed. Open the dashboard — doc_refund_p1 should show a “moved up 1” badge, doc_cancel_p0 a “moved down 1” badge.
Cleanup: ./stop.sh — also removes the generated dashboard.html.
Architecture, in more detail
Two tokenizers exist in this lesson for a specific reason:
tokenize()(from Day 3/Day 4) strips stopwords, used for BM25 and embeddings, where individual word presence is what matters.tokenize_ordered()(new today) keeps every word, including stopwords, because bigrams need genuine adjacency — dropping “at” from “cancel at any time” would turn it into “cancel any time,” destroying the exact phrase match this lesson depends on.
Reranker.rerank() takes HybridResult objects directly from HybridSearcher.search(), computes bigram_overlap_score() for each, and re-sorts. It records hybrid_rank (the incoming order) and final_rank (the outgoing order) on every result, which is what lets the dashboard show movement rather than just a final list.
Why this specific query demonstrates the point honestly
The query “cancel at any time” was chosen after testing several candidate queries and checking the actual output — not written first and assumed to work. Most queries tested left the hybrid order unchanged after reranking; this one produces a genuine flip because it’s constructed from an exact phrase that appears verbatim in one passage (doc_refund_p1: “...but you can cancel at any time”) and only in scrambled form in another (doc_cancel_p0: “...cancel your subscription at any time”). Reranking doesn’t guarantee a rank change on every query — it only matters when phrase structure and bag-of-words agreement genuinely diverge, which is exactly what happened here.
Test coverage
Working Demo Link:
Troubleshooting
ModuleNotFoundError: No module named 'lesson_code'— run tests from insidelesson_05_package.No rank change visible for a query you try yourself — expected for most queries; the demo query was specifically chosen to show a genuine flip, not because reranking always changes the order.
dashboard.htmldoesn’t open automatically —start.sh‘s auto-open is best-effort and silently skipped in headless environments; open the file manually.
Real-World Connection
This two-stage pattern — a cheap, broad first pass followed by an expensive, narrow second pass — is the standard shape of production retrieval systems, usually with a trained cross-encoder model instead of today’s bigram heuristic. The principle is identical either way: don’t pay the cost of precise scoring on every passage in the corpus, only on the handful that already made it through the first cut.
Next Steps
Day 6 begins Phase 0’s agent build: the QueryUnderstandingAgent, the first of the five components that will call this retrieval stack as part of a complete pipeline.