Today we build:
A BM25 keyword scorer — the same ranking function real search engines use
A hybrid ranker that blends it with Day 3’s vector similarity
The fix for misranking, and an honest look at where hybrid search has its own blind spot
Why This Matters
Vector similarity alone ranked an unrelated passage above the actual refund policy for a refund question — a real failure, not a contrived one. Vectors capture fuzzy semantic overlap; they don’t guarantee the passage containing the exact term you asked about ranks first. Keyword search has the opposite problem: it can’t tell “cancel” and “terminate” mean the same thing. Today we stop choosing between them.
Core Concept
BM25 scores a passage against a query using term frequency, weighted by how rare that term is across the whole passage set — a term every passage contains tells you nothing; a term only one passage contains is a strong signal. This is the same core algorithm behind most production keyword search, not a simplified stand-in for it.
Blending BM25 with vector similarity isn’t as simple as adding the two scores — they live on different scales. So both get min-max normalized into a 0–1 range first, then combined with a weighted average controlled by
alpha(0.5 splits the weight evenly). The result: a passage that wins decisively on either signal still ranks well, and a passage that’s mediocre on both doesn’t sneak to the top the way it could with vectors alone.
Run today’s code against the refund-policy query from Day 3, and the fix is visible directly:
doc_refund_p0— the actual refund policy — now ranks first, winning on both keyword match and vector similarity. That’s the win. But hybrid search isn’t a universal fix — it’s two imperfect signals combined, not one perfect one. A query built around a single rare word that happens to appear in an unrelated passage can still pull BM25 (and therefore the blend) in the wrong direction. Today’s code makes that failure mode visible too, not just the success.
Architecture
HybridSearcher owns both scorers: a BM25 index built once over the passage set, and Day 3’s embed_text() reused unchanged. search() scores a query against both, normalizes each score list independently, then blends them into one ranked list.
Implementation
GitHub Link
https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_04_package/lesson_04_package
def search(self, query: str, top_k: int = 3) -> list[HybridResult]:
keyword_scores = self.bm25.score(query)
vector_scores = [cosine_similarity(embed_text(query), v) for v in self.passage_vectors]
norm_keyword = normalize(keyword_scores)
norm_vector = normalize(vector_scores)
combined = [
self.alpha * norm_vector[i] + (1 - self.alpha) * norm_keyword[i]
for i in range(len(self.passages))
]
...alpha is a real tuning knob, not a formality — Phase 1’s eval harness (starting Day 13) is what eventually tells you the right value for your own corpus, rather than guessing.
Build, Test, Verify
chmod +x start.sh stop.sh
./start.shstart.sh installs dependencies, builds the Docker image if available, runs the lesson, runs the tests, and generates dashboard.html — open it in your browser to see the result, not a terminal log. Each passage renders as a card with three color-coded score bars: combined (green), keyword/BM25 (orange), and vector (blue), ranked top to bottom.
For this lesson’s query, doc_refund_p0 — the actual refund policy — renders at the top with all three bars near full, while the two unrelated passages visibly trail behind it. Tests: 8 passed.
./stop.shImplementation Guide
Package contents
lesson_04_package/
├── lesson_code.py # BM25, HybridSearcher, normalize, generate_dashboard (+ Day 3's embed_text, cosine_similarity)
├── test_lesson.py # 8 tests verifying BM25, hybrid blending, and dashboard output
├── requirements.txt # pytest only
├── Dockerfile # minimal python:3.11-slim image
├── start.sh # install deps, build, run, test, verify — one command
├── stop.sh # cleanup — removes .venv, dashboard.html, caches, Docker image
└── README.md # quick-start referenceRun it
cd lesson_04_package
chmod +x start.sh stop.sh
./start.shYou should see dashboard.html generated in the directory, and 8 tests passed. Open the dashboard in your browser — doc_refund_p0 should render at the top with the fullest bars across all three scores for the refund-policy query.
When you’re done: ./stop.sh — also removes the generated dashboard.html.
Architecture, in more detail
BM25.__init__() builds a document-frequency index once, over the whole passage set — how many passages contain each term. BM25.score(query) uses that index to score every passage against a query’s tokens, using the standard BM25 formula (k1=1.5, b=0.75 — the commonly used defaults).
HybridSearcher wraps a BM25 instance and a set of Day 3 embeddings computed once at construction time. search() scores both, min-max normalizes each list independently (so neither signal dominates just because its raw numbers happen to be larger), and blends them with alpha.
Why normalize before blending, specifically
BM25 scores and cosine similarities live on completely different numeric ranges — BM25 can produce values like 3.7, cosine similarity is bounded to [-1, 1]. Adding them directly would let BM25’s larger raw numbers dominate the blend regardless of alpha. Min-max normalization rescales each list to [0, 1] independently first, so alpha actually controls the blend the way it’s meant to.
Reading the real results honestly
For the refund-policy query, hybrid search works cleanly — doc_refund_p0 wins on both signals and ranks first. That’s a genuine fix for Day 3’s misranking, verified by running the code, not asserted.
It’s worth testing other queries yourself, because hybrid search isn’t a universal fix. BM25’s IDF weighting means a single rare shared term can still dominate a score the way hash collisions dominated Day 3’s vector similarity — the failure mode changes shape, it doesn’t disappear. This is exactly why Phase 1 (starting Day 13) builds a real eval harness instead of eyeballing a handful of example queries: with only 4 passages, any single example can look great or look bad by chance. A golden dataset and real pass-rate tracking is what actually tells you whether alpha=0.5 is the right choice for your corpus.
Working Demo Link
Test coverage
Troubleshooting
ModuleNotFoundError: No module named 'lesson_code'— run tests from insidelesson_04_package.dashboard.htmldoesn’t open automatically —start.shtriesopen/xdg-openas a best-effort convenience and silently skips if neither is available (common in headless/sandboxed environments); just open the file manually in any browser.Different rankings than expected for a query you try yourself — expected; this is a 4-passage toy corpus, and small corpora amplify exactly the quirks described above. Don’t read too much into any single query’s result at this scale.
Wondering why
alphaisn’t tuned automatically — that’s intentional. Automatic tuning needs an eval harness to measure against, which doesn’t exist until Phase 1.
Real-World Connection
This is the default retrieval architecture in most production search-and-answer systems for exactly this reason: neither signal alone is trustworthy enough to ship. A support system relying on vectors alone will occasionally miss the passage with the exact policy number a user asked about; one relying on keywords alone will miss a passage that says the same thing in different words. Combining them is standard practice, not an advanced technique.
Next Steps
Day 5, we add reranking — a second pass over the top candidates that can use more expensive, more accurate scoring now that hybrid search has already narrowed the field.