Today we build:
A document ingestor that turns raw text into uniform, retrievable passages
A fixed-size chunking strategy with overlap, so context doesn’t get cut mid-thought
The passage store that Day 4’s retrieval stage will search over
Why This Matters
Yesterday’s
RetrievalStagewas a hardcoded dictionary — real answers to three fake questions. Today it becomes real, because real documents don’t arrive in question-answer pairs. They arrive as long, unstructured text: policy docs, support articles, transcripts. Before any search can work, that raw text has to become a set of small, consistent units a retrieval system can actually compare against a query. Get chunking wrong here, and every later phase inherits the mistake — no eval harness or LLM judge can fix an answer built from a badly cut passage.
Core Concept
A document is too big to search directly. If you embed and compare a 4,000-word support article against a 12-word question, the match is diluted — the one relevant sentence gets buried in three thousand irrelevant ones. So we split documents into passages: smaller chunks a retrieval system can score individually.
The chunking decision has a real trade-off. Chunks too large re-introduce the dilution problem. Chunks too small lose context — a passage that ends mid-sentence, or separates a term from its definition, can’t be understood on its own even if it’s retrieved correctly. Today we use fixed-size character chunking with overlap: each passage shares a small window of text with its neighbor, so an idea that spans a chunk boundary still appears whole in at least one passage.
Every passage also carries metadata: which document it came from, and where in that document it sits. This isn’t optional bookkeeping — Day 4’s retrieval stage needs it to return results a user can trust, and Day 3’s embeddings need a stable ID to attach a vector to.
Architecture
Ingestion sits before the pipeline, not inside it: it’s a one-time (or periodically re-run) process that produces the passage store Stage 2 (Retrieval) will query. Raw documents go in one end; a flat list of Passage objects — each with an ID, source document, character offsets, and text — comes out the other.
Implementation
GitHub Link
https://github.com/sysdr/ai-reliability-engineering/tree/main/lesson_02_package/lesson_02_package
Two data contracts, matching the discipline from Day 1:
@dataclass
class Document:
doc_id: str
source: str
raw_text: str
@dataclass
class Passage:
passage_id: str
doc_id: str
text: str
char_start: int
char_end: intThe ingestor does one job per method: load_documents() reads raw text in, chunk_document() splits one document into overlapping passages, and ingest_all() runs both across every document and returns the full passage store.
Build, Test, Verify
chmod +x start.sh stop.sh
./start.shExpected output:
Loaded 3 documents
Document 'refund_policy.txt' -> 3 passages
Document 'cancellation.txt' -> 1 passages
Document 'pricing.txt' -> 1 passages
Total passages ingested: 5
Passage store saved -> passages.json
== Running tests ==
5 passed./stop.shImplementation Guide
Package contents
lesson_02_package/
├── lesson_code.py # Document, Passage contracts + DocumentIngestor
├── test_lesson.py # 5 tests verifying chunking behavior
├── requirements.txt # pytest only
├── Dockerfile # minimal python:3.11-slim image
├── start.sh # install deps, build, run, test, verify — one command
├── stop.sh # cleanup — removes .venv, passages.json, caches, Docker image
└── README.md # quick-start reference
Run it
cd lesson_02_package
chmod +x start.sh stop.sh
./start.shstart.sh creates a virtual environment, installs requirements.txt, builds the Docker image if Docker is present, runs lesson_code.py, then runs test_lesson.py. You should see 3 documents ingested into 5 total passages, a passages.json file written to disk, and 5 passed.
When you’re done: ./stop.sh — removes the venv, the generated passages.json, all caches, and the Docker image if one was built.
Architecture
Two dataclasses carry the data:
Type Fields Produced by Document doc_id, source, raw_text load_documents() Passage passage_id, doc_id, text, char_start, char_end chunk_document()
DocumentIngestor.ingest_all() is the entry point: it calls load_documents() once, then chunk_document() once per document, and returns a flat list of Passage objects — the passage store.
Why fixed-size chunking with overlap, specifically
chunk_size=220 characters keeps each passage small enough that a single topic dominates it, without cutting it down to a fragment.
overlap=40 characters means each new chunk starts 40 characters before the previous one ended. If a sentence spans the boundary between chunk N and chunk N+1, it appears complete in chunk N+1 even though it was cut off in chunk N.
Short documents (shorter than
chunk_size) produce exactly one passage — there’s nothing to split, and the loop exits after the first pass sinceend == len(text).
This is deliberately the simplest correct chunking strategy. Production systems often chunk on sentence or paragraph boundaries instead of raw character counts — that’s a refinement worth making later, but it changes chunk_document()‘s internals only. The Passage contract and everything downstream stays the same, which is the entire point of the typed-boundary discipline from Day 1.
Working Demo Link:
Test coverage
Troubleshooting
ModuleNotFoundError: No module named 'lesson_code'— run tests from inside thelesson_02_packagedirectory.Different passage counts than expected — passage counts depend on the exact
chunk_sizeandoverlapvalues inDocumentIngestor(); if you change them, the counts in the article and README will no longer match your run — that’s expected, not a bug.passages.jsonalready exists from a previous run —start.shoverwrites it each run;stop.shremoves it during cleanup.
Real-World Connection
This is the step most teams skip past without thinking about — and the one that quietly determines whether retrieval works at all. A support-triage system that chunks a refund policy document badly will confidently retrieve a passage that’s missing the one sentence that mattered, and no amount of prompt tuning downstream fixes that. Chunking strategy is infrastructure, not a detail.
Next Steps
Day 3, we turn these passages into embeddings — numeric vectors that let the retrieval stage compare a question against thousands of passages by meaning, not just keyword overlap.