Lesson 17: Deterministic Checks — The Fast, Cheap First Layer

Free Resources: Build AI Agents That Remember Users Across Sessions

Your agent can follow a rule perfectly at the start of a conversation and miss it three turns later. The model is stateless. Memory is a separate layer and has to be designed intentionally.

We put together a free resource hub for data engineers and ML practitioners building agents that remember users across sessions:

• Explainer videos on what agent memory is and how to design it

• Tutorials for adding persistent memory to CrewAI, mem0, and LangChain

• A full offline RAG implementation running on edge hardware with no internet required

Organized by level. Free. All implementations include working code.

---

Today we build:

• Seven independent, named check functions, replacing Days 14–16’s informal inline “reasons” list

• Two new checks — citation validity and non-empty answers — that pass today but exist for defense in depth

• A per-check breakdown across all 17 golden cases, not just a single pass/fail flag per case

Why This Matters

Every check written since Day 14 lived inside one function, score_case(), with content presence, prohibited phrases, and citation requirements all tangled into a single loop appending strings to a reasons list. That worked at 15 cases. It doesn’t scale into a real eval harness, where you need to answer questions like “which specific check fails most often” or “can I run just the citation check in isolation” — questions the tangled version can’t answer without re-reading the whole function each time.

Core Concept

Seven functions, each taking a PipelineTrace and a GoldenCase and returning exactly one CheckResult — the same name/passed/reason shape Day 8’s CriticAgent already established for the pipeline itself. check_scope_match and check_category_match verify classification; check_content_present and check_content_absent verify the answer contract from Day 13; check_citation_present and the new check_citation_valid verify sourcing; check_answer_not_empty is the simplest possible sanity check, run independently rather than folded into content presence.

Running all seven against all 17 golden cases produces a genuinely more informative picture than a single pass/fail ever could. golden_010 (”I want to unsubscribe”) fails exactly content_present and content_absent — the answer exists, has a citation, and is even correctly categorized, it just contains the wrong facts. golden_005 fails five of seven checks at once — scope_match, category_match, content_present, citation_present, and answer_not_empty — because a misrouted out_of_scope classification cascades into every downstream check having nothing to verify. Both cases show as “failed” in a boolean summary; only the per-check breakdown shows they’re failing for completely different reasons.

The two new checks are worth being honest about: citation_valid (does a cited passage ID actually exist in the corpus) and answer_not_empty (independent of content correctness, is there any text at all) both pass on every one of the 17 current cases. They’re not catching anything today. They exist because the class of bug they’d catch — a citation to a passage ID that’s been renamed or removed, or a synthesis path that returns whitespace — is different in kind from a wrong-content bug, and a checks layer that only tests for problems already known to exist isn’t really testing anything.

Architecture

ALL_CHECKS is a plain list of functions — adding an eighth check means writing one function and appending it to the list, nothing else changes. run_checks() runs the full list against one case; run_all_checks() runs it across a dataset. ScoredCase.passed is a property, not a stored field, computed fresh from all(c.passed for c in self.checks) every time it’s read.