Today we build:
A weighted composite score across Day 17’s seven checks, plus a critical-check override
A finding that composite scoring changes nothing on the real 17-case dataset — every real failure is critical
One clearly labeled synthetic case, built specifically to prove the scorer’s actual behavior
Why This Matters
Day 17’s seven checks judged a case with a strict AND — one failed check, of any kind, fails the whole case. That treats a stale citation the same as a factually wrong answer, which isn’t how severity actually works. A composite scorer should tolerate minor, low-stakes failures while remaining absolute about the checks that matter — but “should” isn’t good enough without checking whether it actually behaves that way.
Core Concept
Each of the seven checks gets a weight (
content_presentandcontent_absenthighest at 0.25 and 0.20,citation_validandanswer_not_emptylowest at 0.05 each, summing to 1.0), and two of them —content_present,content_absent— are marked critical: failing either one forces an automatic fail regardless of the weighted score. A case can lose points on citation formatting and still pass; it can never pass with wrong content, no matter how well everything else scores.Running this against all 17 real golden cases produced an honest, slightly anticlimactic result: composite scoring changes nothing.
12/17pass, identical to Day 17’s strict count, because every one of the 5 real failures hits a critical check —golden_002,golden_005,golden_010, andgolden_013all failcontent_presentorcontent_absentdirectly, andgolden_008failscontent_presenttoo. There is no real case in the current dataset that fails only a low-weight check. Reporting that finding honestly, rather than picking different weights until something looked more interesting, is the actual discipline here.To prove the composite scorer does what it’s supposed to do, one synthetic case was built — the same technique Day 8’s corrupted answer and Day 13’s rephrased synthesis used: take a real, passing case (
golden_001) and deliberately break exactly one non-critical check (citation_valid, simulating a stale citation), leaving everything else untouched. Strict all-pass fails it outright, since one check failed. The composite scorer scores it0.95— above the0.85threshold, no critical check touched — and passes it. That’s the actual, verified divergence the real data didn’t happen to produce.
Architecture
CompositeScorer.score() takes a ScoredCase from Day 17 and returns a CompositeResult: the weighted score, whether any critical check failed, and a final pass/fail verdict combining both conditions. build_synthetic_partial_failure_case() is clearly named and clearly labeled in every output — the dashboard marks it “SYNTHETIC CASE — NOT A REAL PIPELINE RUN” in its own visually distinct section, never mixed into the real 17-case table.
Implementation
def score(self, scored_case: ScoredCase) -> CompositeResult:
composite_score = sum(CHECK_WEIGHTS[c.name] for c in scored_case.checks if c.passed)
critical_failure = any((not c.passed) and c.name in CRITICAL_CHECKS for c in scored_case.checks)
verdict = "pass" if (composite_score >= PASS_THRESHOLD and not critical_failure) else "fail"
return CompositeResult(scored_case, composite_score, critical_failure, verdict)
Build, Test, Verify
chmod +x start.sh stop.sh
./start.shstart.sh installs dependencies, builds the Docker image if available, runs the lesson, runs the tests, and generates dashboard.html. The real results table shows all 17 cases with their composite score and critical-failure flag; a separate, dashed-border section below shows the synthetic case’s side-by-side comparison. Tests: 7 passed, including a direct check that the synthetic case’s strict result is False while its composite verdict is "pass".
./stop.shImplementation Guide
GitHub Link
https://github.com/sysdr/ai-reliability-engineering-p/tree/main/lesson_18_package/lesson_18_package
Package contents
lesson_18_package/
├── lesson_code.py # CHECK_WEIGHTS, CRITICAL_CHECKS, CompositeScorer, build_synthetic_partial_failure_case
├── test_lesson.py # 7 tests verifying weights, critical override, and the synthetic divergence
├── requirements.txt # pytest only
├── Dockerfile # minimal python:3.11-slim image
├── start.sh # install deps, build, run, test, verify — one command
├── stop.sh # cleanup — removes .venv, dashboard.html, caches, Docker image
└── README.md # quick-start reference
Run it
cd lesson_18_package
chmod +x start.sh stop.sh
./start.shYou should see 12/17 pass at threshold 0.85, an explicit note that this matches Day 17’s strict count, the synthetic case’s divergent result printed directly, dashboard.html generated, and 7 tests passed.
Cleanup: ./stop.sh — also removes the generated dashboard.html.
Architecture, in more detail
CHECK_WEIGHTS is a plain dict summing to 1.0, asserted at module load time (assert abs(sum(CHECK_WEIGHTS.values()) - 1.0) < 1e-9) so a typo in the weights fails loudly and immediately rather than silently producing scores that don’t mean what they claim to mean. CRITICAL_CHECKS is a set of check names, not weights — critical status is binary (a check either can override the composite score or it can’t), separate from how much it contributes when it passes.
CompositeScorer.score() computes the weighted sum only from checks that passed — a failed check simply contributes its weight to what’s missing, rather than requiring separate penalty logic. The critical override is checked independently, as a second, unrelated condition that can veto an otherwise-passing score.
Why the honest 12/17 finding matters more than it might seem
It would have been easy to quietly pick check weights and a threshold that produced a more dramatic-looking divergence on the real data, or to only show the synthetic case without mentioning that real data doesn’t need it. Neither happened here. The lesson runs composite scoring against all 17 real cases first, reports plainly that the result is identical to Day 17, and only then introduces a synthetic case — clearly labeled as synthetic in the code, the console output, and the dashboard — to demonstrate the capability the real data doesn’t exercise. A scoring system’s tolerance is either real or it isn’t; asserting it without a concrete example that requires it would be exactly the kind of unverified claim this course has avoided since Day 1.
Why build_synthetic_partial_failure_case starts from a real passing case
The synthetic case is built by copying golden_001‘s actual, real check results and flipping exactly one — citation_valid — from pass to fail. This is deliberate: starting from a genuinely passing real case and changing the smallest possible thing isolates the composite scorer’s behavior to that one variable. A synthetic case built from scratch, with hand-picked check results for every field, would be less convincing — it’s easier to trust a controlled single-variable change than a fully invented scenario.
Test coverage
Working Demo Link:
Troubleshooting
ModuleNotFoundError: No module named 'lesson_code'— run tests from insidelesson_18_package.Changing
CHECK_WEIGHTSbreaks the module on import — the sum-to-1.0 assertion is intentional; rebalance all weights together, not just one.dashboard.htmldoesn’t open automatically —start.sh‘s auto-open is best-effort and silently skipped in headless environments; open the file manually.
Real-World Connection
Reporting “the new scoring method didn’t change anything on real data yet” is a less exciting result than reporting a dramatic improvement, and it’s also the honest one — and honest negative results are what make the positive ones credible later. A composite scorer whose tolerance has never actually been exercised isn’t proven; constructing the one case that exercises it, and saying plainly that it’s constructed, is what separates a demonstrated capability from an assumed one.
Next Steps
Days 19 through 21 build the eval runner that connects this scorer to the live pipeline continuously, tracking pass rate over time instead of a single point-in-time snapshot.
The hard-fail gate is a strong design choice—weighted scores can hide a single catastrophic miss. How do you handle false positives in those critical checks?