Make the code-check versus LLM-judge lesson visible earlier in the Anthropic workshop #13

Thank you for the approachable Sparkles workshop. Feedback from attending the course: the distinction between passing structural checks and actually helping the user deserves to be a headline takeaway.

Module 2.4 already explains this well, including the refund example and the side-by-side comparison. This suggestion is about making that existing lesson easier to discover, especially for attendees who finish the code evaluator but do not reach or deploy the optional judge endpoint.

Concrete example

The bundled fourth row in anthropic/sparkles-evals/sample_runs.jsonl tells a customer with damaged cupcakes that all sales are final. The store policy allows a full refund or replacement for damaged orders reported within 48 hours with a photo.

The row receives 1.00 from grade_sparkles.py because all five required HTML elements are present and the receipt passes its checks. The grader never reads the customer's query or the agent's response. Module 2.4 shows how the policy-grounded Claude judge can catch that different class of failure. The documented model score is illustrative, not a new judge run performed for this feedback.

Suggested small contribution

1. Put this example and a link to Module 2.4 near the workshop introduction as a core learning objective: a passing score only covers what the evaluator actually observes.
2. After a code-only evaluation, print a short scope note: "These scores check kiosk structure and receipt fields; they do not evaluate answer correctness or policy adherence. Compare the refund example in Module 2.4."
3. Provide a short discussion exercise using the existing comparison table when the judge endpoint is unavailable, clearly labeling the results as illustrative.
4. Add a brief production takeaway: keep deterministic checks for exact constraints and verified tool outcomes; use a grounded LLM judge for contextual response quality, and validate its rubric against human-scored examples. An answer claiming it checked stock is not itself evidence that a stock tool ran.

The proposal is to teach complementary coverage, rather than suggest that rules always fail or that an LLM judge guarantees correctness. Rules can miss contextual failures outside their scope, and model judges have their own blind spots and variability.

Would a focused documentation change plus the code-only output note be welcome?