Repository navigation
feat(evals): add LLM-as-judge scoring #8514
Description
Problem
The eval suite scores answers with substrings and regexes. Those measure
phrasing, not correctness: across three live runs every failure was a valid
paraphrase or an over-specific assertion, not a wrong answer. Open-ended
behavior — is the answer grounded in the tool results? does it answer the
question? — cannot be scored this way at all.
Proposal
Add LLM-as-judge scoring alongside the deterministic checks.
- A judge model scores an answer against a weighted rubric (grounding,
completeness, …) and returns structured numbers. runScenariogains an optionaljudge; when present it adds ajudgecheck.- The judge transport is an injectable OpenAI-compatible completion, so a
recorded transcript can replay it deterministically in CI.
Location: apps/sim/evals/agent-tool-use/judge.ts.
Acceptance criteria
-
judgeAnswerscores a rubric and returns a weighted verdict - Malformed/missing-score judge responses are rejected, scores clamped
- Key-free unit tests for parsing, weighting, and clamping
- A live test proves a grounded answer outscores an invented one
-
runScenariocan attach a judge check
Activity
ViridianAIQualityAssurance commentedon Oct 1, 2026
This looks like the right direction. One thing I'd make explicit before a judge score is allowed to influence a regression/release decision is judge comparability, not just judge availability.
The recorded-transcript replay is useful for transport/parser determinism, but it does not establish that two judge-produced scores are comparable if the judge identity or evaluation contract changes. I would persist a small identity envelope with each result:
Then fail closed on comparative claims when a material field differs. A candidate score of 0.79 vs a baseline of 0.71 is still a valid observation, but it should not automatically become evidence of improvement if one arm was judged under a materially different evaluator.
I'd also add one frozen, human-adjudicated calibration slice. “Grounded answer outscores invented answer” is a good sanity check; a small fixed reference set gives you a way to detect judge drift rather than treating the judge as ground truth.
I work on Viridian, an experimental-assurance project, so disclosure: this is directly in the area we're building around. We published a synthetic example of this exact failure mode (score goes up, evaluator identity changes, comparative claim becomes
INSUFFICIENT_EVIDENCE):https://github.com/ViridianAIQualityAssurance/Viridian/blob/main/docs/experimental-assurance-audit-model-comparison.md
If you ever want this specific judge/comparability boundary reviewed as one bounded case, the written intake is here:
https://docs.google.com/forms/d/e/1FAIpQLScRhl7hZfzJbxi2ittQ2nNGawK9X_c67ERjhBrIVsmPuxsGxg/viewform