[ndisidore]

Lets gadgets bind Cloudflare's Clef and Clef Flash. These are decision models: JSON state + typed questions in, a probability per allowed answer out. No chat, tools or text output, so they can't back a chat, a spawned agent or LanguageModelBinding.run(). Gadgets get a new DecisionModelBinding instead:

const { intent } = await env.DECISION_MODEL.decide({
  state: { message: "My order never arrived." },
  questions: { intent: { type: "choice", instructions: "What does the customer want?",
    criteria: { refund: "Money back or a replacement", other: "Anything else" } } },
});
// { type: "choice", choice: "refund", probabilities: { refund: 0.96, other: 0.04 }, confidence: 0.86 }

• ModelKind ("language" | "decision") is part of the shared API. SUGGESTED_MODELS entries carry kind, and modelKind(provider, model) reads it.
• Reuses the AI Model connection. newAiModelGatekeeper() mints a DecisionModelGatekeeper when the model's kind is "decision". Creation spec, sharing and analytics unchanged.
• Never offered to chats or spawners: listModels(kind = "language") lists one kind, so listModels("decision") returns the decision models and a future kind needs no new method.
• Blueprint aiModel bindings record modelKind: "decision", so installers pick among decision models.
• state is any (JSON-compatible), as JSON.parse() returns; Workers AI also takes plain text there.
• Routing follows getModel(): direct with the user's Cloudflare creds, or through the platform AI Gateway. On the WORKERS_AI binding that's binding.run(model, input, {gateway}), since the gateway's passthrough refuses /run.
• Admin "Disabled" stops bound gadgets. No quota or cost accounting, same as the LLM binding today.

Known gap: doesn't work on deployments that reach AI Gateway over HTTPS (gateway in another account). The gateway supplies Workers AI's credential only for chat, and reusing the gateway token for /run gets a 401 even when it has Workers AI permissions. Checked live on three accounts. Docs say to disable Clef there; I raised this with the AI Gateway team and am waiting to hear back. Deployments on the default binding route are unaffected.

Naming: "decision model", the industry term and the one Cloudflare's Clef docs use. pi calls these "classifiers" (type: "classifier", classify()), so its own Classifier* names remain only where the code calls pi.

Prompt (against pi): "Please look at the last 10 commits in this repo and classify as either a new feature, a bugfix, docs changes, a refactor, or none of the above"

[ask-bonk]

LGTM!

github run

[github-actions]

Preview: pr662-feat-classifi-0f0dbda7

https://pr662-feat-classifi-0f0dbda7-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run

[devin-ai-integration]

Devin Review found 1 new potential issue.

[ask-bonk]

LGTM!

github run

[github-actions]

Eval results

Verdict: ⚪ Unchanged. No task moved beyond what 10 runs can tell apart from noise.

Task	Score	Δ score	Fisher test	Cache hits	Avg min	Avg steps
change-calendar	90% → 70%	−20 pp	p = 0.58	95% 0 pp	2.8 → 3.0	26.7 → 23.6
chess	80% → 100%	+20 pp	p = 0.47	98% → 97% −1 pp	6.9	64.5 → 52.3
incident-desk	100% → 90%	−10 pp	p = 1.00	97% → 96% −1 pp	3.6 → 3.5	38.6 → 29.4
worker-logs	100% → 90%	−10 pp	p = 1.00	95% → 96% +1 pp	3.5 → 3.1	26.4 → 26.5

Failed checks

Task	Check	Failed
change-calendar	t1 schedules-and-lists-windows	0/10 → 1/10
change-calendar	t1 rejects-invalid-windows-without-changing-anything	0/10 → 1/10
change-calendar	t1 overlap-is-per-service-and-touching-is-allowed	0/10 → 1/10
change-calendar	t2 plan-is-a-document-listing-exactly-the-weeks-windows	0/10 → 1/9
change-calendar	t4 names-the-booked-windows-the-new-rules-reject	1/10 → 0/8
change-calendar	t4 earliest-available-applies-every-rule	0/10 → 1/8
change-calendar	t4 earliest-available-matches-the-reference	0/10 → 1/8
change-calendar	t4 earliest-available-slots-are-bookable	0/10 → 1/8
change-calendar	t4 earliest-available-sees-new-bookings	0/10 → 1/8
chess	t1 agrees-with-the-oracle-on-the-hard-positions	2/10 → 0/10
chess	t1 agrees-with-the-oracle-on-perft-positions	2/10 → 0/10
chess	t1 agrees-with-the-oracle-through-random-games	2/10 → 0/10
incident-desk	t1 opens-acknowledges-and-resolves-in-order	0/10 → 1/10
incident-desk	t1 simultaneous-acknowledges-yield-exactly-one-owner	0/10 → 1/10
incident-desk	t1 simultaneous-opens-of-one-id-admit-exactly-one	0/10 → 1/10
incident-desk	t1 board-lists-by-severity-then-age	0/10 → 1/10
worker-logs	t1 ingests-resets-and-summarises-per-worker	0/10 → 1/10
worker-logs	t1 hourly-buckets-cover-every-hour-including-empty-ones	0/10 → 1/10
worker-logs	t1 ranges-are-half-open-and-filter-by-worker	0/10 → 1/10

Run · trajectories and raw results

[ask-bonk]

🔬 Eval runs review

Performance

Across 10 runs per task, main → PR pass rates were change-calendar 90% → 70%, chess 80% → 100%, and incident-desk and worker-logs 100% → 90%; comparison.json calls all four unchanged, with pass-rate p-values ≥ 0.47. Mean run costs fell respectively from $0.0206 → $0.0184, $0.0552 → $0.0475, $0.0274 → $0.0215, and $0.0224 → $0.0192, but variation was substantial and several candidate failures stopped early; cache hits stayed around 95–98%, with no significant hit-rate or break-rate changes. Calendar lost the most runs, while chess remained the largest cost and step consumer: 64.5 → 52.3 model steps at roughly 6.9 minutes on both sides, with PGN escaping repairs and exact-text edit retries adding detours.

⚪ VERDICT: NO REGRESSION FROM THIS PR

The candidate failures trace to generated gadget mistakes or an unchanged verifier, not the decision-binding additions or blueprint labels, and the pi upgrade has no demonstrated beyond-noise improvement or regression in these runs.

Triage

Failure modes

• Rejecting valid timestamp spellings · change-calendar 1/10 · model error · this PR: no — Trial 4, turn 1, writeFile(server.js) required timestamps to equal toISOString(), rejecting valid inputs without .000; its executeCode smoke test used only .000Z timestamps and therefore missed the defect.
• Undefined slot-search helper · change-calendar 1/10 · model error · this PR: no — Trial 6, turn 4, editFile(server.js) introduced calls to ceilQuarterHour without defining it. Subsequent executeCode verified the booking-audit reply, not earliestAvailable, leaving every search check throwing; baseline searches passed.
• Human-readable headings rejected · change-calendar 1/10 · verifier · this PR: no — Trial 7, turn 2, executeCode(setDocument) produced the correct windows under “API Gateway”, “Billing”, and “Edge Cache”. The prompt asks for service headings without requiring literal slugs, but the unchanged verifier requires headings to contain api-gateway and edge-cache; baseline documents used those slugs.
• Empty SQL lookup treated as nullable · incident-desk 1/10 · model error · this PR: no — Trial 8, turn 1, writeFile(server.js) used .one() for the duplicate-ID lookup before inserting a new incident, so the empty lookup threw and nothing opened. There was no RPC smoke test; baseline implementations handled absent rows with arrays or conditional inserts.
• Oversized SQL insert batches · worker-logs 1/10 · model error · this PR: no — Trial 9, turn 1, writeFile(server.js) batched 100 events with six bindings each, exceeding the runtime’s SQL-variable limit. Its executeCode test ingested only three events successfully; the verifier’s larger ingest failed and left downstream reports empty, whereas baseline row-at-a-time inserts worked.

Tool errors

• editFile: No matching text was found in the file. · chess 5 → 11, incident-desk 1 → 2, worker-logs 7 → 1 · model error — Agents quoted text that differed from the file, notably double-escaped regexes on both sides; candidate chess trial 3 recovered by grepping halfText and using the literal source.
• editFile: Multiple matches were found. The text to match must be unique. · chess 6 → 0, incident-desk 9 → 4, worker-logs 2 → 1 · model error — Baseline chess retried repeated method-call fragments; candidate incident-desk trial 7 submitted the same repeated SQL fragment twice, then recovered by including surrounding statements.
• editFile: Validation failed for tool "editFile": · change-calendar 1 → 1, chess 2 → 1, incident-desk 7 → 3, worker-logs 1 → 5 · model error — Calls omitted required fields or used the wrong argument names. Candidate worker-logs trial 2 omitted filename, then supplied it successfully; baseline calls made similar omissions.
• readFile: File does not exist. · chess 2 → 2, incident-desk 4 → 2 · model error — Agents attempted to read server.js and client.js immediately after creating empty gadgets, despite the prompt explaining that scratch gadgets have no files; they recovered by writing them.
• editFile: CHESS has no file named "CHESS". · chess 2 → 1 · model error — The binding name was also passed as filename. Candidate trial 2’s turn-3 documentation edit succeeded after changing filename to README.md; the baseline likewise corrected the filename.

What to do

• Optional follow-up in packages/workshop-evals/evals/change-calendar.eval.ts: normalize service headings for spaces versus hyphens before comparison, so defensible display names do not count as failures.
• Optional follow-up in packages/workshop-backend/src/agent.ts: add guidance to exercise newly added RPCs with representative inputs before declaring completion, including realistic batch sizes, and clean up probe data afterward. These runs do not require a regression fix in this PR.

github run

[ndisidore]

@kentonv actioned against all your feedback and ready for re-review.

[kziemski]

🎉