Model Policy

Every model call in the agent names a role, never a model id. A resolver turns the role into a model (and a fallback chain) from the agent's settings, the tenant's policy and platform defaults, with health, canary and budget rules applied on top. Switching models — per agent, per business, or for one role only — is a settings change, not a code change.

Code: manage/src/agent/model-policy.ts (resolver, pure), model-health.ts (health window), the eval runner in index.ts.

Roles

Role	Used for	Platform default
chat	plain conversational turns	@cf/meta/llama-4-scout-17b-16e-instruct
tool	turns where action tools are loaded (anything beyond the core category)	derives from chat
escalation	one re-run of a turn that went wrong (unbacked claim, failure cap)	none unless set; otherwise the next link of the tool chain
decision	typed decisions — routing, gates, task close, claim check, browser_task	@cf/cloudflare/clef → clef-flash
extract / distill	structured JSON jobs, learned-skill distillation	kimi-k2.6 → glm-4.7-flash
classify	the app-intent classifier	llama-3.3-70b → glm-4.7-flash
reflect	scheduled reflection	glm-4.7-flash → llama-4-scout
vision	screenshot / image description	the chat model when it has vision, else Llama 4 Scout
coding	Code Mode generation	the chat model when it can code, else Gemma 4

Resolution order

For a role, the first layer that yields an eligible model wins:

1. Turn override — evals and retries pin a model for one turn.
2. Agent settings — settings.llm.roles.<role>. The older fields model, toolModel, escalationModel, visionModel, codingModel are read as aliases, so existing agents behave as before.
3. Tenant policy — tenants.policy.models.<role>, pushed by the product over the platform API. This is how a business plan sets models for all of its agents.
4. Platform defaults — the table above.

Each layer may give a chain: ["@cf/zai-org/glm-5.2", "@cf/moonshotai/kimi-k2.6"] means GLM-5.2 first, Kimi on failure or escalation.

Eligibility

An entry is skipped — with the reason recorded — when the model

• lacks the role's capability (tool/escalation need tool calling, vision needs vision, coding needs coding);
• is external (openai/…, anthropic/…, google-ai-studio/…) and the agent has no AI Gateway settings;
• is unknown to the catalogue (Workers AI ids without a capability requirement are allowed through);
• is currently unhealthy (below).

GET /api/agents/:id/models/policy returns every role's resolution with its source, chain and skipped entries, plus per-model health. The LLM Models page shows the same table.

Health

Each agent keeps a rolling window of the last 20 calls per model in Durable Object storage: success, latency, last error. A model with at least 3 failures at a failure rate of 50% or more — or a median latency above 45 s over a full window — is skipped by the resolver for 10 minutes, then tried again. A Kimi turn that hangs for 75 s and returns nothing therefore moves the tool chain to its next link on its own; nothing needs flipping.

Canary

A role entry may name a candidate for a share of traffic:

{ "roles": { "tool": { "model": "@cf/zai-org/glm-5.2", "fallbacks": ["@cf/moonshotai/kimi-k2.6"],
                       "canary": { "model": "@cf/google/gemma-4-26b-a4b-it", "percent": 10 } } } }

Assignment is sticky per conversation (a hash of role + conversation id), so a thread stays on one arm. The configured chain remains behind the canary as its fallback, and model_switch rows carry source: "canary". Canaries are suppressed while the budget is tight.

Budget tier

settings.llm.budget.dailyUsd sets a daily cap. Today's spend is estimated from token_usage and the pricing table; past 80% of the cap, the chat, tool and escalation chains are reordered cheapest-first (source: "budget"). Background roles (distillation, reflection, decisions) are unaffected. The hard stop remains the prepaid credit balance.

Switching inside a turn

The loop resolves tool once per turn and switches to it when action tools are loaded; a model_switch decision-log row records from, to and source. After an unbacked claim or the tool-failure cap it escalates once — to the escalation model if set, else the next link of the tool chain — and never escalates during an eval.

Evaluating a candidate

POST /api/agents/:id/models/eval { "model": "@cf/zai-org/glm-5.2", "role": "tool" } runs six representative business tasks (calendar with ISO and prose dates, CRM contact, a four-step task, a lead with a linked contact, a plain question) on the candidate with a dry-run executor: every tool returns a canned success, nothing is created. Each task is scored on whether the expected tools were called, nothing forbidden was called, and the reply did not claim an action without a tool (the claim check's regex). Runs are stored in model_evals and listed by GET /models/evals; the LLM Models page has a Run eval control.

The first evaluation (2026-10-01, adappter) is what the defaults rest on: GLM-5.2 7/7 clean, Kimi K2.6 5/7, glm-4.7-flash 7/7 only because the claim check rescued three fabricated replies, Gemma 4 and GPT-OSS-120B failing the multi-step task.

Pricing

model-pricing.ts carries USD per 1K tokens for every catalogue model and feeds both the budget tier and credit debits. Clef is $0.24 per M input tokens, Clef-flash $0.09 (no output charge).