Swap in a new frontier model, point it at your warehouse, and let it answer revenue questions. That's the pitch behind most “chat with your data” products in 2026, and it's also how a plausible-sounding SQL query quietly overcounts a metric for a board deck. The failure mode isn't that the model is bad — it's that nobody checked whether this specific model, on this specific class of question, has ever been good enough to trust.
Models are not interchangeable parts
Most agent platforms treat the underlying LLM as a config value: swap the model string, redeploy, done. That works for a chatbot that drafts emails. It does not work for an agent that writes SQL against a production warehouse and hands the result to a VP as ground truth. A model that's excellent at conversational reasoning can still misresolve a self-join, silently drop a WHERE clause a business user implied but didn't say, or hallucinate a column that sounds right. The gap between “sounds confident” and “is correct” is exactly where a semantic layer earns its keep — but only if you know which models are allowed near it in the first place.
A registry, not a config flag
Inside DataAgents, every model we route production traffic to is backed by a capability registry: a record, derived from reliability scorecards, of what that specific provider/model pairing has actually demonstrated it can do. A model doesn't get a single yes/no. It gets a tier per task class:
- Certified — cleared for the task tier, routes automatically.
- Restricted — allowed only behind a narrower guardrail (e.g. read-only, single-table queries).
- Unverified — no scorecard on file yet; the router will not send it production traffic.
A model can be certified for “summarize this dashboard in plain English” and simultaneously restricted for “write the SQL that computes contribution margin across three joined fact tables.” Capability is scoped to what was actually tested, not to the model's reputation.
How a model earns a tier
Scorecards run each candidate model against a fixed set of required test tiers before it's eligible for any tier above unverified — semantic SQL correctness against known-good query results, alias resolution across joined tables, resistance to inventing columns or tables that don't exist, and consistency across repeated runs of the same question. A model has to clear every required tier for a task class, not just score well on average, before the registry marks it certified for that class. This is a hard gate, not a soft signal: if a model isn't certified for a task tier, the router doesn't fall back to it quietly with a lower confidence score attached — it doesn't get routed there at all.
{
"provider": "anthropic",
"model": "claude-sonnet-5",
"task_tiers": {
"summarization": "certified",
"single_table_sql": "certified",
"multi_join_semantic_sql": "restricted",
"schema_migration_planning": "unverified"
},
"last_scorecard_run": "2026-07-12T00:00:00Z"
}Certification isn't a substitute for validation
The registry decides which model is allowed to attempt a task. It doesn't decide whether that attempt was correct — that's a separate, mandatory step. Every query a certified model produces still passes through an alias-aware semantic SQL validation contract before it touches the warehouse: column and table references are checked against the actual schema and the semantic layer's definitions, not just against what the model claims exists. A certified model that writes a plausible but wrong query still gets caught at validation time. Certification narrows which models get to try; validation catches the ones that tried and got it wrong anyway. Neither one alone is enough.
Why this matters more in 2026 than it did a year ago
New frontier models ship every few weeks now, and the temptation to route to “whatever's newest” is constant — newer usually means better on general benchmarks. But general benchmarks don't tell you whether a model correctly handles your specific fact-table grain, your specific definition of “active customer,” or your specific history of ambiguous column names. Treating model access as a governed resource — the same way you'd govern warehouse credentials — means a model upgrade is an event that produces a new scorecard, not a silent swap that changes what your dashboards say overnight.
The takeaway
If your data agent can name the model that answered a question, it should also be able to say why that model was allowed to. “It's a good model” is not an access control policy. A scorecard, a tier, and a hard gate are.