A tenant's nightly dbt run finished with 236 of 239 models materialized. Three marts failed — one of them generated by an agent that had referenced a column an upstream source dropped that week. The run exited non-zero, the parent pipeline logged "dbt run failed," the publication gate refused the revision, and the 236 tables that had actually built sat finished-and-untouched in object storage while the tenant waited for a full rerun.
Nobody lost data. Everybody lost a day. And the part that should bother you is that the pipeline had all the information it needed to do better — it just threw it away one line above the decision.
The exit code is one bit. The outcome has three values.
dbt already does the right thing here. When a model fails, dbt does not roll back the models that succeeded — they are built, committed, and queryable. It then tells you exactly what happened in its summary line:
12:04:31 Finished running 239 models in 0 hours 8 minutes and 12.44 seconds.
12:04:31
12:04:31 Completed with 3 errors and 0 warnings:
12:04:31
12:04:31 Database Error in model mart_partner_revenue (models/marts/mart_partner_revenue.sql)
12:04:31 column "partner_tier" does not exist
12:04:31
12:04:31 Done. PASS=236 WARN=0 ERROR=3 SKIP=0 NO-OP=0 TOTAL=239Then the orchestration layer collapses all of that back down to a boolean:
result = subprocess.run(dbt_cmd, capture_output=True, text=True)
if result.returncode != 0:
raise RuntimeError("dbt run failed") # 236 successful models, gone
publish_revision(tenant_id)PASS=236 ERROR=3 and PASS=0 ERROR=239 produce the same exception, the same alert, and the same remediation path. One of those is three broken marts. The other is a warehouse that never connected. Treating them identically is not conservative — it is just uninformative, and it costs a full rebuild every time a single model regresses.
This gets worse, not better, when models are agent-generated. An autonomous pipeline that proposes new dbt models will occasionally propose a bad one; that is the expected steady state, not an incident. If one bad model can void an eight-minute run for 238 good ones, the blast radius of routine agent error scales with the size of your project.
Make partial materialization a typed outcome
The fix is not a flag that says "ignore failures." It is a third outcome with its own structure, raised from the place that can still see the evidence — dbt's run_results.json and the Done. summary — so the caller never has to re-scan stdout to make a decision.
class PartialMaterializationError(Exception):
"""dbt built some models and failed others.
Carries structured fields so callers can distinguish a few broken marts
from an infrastructure failure, and decide whether to publish the
successful subset or block publication entirely.
"""
def __init__(
self,
message: str,
*,
combined_output: str,
succeeded_models: list[str],
failed_models: list[str],
total_models: int,
) -> None:
super().__init__(message)
self.combined_output = combined_output
self.succeeded_models = succeeded_models
self.failed_models = failed_models
self.total_models = total_modelsThe names matter as much as the counts. succeeded_models is what downstream consumers are allowed to trust; failed_models is what has to be quarantined and reported. combined_output stays attached for diagnostics so nobody has to go digging through logs to find the one Database Error in eight minutes of output.
Parsing is boring and that is the point. run_results.json gives you per-node status with no ambiguity; the Done. line gives you the totals to cross-check against. Take the last Done. in the combined output — a batched run emits one per batch, and only the final one describes the whole project.
def succeeded_model_names(project_dir: str) -> list[str]:
path = os.path.join(project_dir, "target", "run_results.json")
if not os.path.exists(path):
return []
with open(path) as f:
results = json.load(f).get("results", [])
return [
r["unique_id"].split(".")[-1]
for r in results
if r.get("status") in ("success", "pass")
and r["unique_id"].startswith("model.")
]The outer flow catches the typed error and returns an outcome the parent pipeline can branch on, instead of an exception it can only log:
{
"status": "partial_success",
"succeeded_models": ["stg_orders", "dim_partner", "..."],
"failed_models": ["mart_partner_revenue", "mart_tier_rollup", "mart_churn_risk"],
"total_models": 239,
"published": true
}Three rules that keep "partial" from becoming "sloppy"
1. Zero successes is not partial. It is a hard failure.
If succeeded_models is empty, there is nothing valid to publish, and the all-or-nothing gate should stay exactly as strict as it was. This matters more than it sounds: a dbt invocation that dies before it compiles — bad profile, unreachable warehouse, syntax error in dbt_project.yml — can produce ERROR=0 and no successful nodes at all. Without a non-empty check, that infrastructure failure gets classified as a publishable partial subset with zero models in it. Require at least one real success before you are allowed to call something partial.
2. Publishing a subset means declaring the subset.
A partial publish that quietly lands is worse than a blocked one, because the consumer sees a green pipeline over a warehouse that is missing three marts. The failed_models list has to travel with the revision — into the run record, the tenant-facing status, and whatever an agent reads before it answers a question. Partial success is only safe when it is legible downstream.
3. Strict mode stays available, off by default.
Some operators genuinely cannot ship a revision that is missing a model a customer expects to see — a contractual report, a regulatory extract. Give them one environment variable that restores all-or-nothing (BLOCK_PARTIAL_MATERIALIZATION_PUBLISH=true), and if you are replacing an older escape hatch, keep the old name working as an alias. Operators who wired a flag into their runbook two years ago should not lose the behavior because you renamed the concept.
What an agent does with a partially-built warehouse
Publishing the subset is the pipeline half of the problem. The other half is what happens when something asks a question of a warehouse where three marts are one revision behind.
This is where partial materialization stops being a dbt detail and becomes a trust problem. A query against mart_partner_revenue still succeeds — the table exists, it is just yesterday's. Nothing errors. The failure looks exactly like a normal answer, which is the same shape as every other silent data failure: the system is confident and the number is stale.
A run that half-succeeded produces a warehouse where correctness is per-table. Any consumer that reasons per-run will be wrong about half of it.
So the model list has to be enforced at answer time, not just recorded at build time. Before an agent uses a table, it checks the table against the last revision's outcome:
def check_freshness(tables: list[str], revision: Revision) -> list[str]:
"""Return the subset of `tables` not materialized in the current revision."""
if revision.status == "success":
return []
return [t for t in tables if t in revision.failed_models]
stale = check_freshness(plan.referenced_tables, current_revision)
if stale:
answer.add_caveat(
f"{', '.join(stale)} did not rebuild in the latest run "
f"({current_revision.finished_at:%Y-%m-%d %H:%M}); figures below "
f"reflect the previous revision."
)Three things fall out of that, and only the first is obvious. Questions that touch none of the failed models get answered normally — which is most of them, and the entire point of publishing the subset. Questions that touch a failed model get a real caveat naming the table and the age. And questions where the failed model is load-bearing enough that a caveat is not honest — a revenue total that would silently exclude a tier — should refuse and say which rebuild they are waiting on.
That is a strictly better failure mode than the one we started with, where the whole revision was blocked and every question got the previous warehouse with no caveat at all.
The general shape
Partial success is the outcome most pipelines refuse to model, because modeling it means admitting that "did it work" is not a yes/no question. The cost of that refusal is paid in full rebuilds, in queues behind runs that mostly succeeded, and in agents that cannot tell which half of their warehouse is current.
The pattern generalizes past dbt. Anywhere a batch job produces per-item results and you compress them into a process exit code, you are throwing away the only information that would let the next step act intelligently. Parse the per-item outcome. Give it a type. Let the caller decide. And whatever subset you do publish, publish the list of what is missing alongside it — because the consumer that reasons per-run over a warehouse that failed per-table is going to be confidently, specifically wrong.