Letting an agent fix failed pipelines: a step-by-step build of Snowflake's 16-minute repair loop

A pipeline fails at night. Someone on call gets paged, reads the logs, finds the root cause, writes a fix, tests it and opens a pull request, and the whole thing often takes hours. On October 19, Zachary Blackwood of Snowflake’s internal data team described how they handed this job to an agent. Their pipelines were growing faster than the team, and they wanted more than an alert: an agent that could explain the problem and have the fix ready.¹

Below, I turn that setup into a tutorial. First the results and their limits, then what you need before you start, then the build step by step, and finally what it buys you, what a rollback costs and what the post leaves out.

A PR in 16 minutes, but a person still presses merge

The reported results cover only the pipelines this team manages, over five months:¹

Under 16 minutes, on average, from detecting a failure to opening a pull request for review.
Near-complete triage of monitored pipeline failures.
“Hundreds” of PRs merged and more than 500 pipelines improved.
“Hundreds” of credits a day (Snowflake’s billing unit) no longer wasted.

These are the vendor’s own numbers. There is no success rate, no count of rejected PRs and no failure case. “Hundreds of PRs” and “500+ pipelines” have no denominator either, so we can’t tell what share of all failures they represent.

The ground rule is stated plainly: the agent proposes the repair, and a person decides what ships.¹ Everything else in the design follows from it. What the agent produces is a PR waiting for review; it never edits production directly.

Five things to have in place first

The post doesn’t list prerequisites. These five are inferred from what it describes, and copying the setup without any one of them will cause trouble.

Pipeline code in git, with a named owner for every pipeline. The agent’s output is a PR, so the code has to live in a repository. The post’s workflow config names the repository and branch (main) and has an owner field.¹

Failures that are detected and can trigger a run with the pipeline’s ID. The post calls CALL RUN_WORKFLOW('fix-broken-pipeline', '{"pipeline_id": "12345"}').¹ If your scheduler can only send a “job failed” email, change it to emit an event that can call a workflow with parameters.

Least privilege for the agent. It can read production logs and code and write to a branch, but it cannot deploy; CI deploys after the merge. The post says nothing about permissions. This one is my addition, and it is how “a person decides what ships” shows up in access control.

Somewhere to test a fix away from production. The post lists testing fixes “before anything touches production” as a requirement but doesn’t say how.¹ A dev database, a staging environment or a zero-copy clone of production tables all work, as long as the test results end up in the PR where the reviewer can see them.

Someone on call to review, and a budget cap on every run. The post notifies a Slack channel and an email address, and each run is capped at 50 turns and 30 minutes.¹ Without a reviewer, the whole thing quietly turns into an agent that edits production.

Four steps to build it: trigger, isolate, configure, open a PR

FIG.1Snowflake's pipeline repair loop end to end. The agent stops at a PR waiting for review; a person decides what ships. Limits come from the post's example config.

Step 1: two triggers. When a pipeline fails, its ID starts the repair workflow directly. The second trigger is proactive: a scheduled task runs a code-smells workflow every 12 hours to look for performance problems and bad patterns, written as CREATE TASK ... SCHEDULE = '12 hours' AS CALL RUN_WORKFLOW('code-smells').¹ The first trigger fights fires. The second one finds problems that nobody gets paged about but that burn money every day.

Step 2: decide where the agents run. The post compares three options.¹

FIG.2The three ways Snowflake considered running its agents. Option 2 runs today; option 3 depends on a feature still in public preview.

The first design was a single long-running server exposing a REST API, but one machine caps how many agents can run at once. The team moved to a separate job per agent, so many agents run in parallel with no single bottleneck. Next, they plan to use the new EXECUTE AGENT TASK from CoCo automations, now in public preview, which starts a run with one SQL call and removes servers and shared compute pools from the picture.¹

Step 3: write the workflow config. The post shows a JSON config. Item by item:¹

Instructions: a set of templated instruction files in the repository that tell the agent how to investigate and fix this kind of problem.
Model: opus.
max_turns: 50: at most 50 rounds between the agent and its tools.
repos: which repositories and branches it may touch; the example uses main.
timeout_minutes: 30: a run stops after 30 minutes.
Notifications: a Slack channel (#data-eng-alerts) and email addresses, plus an owner field with the responsible person’s email.

Simple workflows can be chained into bigger ones, with human feedback in between.¹

Step 4: open a PR with full context. The PR should state the symptom, the root cause, what changed and the test results. A reviewer should be able to decide from the PR alone, without going back to the logs.

The payoff comes from one job per agent plus a PR gate

Together, those two choices give you four things.

Parallelism. On a day when failures pile up, each one gets its own job instead of queueing for a server.
A cost ceiling. With 50 turns and 30 minutes per run, the worst case for a single run is known, and the worst case for a day is that ceiling times the number of failures.
A built-in audit trail. The PR records who approved what, what changed and why, ready for an auditor.
Savings from the scan, perhaps the larger share. The daily credits saved come from continuous scanning for waste, with no alert needed.¹

Where the gate sits matters. Orchestra’s comparable template asks for approval of the agent’s suggested fix in Slack, then a second agent implements it, and the docs say it “applies changes without further review.” If no approvers are set, anyone with pipeline run permissions can approve.³ Both setups have “human approval,” but Snowflake’s reviewer approves written and tested code, while Orchestra’s approves a description of a fix. The first carries far less risk.

Reverting a PR is cheap; reverting data already in tables is not

This section answers two questions: can you undo a fix, and what does undoing cost? The post doesn’t mention rollback at all, so what follows is my design.

Can it go both ways? At the code level, yes. A fix is a PR, so undoing it is a revert PR that reverses the merge. The agent can open that revert PR too, and it goes through the same review and the same CI deploy. Fixing and undoing take the same path, so every undo leaves a record.

The cost depends on how far the fix has travelled.

FIG.3What a rollback costs depends on how far the fix got. Reverting code is nearly free; once downstream has used bad data, it is the most expensive case. Retention periods come from the Snowflake docs; the timeline is not to scale.

Code changed, not run yet. Merge the revert PR and you are done, at almost no cost.
It ran and changed tables. Now the tables have to go back to their state before the fix. Snowflake’s Time Travel lets you query or clone a table as it was at a past point, located by timestamp, offset or a statement’s query ID; BEFORE(STATEMENT => ...) returns the data from just before a given statement ran.² History has a retention limit, though. Every account gets 1 day by default, Standard Edition can’t go higher, and Enterprise and above can keep up to 90 days for permanent tables. After that, data moves into Fail-safe, 7 days in the docs’ example, during which only Snowflake can recover it; you can no longer query or clone it yourself. Longer retention also means a bigger storage bill.²
Downstream already used the bad data. Reports went out, models were trained, other teams’ tables were computed from it. Rolling back now means recomputing everything downstream and explaining the change to the people who rely on it. This is the most expensive case and the hardest to estimate in advance.

Three habits keep the cost down:

Make the PR template require two things: which tables the fix will change, and how to undo it.
When a fix includes a backfill that rewrites history, record the backfill statement’s query ID first, so BEFORE(STATEMENT => ...) can take you back to just before it.
Lengthen Time Travel retention on critical tables, and budget for the extra storage.

This matches what I argued in the post on judge models: a wrong decision in a gateway costs one call, but once it is written into a table it becomes history you have to rewrite and explain.

What the post leaves out is yours to fill in

There is no success rate, rejection rate, testing method, permission model or failure case in the post. An outside study helps calibrate expectations.

A January 2026 study looked at 8,106 fix-related pull requests opened by AI coding agents. Overall, 65.0% were merged, 26.1% were closed without merging and 8.9% were still open.⁴ Merge rates varied widely by agent:

FIG.4Merge rates of fix-related PRs opened by five AI coding agents, from a study of 8,106 PRs in general open-source projects, not data pipelines.

The authors read 326 unmerged PRs by hand: 22.1% were superseded by another PR, 18.1% failed tests and 15.3% were wrong or incomplete fixes.⁴ These projects aren’t data pipelines, so the numbers don’t transfer directly. The direction is clear, though: a real share of agent fixes fail their tests or miss the root cause. The review gate earns its keep.

A rollout checklist: one pipeline, one month

Go through the five prerequisites and fill whatever is missing first.
Pick one pipeline that fails often and affects few people, and start there.
For the first month, turn on diagnosis and PRs only, not the proactive scan, and watch the rejection rate and how long each review takes.
For every PR, record four numbers: time from failure to PR, turns used, merged or not, and whether it was reverted later.
After a month, replace the vendor’s figures with your own four numbers before deciding whether to widen the scope.

A PR in 16 minutes is a good start. The part worth learning from is where the agent’s work ends: at a PR that a person can review and undo.