I used to spend a couple of days at a time refactoring the worst parts of our dbt project. With AI, the implementation got cheaper. And my own ability to scope, verify and review the changes became the bottleneck.
Refactoring was particularly bad for this. I tend to make quite large structural PRs, and the bigger they got, the harder they were to verify. So usually I opened these PRs and they didn’t go anywhere.
When I learned we’ll have to onboard somebody new to our dbt project, I was reminded of the six years of accumulated tech debt we have, and decided now was the time to change that.
Around the same time, Aiven, where I work, acquired Flow AI. They’d been building a harness for running analytical agents reliably on enterprise data. I’d heard plenty about agent harnesses by then and just wanted to understand the concept. The best way to do so was by building one. I didn’t use anything from Flow.
The only safe way to do this refactor was to make each change very easy to review. I didn’t want to put extra load on the team due to this refactor, and we shouldn’t have any incidents come up due to changes made by me. I’ve been using Claude Code for months and it can 100% make these changes. But the overhead for me to manage the agent sessions, worktrees, PRs and verification is too much to do manually.
So about two weeks ago I built a harness. I write the campaigns: the goals, the scope, the success criteria. Claude Code does the actual work. The harness orchestrates everything.
In nine days it opened 255 pull requests against our dbt project. I read all of them and merged 223. The median change was 25 lines across 2 files.
What the loop actually does
A harness sounds a bit mysterious, but it’s just a Python script that runs agents. I’d tried to build a harness a couple of times before and never got very far. It quickly became too complicated to understand. This time I focused on the most minimal setup: pick up one task, implement that, validate it, make a PR and log its work.
I call one run a tick. From there it was mostly trial and error: making sure each tick logged enough information for the next one to continue. The next tick starts from the log again rather than relying on whatever the previous Python process happened to have in memory. There’s no separate state to maintain. State is derived by replaying the log. Before doing something, the harness records what it’s about to do. If a run dies halfway through, the next run can see that and work out what happened. The harness is deterministic. Only the agent varies between runs. Same log, same pull request state, same decision, so when something goes wrong I can tell whether the harness made a bad decision or the agent did bad work.
The names I ended up with, since I use them throughout the article:
Campaign. One YAML file. One piece of related work, split into steps that run one at a time.
Step. One change, with its own target and its own verification. The unit of work: one reviewable PR.
Sweep. One pass across all campaigns, in priority order. Campaigns advance in parallel, up to twenty at a time, with at most one agent per campaign.
Tick. One decision about one campaign: launch, wake, wait, terminate or escalate. Then it exits. A step usually takes several ticks: one to launch it, more to deal with what comes back.
Agent. One Claude Code run doing a single step, in its own git worktree.
Journal. The append-only log. State is rebuilt by replaying it rather than stored separately.
The agents need to survive interruptions too, like me closing my laptop or my Google authentication expiring. Building models and comparing them against production is usually the slow part of a step. That also makes it the point where a run is most likely to die or hit its timeout. 60 runs have been interrupted so far. So, to solve that, each step is instructed to commit its work before validation. If a run dies halfway through a build with the change still sitting uncommitted in a worktree, that work is gone and the next attempt starts from nothing. If it’s committed first, the branch already holds the code and the next attempt only has to repeat the verification.
A tick has five things it can decide to do:
Launch. There’s work ready to go, so create a fresh git worktree, give the agent its brief, and let it work.
Wake. There’s already an open pull request and something has changed. CI finished, an automated reviewer commented, a human asked something, or a data comparison came back. Send the agent back in to deal with that specific thing.
Wait. Something is still running or waiting for an answer.
Terminate. The change is ready for me to read, or the step is being given up on.
Escalate. The agent found something real that isn’t this step’s job to fix.
Most of these decisions are straightforward, except for the wake. Say Copilot reviews the PR and leaves a comment. The next tick sees it, wakes the agent, and the agent responds. Then another tick runs and sees the same comment again. The harness wakes the agent again, gets another answer, posts it again, and can keep doing that forever.
So these comments are logged against the commit they belong to. Once one has been answered at that commit, it stays answered until the commit changes.
When it doesn’t go as expected
Two things can go differently than planned, and they’re not the same. A punt is the agent saying the current step is unreachable. An escalation goes beyond a single step: it can address a campaign or the harness itself. It’s rarely a code change. A step can do both: punt its own work and still escalate something it found.
It’s important for the agent to be able to refuse to do its work. It can stop and say: This doesn’t make sense, so I’m not doing it. If the thing blocking the step changes later I can explicitly reopen it, but I don’t want the harness retrying the model until it eventually produces something. 38 steps have punted so far, some even multiple times.
Those refusals contain tons of information. If an agent can make the change but can’t prove that the result is equivalent, that tells me something about the model, the data around it, or the verification I gave it. A common example was array ordering. Two outputs were equivalent, but nondeterministic ordering made the parity check report a difference.
And if an agent can’t decline the work, eventually it will try to force an answer. I don’t want that.
Escalations I found through trial and error. An agent found a follow-up while working on a step. It correctly decided that fixing it would widen the scope of the PR, and wrote what it had found in a comment instead. The problem was that nothing actually reads every PR comment looking for future work, so the finding got lost. The agent had done exactly what I wanted and I still lost the information.
So agents write down what they found, why the current step shouldn’t own it, and what they think should happen next. The harness writes that to the journal and keeps surfacing unresolved escalations. Escalations became a way for the harness itself to evolve, not just the campaigns.
Over the course of the 255 PRs, there have been 138 escalations. Most of them don’t come to me. There’s an orchestrating agent above the individual step agents that picks them up. If it can resolve one confidently, it does and records why. If it can’t, it assembles the context and puts the decision in front of me. It could also use them as feedback about what functionality was missing.
One escalation last week came from an agent that couldn’t trust its own verification because a comparison had silently run against a stale table instead of production. The model it was migrating wasn’t the problem, so changing that model would have been the wrong fix. I added a pre-comparison check to the shared instructions instead. Every campaign gets it now, including campaigns I haven’t written yet.
What broke
The most difficult problems haven’t been agents making obviously bad changes. They’ve been cases where the system told me something completely reasonable that happened not to be true.
One involved punts. Agents run in fresh worktrees. Claude Code doesn’t trust a directory it hasn’t seen before, and at one point that meant the repo’s allow-list entries weren’t being applied. Those included the permissions needed for dbt, warehouse queries, commits and PR creation. Nothing crashed or obviously failed. The agent tried to do the work, got refused when it reached the tools it needed, and because the run was non-interactive there was nobody to ask. It concluded, quite reasonably, that it couldn’t verify the change and punted.
When you’re sitting in front of Claude Code, a missing permission is obvious. When nobody is there, the agent has to explain the failure to itself.
In the log, that looked exactly like a good punt: “Could not establish parity” looks the same whether the data actually disagrees or the tool needed to establish parity never ran. The reason an agent declined to act is useful information, and in this case that reason was simply false.
Preconditions caused a similar problem. Some campaign steps have gates that need to pass before the work starts. Originally, if a gate refused the step, that refusal was terminal, so from the harness’s point of view the work was finished. The first four times one of those gates fired for real, all four were the gate being wrong rather than the work being invalid. Nothing errored. The campaign just looked like it had nothing left to do. So now I treat a refusal as a reason to inspect the gate before assuming it tells me something true about the work.
I didn’t run into many problems like these while I was sitting in front of Claude Code using it interactively. They started appearing when I stopped being there.
Human review
I still read every pull request before it merges. I built around that assumption from the start. What’s important though is all the pre-work done before that: a pre-scoped task, an isolated worktree, explicit verification against production, every difference triaged and explained, feedback from Copilot, all comments addressed, and then me making the final decision.
None of that makes the agents safe. A 25-line mistake I can understand in a couple of minutes is a very different problem from a large AI-driven refactor across hundreds of lines and tens of files.
It also changes what I need from the model. A lot of the domain judgment has already happened when the campaign was created. The files were identified, the allowed patterns were specified, the verification bar is explicit, and the reasons to give up are written down.
The agent still has to work out how to implement the change. It doesn’t have to invent what a good migration means.
Because of that, these migrations can run on Sonnet rather than Opus. I did let Sonnet pull in Opus as an advisor when it needed help.
Between runs
A lot of the discussion around coding agents is still about what happens inside a run. Which model, which prompt, which skills, how much context, how much autonomy.
I care about those things too, but I spent very little time on them as I built this. Most of the engineering ended up happening between runs.
What work is allowed into the loop? How big can one unit of work get before I stop wanting to review it? What counts as success? What happens when the evidence is ambiguous? When should the agent retry, and when should it stop? When does something need to come back to me? And how do I know afterwards what actually happened?
Claude Code wrote the SQL, ran the checks, responded to reviewers and fixed what they found.
That turned out to be the easy part.
What I built was everything around it: deciding what work should enter the loop, keeping that work small, feeding the agent what came back from CI and review, giving it a way to refuse things it couldn’t verify, keeping track of what had already happened, and making the result cheap enough for me to review.
The better coding agents get, the less time I expect to spend wondering whether they can make the change. I’m much more interested now in what happens before they start, what we require before they stop, and what happens when nobody is sitting there watching.
What I’m trying next
So far this has only run on my laptop. We’re looking at open source solutions that could replace the whole harness and be centrally managed in the cloud, but honestly I’m not sure. For now, packaging it up and sharing it with the team seems to be the right way.
And we’ll be exploring other use cases than just refactoring. This is perfect for building out new models as well, and for cost optimisation.
The bottleneck is now figuring out campaigns and reviewing small changes. I know some companies judge the risk of PRs and approve the low-risk ones. Perhaps that’s what we’ll try as well. I strongly believe that in the long run it’s not feasible for humans to review every change anymore. What that looks like, I don’t know.
A note on how I’ve written this: the harness has accumulated some ideas that have proper names in workflow and distributed systems: replay, leases, idempotency, durable execution. I didn’t start with any of those. I started with a Python file that launched agents, then kept fixing the ways it broke.
I’ve tried to explain it in that order here too. The concrete problem first, the mechanism I ended up with second, and the name for it only when the name is useful. This is not meant to be a general architecture for agent systems. It’s one harness, for one repo, run by one person, and it’s less than two weeks old.
The full spec for the harness is in a gist: https://gist.github.com/StijnZanders/f9cb894acde733b2fb839f04772245c4.