I used to spend a couple of days at a time refactoring the worst parts of our dbt project. With AI, the implementation got cheaper. And my own ability to scope, verify and review the changes became the bottleneck.

Refactoring was particularly bad for this. I tend to make quite large structural PRs, and the bigger they got, the harder they were to verify. So usually I opened these PRs and they didn’t go anywhere.

When I learned we’ll have to onboard somebody new to our dbt project, I was reminded of the six years of accumulated tech debt we have, and decided now was the time to change that.

Around the same time, Aiven, where I work, acquired Flow AI. They’d been building a harness for running analytical agents reliably on enterprise data. I’d heard plenty about agent harnesses by then and just wanted to understand the concept. The best way to do so was by building one. I didn’t use anything from Flow.

The only safe way to do this refactor was to make each change very easy to review. I didn’t want to put extra load on the team due to this refactor, and we shouldn’t have any incidents come up due to changes made by me. I’ve been using Claude Code for months and it can 100% make these changes. But the overhead for me to manage the agent sessions, worktrees, PRs and verification is too much to do manually.

So about two weeks ago I built a harness. I write the campaigns: the goals, the scope, the success criteria. Claude Code does the actual work. The harness orchestrates everything.

In nine days it opened 255 pull requests against our dbt project. I read all of them and merged 223. The median change was 25 lines across 2 files.

What the loop actually does

A harness sounds a bit mysterious, but it’s just a Python script that runs agents. I’d tried to build a harness a couple of times before and never got very far. It quickly became too complicated to understand. This time I focused on the most minimal setup: pick up one task, implement that, validate it, make a PR and log its work.

I call one run a tick. From there it was mostly trial and error: making sure each tick logged enough information for the next one to continue. The next tick starts from the log again rather than relying on whatever the previous Python process happened to have in memory. There’s no separate state to maintain. State is derived by replaying the log. Before doing something, the harness records what it’s about to do. If a run dies halfway through, the next run can see that and work out what happened. The harness is deterministic. Only the agent varies between runs. Same log, same pull request state, same decision, so when something goes wrong I can tell whether the harness made a bad decision or the agent did bad work.

The names I ended up with, since I use them throughout the article:

• Campaign. One YAML file. One piece of related work, split into steps that run one at a time.

• Step. One change, with its own target and its own verification. The unit of work: one reviewable PR.

• Sweep. One pass across all campaigns, in priority order. Campaigns advance in parallel, up to twenty at a time, with at most one agent per campaign.

• Tick. One decision about one campaign: launch, wake, wait, terminate or escalate. Then it exits. A step usually takes several ticks: one to launch it, more to deal with what comes back.

• Agent. One Claude Code run doing a single step, in its own git worktree.

• Journal. The append-only log. State is rebuilt by replaying it rather than stored separately.

The agents need to survive interruptions too, like me closing my laptop or my Google authentication expiring. Building models and comparing them against production is usually the slow part of a step. That also makes it the point where a run is most likely to die or hit its timeout. 60 runs have been interrupted so far. So, to solve that, each step is instructed to commit its work before validation. If a run dies halfway through a build with the change still sitting uncommitted in a worktree, that work is gone and the next attempt starts from nothing. If it’s committed first, the branch already holds the code and the next attempt only has to repeat the verification.

A tick has five things it can decide to do:

• Launch. There’s work ready to go, so create a fresh git worktree, give the agent its brief, and let it work.

• Wake. There’s already an open pull request and something has changed. CI finished, an automated reviewer commented, a human asked something, or a data comparison came back. Send the agent back in to deal with that specific thing.

• Wait. Something is still running or waiting for an answer.

• Terminate. The change is ready for me to read, or the step is being given up on.

• Escalate. The agent found something real that isn’t this step’s job to fix.

Most of these decisions are straightforward, except for the wake. Say Copilot reviews the PR and leaves a comment. The next tick sees it, wakes the agent, and the agent responds. Then another tick runs and sees the same comment again. The harness wakes the agent again, gets another answer, posts it again, and can keep doing that forever.

So these comments are logged against the commit they belong to. Once one has been answered at that commit, it stays answered until the commit changes.

When it doesn’t go as expected

Two things can go differently than planned, and they’re not the same. A punt is the agent saying the current step is unreachable. An escalation goes beyond a single step: it can address a campaign or the harness itself. It’s rarely a code change. A step can do both: punt its own work and still escalate something it found.

It’s important for the agent to be able to refuse to do its work. It can stop and say: This doesn’t make sense, so I’m not doing it. If the thing blocking the step changes later I can explicitly reopen it, but I don’t want the harness retrying the model until it eventually produces something. 38 steps have punted so far, some even multiple times.

Those refusals contain tons of information. If an agent can make the change but can’t prove that the result is equivalent, that tells me something about the model, the data around it, or the verification I gave it. A common example was array ordering. Two outputs were equivalent, but nondeterministic ordering made the parity check report a difference.

And if an agent can’t decline the work, eventually it will try to force an answer. I don’t want that.

Escalations I found through trial and error. An agent found a follow-up while working on a step. It correctly decided that fixing it would widen the scope of the PR, and wrote what it had found in a comment instead. The problem was that nothing actually reads every PR comment looking for future work, so the finding got lost. The agent had done exactly what I wanted and I still lost the information.

So agents write down what they found, why the current step shouldn’t own it, and what they think should happen next. The harness writes that to the journal and keeps surfacing unresolved escalations. Escalations became a way for the harness itself to evolve, not just the campaigns.

Over the course of the 255 PRs, there have been 138 escalations. Most of them don’t come to me. There’s an orchestrating agent above the individual step agents that picks them up. If it can resolve one confidently, it does and records why. If it can’t, it assembles the context and puts the decision in front of me. It could also use them as feedback about what functionality was missing.

One escalation last week came from an agent that couldn’t trust its own verification because a comparison had silently run against a stale table instead of production. The model it was migrating wasn’t the problem, so changing that model would have been the wrong fix. I added a pre-comparison check to the shared instructions instead. Every campaign gets it now, including campaigns I haven’t written yet.

What broke

The most difficult problems haven’t been agents making obviously bad changes. They’ve been cases where the system told me something completely reasonable that happened not to be true.

One involved punts. Agents run in fresh worktrees. Claude Code doesn’t trust a directory it hasn’t seen before, and at one point that meant the repo’s allow-list entries weren’t being applied. Those included the permissions needed for dbt, warehouse queries, commits and PR creation. Nothing crashed or obviously failed. The agent tried to do the work, got refused when it reached the tools it needed, and because the run was non-interactive there was nobody to ask. It concluded, quite reasonably, that it couldn’t verify the change and punted.

When you’re sitting in front of Claude Code, a missing permission is obvious. When nobody is there, the agent has to explain the failure to itself.

In the log, that looked exactly like a good punt: “Could not establish parity” looks the same whether the data actually disagrees or the tool needed to establish parity never ran. The reason an agent declined to act is useful information, and in this case that reason was simply false.

Preconditions caused a similar problem. Some campaign steps have gates that need to pass before the work starts. Originally, if a gate refused the step, that refusal was terminal, so from the harness’s point of view the work was finished. The first four times one of those gates fired for real, all four were the gate being wrong rather than the work being invalid. Nothing errored. The campaign just looked like it had nothing left to do. So now I treat a refusal as a reason to inspect the gate before assuming it tells me something true about the work.

I didn’t run into many problems like these while I was sitting in front of Claude Code using it interactively. They started appearing when I stopped being there.

Human review

I still read every pull request before it merges. I built around that assumption from the start. What’s important though is all the pre-work done before that: a pre-scoped task, an isolated worktree, explicit verification against production, every difference triaged and explained, feedback from Copilot, all comments addressed, and then me making the final decision.

None of that makes the agents safe. A 25-line mistake I can understand in a couple of minutes is a very different problem from a large AI-driven refactor across hundreds of lines and tens of files.

It also changes what I need from the model. A lot of the domain judgment has already happened when the campaign was created. The files were identified, the allowed patterns were specified, the verification bar is explicit, and the reasons to give up are written down.

The agent still has to work out how to implement the change. It doesn’t have to invent what a good migration means.

Because of that, these migrations can run on Sonnet rather than Opus. I did let Sonnet pull in Opus as an advisor when it needed help.

Between runs

A lot of the discussion around coding agents is still about what happens inside a run. Which model, which prompt, which skills, how much context, how much autonomy.

I care about those things too, but I spent very little time on them as I built this. Most of the engineering ended up happening between runs.

What work is allowed into the loop? How big can one unit of work get before I stop wanting to review it? What counts as success? What happens when the evidence is ambiguous? When should the agent retry, and when should it stop? When does something need to come back to me? And how do I know afterwards what actually happened?

Claude Code wrote the SQL, ran the checks, responded to reviewers and fixed what they found.

That turned out to be the easy part.

What I built was everything around it: deciding what work should enter the loop, keeping that work small, feeding the agent what came back from CI and review, giving it a way to refuse things it couldn’t verify, keeping track of what had already happened, and making the result cheap enough for me to review.

The better coding agents get, the less time I expect to spend wondering whether they can make the change. I’m much more interested now in what happens before they start, what we require before they stop, and what happens when nobody is sitting there watching.

What I’m trying next

So far this has only run on my laptop. We’re looking at open source solutions that could replace the whole harness and be centrally managed in the cloud, but honestly I’m not sure. For now, packaging it up and sharing it with the team seems to be the right way.

And we’ll be exploring other use cases than just refactoring. This is perfect for building out new models as well, and for cost optimisation.

The bottleneck is now figuring out campaigns and reviewing small changes. I know some companies judge the risk of PRs and approve the low-risk ones. Perhaps that’s what we’ll try as well. I strongly believe that in the long run it’s not feasible for humans to review every change anymore. What that looks like, I don’t know.

A note on how I’ve written this: the harness has accumulated some ideas that have proper names in workflow and distributed systems: replay, leases, idempotency, durable execution. I didn’t start with any of those. I started with a Python file that launched agents, then kept fixing the ways it broke.

I’ve tried to explain it in that order here too. The concrete problem first, the mechanism I ended up with second, and the name for it only when the name is useful. This is not meant to be a general architecture for agent systems. It’s one harness, for one repo, run by one person, and it’s less than two weeks old.

The full spec for the harness is in a gist: https://gist.github.com/StijnZanders/f9cb894acde733b2fb839f04772245c4.

Our analytics engineers can extract their own data sources now. A year ago I wouldn’t have wanted them to.

Not because they couldn’t write the code. Doing extraction properly meant knowing too many things outside their normal area: authentication, pagination, memory, schema changes, retries, monitoring.

So we built the framework instead. It runs on dlt, and most of that knowledge sits in the framework now instead of in a person.

We’ve been doing the same in analytics engineering. Our standards and methodologies increasingly live in skills the agents follow, so analysts and data engineers can contribute to the same codebase safely.

Same with our semantic layer. The approach we take is in a skill, and we even have one that walks you through whether you need a new topic or just some smaller changes.

None of it is airtight. “Safely” is carrying a lot of weight in those sentences. Someone still reads everything before it lands, and that part hasn’t got cheaper.

We originally built these frameworks to make our own work easier and more maintainable. But with AI, they started doing something more interesting: raising the floor for everyone else.

The boundary moved from both sides

We use Omni. Our stakeholders ask questions in natural language against our governed semantics, or build dashboards themselves on the same models our analysts use.

The quality varies, of course, and a lot of what they build is quite niche. But it changes the role of a data analyst.

Many analysts I’ve known have mostly been building dashboards. Not going deep into figuring out how to change the processes of their stakeholders. That work isn’t going away. It’s just not the scarce part anymore.

When your stakeholders can do all the basics themselves, you need to go further into their area to provide more value. Understand how the process actually works. Know which questions matter before anyone brings you one.

Which was arguably the job all along. Some analysts just got stuck building dashboards.

Execution alone is not enough

I don’t think this is specific to data. As AI makes more of the basic execution cheap, I see three directions for knowledge work. I see them most clearly in data, because that’s where I’ve been doing a lot of the thinking, and I can watch the bottlenecks move.

Breadth. Do more of the adjacent work yourself. An analytics engineer setting up extractions. A data engineer building entity models. An analyst updating governed semantics. You’re still deep somewhere. You have to be. But the threshold for working in the areas around you gets lower.

Depth. Understand the domain much more deeply. Waiting for stakeholders to bring you context isn’t enough. You need to understand their world well enough to form your opinion of what matters.

Leverage. Raise the floor so other people can do more. Turn what you know into frameworks, skills, standards and systems that let other people do work that previously needed you.

This might sound like T-shaped people. I think it’s different.

The horizontal part of the T was mostly about knowing enough to work with other disciplines. Breadth here means actually doing some of that work. And depth here isn’t deeper expertise in your own discipline. It’s deeper understanding of the domain you’re applying it to.

The people I find most valuable increasingly combine these.

Breadth and depth: understand the problem deeply and build most of what it takes to solve it.

Breadth and leverage: work across the stack and build things that make everyone else better.

Depth and leverage: understand a domain well enough to encode how it works into semantics, workflows and systems other people can use.

We’re looking for a process automation engineer this summer. Writing the role, it became obvious the person we really wanted needed all three: enough depth to understand how a finance process actually runs, enough breadth to work across the systems involved, and enough leverage to turn the solution into something reusable.

I don’t think everyone needs all three. Expertise still matters, and business users can’t suddenly extract data sources or build reliable data models because they installed Claude Code.

But the boundaries are getting blurrier.

The people I’d bet on are getting wider, going deeper, or raising the floor for everyone else.

Increasingly, two out of three.

A large ARR drop showed up in one of our dashboards. A data analyst in the FP&A team caught it on a Saturday. I’d shown him our new triage skill three days earlier.

He ran /triage on the issue and posted a full root cause analysis. Evidence, root cause, category, the most likely explanations. It showed the source data had stopped extracting: hourly records normal, then a sharp drop, then near zero.

I didn’t see any of it until Sunday morning. I was packing for the airport, a company offsite in Croatia starting that day. Nobody on my team was around. So instead of waiting, he took the first step himself.

At the airport, over lunch, fighting a dying laptop battery, I copied his root cause analysis into Claude Code. I asked it to investigate further and open a PR for the fix. I merged it that day.

I only learned afterwards that he’d had the same fix in mind. He’d asked Cursor to draft the PR. He just didn’t feel confident enough to ship it.

He had the knowledge. The skill gave him that. What he didn’t have was the authority to merge it, or the appetite to own it if it broke. The method had transferred. The mandate to act on it, and the responsibility for getting it wrong, had not. That’s the part I never thought to package, maybe because I’m not sure it packages at all.

My first reaction was: this works. My second, honestly, was disappointment that he didn’t just ship it.

What I want is to wake up to “I found the problem, here’s the fix,” review it, and have it done. I don’t need to own every fix. But handing someone the method is not the same as handing them the authority and the confidence to use it.

I could have written these skills earlier.

The reason I didn’t is uncomfortable. When a messy data issue shows up, people ask me, and I can usually figure it out.

I’ve been around long enough to know where the weirdness hides. Late sources. Models that oversimplify reality. Definitions that changed three times. Tables people trust too quickly. I move fast because I have the muscle memory: find the issue in aggregate, drill down to the lowest useful granularity, fix it there, repeat until the remaining movement is explained.

Not everybody has that, and that’s fine. Some people are stronger architecturally. Others are better at solving a problem once it’s scoped. What I’m good at is the scoping, the part where you haven’t yet figured out what kind of problem you’re looking at.

It feels useful because it is useful. The next issue comes in, it looks different enough that someone asks again, I answer again, and the work moves.

AI changed that. You can distribute a skill like this far more easily than before. The knowledge was never secret. The problem was that I was still the easiest way to get to it. I was the interface. Now there’s a realistic path to packaging what I do into something people can invoke without me in the room.

That’s a strange thing to do. You take something that lived in your head and make it usable without you. Your shortcuts, your checks, your feel for when something is probably fine versus when you need to keep digging.

Part of me wants to call that progress. Another part knows that when you’re the person everyone asks, your value is obvious. You know the weird edge cases, which source to distrust, what normal looks like. That’s expertise. It’s also status. Writing the skill means the next person gets further before you show up. In the best case, you’re not needed at all.

You built the road. People drive on it. Nobody thanks the road.

That’s the job, though. At least it should be.

But the skill didn’t remove the bottleneck. It moved it. The analyst got all the way to the fix and still didn’t ship. What stopped him wasn’t knowledge. It was that the fix wasn’t yet his to make.

The second story showed me the other half of it.

A billing issue came back, one we thought we’d fixed. It landed in the evening and I picked it up around ten the next morning. We were mid-migration between billing systems, and the complexity sat in the gap between them.

I had the triage skill, so I used it. I ran it, it didn’t find it, I told it to look harder. Go deeper. Continue. There were so many factors in play that it never figured it out.

It narrowed things down. I got a CSV of where the issue might be. I handed it to a colleague for a fresh pair of eyes and asked them to go deeper. They’re an AI skeptic, more of a data artisan. They took the manual route and found it.

The skill narrowed the search space. The last step still needed someone who distrusted the answer enough to keep looking.

Here’s the part I’m not proud of. My gut said it didn’t add up. The explanation we landed on would have implicated every monthly run, and the last few had been clean. I shipped the fix anyway because I was out of time. The skill scaled the diagnosis. It couldn’t make the judgment call, and under pressure my judgment lost to the calendar.

Two problems, two different gaps. The analyst had the diagnosis but not the authority to act on it. I had the authority and let a deadline overrule my own judgment.

Packaging what I knew didn’t make me unnecessary. It pulled apart what had always been bundled in one person: the knowledge, the judgment to know when an answer is wrong, the authority to act when it’s right, and the responsibility for what happens next. The skill could carry the knowledge. The rest defaulted to one person. To me. Again.

So the job isn’t answering more questions. It’s moving the judgment, the authority, and the responsibility toward the people who hit the problem first. That’s slower and less satisfying than writing a skill, and it’s the actual work.

You don’t stop being the bottleneck. You just move it. The work is making sure the same decisions don’t keep routing back through you behind a new interface.

There are two finance books on my desk. I bought them six weeks ago. Neither one is open.

We had to assess whether to replace one of our finance tools with an AI-native version. To judge that, I had to understand what we were replacing. That’s where I keep getting stuck.

The pressure moves faster than the thinking

A colleague told me a few weeks ago how investors had put it at another company:

Use AI. Find use cases. Personal agents in production by September. Or…

That exact phrasing is not everywhere yet, but the pressure underneath it is. Boards want movement. Leadership wants proof the company is not falling behind. The urgency is real. It is not crazy to look at what AI can already do and conclude your company should be moving faster.

But “use AI” is not a strategy. It is urgency without direction. A strategy says what should change and what should stay human. “Use AI” doesn’t answer that. It moves the question to someone else. Every team is asked to find use cases and show progress.

I knew the numbers, not the work

I’ve sat close to Finance since I joined Aiven, mostly through the data around it. I knew enough gotchas in the revenue data to feel like I understood the work. That made me overconfident.

Looking back, I think I knew it wasn’t enough. I was hoping it wouldn’t matter.

The numbers are the surface. Underneath them is the work itself: what Finance needs to produce, and where the manual work still hides. Almost none of it sits in a dashboard. Most of it lives in handoffs and tribal knowledge.

Nobody has the whole picture. Every person you ask gives you a different version depending on which part of the process they touch. You assemble it person by person, exception by exception, and the full picture only exists in the overlap between conversations.

That’s why the books are still on my desk. I bought them thinking I could catch up. Then I realized a finance textbook isn’t going to teach me what our team actually needs to produce, or which parts of the current process should disappear.

The vendor came in for a day and asked how things work on our end. They showed us what the tool does. What none of us could answer was whether it actually fits the work we run, not in a few hours.

The gap didn’t show up during the demo. It showed up when we started scoping the POC and had to think about requirements and success criteria. That’s when we realized how much of the process we couldn’t clearly describe.

The one that works

We already have an agent triaging data issues from our observability tool. I wrote the skills for it myself. Issue comes up, triage runs, it queries BigQuery, reads our dbt codebase, pulls PRs from GitHub, and figures out whether it’s a platform issue or a data quality issue. Then it routes the issue to a data engineering agent or an analytics engineering agent. Before that, every issue sat in a queue until someone on my team had time to look.

I could describe that process in thirty seconds. I knew the routing logic because I have been doing it myself for years. Putting it in a skill was straightforward. The agent works because the process was already clear enough to teach.

Same company, different team. One process I could automate because my team owns it. The other I can’t even scope properly because the knowledge isn’t mine.

The hard part comes before the tool

When you understand what the work has to produce, you can stop treating the current process as the thing to automate. You know the output and where judgment still matters. You can see where an agent might fit and where a human still needs to stay.

The danger is that you can move forward without that understanding. Buy the tool, push the agent live, and you have automated a half-understood version of the work. That is worse than the old tool, because the old tool did not pretend.

“Use AI” sounds like a tools problem. Tools problems become vendor problems. By the time the vendor is in the room, the harder question has been skipped.

The pressure to use AI isn’t going away. It shouldn’t. But I’m starting to think the real work isn’t picking the tool. It’s becoming the person who understands the work well enough to know what should change.

For the triage agent, I was already that person.

For finance, I’m not there yet. The books on my desk aren’t going to change that.

The next conversation is not with the vendor. It is with the people who know where the clean process breaks. If I cannot describe the work without the tool, I’m not ready to decide what it should replace.

The pile becomes the world

We’re in the middle of a migration. A small subset of customers is on the new system, but most are still on the old one. Nothing in the billing topic made that parallel state obvious. The old systems were in the knowledge repo, but they didn’t show up as part of the answer to how billing works today. So the model resolved the pile into one clean truth: the version with the most documentation.

A couple of weeks ago, I had a few spare hours on my hands and Karpathy had just posted his viral GitHub gist about the LLM wiki: basically, a way to build a knowledge base readable by agents. This is something I’ve been tinkering with in my role at Aiven anyway: how much of the company can we make understandable by AI? So I did the obvious thing and tried to build one for our internal context. I pointed AI at everything I could reach: Slack channels I had access to, Coda pages, dbt models, Omni metric models, system data, ownership notes, whatever I could feed into a script during one sitting. The output was a neat Obsidian repo, deployed to GitHub Pages as a Quartz site, with an MCP server on top. I could query it through Cowork and it felt like magic.

It broke on billing

The answer was confident. It explained the integrations, the ownership, the gotchas. It had a very good read on the new system. It even understood our new ARR model, which we had built ahead of go-live, even though that model wasn’t live yet. If a new hire had read that summary, they’d have come away with a clean mental model of how billing works at our company. That mental model would have been wrong about almost every customer we have today.

The model wasn’t hallucinating. It was reading the artifacts I’d given it and building the version of the company those artifacts most clearly described. And those artifacts were fresh. As fresh as they could be. Updated that day, backed by an active Slack channel, architecture docs, ticket trails, metric definitions. The problem was that most of them described the new billing system, because that’s the one people were actively building. The old one was just running. There were traces of it in the repo, but almost nothing that said: this is still the system most customers are on. That kind of information disappears in most companies. Not because anyone deletes it, but because nobody wakes up thinking: today I should document the system that has been running for years.

So it picked the version with the most signal and treated it as how things work today.

The map had no edge

This is how we write things down, anyway. Documents accumulate around the work that’s currently getting attention. Operations that just run don’t generate much of anything. And once you give that pile to a model, the pile becomes the world. It doesn’t know what is missing. It doesn’t know which parts live only in people’s heads. A human can catch one bad answer, sure. But that doesn’t fix the failure mode. The same thing happens everywhere, big and small: the model treats the documented version as complete, even when it’s only the part that happened to leave a trail.

The deeper failure isn’t that the answer was wrong. It’s that the model picked one version at all.

The honest answer to my billing question wasn’t the new system or the old one. It was that two systems are running in parallel. They don’t agree, and the right answer depends on which customer you’re asking about. The model had enough signal to hesitate. Instead it picked the cleaner story and presented it as how things work.

The complete wiki will never stay complete

A better wiki could have fixed this answer. In principle, sure. Someone could maintain a perfect billing page that captured everything: what is intended, what is running, the manual work filling the gaps, who handles the exceptions, and why they exist. The model would do better. This specific failure would probably go away.

But that is the point. The answer depends on someone maintaining the boring middle forever. The messy transition state. Which customers are where. Which model is live. Which process is designed, and which process people actually follow because the designed one doesn’t cover reality yet.

That page doesn’t exist for long unless someone keeps doing the invisible maintenance work. Not because people are careless, but because nobody wakes up thinking: today I should document the gap between the system and the business. Every company runs on that layer of human work. People know which exception still matters. They know who to ask. They know which dashboard is technically right but practically misleading. Most of that never becomes an artifact.

So the model read the pile and treated it as the complete picture. The failure is that it had no way to know where the map ended.

The next version has to assume the complete wiki will never stay complete. Whatever it is, it has to work with partial sources, conflicting sources, and suspiciously clean sources. When the company is in the middle of changing, it should not collapse that into one answer. It should say which layer it’s reading from, what seems live, what seems planned, and where it can’t see enough to be confident.

That is the next thing I want to figure out: not how to make the pile bigger, but how to show where it ends.

Greenstep HQ in Espoo, halfway through a hackathon. We had a fraud triage tool working. Four AI agents scored each customer. A human swiped through the queue with arrow keys. Our CFO walked over and asked why we were only thinking about defending revenue. Why not grow it too?

Same customer signals, different question. A usage spike is either fraud risk or expansion intent depending on the lens. We added expansion signals in the last hour and placed second out of 70 teams.

But we only applied the triage UI to the fraud side. The expansion signals sat in a dashboard.

Two weeks ago at an offsite dinner with people from our GTM team, we ended up talking about outreach. Cold calls, emails, all the usual pain. How hard it is to hit the right note. How AI-drafted messages are often close, but slightly off.

It clicked later. Same idea, but applied to leads.

It matters even more there than in fraud. Your average VP gets buried in outreach. AEs send dozens of emails a day and most of them go nowhere. If it’s not specific, it’s noise. AI can draft something decent. But a human can scan the draft, catch the wrong tone, and swap “your team’s usage is growing” for “you’re two weeks from hitting your API limit.” That’s what makes someone actually reply.

Everyone picks the wrong extreme

A company decides to “use AI” and usually picks one of two dead ends. Full automation and hope nothing breaks. Or give everyone a Claude subscription and some prompts and call it a day.

We keep asking AI to make the final call. It shouldn’t be making the call. It should be preparing the call for us.

People kept asking why we didn’t just fully automate the fraud tool. If the agents are right 85% of the time, what happens in the other 15%? In fraud, that’s suspending a legitimate account. In outreach, that’s sending a tone-deaf email to your biggest expansion opportunity.

Five seconds instead of fifteen minutes

So I built a demo. The version we should have built for the expansion opportunities at the hackathon.

Each card: company name, four signals, a draft email the AI wrote, and a send button. Press right to send, left to override, down if you want to see why the AI suggested this outreach. Most decisions take two seconds. Glance at the signals, read the subject line, press right. The ones where something feels off, you press down, read the evidence, maybe edit the draft inline.

I didn’t expect how little information you need when it’s laid out right. The fifteen minutes people spend digging through dashboards and CRM notes, that’s the AI’s job now. The human’s job is the five-second gut check.

The swipe flow: approve, override, or ask why.

Overrides are the point

This is the part most people miss. The override button matters more than the send button.

When someone presses left instead of right, that’s training data. You know which account, which signals, which human disagreed. Track enough of those and patterns show up. One agent gets overridden 40% of the time on borderline cases. Enterprise drafts get edited twice as often as mid-market ones. A specific AE overrides everything under a score of 70 because they know something the model doesn’t.

The override rate should drop over time. Not because people stop paying attention, but because the system starts to learn what they would have done anyway.

Right now it’s just a UI. I’ll update this post once we’ve put it to work and have real override data to look at.

What I’m still working out

Since the hackathon I notice the pattern everywhere. Support ticket routing, invoice approvals, screening resumes. Any queue where a human applies a gut check probably has some version of this hiding inside it.

Can overrides actually train the system over time? I think so, but I haven’t tested it yet.

But the thing I keep coming back to is that most of the time goes into the investigation, not the actual decision. Once you have a suggestion and the evidence sitting right there, the decision takes about five seconds.

I wrote up the implementation notes for the pattern here.

Crab Week, our company offsite in Šibenik, Croatia. We were standing near the hotel lobby talking about AI. I showed them Paperclip, a tool I’ve been building with. One agent delegating work to a dozen others, creating PRs while we watched. Nobody said anything. I’d been expecting questions. How does it handle errors. How do you keep the agents focused. We moved on, talked about other things. But what I kept thinking was: how do I get them to see what this could do for them, not to them.

Most people think AI threatens jobs. More often, it threatens identities.

In the days after, I kept having similar conversations. A colleague told me they were thinking about leaving for an AI-native company before they miss the boat. I thought they might be right. We’re not moving fast enough. But they weren’t really chasing speed. They knew they had to reinvent, and didn’t think they could do it here. They wanted to be somewhere figuring out what work actually becomes.

Managers know this feeling

I’ve been struggling with this for years.

It reminds me of when I became a manager. You get promoted because you’re good at the work. Then you’re supposed to stop doing it. Except you’re also the one who knows why it works in the first place, and that doesn’t transfer in a handover doc.

Our data platform team is four people. I still fix things at odd hours that I should be delegating. It’s faster, I know the code, and I tell myself I’m just unblocking the team.

That’s the tension. Let go and you lose touch. Hold on and you become the bottleneck.

AI just did this to everyone at once. And now everyone’s asking the same question managers have been asking for years: if I’m not the one doing the work, what am I?

So I made my team try it

I was doubting all week whether I should have the team try it. But after all the hype during our offsite about practical use of AI, it felt right. They needed to see Paperclip for themselves.

Four data and analytics engineers, each managing their own agents. Within an hour they’d hired a dozen agents and couldn’t get them to work together. One person wanted more governance. Another wanted to move faster and hire more. They clashed. They ended up acting as a board of directors, arguing about the goals of their company.

Nobody was writing SQL. Everyone was managing. The agents kept going, PRs kept getting created, needing more guidance than anyone expected. My team was the bottleneck for direction and quality control. The agents didn’t have enough of our tribal knowledge, the same way a new hire doesn’t.

I let it sit for a while after that. Let them think about what they’d just experienced. We’re not going all in overnight. Though we already have an agent triaging data issues from our observability tool, freeing us up to focus on what actually needs us.

The impossible balance

I’ve always identified with getting things done fast. I still do. AI makes me even faster.

But that’s also the trap.

If I only use it to move faster myself, nothing really changes. I’m still the bottleneck. Just a faster one.

The harder part is getting the team there too.

That means letting go of the parts of the job that feel safest. Writing the query. Fixing the issue. Knowing the code path. Being the person who can always make it work.

Those things still matter. But they can’t be the whole job anymore.

In the Paperclip exercise, the team didn’t need to write more SQL. They needed to decide what the agents were trying to do, what good looked like, where to slow down, and when to stop.

That’s the part I’m still figuring out.

You can find Paperclip on GitHub