For a while, I’ve been struggling to explain why our data platform and enterprise IT belong in the same team.
I started at Aiven as a data engineer, helping build our data warehouse from scratch. Over the years, I ended up managing the data team, and more recently, enterprise IT became part of my responsibility. We’re now expanding further into AI automation and internal product engineering.
These disciplines have quite different histories and ways of working. Data teams have traditionally focused on making information available and helping people understand the business, while enterprise IT has concentrated on the systems people use to actually run it.
I’ve been trying to figure out how these responsibilities fit together, beyond the practical advantages of having them under the same management.
Over the past few months, a clearer picture has been crystallizing.
We’ve spent years making the company observable. Now I want us to close the loop and help make it better.
And I think AI is changing the economics enough that this is becoming a realistic ambition for a team our size.
When the engineering gets cheap
Something remarkable is happening to the work involved in running our data platform.
An extraction that previously took us two or three days can sometimes be implemented in thirty minutes using dlt and AI. Extending existing data models has become considerably easier too, particularly when we have well-established patterns that agents can follow.
At the same time, the effort required to maintain what we’ve already built is falling dramatically. We’ve developed an internal agent called Luigi that investigates data quality issues, examines our code, looks for optimization opportunities, and proposes changes. It increasingly discovers work for itself, including problems and improvements that nobody explicitly turned into tickets.
We’ve been adding responsibilities for cost optimization, unused models and columns, and other kinds of technical debt. Some of the cleanup we’re doing now would previously have been difficult to justify spending engineering time on.
The agent isn’t operating without supervision. Changes still need to be tested, reviewed, and verified, and getting an initial extraction working in fifteen minutes is very different from operating it reliably for years. We’re also spending plenty of effort improving the agent and the systems around it.
But the direction is becoming difficult to ignore. Maintaining the platform is requiring progressively less human attention, while extending it along established paths is becoming remarkably cheap.
I can increasingly imagine a situation where standing still requires very little engineering effort, and most routine development happens almost on demand.
For a team that has always had more requests than capacity, that’s an enormous change.
We could use it to build many more pipelines, add more models, and produce more dashboards. Some of that will certainly happen, and making trustworthy data easier to access is valuable in its own right.
But I’m much more interested in what we can do with the capacity we’re gaining.
For years, we’ve had to make choices about which parts of the business were worth understanding through data. Connecting several operational systems, modeling the underlying processes, and maintaining those models afterwards was often too expensive to justify.
When that work becomes dramatically cheaper, a much larger part of the company becomes worth investigating.
And that’s where things get interesting.
We’ve only been doing half the work
One of the areas we’ve invested heavily in at Aiven is our Weekly Business Review. We’ve made substantial progress in giving leadership visibility into how the company is performing, with shared metrics and better discussions about what’s happening.
I’m proud of that work, but I’ve also become increasingly interested in what sits underneath those metrics.
Knowing that a particular outcome improved doesn’t necessarily mean understanding what made it improve. We might be able to explain the change at a high level without really understanding how the underlying work happened.
Consider an operational process that involves several teams and moves through five different systems. Each system might tell us something about its own activities, but understanding the complete process is much harder.
Where does work spend most of its time waiting? Which exceptions repeatedly require intervention? Why do some cases move smoothly while others get stuck? Are there unnecessary handoffs that we’ve simply learned to live with?
Historically, answering those questions could require quite a significant investment in data engineering. In some cases, the necessary events aren’t even recorded properly.
AI is beginning to change what we can afford to instrument and investigate. I think that should encourage data teams to go much deeper into how their companies actually operate, rather than concentrating primarily on a relatively small collection of outcome metrics.
There’s an enormous opportunity for data science here too. Understanding why something happens, predicting where problems are likely to emerge, and evaluating whether an intervention actually caused an improvement are considerably harder problems than producing another report. Better instrumentation makes those questions more tractable, while AI can help with much of the analytical work.
But imagine we become exceptionally good at all of this. We can reconstruct a process, identify where it breaks down, and establish with reasonable confidence what needs to change.
What happens next?
Traditionally, the data team might deliver its findings, recommend an improvement, and leave the implementation to somebody else. Sometimes that makes perfect sense, but I’ve become increasingly uncomfortable with treating it as the natural boundary of our responsibility.
If we understand why a process isn’t working, and we have the engineering capabilities to improve it, why would we stop at the recommendation?
This is where bringing enterprise IT, data, and automation engineering together starts to make much more sense to me.
We can instrument a process, investigate where the friction is, and work with the people involved to understand why things happen the way they do. Then we can help redesign the process, build the integrations or applications it needs, and measure whether the changes actually worked.
That last step matters. Without it, we don’t know whether we’ve improved anything or simply introduced another way of doing the same work.
I want us to be able to repeat this continuously, using what we learn from one improvement to identify the next.
That’s the loop I want our team to help close.
Understanding a process isn’t the same as redesigning it
There’s an important complication here, though.
Knowing how to operate a business process is not necessarily the same skill as knowing how to redesign it. And being good at engineering doesn’t automatically make you good at either.
The people working in Finance, Sales, or other parts of the business understand their responsibilities, their constraints, and the details of their work in ways that we simply don’t. They also have the business judgment needed to make decisions about what matters.
But they don’t necessarily know what’s newly possible with AI, software, and integrations. An existing workflow might feel inevitable simply because the current systems have always required people to work that way.
This is why I think process redesign needs to be collaborative. It requires people who understand the work and people who can challenge the technical constraints that have shaped it, with enough shared understanding to question whether the process should continue in its current form.
We’re already encountering some interesting challenges as we move in this direction.
In Finance, for example, there was initially some understandable hesitation around the prospect of changing how work gets done. More recently, the situation has been quite different. There are so many ideas for improvements and automation that we’re struggling to keep up with the demand.
That’s encouraging, but it also exposes a problem with how we’re currently organized.
We can keep taking requests, prioritizing them, and implementing changes as quickly as our capacity allows. And perhaps many of those individual requests are exactly what we should be working on.
What I don’t yet know is whether fulfilling that backlog gives us enough opportunity to understand and improve the bigger processes those requests belong to.
If we’re continuously occupied delivering individual improvements, we may struggle to create the space to examine a process end to end, consider different ways of organizing the work, and determine which changes would have the greatest effect.
I don’t have a neat solution for this yet. It’s one of the reasons we’re hiring senior AI automation engineers who can work directly with business teams, understand their processes, and help drive changes rather than simply implement technical requests.
But hiring more engineers can’t be the entire answer.
Our job is also to make everyone else more capable
We’ve always approached our data platform as a combination of providing shared capabilities and tackling difficult problems ourselves.
We don’t want every analyst or business team to depend on a central engineer whenever they need something new. We establish patterns, build reusable models, and make it easier for others to work independently.
At the same time, we take on the hardest problems, where there isn’t an established way to do something yet. What we learn from solving those problems should make the platform better for everyone else.
I think we need to extend that philosophy to automation and internal software.
Our team can work with business functions on some of the most complicated processes, where multiple systems are involved or where redesign requires substantial engineering work. We can experiment with new approaches, figure out what works, and turn those learnings into reusable capabilities.
But the goal shouldn’t be for every process improvement in Aiven to become a ticket in our backlog.
If somebody in Finance can make a relatively straightforward change themselves, with access to the right data, integrations, AI tools, and engineering foundations, I’d much rather enable them to do that than have them wait for our team.
This is also why we’re hiring a staff internal product engineer. As more people build internal applications, we need foundations that make those applications secure, maintainable, and straightforward to operate. Otherwise, we risk exchanging a central delivery bottleneck for a growing collection of software that nobody can reliably maintain.
I don’t expect everybody in the company to become a software engineer. People have different skills, interests, and responsibilities, and there are good reasons why some changes need specialist oversight.
But I think we can move the boundary of what people can safely do themselves much further than it is today.
That creates a second improvement loop for our team.
We work on difficult problems with the business, learn what approaches are effective, and make those approaches easier for other people to use. As more teams become capable of implementing their own improvements, our engineers can spend more time on the problems that still require deeper expertise.
And what we learn from those problems feeds back into the capabilities everyone else uses.
It’s a way to scale the impact of a relatively small team without making that team responsible for executing every change across the company.
The seams are becoming the expensive part
There’s another consequence of cheaper engineering that I’m only beginning to understand.
As the effort required to build integrations, automate tasks, and develop internal applications falls, the friction between teams and systems becomes much more visible.
We might be able to automate an individual activity in an afternoon, while the process around it still involves days of waiting, several handoffs, and decisions bouncing between departments.
Each team may be doing its own job perfectly well. The problem is that nobody has a good view of the entire process, or enough influence over it to make meaningful changes.
I’m increasingly wondering whether this is where some of the largest opportunities for improvement will be found.
We’ve become quite good at assigning ownership to metrics. Someone is responsible for a conversion rate, a processing time, or an operational target. But how meaningful is that responsibility if the person can’t change the things that determine the result?
The same applies to process ownership. Being accountable for a process on paper isn’t particularly useful if every improvement requires negotiating with five different teams, each optimizing its own part.
A process owner needs more than a name next to a metric. They need the authority, capabilities, and organizational support to improve how the work actually happens.
And as automation makes more of the execution straightforward, that ability to act could become an increasingly important constraint.
I’m starting to think companies may need to organize themselves more deliberately around end-to-end processes, although I’m not yet sure what that should look like structurally. I don’t think the answer is necessarily to dissolve departments and reorganize everyone around individual workflows.
We still need deep expertise, shared infrastructure, and people who understand the systems underneath the process.
But somebody needs to be able to see and improve the whole thing.
This is also why I think the role of leadership is so important. Our metrics can help identify where improvement efforts should be focused, but redesigning a process doesn’t stop at changing systems or integrations. Sometimes it requires changing responsibilities, decision rights, or the way teams collaborate.
You can’t always remove an organizational handoff by building an API.
And you can’t close the improvement loop if nobody has the authority to act on what the data tells them.
I don’t want our team to own every process in the company. I want us to help the people who do own those processes become genuinely capable of improving them.
For the most important or difficult problems, that means working alongside them to redesign the process and implement the changes. For others, it means providing the foundations, guidance, and tools that let them make improvements independently.
The more effectively we do that, the less improvement should depend on our own team’s capacity.
The difficult part isn’t disappearing
I don’t want to pretend we’ve already figured out how to operate this way throughout Aiven.
Our most developed example of autonomous improvement is still our own data platform. Luigi is doing real work, and we’re learning how to give agents ongoing responsibilities, handle changes safely, and verify outcomes. We’re expanding our capabilities in internal software and automation, but closing business improvement loops across the company is still something we’re building toward.
We’re also discovering that cheaper execution brings its own problems.
Despite the amount of cleanup and optimization we’re now able to do, our dbt project is still more complicated than I’d like. Agents can make hundreds of locally sensible changes without questioning whether the overall architecture makes sense. They can follow an established pattern extremely well, even when that pattern is part of what makes the system difficult to understand.
I suspect we’ll encounter the same problem with business processes.
We could automate an enormous amount of existing work without ever questioning whether that work is necessary. We could build new applications around every operational inconvenience and end up with a company that’s running more software than ever, without being particularly easier to operate.
Sometimes the right answer will be an integration or an agent. Sometimes it’ll be changing who makes a decision, removing an approval, or allowing one person to take responsibility for a process that previously involved several teams.
Those choices require judgment. They also require a good understanding of the consequences for the people doing the work.
And this is where I think the human side of engineering becomes much more important.
For years, we’ve tried to protect engineers from meetings so they can concentrate on implementation. I’ve done the same, and I still don’t want anyone spending their day in unnecessary meetings.
But as implementation gets cheaper, I increasingly want our engineers spending more time talking to people, understanding their problems, and working through how things should change.
An agent can help analyze historical conversion rates and implement a definition of a sales-ready lead. Deciding what that definition should be, and reaching agreement between Sales and Marketing, requires a different kind of work.
The better we become at automating implementation, the more capacity we should have for those conversations.
Our metrics should increasingly help us understand where to focus improvement efforts, but making those improvements happen requires much more than engineering. We need people who can take ownership of end-to-end processes, business leaders willing to support changes that cross departmental boundaries, and an organization that gives those owners the ability to act.
Otherwise, we risk building increasingly sophisticated technology around the same old ways of working.
I’m not sure we’re a data team anymore
This week, I’m joining a panel with MotherDuck and dltHub at Helsinki Data Week to discuss the future of data teams. Preparing for it has made me reflect on how much my own thinking has changed.
A few years ago, I would probably have talked about better data infrastructure, self-service analytics, and making trustworthy information available to everyone. I still believe those things matter enormously.
In fact, as more employees build their own applications and agents, the data platform may become an even more important foundation for how the company operates. The consumers of that data are no longer limited to analysts and dashboards, and we need to make sure the information they use remains reliable and consistent.
But I no longer think providing that foundation and delivering good analysis is an ambitious enough endpoint for our team.
The engineering required to maintain and extend the platform is becoming dramatically cheaper. That gives us an opportunity to understand far more of how the business operates, and to use our combined data, IT, and engineering capabilities to actually improve it.
I want us to tackle the difficult problems, figure out better ways of working, and make what we’ve learned available to the rest of the company so they can improve their own processes too.
We’re still early in that transition, and I’m sure we’ll need to change how we organize ourselves as we learn what works. But over the past few months, the reason for bringing data engineering, analytics, enterprise IT, and automation together has become much clearer to me.
I want us to close the loop between understanding how the company operates and changing how it operates, and then help everyone else close that loop too.
I’m not sure yet what to call that team. But that’s the kind of responsibility I want us to take on.