In this edition of The Operators Guide to AI I want to share the most important lessons we’ve learned from building agents in production at scale for global enterprises like Airwallex, Worldpay, and IG.com over the last three years.
Here are the ten lessons we learned:
Start with one use case and become excellent at it.
A working demo accounts for about 10% of the effort. Budget the rest for accuracy, consistency, recovery, evaluation and scale.
Deterministic workflows make known work easier to operate, but add code to maintain and customize.
Measure flexibility by how quickly the system can support a new task.
Business logic encoded in the harness limits the model’s judgment and is a failure waiting to happen.
Configuration only expands the combinations available within the boundaries of the existing system.
Give agents a workspace where they can inspect files, write programs and preserve state.
Use benchmarks to measure capability and real workloads to measure recovery, consistency, auditability and cost.
Route each stage to the least expensive model that can perform it well.
Let agents explore unfamiliar work, then preserve successful paths as constrained procedures.
This is the story of how we learnt them.
Lego Lamborghinis
In November 2023, our team of five gathered at a WeWork near Mill Valley, north of San Francisco. A bookshelf in the common area concealed a secret door. Behind it were a speakeasy and a room for listening to records. It was difficult to tell whether people went there to work or escape work, which made the place well suited to a company offsite.
My co-founder Miguel and I had occasionally worked from this building during Covid, when we were both at Brex. On one of those days, I told him that if I ever started a company, I would want to do it with him.
Now we were back at the same WeWork with Parcha Labs, the company we had started together.
The offsite began with me presenting the team with five identical white Lego Lamborghini race cars, unassembled. The activity was originally planned as a race to see who could assemble a Lamborghini first.
At the time, Parcha had five design partners. Each wanted a different agent. We had five team members building five different agents at once, held together by a common platform and a growing collection of custom code. I had been reflecting on this the night before and decided to change the Lego challenge: Build all five cars together as a factory in under an hour.
One person would open every box and arrange the Lego pieces. Another would complete the same stage on all five cars before passing them along. Another would be the factory foreman managing the whole production process. The goal was still to finish five Lamborghinis, but the team would build them through one shared system.
After we finished all five cars, I asked the team what the exercise taught us about how we could move faster as a company. The answer was uncomfortable but obvious: We were doing too many things and needed to focus our agent building efforts, even if that meant we couldn’t serve all of our design partners.
The next day I let two design partners know we couldn’t continue working on their use cases. We would focus on one agent type, Know Your Business reviews, instead of five.
That decision began three years of architectural iteration on AI agents. We moved from agent loops to controlled workflows, from controlled workflows to configurable agents, and eventually towards autonomy with oversight. Each version fixed the most painful failure of the one before it. Each also created the conditions for the next iteration.
Lesson 1: Start with one use case and become excellent at it.
The first agent
We started Parcha in 2023, when giving a language model tools and asking it to complete a multi-step task still felt reckless.
Our first system resembled today’s general agents but with far less intelligence. We gave the model a standard operating procedure, a goal and a collection of tools. It wrote a plan, kept notes in a scratchpad and worked through the task until it produced an answer.
The demos were remarkable. The same architecture could investigate a business, inspect a website, search for sanctions, identify owners and produce a due-diligence report. Changing the task often required little more than changing the instructions and available tools.
The demos hid the production failures.
An agent might skip a step that it had completed correctly the previous hundred times. It could call the same tool twice, lose an important fact as its context grew or invent a tool that did not exist. A job could run successfully for several minutes, hit one provider error near the end and lose its progress. The route through a task changed from run to run, which made failures difficult to reproduce and improvements difficult to measure.
Our customers needed to use the system for regulated work, so these errors carried consequences. A compliance team could not tell an auditor that the agent had been in an unusual mood.
We wrote about these lessons in Agents aren’t all you need. The central problem was that an agent carried too much responsibility. It had to plan the work, remember what it had done, choose every tool, manage dependencies and recover from failures. A small mistake early in the run could change everything that followed.
We began taking decisions away from the agent.
Lesson 2: A working demo accounts for about 10% of the effort. Budget the other 90% for consistent results, recovery, evaluation and scale.
The agent on rails
We decomposed the work into specialized components. One searched. Another extracted facts. Another applied a policy. A workflow controlled the order in which they ran and the conditions under which each could retry.
This architecture was easier to operate. We could see which component failed, retry it without restarting the entire job and compare results across versions. The same input followed the same path, which made evaluation and incident response possible.
The model still performed the judgment-heavy parts of the task. Our code controlled the sequence.
This division of labour worked for the tasks we knew. Reliability improved, customers trusted the results and we could process far more work. The system began handling thousands of real cases rather than carefully selected demonstrations. It also had the advantage of being much faster because tasks could be run in parallel when there was not longer an agent loop.
But new tasks exposed the architecture’s limits.
A new task required us to choose its components, map the data flow and define how each step should fail. As the number of supported use cases grew, so did the web of loaders, prompts, schemas, queues and conditional logic connecting them.
We had turned the agent into a train. It stayed on the track beautifully. Laying a new track was becoming an engineering project.
Lesson 3: Deterministic workflows make known work easier to operate. Each decision encoded in software adds code that you must maintain and customize for clients.
The Thanksgiving sprint
In November 2024, a large US crypto company asked whether we could build a new agent to scan the internet for negative news related to specific users of theirs. It was a process required by anti-money laundering regulations when a transaction was flagged as suspicious.
The company had more than 38,000 transaction-monitoring cases to review before year-end. Its investigators needed our first outputs on December 2.
We agreed to help, knowing the request reached beyond the agent we had built.
We had designed the existing system around businesses. This job required us to investigate people, ingest unfamiliar customer data, search news and the broader web, generate PDFs and return the results through a temporary integration. We needed new identity logic, new loaders, new data objects, new prompts, new evaluation sets and a dedicated environment.
By November 18, it had become an all-hands project. We spent the Thanksgiving break building it.
The team delivered the first reports on time and helped the customer work through a backlog that would have been punishing to clear by hand.
During the sprint, we saw how much custom machinery we had accumulated. Adding one new agent type meant building a new data pipeline and another collection of task-specific tools. The rails had made the existing jobs reliable, but every new destination demanded another line.
Lesson 4: Time-to-support a new task is a better measure of flexibility than the number of configuration options in the product.
One incorrect line of code
On December 18, we found a problem in the negative news pipeline.
The search process called for a Google News query using a set of adverse-media keywords, followed by a broader Google Search in each case. The implementation contained one incorrect condition. It ran Google Search only when Google News returned no results.
If Google News found one thin or irrelevant result, the wider search never happened, which could result in excluding important negative news articles.
The pipeline did not crash. It completed and produced reports. From the infrastructure’s point of view, nothing was wrong. The mistake lived inside a deterministic branch of application logic, so every affected job made the same bad decision.
We paused the work, deployed a fix and began rerunning samples to assess the impact. We had to redo thousands of jobs.
The issue here was resilience. An agent working in a loop could inspect weak evidence, decide that the search was incomplete and try another query or source. Our deterministic system had no mechanism for questioning its own successful execution.
It followed the workflow exactly. The workflow was wrong.
The mistake was the software equivalent of assembling the airbags in a car incorrectly. Every job that had passed through that line of code had to be recalled, inspected and rerun. The error cost us five figures in refunds.
The model could have followed the original instructions if we had allowed it to judge the evidence and choose another search route. Our custom code prevented it from doing so. We had added rails to improve reliability, then encoded enough reasoning in the harness for one incorrect condition to shape thousands of results.
After the incident, we increased monitoring tenfold across every critical path in the harness. We improved traceability so we could identify affected jobs faster, inspect the system’s decisions and rerun the work safely. At production scale, detecting a bad result quickly matters as much as preventing one.
Lesson 5: Business logic encoded in the harness limits the model’s judgment and is a failure waiting to happen. Instrument critical paths so your team can find failures, explain them and rerun affected work safely.
Agent Hub
Our next iteration was Agent Hub, a configurable version of the system intended to serve smaller companies we had often been forced to turn away for lack of resources to customize their agents.
Customers could choose steps, change criteria, connect data and customize the output. A company could create an agent without asking our engineering team to build a new product.
Agent Hub expanded what the platform could support. Its flexibility still came from a catalogue of templates and components we had already built. Customers could rearrange the rails and change how the train ran. They remained limited to the destinations we had anticipated.
We were getting better at manufacturing variations of known work. New categories of work remained expensive to support.
Lesson 6: Configuration expands the combinations you can offer within the boundaries of your existing system.
Claude in a box
On June 6, 2025, our team crowded into a conference room at our San Francisco office for a half-hour call with Boris Cherny, creator of Claude Code.
I had met Boris at an Anthropic event a couple of weeks earlier. We happened to sit next to each other, and during the conversation I learned that he had created Claude Code. Several people at Parcha were already using it daily, so I asked if he would speak with the team.
Claude Code was still new. Most people outside engineering had never heard of it. Inside our office, it was already changing how we worked.
Boris joined over Zoom while the team dialed in from our office. We had prepared a long list of questions about prompts, context management, evaluations and parallel agents. Within minutes, the conversation became a rapid exchange of shortcuts and discoveries.
He told us that the first version of Claude Code had little to do with coding. It took screenshots of his music player, recommended songs and wrote notes into Apple Notes. Then he gave it access to Bash.
Claude began running commands to answer questions. Soon it was editing files. That was when he realized there might be a product there.
The architecture sounded simple. Claude had a goal, access to a computer and enough freedom to keep working. It stored its progress in files. It could inspect what it had done, recover context from previous sessions and try another approach when the first one failed.
Boris said he had known little about orchestration frameworks when he began. For each decision, he chose the simplest implementation that produced the experience he wanted.
We noticed the contrast with the system we had spent two years building.
An agent failure led us to add a rail. A customer variation led us to add a template or configuration option. The system became more dependable, while its growing codebase carried more of the reasoning that should have belonged to the model.
Near the end of the call, Miguel asked whether several specialized Claude instances could work together on the same task. Boris questioned why the model needed to inherit human job categories at all. A capable agent could use whatever skills the task required.
His answer exposed the constraint in Agent Hub.
We had made the rails configurable. We had not escaped them.
The call ended after thirty minutes. Miguel told Boris that Claude Code allowed our seven-person company to operate more like a company of twenty. We left with an idea that would reshape our architecture.
Claude Code gave a general model a computer, a workspace and a loop. We decided to apply that structure to our agents.
We began building what we the first version of Grep as it exists today, and called the architecture Claude in a box.
Each research job ran inside its own isolated environment. The agent could search, inspect documents, write small programs and leave evidence behind. We gave it only the skills, tools and data required for that job.
The computer became part of the reasoning system.
Lesson 7: Give the agent a workspace where it can inspect files, write small programs and preserve state. That reduces how much orchestration the application must encode in advance.
From diligence to deep research
Our first goal was narrow: build a due diligence agent. We chose this use case because a lot of compliance work starts with diligence of sort or another, whether its during a know-your-customer workflow or while investigating an alert.
A user could provide a company, person or question, and the agent would decide how to investigate it. It could search the web, inspect source documents, transform data and assemble a report. The workflow no longer had to be mapped in advance for every task.
During the holidays, we widened the problem. The agent became useful for diligence outside compliance, then for general deep research. We began testing whether the same architecture could handle other high-stakes research tasks.
The agent handled exceptions that had required new code in the old system. If one search returned thin results, the agent could try another query or source. If a document arrived in an unfamiliar format, it could write a small program to extract it. If an intermediate result looked wrong, it could inspect its own work before continuing.
We still supplied structure. We created 25 different “experts” for users could choose from, each with its own collection of skills, tools and instructions. Our application defined the operating environment while the model decided how to move through it.
We then tested Grep’s research quality against public benchmarks.
On DRACO, Grep scored 78.6%, ahead of Perplexity Deep Research at 70.5% and Gemini Deep Research at 59%. On DeepSearchQA, it reached 84.5%, above Perplexity’s reported 81.9% and Google’s 66.1%. It later scored 56.27 on DeepResearch Bench.
By April 2026, Grep held the top result across three major independent deep-research benchmarks, ahead of systems from Perplexity, Google, Anthropic, Nvidia and OpenAI. We published the methodology, scoring code and results so that people could inspect the claims themselves.
For a small team that had begun with five Lego cars on a table, beating systems from much larger companies felt extraordinary.
The benchmarks measured capability under controlled conditions.
Benchmarks told us that the agent could perform difficult research under controlled conditions. They could not tell us whether we could run it thousands of times, recover cleanly from provider failures, explain its decisions to customers or afford the compute bill.
Production volume would test the parts that benchmarks missed.
Our customers sent messy documents, ambiguous instructions and large batches of real work. They expected the thousandth report to be as dependable as the first. When something failed, they needed us to explain what happened and resume the job without starting again.
A state-of-the-art agent that could not survive production volume was still a demonstration.
Lesson 8: Use benchmarks to measure capability. Use real workloads to measure recovery, consistency, auditability and cost.
The month the numbers stopped working
At the end of March, I pulled together our AI spending across Anthropic’s API, Microsoft Azure and the coding tools the team used every day.
The total was close to $80,000 for one month.
Some of that spend had helped us build the product. The rest came from running Grep itself. We were spending more in tokens that we were earning in revenue.
I checked the numbers again. Then I annualized them.
At that rate, we were heading towards an annual token bill of almost $1 million. Model costs exceeded our headcount costs, and each run lost money.
My heart skipped a beat. I remember thinking that the company was going to die if we did not get the costs under control.
We had built one of the strongest deep-research agents in the world. Each run relied heavily on the most capable and expensive models available. Moving a few stages from Opus to Sonnet helped, but the savings were incremental. We needed a different execution model.
Miguel began experimenting with a stripped-down harness for lower-effort research. Within a week, he had a prototype that could use cheaper open models for substantial parts of a task.
It cost roughly one-fifth as much to run.
Some outputs were better as well.
The harness gave each model a smaller, clearer piece of work. It preserved state in the workspace, routed different stages to different models and allowed the agent to recover when something failed.
When I saw the early results, I texted Miguel: “This is the product.”
Lesson 9: Frontier models can produce a capable agent fast. A scalable agent routes each stage to the least expensive model that can perform it well.
The harness becomes the product
We had spent three years building agents for mission-critical work. Customers valued the research, but they depended on everything required to run it reliably: secure access to their systems, controlled execution, monitoring, recovery, audit trails and predictable costs.
Customers were paying us to operate the agents, recover failures and account for each result.
Grep is now the easiest way to create and deploy managed agents that scale repetitive mission-critical work. A customer defines the job, provides its operating procedures. We choose the relevant skills from our library of over 250 or create new ones where needed and add tools the agent needs from over 100 trusted data sources. The harness controls execution.
For unfamiliar tasks, an agent could plan, investigate and adapt. As successful runs began following the same path, the harness could preserve that path as a cheaper, repeatable workflow. When an exception appeared, the agent could step back in rather than allowing the job to fail silently.
The new design drew on each version of our architecture.
Our first agents had freedom without enough control. The production pipelines had control without enough resilience. Agent Hub made those pipelines configurable, but every configuration still depended on a path we had anticipated.
The new harness could explore a task, learn from successful runs and reuse what became predictable. It kept credentials outside the agent’s environment, limited access to approved tools and retained the evidence needed to inspect each result.
The harness had become the product because the hardest part of production agents was running the work safely and reliably at the volume a real business required.
Lesson 10: Let agents explore unfamiliar work. Preserve successful paths as constrained procedures once the work becomes predictable.
The factory of factories
Three years ago, five of us sat around a table building five Lego Lamborghinis as a factory. We thought the lesson was standardization: stop building five different products and create one shared system.
That decision helped us scale. We then carried the metaphor too far, encoding more of the work into each station of an elaborate production line. One incorrect instruction forced a recall, while our machinery prevented newer models from applying their judgment.
Claude in a box gave the models room to reason. The factory could build remarkable cars, though each one remained expensive.
Our current architecture gives agents enough freedom to handle work we did not anticipate. The harness supplies the monitoring, traceability, recovery and cost controls required to run that work thousands of times.
We still think of ourselves as building a factory. It must learn how to build a new car, turn that work into a repeatable process and produce thousands without losing track of any one of them. The only difference is now it’s a factory of factories.
The ten lessons learned
Start with one use case and become excellent at it.
A working demo accounts for about 10% of the effort. Budget the rest for consistency, recovery, evaluation and scale.
Deterministic workflows make known work easier to operate, but every encoded decision adds code to maintain and customize.
Measure flexibility by how quickly the system can support a new task.
Each decision encoded in the harness limits the model’s judgment. Instrument every critical path.
Configuration expands the combinations available within the boundaries of the existing system.
Give agents a workspace where they can inspect files, write programs and preserve state.
Use benchmarks to measure capability and real workloads to measure recovery, consistency, auditability and cost.
Route each stage to the least expensive model that can perform it well.
Let agents explore unfamiliar work, then preserve successful paths as constrained procedures.
If you are deploying agents for mission-critical work, I would like to hear what is working and where they fail. You can book time here.
About the author:
I’m AJ Asver, co-founder and CEO of Grep. I’ve spent the past three years building and deploying AI agents for high-stakes work inside large companies. I write The Operator’s Guide to AI to share what we learn when those systems meet real workflows, customers, costs, and production constraints.
Lesson 5 lines up with a pattern I keep seeing across the operations-automation data: 84 percent of organizations have no documented workflows, and one case study found 47 undocumented business rules hiding behind a single screen control. Your Google Search line is that same finding at n=1, just discovered by a refund bill instead of an audit.
The costly part wasn't the bad line, it was that nobody could see the logic well enough to catch it first. That's the harder cost behind lesson 3's per-task tracks too: every new track is another place for an unreviewed rule to hide, and review capacity doesn't scale with them.