生産環境の耐久性と高速評価を両立するAIワークフローのパターン

要点

• AI workflows have two needs that trade off directly. Running reliably in production requires persisting and distributing every step so it survives crashes, deploys, and restarts. But that same machinery is what makes runs too heavy for the fast, throwaway loop you need to check an LLM's output quality. The properties that buy durability are the ones that kill iteration speed.

• You can serve both needs by writing the workflow as pure business logic that doesn’t know where it runs, then plugging in the runtime, so the exact same logic runs unchanged in production and evals.

• When there is only one version of the logic, the version that goes through eval is guaranteed to match the one that ships. This approach removes a whole class of bugs caused by different versions of the logic drifting apart over time.

• Keeping the logic agnostic of where it runs can’t rely on developer discipline. The architecture must make the right way the easiest way to write a workflow. Whenever someone writes non-agnostic code, the build must fail.

• The decoupling isn’t free. The orchestration loses direct access to each runtime’s native features. Every new capability has to be wired through the agnostic layer. This design only pays off when a project genuinely needs both production reliability and fast evaluation.

An AI workflow is a sequence of steps chained together to complete a task, where one or more of those steps consists of calls to a large language model (LLM). The logic that combines these steps (their ordering and branching) is hereby referred to as the workflow’s orchestration logic.

These workflows carry the same production requirements any long-running distributed system has had for a decade: They need to survive deploys and crashes, retry idempotently, and scale horizontally. Workflow engines solved this class of problem years ago.

What sets AI workflows apart is that an LLM step’s output quality can drift with every prompt tweak or model change, so it has to be checked with evals, offline runs of the workflow against a labeled dataset to score its outputs.

That, in turn, demands something the classic workflow engine wasn't designed for: a fast evaluation loop that is cheap enough to rerun hundreds of times.

These two requirements pull in opposite directions. Production durability wants a heavyweight, persistent, distributed runtime. Eval iteration wants a lightweight, ephemeral, in-process loop you can rerun in seconds. Most stacks are built around one of these runtimes.

Durability-first runtimes do offer test environments, but they need to stand up sandboxes, task queues, and test servers for an evaluation loop that needs none of it. This article describes the pattern we used to remove that trade-off.

The pattern comes out of Brex's AI workflow platform. The platform is written in TypeScript and maintained by a team of five engineers. Its workers run on Brex's Kubernetes cluster and connect to Temporal Cloud, the managed Temporal offering, to execute long-running agents.

Agents reach LLMs through the Vercel AI SDK, which routes to an internal LLM Gateway that centralizes rate limiting and authentication. Evals run on our own in-house platform.

The Trade-Off Between Production Durability and Fast Offline Evals

Let’s start with workflow engine features. Durable execution requires that the result of every step is persisted before the next step runs. If the process crashes, is redeployed, or is rescheduled onto a different worker, the engine replays history and resumes exactly where it left off. State outlives any single process. That is precisely what you want for a deep research agent that runs for an hour across dozens of LLM calls and tool invocations: You cannot afford to lose forty minutes of work because a pod was recycled.

Now look at eval iteration features. You are tuning a prompt or a branching decision. You want to change one line, load a dataset of a few hundred examples, and see aggregate scores. The loop is local, in-process, and ephemeral. Nothing should be persisted or scheduled across a cluster. Nothing should survive the run. You want to mock the dependencies so that their outputs are held fixed, isolating the LLM (the piece you’re actually evaluating) as the only thing that varies between runs. The loop stays cheap enough to run hundreds of times an hour.

Running your evals *through* a workflow engine is a category mismatch. You inherit persistence, task queues, workers, and replay semantics. This is overhead that fights the tight loop you need. Conversely, running production *through* an eval harness gives you none of the durability guarantees on which your hour-long agent depends.

They are different runtimes solving different problems. Your orchestration should not have to marry either one. In practice, though, it almost always does.

Most Stacks Force You to Pick One

The reason teams end up married to a runtime is that mainstream tools couple orchestration to the runtime by design. Agent frameworks such as LangGraph and Mastra express orchestration directly in their own SDKs. Your control flow becomes graph nodes and edges, or the framework’s DSL. The orchestration logic and the framework are the same artifact. To evaluate that logic, you run the framework; to serve it, you run the framework. There is no orchestration that exists independently of the framework.

As a concrete example, here is the classifyBusinessAgent (the same one we’ll rewrite later) as a Mastra workflow:

import { createWorkflow, createStep } from "@mastra/core/workflows";
import { z } from "zod";

const enrichWithWebData = createStep({
  id: "enrich-with-web-data",
  inputSchema: z.object({ businessName: z.string(), website: z.string() }),
  outputSchema: z.object({ businessName: z.string(), webContext: z.string() }),
  execute: async ({ inputData }) => {
    // ...
  },
});

const classify = createStep({
  id: "classify",
  inputSchema: z.object({ businessName: z.string(), webContext: z.string() }),
  outputSchema: z.object({ category: z.string() }),
  execute: async ({ inputData }) =>
    // ...
});

// Ordering lives in Mastra's builder, not in plain control flow. The steps are
// Mastra objects, and the workflow only exists once committed to its engine.
export const classifyBusinessWorkflow = createWorkflow({
  id: "classify_business",
  inputSchema: z.object({ businessName: z.string(), website: z.string() }),
  outputSchema: z.object({ category: z.string() }),
})
  .then(enrichWithWebData)
  .then(classify)
  .commit();

Everything here is Mastra. Nothing runs until .commit() hands the graph to Mastra’s engine. To evaluate this logic, you stand up that same engine.

Workflow engines such as Temporal go the other way, allowing you to write orchestration in a general-purpose language, but constraining how you write it. Temporal workflow code must be deterministic, so you cannot call Date.now() or perform I/O directly inside the orchestration. All that has to be pushed into the workflow steps. There are payload size limits on what crosses the workflow boundary. The constraints exist to ensure replayability, but their existence demands that the orchestration is written *against* the engine’s rules.

Brex's onboarding agents run in production on Temporal, but need their LLM-driven decisions tuned continuously. The naive way to evaluate them is to reimplement each agent in a separate eval runtime, but that approach leaves two copies of the same logic, which opens the door to eval-prod skew, because the copies can drift.

Eval-prod skew is the failure mode this pattern is designed to make impossible and is achieved by implementing runtime-agnostic workflow orchestration, which breaks most frameworks’ assumption that runtime and orchestration are a single artifact.

Runtime-agnostic Orchestration

The core move is to stop writing orchestration *for* a runtime and start writing it against an interface that the runtime satisfies.

Figure 1. The portable core and its adapters. Source: created by the author.

The orchestration and its Steps interface are identical across runtimes; only the injected plugins and the runtime underneath change.

A complete, runnable version of the pattern is available in the repository. It holds a runnable agent named ClassifyBusinessAgent that is wired to production and eval runtimes. Its src/ folder is split in three ways. The agents/ folder is the only one an agent author touches, with one folder per agent holding the orchestration and the concrete steps that implement it. The platform/ folder holds the primitives an agent is defined against, plus the two runtime adapters from the image above. Finally, bin/ holds the entry points, such as the production worker and the eval loop.

This top-level split is what backs up the argument behind the pattern. Adding an agent requires writing in agents/ only. The platform/ does not change and neither runtime learns anything about the new agent.

Concretely, an agent’s orchestration is a plain function. Its only dependency is a typed Steps interface that names the agent’s meaningful operations. Nothing runtime-specific is imported: no Temporal, no eval framework, and no Node.js built-ins.

// The contract the orchestration depends on. Nothing here is runtime-aware.
export interface ClassifyBusinessSteps {
  enrichWithWebData(website: string): Promise<string>;
  classify(businessName: string, webContext: string): Promise<string>;
}

// No workflow engine or eval framework imports.
export const classifyBusinessAgent = defineAgentHandle({
  name: "classify_business",
  description: "Classifies a business given its name and website.",
  orchestration: async (
    steps: ClassifyBusinessSteps,
    input: { businessName: string; website: string },
  ) => {
    const webContext = await steps.enrichWithWebData(input.website);
    return steps.classify(input.businessName, webContext);
  },
});

The orchestration reads like business logic: Enrich, then classify. It has no opinion on whether enrichWithWebData is a Temporal activity dispatched to a worker or an in-process call that returns fixture data. A developer writing a new agent never touches a runtime.

The side effects live in a concrete Steps implementation. This is where real work happens. Here it receives dependencies, such as the web scraper and the LLM client, via injection (see ClassifyBusinessStepsImpl).

In production, the plugins hit real services. In evals, the plugins return fixtures. The orchestration cannot tell the difference, which is exactly the property we want.

Keeping Orchestration Portable

Portability is not free; it has to be enforced or it erodes the first time someone reaches for a convenient shortcut. Two rules matter most:

• No hidden non-determinism in orchestration

No wall-clock reads, no random values, no direct I/O. Anything non-deterministic is a Steps method, so it becomes a point where the runtime takes control.

• No Node.js or runtime-specific APIs in orchestration

The orchestration and the Steps interface must import nothing that ties them to a process model. Node-only modules (HTTP clients, file/CSV parsers) belong in StepsImpl, never in the orchestration or the interface.

These rules allow the same orchestration to replay safely in production and evals. We make the portable shape the path of least resistance: The function, defineAgentHandle, gives you nothing but *steps* and *input* to work with, so the natural way to write an agent is already correct.

Production and Eval Adapters

With orchestration reduced to a function over an interface, a runtime adapter is just "supply a *Steps* and call the function". Two adapters carry the load: Temporal for production durability and an in-process adapter for evals.

The Temporal Adapter

Temporal splits code into two worlds: *activities* (workflow steps), which run in a normal Node.js process and may do I/O, and *workflow* code, which runs in a deterministic sandbox, whose only way to affect the outside world is to dispatch an activity. Our adapter cleanly maps the pattern onto that split: Each *Steps* method becomes an activity and the orchestration runs inside the sandbox.

Figure 2. The Temporal adapter sequence diagram. Source: created by the author.

A steps.foo(...) call in the sandbox is dispatched as a durable activity on the worker. The per-agent Proxy readds the agentName prefix, so the orchestration stays unaware of the runtime.

The worker side runs in a regular Node process. It instantiates each agent’s concrete Steps with the real plugins and registers every method as a Temporal activity flattened into a single dictionary under names prefixed by the agent (classify_business_enrichWithWebData, for example), so that two agents defining the same method name never collide. The full source code for the worker is in worker.ts.

The orchestration side is what runs in the sandbox. It never imports StepsImpl, so nothing here transitively pulls in Node modules. This is the build-time enforcement mentioned earlier. If the orchestration accidentally depended on a Node-only module, this bundle would fail to build. The full source code for the orchestration code is in workflows.ts.

The Temporal adapter supplies an agent’s orchestration function with Temporal activities as the implementation of the Steps interface. Each time the orchestration calls a method, a *Proxy* intercepts the access and prepends the agent name, mapping the plain method to the prefixed activity the worker registered (so steps.enrichWithWebData maps to classify_business_enrichWithWebData).

Once a client starts agentWorkflow, every steps.foo(...) call inside the orchestration is dispatched as a Temporal activity with retries, timeouts, and replay on redeploy. The orchestration code is completely unaware that any of it is happening.

This is the layer that hardened our long-running agents. They run about one hundred times a day and take twenty to sixty minutes each, spanning dozens of LLM and tool calls. Under the old setup, a pod recycle, deploy, or timeout anywhere in the run would wipe it all and nearly four percent never completed. Now, when a worker dies mid-run, Temporal replays history and resumes at the last completed step; completion has held at 99.9 percent over the last few months.

The Eval Adapter

The eval adapter is dramatically smaller. There is no sandbox, no worker, and no task queue. It runs the *same* orchestration in-process with a *Steps* instance whose plugins return fixtures instead of hitting real services. Check it yourself at run-eval.ts.

// Runs orchestration in-process with a real Steps instance, but with plugins
// that return fixture data. Same orchestration as production — byte-for-byte —
// only the runtime underneath differs.
export async function runEval<Input, Output, Steps>(
  handle: AgentHandle<Input, Output, Steps>,
  StepsClass: new (plugins: Plugins) => Steps,
  input: Input,
  fixtures: Record<string, string>,
  llm: Llm,
): Promise<Output> {
  const plugins: Plugins = {
    webScraper: new MockWebScraper(fixtures),
    llm, // LLM calls stay real — they are what we are evaluating.
  };
  return handle.orchestration(new StepsClass(plugins), input);
}

Because runEval is just an (input) => Promise<output> function, any eval platform can wrap it as a black box. See braintrust-eval.ts for an example of using Braintrust to run evals.

Braintrust is just an example to illustrate that nothing about the orchestration knows about the eval framework or runtime. Laminar, an internal tool or a plain script, plugs in the same way. This approach lets us experiment cheaply with eval platforms.

Payoffs and Costs

This architecture, like any other in software, comes with trade-offs.

What We Gained

• No eval-prod skew, by construction.

The orchestration you evaluate is the orchestration you ship, so a branch-tuned-in eval cannot silently differ from the one in production.

• Runtime choices became reversible.

Temporal or Restate for execution, Braintrust or Laminar or LangSmith for evals. These are adapter swaps, not rewrites. We changed eval platforms more than once without agents noticing.

• Runtime complexity became a one-time platform cost.

Developers write plain TypeScript against a Steps interface. They do not learn Temporal’s determinism rules or an eval SDK to ship an agent. Onboarding a new contributor is onboarding to an interface, not to a runtime.

• Reliability and business impact followed.

By absorbing transient infrastructure failures, the Temporal adapter raised long-running execution success from about ninety-six percent to 99.9 percent. On the business side, agents built on this platform now produce an automated decision for more than half of onboarding applications.

What We Gave Up

• Direct access to runtime-native primitives.

Orchestration cannot call Temporal signals, queries, or timers directly, because those do not exist in the eval runtime. Anything runtime-native has to be modeled behind the interface, which sometimes forces a less elegant abstraction than the native API.

• Every runtime feature must be added through indirection and propagated.

Exposing a new capability requires designing it into the interface and implementing it across all adapters. New features are platform-wide changes, not one-liners.

• Visual tooling out of the box.

Framework-native graph visualizers and step debuggers assume you wrote to their model. When your orchestration is plain functions over an interface, you give that up unless you build your own.

The pattern earns its keep when you have more than a couple of workflows, real production-durability requirements, and a serious eval practice. For a platform hosting a fleet of agents against regulated decisions, treating the runtime as a plugin behind an interface has been the design decision that lets production reliability and eval velocity stop fighting each other.

About the Author

Mateus Moury

Mateus Moury is a Staff Software Engineer with ten years of experience architecting large-scale distributed systems and high-throughput platforms. He currently leads the Agents Platform at Brex, driving the orchestration, evaluation, and annotation infrastructure required to deploy production AI agents across core financial workflows, including KYC, fraud, underwriting, and sales. He has previously worked at Google and Incognia.

Show moreShow less