Arize AI、AI エージェントの生産環境移行を評価駆動開発で加速する方法を解説

CVS Health teams moved a four-week feature from idea to production in roughly a day and a half, then completed a legacy rewrite in one month after estimating six to nine months. In their Arize Observe 2026 talk, Matt Turner and Lagan Khare explain how specifications, evaluation harnesses, AI observability, guardrails, and cost-per-outcome metrics can make that speed durable.

Enterprise AI projects often stall at the point where a convincing demo has to become a dependable production system. Evaluation, AI observability, governance, and cost controls suddenly carry as much weight as model performance, and teams quickly discover that these capabilities cannot be added casually after the build.

At CVS Health, lead director of architecture, Matt Turner, and executive director of engineering and enablement, Lagan Khare, have spent years developing the practices required to close that gap. Turner said their teams had taken a feature from idea to production in roughly a day and a half, and completed a legacy rewrite in one month after estimating six to nine months. Although the acceleration was dramatic, making it repeatable required a disciplined AI-native software development lifecycle supported by clear specifications, trusted tests, production monitoring, and measurable business outcomes.

Their experience points to a broader lesson for engineering teams: AI delivery speed depends on the systems surrounding the model. As developers generate more code, prototypes, and workflows at machine speed, evaluation-driven development and observability become core parts of the development process. Together, they provide the evidence teams need to grant agents greater autonomy without losing control of quality, cost, or risk.

Why enterprise AI pilots stall before production

Enterprise AI pilots stall when the surrounding operating system of software delivery cannot evaluate, govern, and absorb work at the pace a model produces it. The demo room tends to conceal that mismatch because a polished interaction can prove capability before anyone has settled the harder questions about release criteria, production monitoring, ownership, access, or return on investment.

Turner began the session by asking how many people had seen an AI-assisted demo that looked immediately useful, then followed with a question about systems that had reached production and produced measurable value. Together, those questions framed what he called “pilot purgatory,” a state in which a promising system waits for the organization around it to become ready.

Model improvements do little to resolve the missing discipline. Evaluation, governance, deployment controls, and measurement arrive through deliberate engineering work, while stronger base models can simply deliver more output into an unprepared process. When a pilot sits for more than a quarter, Turner argued, teams should inspect the surrounding system before assuming that another model release will rescue the project.

Khare approached the same problem through durability, which she described as the operating model that continues to hold after a demo has established capability and a deployment has shown that the team can ship. Durable value emerges when that model survives growing usage, shifting source data, accumulating costs, and newly visible failure modes.

That distinction changes the question developers should ask. Production becomes an engineering path that teams need to design before the first agent begins making consequential decisions.

What is an AI-native software development lifecycle?

An AI-native software development lifecycle uses agents across individual development, team coordination, and production validation while preserving human ownership of intent, evidence, and release decisions. Turner described three layers of discipline, each of which creates a new form of leverage and a new class of failure.

Layer 1: AI coding tools and specifications accelerate individual developers

Personal productivity is the usual on-ramp because coding assistants produce immediate, tactile gains. An engineer can explore a codebase, scaffold a service, generate a working spike, and draft tests before a conventional handoff would have begun. At CVS Health, general coding tools sit alongside shared Model Context Protocol (MCP) servers, source-control and ticketing integrations, and reusable skills that encode repeatable practices.

One internal skill, called Clarify, helps align product and engineering intent before implementation starts. Its purpose reflects a larger shift toward specifications that agents and humans can evaluate: the specification becomes a durable source of truth containing structured requirements, edge cases, acceptance criteria, and system context.

“The spec is as much for the engineer as it is for the agent building it,” Turner said.

That obligation matters because an agent can produce code that compiles while misunderstanding the feature. A specification forces the engineer to articulate the intended behavior before delegating the work, which gives reviewers a stable artifact against which they can judge the implementation. As a library of well-formed specifications grows, new features become faster to scaffold because previous context, contracts, and decisions remain available.

Turner has also seen teams prototype live while stakeholders remain in the room. Because the working system arrives while the conversation is still active, product, design, and engineering can test assumptions before they harden into a backlog item or an architectural commitment. The prototype becomes an instrument for collective reasoning, allowing the team to validate the idea before a conventional post-meeting handoff would return.

The same acceleration creates traps. Vibe coding encourages developers to approve code they do not understand, while agent-generated tests may confirm the assumptions already embedded in the implementation. Large tasks can overwhelm the available context until the agent loses coherence, so engineers need to decompose work into smaller, executable units with clear intent. Shared tooling also needs enough consistency for practices to compound across teams, since isolated tool choices prevent one group’s learning from becoming organizational infrastructure.

Layer 2: Shared context and dependency maps align engineering teams

Once individual output rises, coordination becomes the pacing system. Product intent, design decisions, service contracts, and dependencies must leave people’s heads and enter artifacts that humans and agents can both access, because undocumented knowledge becomes an immediate ceiling on agent performance.

Specifications gain a second function at this layer. They align product, design, and engineering around the same acceptance criteria, while revealing gaps before generated code turns them into rework. Turner cautioned against polishing every spec into a monument because the useful artifact only needs enough structure to support fast iteration, with a scope small enough for both the team and the agent to reason about clearly.

Dependency mapping becomes equally important because agentic changes can cross service boundaries faster than organizational memory can track them. “If an agent doesn’t know what its change is going to break, it will break it,” Turner says.

A dependency map gives the agent a view of the blast radius before it changes a contract, a data flow, or a shared component. Without that context, the time saved during generation can disappear into cleanup, incident response, and cross-team negotiation.

AI can also compress the distance between roles, allowing product managers, designers, and engineers to work in parallel around the same prototype and specification. Turner heard several teams describe this emerging identity as “builders,” a term that captures how the boundaries between defining, designing, and implementing a feature can become more porous.

The organizational constraint does not vanish with the handoff. Turner described features that developers completed in hours and then watched stall for weeks because user validation, product review, or dependent teams could not move at the same speed. AI amplifies the coordination system already in place, so healthy teams gain more parallel capacity while brittle silos collide earlier.

Layer 3: Evaluation and observability earn agent autonomy

Production autonomy depends on evidence that the system behaves correctly and can be stopped when its behavior changes. Turner framed this layer as the point where trusted tests, agent observability, staged rollouts, behavioral dashboards, and rollback mechanisms become prerequisites for speed.

Deterministic tests provide the foundation, although coverage percentages alone offer little comfort when the tests fail to represent meaningful behavior. AI can propose cases and draft test code, while humans remain responsible for verifying that those tests catch failures the business and the user would care about.

Agentic workflows add nondeterminism, which requires evaluators that can inspect whether outputs and end-to-end behavior satisfy the intended outcome. A pull request may look reasonable line by line while producing an unexpected behavior after deployment, so review shifts toward intent, system impact, and the evidence supplied by traces and evaluations.

User validation belongs in the same loop. Faster shipping creates value only when users can confirm that the system solves the intended problem, and a rapid implementation without rapid feedback can turn acceleration into efficiently produced waste.

When teams increase generation volume before building this layer, the resulting failures become harder to locate and more expensive to contain. In a regulated environment, an undetected error can also become a compliance event, which means explainability and audit trails influence the ceiling on autonomy as directly as model capability does.

What should an AI evaluation harness include?

An AI evaluation harness should include golden datasets, regression suites, and continuous production monitoring that together define acceptable behavior, catch regressions, and reveal drift. The evaluation harness gives teams a repeatable way to answer whether an agent is good enough to ship and whether it remains good enough after release.

“Build the harness before the agent,” Khare asserts.

Khare compared evals for generative AI systems to unit tests in conventional software engineering. The analogy becomes especially useful when teams place evaluation at the beginning of development, because the cases and thresholds force them to define quality before prompt tuning and model selection consume the conversation.

Golden datasets and regression suites define release quality

Golden datasets encode the cases that matter, including expected behavior, known edge cases, and failures that would make a release unacceptable. Regression suites then preserve those lessons by rerunning the cases whenever the agent or its surrounding system changes.

This practice turns evaluation into a design activity. When a team has to decide which examples deserve permanent protection, it exposes disagreements about intent while those disagreements remain inexpensive to resolve. The resulting suite also gives reviewers a body of evidence that can support a release decision without relying on a handful of manually selected demos.

Human judgment remains essential because a weak evaluator can reward the same shallow behavior that a weak test would permit. Teams need to review the cases, scoring criteria, and failure thresholds with the same seriousness they apply to production code, especially when an agent’s output has several acceptable forms.

Continuous evaluation detects model, data, and user drift

Continuous evaluation extends the harness into production, where models, source data, and user behavior can all drift. A launch score captures a moment; an operational evaluation program reveals whether that score changes over the next 30 days and which slices of traffic account for the movement.

That history allows developers to separate a broad regression from a localized failure, such as a tool-call issue, a newly ambiguous user request, or a data-quality problem in one source. Traces and production samples can then feed new cases back into the golden dataset, which lets the harness mature alongside the system.

This feedback loop gives AI agent evaluation its production value. In evaluation-driven development, the harness becomes the mechanism through which teams learn what the agent does under conditions the original test set did not anticipate, while each newly observed failure sharpens the next release decision.

How should teams measure AI agent cost and ROI?

Teams should measure AI agent economics through cost per outcome and connect that figure to a business KPI such as cycle time, error rate, cost per transaction, or time to resolution. Token usage remains useful for infrastructure analysis, although it cannot capture the retries, rework, escalations, and review burden created by an unreliable workflow.

Cost per outcome captures retries, rework, and review

Khare argued that the meaningful unit is cost per outcome because a cheaper model can produce a more expensive operating system when its outputs require additional cleanup. A model that saves pennies during inference may add dollars downstream when a human has to verify, correct, rerun, or escalate the work.

The unit should match the job the agent performs. A support workflow might track cost per resolved inquiry, a document workflow might track cost per processed document, and an internal automation might track cost per completed task. Those measures incorporate the full path to a usable result, which makes them more informative than a monthly token bill detached from task success.

Production planning should therefore include per-feature spending ceilings, fallback paths, and observability on spend. These controls allow engineering and finance to understand how costs behave under real volume, where retries and exceptions can make pilot economics misleading.

Business KPIs belong in the AI intake process

A production candidate should enter development with a business metric already attached. Khare named cycle time, throughput, documentation turnaround, error rate, cost per transaction, and time to resolution as examples that can connect an AI system to operational value.

When the metric appears during intake, engineering can design the evaluation suite and production monitoring around the result the organization expects. The same metric then supports prioritization, release decisions, and post-launch analysis, which prevents ROI from becoming a story assembled after the budget has already been spent.

Khare’s rule was simple: the metric comes before the model. Finance belongs in this conversation early because enterprise scale changes the economics. Although a pilot may look inexpensive while serving a limited audience and routing difficult cases around the system, production exposes the true cost of retries, human review, fallback models, data access, and ongoing monitoring. Cost per outcome gives finance and engineering a common unit for deciding whether the workflow deserves to expand.

Which parts of the agent harness survive better models?

The durable parts of an agent harness are the data, workflow design, evaluation suites, permissions, and trust fabric that remain valuable as model capabilities improve. Teams should scrutinize custom orchestration, retrieval, and routing layers because a future model or API may absorb functions that currently require extensive scaffolding.

“The model will eat your stack,” Khare said, describing how quickly capabilities can migrate from surrounding infrastructure into the model layer.

The observation redirects engineering investment toward the parts of the system that retain value as the model changes. A workflow that captures domain intent, constrains access, records traces, and measures outcomes can continue to create value after the underlying model changes. A brittle layer built around a temporary limitation may become maintenance work with a short half-life.

Data quality, workflow design, evals, and trust remain durable

Data quality moves onto the critical path because agents amplify ambiguity at scale. A messy column that an experienced analyst can interpret through context may lead an agent toward a confident error, and the resulting answer can propagate through a workflow before anyone notices the source of the mistake.

Clean source material, relevant context, and well-defined semantics therefore remain part of production engineering. The work may lack the spectacle of a new model demo, although it determines whether the system can use that model safely.

Workflow design also survives because a model does not inherit the organization’s operating context. Teams still have to decide which problem deserves automation, how responsibilities move between systems and people, where approval belongs, and what evidence should accompany an action. Evaluation suites preserve those decisions as executable expectations, while the trust fabric enforces the permissions and auditability required to operate them.

Why can AI workflow automation create downstream bottlenecks?

AI workflow automation creates downstream bottlenecks when one step accelerates without a corresponding increase in the capacity of the teams and systems that receive its output. Because AI often performs only a small part of a larger process, local throughput can rise while end-to-end completion time becomes worse.

Khare described an example in which Team A’s agent ran continuously and buried Team B under the resulting tasks. The first group could point to a faster local process, yet the organization had merely moved the queue to a place with less capacity.

The pattern is easy to miss during a low-volume pilot, which allows downstream teams to absorb the extra work, whereas production changes the shape of the system. A workflow that generates cases, documents, code changes, or review requests around the clock can overwhelm a step that still depends on a small group of specialists.

Map the full workflow before automating a step

Teams should map the complete workflow, including upstream inputs, downstream queues, approval points, and exception paths, before choosing the automation target. That map reveals where increased volume will land and whether another step will become the new constraint.

The exercise also changes prioritization because the easiest manual task may offer little system-level value when it feeds a scarce reviewer, while a less obvious intervention could remove a handoff, improve source data, or reduce the number of cases that require escalation. Developers need to evaluate the automation across an end-to-end workflow containing users, reviewers, compliance functions, dependent teams, and other software.

This systems view also protects the business metric. When teams measure cycle time or cost per completed task across the full workflow, a local speedup cannot disguise a slower overall process.

How do AI agent guardrails work in regulated environments?

AI agent guardrails in regulated environments combine mechanical controls that constrain what an agent can physically do with text-based controls that make its intended behavior reviewable. Khare argued that both layers belong in the architecture because failures carry asymmetric costs, and a defect that would cause a retry in a consumer application can trigger audit exposure, notification obligations, or a breach in a regulated workflow.

Mechanical guardrails constrain production access

Mechanical guardrails use deterministic controls to enforce boundaries around data, tools, and actions. They include hard checks on infor