急速に進化するAI製品のフィールドガイド

Most AI teams focus on the wrong things. Here’s a common scene from my consulting work:

AI TEAM

Here’s our agent architecture – we’ve got RAG here, a router there, and we’re using this new framework for…

ME

[Holding up my hand to pause the enthusiastic tech lead.]

“Can you show me how you’re measuring if any of this actually works?”

… Room goes quiet

This scene has played out dozens of times over the last two years. Teams invest weeks building complex AI systems, but can’t tell me if their changes are helping or hurting.

This isn’t surprising. With new tools and frameworks emerging weekly, it’s natural to focus on tangible things we can control – which vector database to use, which LLM provider to choose, which agent framework to adopt. But after helping 30+ companies build AI products, I’ve discovered the teams who succeed barely talk about tools at all. Instead, they obsess over measurement and iteration.

In this post, I’ll show you exactly how these successful teams operate. You’ll learn:

How error analysis consistently reveals the highest-ROI improvements

Why a simple data viewer is your most important AI investment

How to empower domain experts (not just engineers) to improve your AI

Why synthetic data is more effective than you think

How to maintain trust in your evaluation system

Why your AI roadmap should count experiments, not features

I’ll explain each of these topics with real examples. While every situation is unique, you’ll see patterns that apply regardless of your domain or team size.

Let’s start by examining the most common mistake I see teams make – one that derails AI projects before they even begin.

1. The Most Common Mistake: Skipping Error Analysis

The “tools first” mindset is the most common mistake in AI development. Teams get caught up in architecture diagrams, frameworks, and dashboards while neglecting the process of actually understanding what’s working and what isn’t.

One client proudly showed me this evaluation dashboard:

The kind of dashboard that foreshadows failure.

This is the “tools trap” – the belief that adopting the right tools or frameworks (in this case, generic metrics) will solve your AI problems. Generic metrics are worse than useless – they actively impede progress in two ways:

First, they create a false sense of measurement and progress. Teams think they’re data-driven because they have dashboards, but they’re tracking vanity metrics that don’t correlate with real user problems. I’ve seen teams celebrate improving their “helpfulness score” by 10% while their actual users were still struggling with basic tasks. It’s like optimizing your website’s load time while your checkout process is broken – you’re getting better at the wrong thing.

Second, too many metrics fragment your attention. Instead of focusing on the few metrics that matter for your specific use case, you’re trying to optimize multiple dimensions simultaneously. When everything is important, nothing is.

The alternative? Error analysis - the single most valuable activity in AI development and consistently the highest-ROI activity. Let me show you what effective error analysis looks like in practice.

The Error Analysis Process

When Jacob, the founder of Nurture Boss, needed to improve their apartment-industry AI assistant, his team built a simple viewer to examine conversations between their AI and users. Next to each conversation was a space for open-ended notes about failure modes.

After annotating dozens of conversations, clear patterns emerged. Their AI was struggling with date handling – failing 66% of the time when users said things like “let’s schedule a tour two weeks from now.”

Instead of reaching for new tools, they: 1. Looked at actual conversation logs 2. Categorized the types of date-handling failures 3. Built specific tests to catch these issues 4. Measured improvement on these metrics

The result? Their date handling success rate improved from 33% to 95%.

Here’s Jacob explaining this process himself:

Bottom-Up vs. Top-Down Analysis

When identifying error types, you can take either a “top-down” or “bottom-up” approach.

The top-down approach starts with common metrics like “hallucination” or “toxicity” plus metrics unique to your task. While convenient, it often misses domain-specific issues.

The more effective bottom-up approach forces you to look at actual data and let metrics naturally emerge. At NurtureBoss, we started with a spreadsheet where each row represented a conversation. We wrote open-ended notes on any undesired behavior. Then we used an LLM to build a taxonomy of common failure modes. Finally, we mapped each row to specific failure mode labels and counted the frequency of each issue.

The results were striking - just three issues accounted for over 60% of all problems:

Excel Pivot Tables are a simple tool, but they work!

Conversation flow issues (missing context, awkward responses)

Handoff failures (not recognizing when to transfer to humans)

Rescheduling problems (struggling with date handling)

The impact was immediate. Jacob’s team had uncovered so many actionable insights that they needed several weeks just to implement fixes for the problems we’d already found.

If you’d like to see error analysis in action, we recorded a live walkthrough here.

This brings us to a crucial question: How do you make it easy for teams to look at their data? The answer leads us to what I consider the most important investment any AI team can make…

1. The Most Important AI Investment: A Simple Data Viewer

The single most impactful investment I’ve seen AI teams make isn’t a fancy evaluation dashboard – it’s building a customized interface that lets anyone examine what their AI is actually doing. I emphasize customized because every domain has unique needs that off-the-shelf tools rarely address. When reviewing apartment leasing conversations, you need to see the full chat history and scheduling context. For real estate queries, you need the property details and source documents right there. Even small UX decisions – like where to place metadata or which filters to expose – can make the difference between a tool people actually use and one they avoid.

I’ve watched teams struggle with generic labeling interfaces, hunting through multiple systems just to understand a single interaction. The friction adds up: clicking through to different systems to see context, copying error descriptions into separate tracking sheets, switching between tools to verify information. This friction doesn’t just slow teams down – it actively discourages the kind of systematic analysis that catches subtle issues.

Teams with thoughtfully designed data viewers iterate 10x faster than those without them. And here’s the thing: these tools can be built in hours using AI-assisted development (like Cursor or Loveable). The investment is minimal compared to the returns.

Let me show you what I mean. Here’s the data viewer built for NurtureBoss (which we discussed earlier):

Search and filter sessions

Annotate and add notes

Aggregate and count errors

Here’s what makes a good data annotation tool:

Show all context in one place. Don’t make users hunt through different systems to understand what happened.

Make feedback trivial to capture. One-click correct/incorrect buttons beat lengthy forms.

Capture open-ended feedback. This lets you capture nuanced issues that don’t fit into a pre-defined taxonomy.

Enable quick filtering and sorting. Teams need to easily dive into specific error types. In the example above, NurtureBoss can quickly filter by the channel (voice, text, chat) or the specific property they want to look at quickly.

Have hotkeys that allow users to navigate between data examples and annotate without clicking.

It doesn’t matter what web frameworks you use - use whatever you are familiar with. Because I’m a python developer, my current favorite web framework is FastHTML coupled with MonsterUI, because it allows me to define the back-end and front-end code in one small python file.

The key is starting somewhere, even if it’s simple. I’ve found custom web apps provide the best experience, but if you’re just beginning, a spreadsheet is better than nothing. As your needs grow, you can evolve your tools accordingly.

This brings us to another counter-intuitive lesson: the people best positioned to improve your AI system are often the ones who know the least about AI.

1. Empower Domain Experts To Write Prompts

I recently worked with an education startup building an interactive learning platform with LLMs. Their product manager, a learning design expert, would create detailed PowerPoint decks explaining pedagogical principles and example dialogues. She’d present these to the engineering team, who would then translate her expertise into prompts.

But here’s the thing: prompts are just English. Having a learning expert communicate teaching principles through PowerPoint, only for engineers to translate that back into English prompts, created unnecessary friction. The most successful teams flip this model by giving domain experts tools to write and iterate on prompts directly.

Build Bridges, Not Gatekeepers

Prompt playgrounds are a great starting point for this. Tools like Arize, Langsmith and Braintrust let teams quickly test different prompts, feed in example datasets, and compare results. Here are some screenshots of these tools:

Arize Phoenix

LangSmith

Braintrust

But there’s a crucial next step that many teams miss: integrating prompt development into their application context. Most AI applications aren’t just prompts – They commonly involve RAG systems pulling from your knowledge base, agent orchestration coordinating multiple steps, and application-specific business logic. The most effective teams I’ve worked with go beyond standalone playgrounds. They build what I call integrated prompt environments – essentially admin versions of their actual user interface that expose prompt editing.

Here’s an illustration of what an integrated prompt environment might look like for a real estate AI assistant:

The UI that users (real estate agents) see.

The same UI, but with an “admin mode”used by the engineering & product team to iterate on the prompt and debug issues.

Tips For Communicating With Domain Experts

There’s another barrier that often prevents domain experts from contributing effectively: unnecessary jargon. I was working with an education startup where engineers, product managers, and learning specialists were talking past each other in meetings. The engineers kept saying, “We’re going to build an agent that does XYZ,” when really the job to be done was writing a prompt. This created an artificial barrier – the learning specialists, who were the actual domain experts, felt like they couldn’t contribute because they didn’t understand “agents.”

This happens everywhere. I’ve seen it with lawyers at legal tech companies, psychologists at mental health startups, and doctors at healthcare firms. The magic of LLMs is that they make AI accessible through natural language, but we often destroy that advantage by wrapping everything in technical terminology.

Here’s a simple example of how to translate common AI jargon:

Instead of saying…

Say…

“We’re implementing a RAG approach”

“We’re making sure the model has the right context to answer questions”

“We need to prevent prompt injection”

“We need to make sure users can’t trick the AI into ignoring our rules”

“Our model suffers from hallucination issues”

“Sometimes the AI makes things up, so we need to check its answers”

This doesn’t mean dumbing things down – it means being precise about what you’re actually doing. When you say “we’re building an agent,” what specific capability are you adding? Is it function calling? Tool use? Or just a better prompt? Being specific helps everyone understand what’s actually happening.

There’s nuance here. Technical terminology exists for a reason – it provides precision when talking with other technical stakeholders. The key is adapting your language to your audience.

The challenge many teams raise at this point is: “This all sounds great, but what if we don’t have any data yet? How can we look at examples or iterate on prompts when we’re just starting out?” That’s what we’ll talk about next.

1. Bootstrapping Your AI With Synthetic Data Is Effective (Even With Zero Users)

One of the most common roadblocks I hear from teams is: “We can’t do proper evaluation because we don’t have enough real user data yet.” This creates a chicken-and-egg problem – you need data to improve your AI, but you need a decent AI to get users who generate that data.

Fortunately, there’s a solution that works surprisingly well: synthetic data. LLMs can generate realistic test cases that cover the range of scenarios your AI will encounter.

As I wrote in my LLM-as-a-Judge blog post, synthetic data can be remarkably effective for evaluation. Bryan Bischof, the former Head of AI at Hex, put it perfectly:

“LLMs are surprisingly good at generating excellent - and diverse - examples of user prompts. This can be relevant for powering application features, and sneakily, for building Evals. If this sounds a bit like the Large Language Snake is eating its tail, I was just as surprised as you! All I can say is: it works, ship it.”

A Framework for Generating Realistic Test Data

The key to effective synthetic data is choosing the right dimensions to test. While these dimensions will vary based on your specific needs, I find it helpful to think about three broad categories:

Features: What capabilities does your AI need to support?

Scenarios: What situations will it encounter?

User Personas: Who will be using it and how?

These aren’t the only dimensions you might care about – you might also want to test different tones of voice, levels of technical sophistication, or even different locales and languages. The important thing is identifying dimensions that matter for your specific use case.

For a real estate CRM AI assistant I worked on with Rechat, we defined these dimensions like this:

features = [

"property search", # Finding listings matching criteria

"market analysis", # Analyzing trends and pricing

"scheduling", # Setting up property viewings

"follow-up" # Post-viewing communication

]

scenarios = [

"exact match", # One perfect listing match

"multiple matches", # Need to help user narrow down

"no matches", # Need to suggest alternatives

"invalid criteria" # Help user correct search terms

]

personas = [

"first_time_buyer", # Needs more guidance and explanation

"investor", # Focused on numbers and ROI

"luxury_client", # Expects white-glove service

"relocating_family" # Has specific neighborhood/school needs

]

But having these dimensions defined is only half the battle. The real challenge is ensuring your synthetic data actually triggers the scenarios you want to test. This requires two things:

A test database with enough variety to support your scenarios

A way to verify that generated queries actually trigger intended scenarios

For Rechat, we maintained a test database of listings that we knew would trigger different edge cases. Some teams prefer to use an anonymized copy of production data, but either way, you need to ensure your test data has enough variety to exercise the scenarios you care about.

Here’s an example of how we might use these dimensions with real data to generate test cases for the property search feature (this is just pseudo-code, and very illustrative):

def generate_search_query(scenario, persona, listing_db):

"""Generate a realistic user query about listings"""

# Pull real listing data to ground the generation

sample_listings = listing_db.get_sample_listings(

price_range=persona.price_range,

location=persona.preferred_areas

)

# Verify we have listings that will trigger our scenario

if scenario == "multiple_matches" and len(sample_listings) < 2:

raise ValueError("Need multiple listings for this scenario")

if scenario == "no_matches" and len(sample_listings) > 0:

raise ValueError("Found matches when testing no-match scenario")

prompt = f"""

You are an expert real estate agent who is searching for listings. You are given a customer type and a scenario.

Your job is to generate a natural language query you would use to search these listings.

Context:

• Customer type: {persona.description}
• Scenario: {scenario}

Use these actual listings as reference:

{format_listings(sample_listings)}

The query should reflect the customer type and the scenario.

Example query: Find homes in the 75019 zip code, 3 bedrooms, 2 bathrooms, price range $750k - $1M for an investor.

"""

return generate_with_llm(prompt)

This produced realistic queries like:

Feature

Scenario

Persona

Generated Query

property search

multiple matches

first_time_buyer

“Looking for 3-bedroom homes under $500k in the Riverside area. Would love something close to parks since we have young kids.”

market analysis

no matches

investor

“Need comps for 123 Oak St. Specifically interested in rental yield comparison with similar properties in a 2-mile radius.”

The key to useful synthetic data is grounding it in real system constraints. For the real-estate AI assistant, this means:

Using real listing IDs and addresses from their database

Incorporating actual agent schedules and availability windows

Respecting business rules like showing restrictions and notice periods

Including market-specific details like HOA requirements or local regulations

We then feed these test cases through Lucy and log the interactions. This gives us a rich dataset to analyze, showing exactly how the AI handles different situations with real system constraints. This approach helped us fix issues before they affected real users.

Sometimes you don’t have access to a production database, especially for new products. In these cases, use LLMs to generate both test queries and the underlying test data. For a real estate AI assistant, this might mean creating synthetic property listings with realistic attributes – prices that match market ranges, valid addresses with real street names, and amenities appropriate for each property type. The key is grounding synthetic data in real-world constraints to make it useful for testing. The specifics of generating robust synthetic databases are beyond the scope of this post.

Guidelines for Using Synthetic Data

When generating synthetic data, follow these key principles to ensure it’s effective:

Diversify your dataset: Create examples that cover a wide range of features, scenarios, and personas. As I wrote in my LLM-as-a-Judge post, this diversity helps you identify edge cases and failure modes you might not anticipate otherwise.

Generate user inputs, not outputs: Use LLMs to generate realistic user queries or inputs, not the expected AI responses. This prevents your synthetic data from inheriting the biases or limitations of the generating model.

Inc