DataOpsのループを閉じる：なぜ私たちがDagster+向けにCompassを構築したのか

Meet Compass — Dagster’s new AI data analyst for Slack. Turn questions into trusted insights, instantly. Try Compass now →Discover What assets do best, an animated, narrated story about how data assets work together. Watch now →

Try Dagster+Sign InDetection isn't the bottleneck anymore. Understanding is. Compass closes the loop by turning Dagster+ operational data into a conversation.

Data teams have more visibility into their pipelines than ever before. Modern orchestration and observability tools capture every run, track every asset, and surface alerts when something breaks. The promise was that all this instrumentation would finally make data platforms legible, no longer black boxes.

We have been building Dagster to help solve the “big complexity” problem with data engineering. But as platforms became more complex, the abstractions we built to address the last wave of complexity needed more support to handle the next.

The central question that a good observability and operational tool should answer is “What is even happening?” and the great ones can answer “What should I do next?”

That gap is what we set out to close with Compass.

Request access to Dagster+ Compass.

In operational analytics, there's a concept called closed-loop analytics: a cycle that captures signals, analyzes patterns, acts on findings, and measures results. The loop only works if each stage flows into the next.

We think about DataOps the same way: as a loop with three parts: Orchestrate → Observe → Act.

Dagster+ handles orchestration. You define your assets, set up schedules and sensors, and Dagster makes sure your pipelines run when and how they should.

Dagster+ also handles observation. Every run is tracked. Every asset has lineage. You can see what succeeded, what failed, what's stale, and what depends on what. The metadata is rich, and it's all there.

But the third part, acting on what you observe, has historically been left as an exercise for the reader. You have the signals. Now go figure out what they mean, decide which ones matter, and determine what to do next.

This is where the observability industry has focused on Mean Time to Detection and made real progress. But detection isn't resolution. Teams still struggle with Mean Time to Resolution because turning an alert into understanding requires manual investigation: opening the right views, filtering to the right time window, cross-referencing across jobs, and pattern-matching in your head.

Compass closes this loop. It turns Dagster+ operational data into a conversation, so you can go from "something seems wrong" to "here's what's happening and what to do next" without leaving Slack.

What the Dagster+ Compass Integration Actually Does

Compass is an AI agent that now understands your Dagster+ environment. You can ask it questions in natural language, and it responds with answers grounded in your actual runs, assets, and metadata.

Compass is designed to guide investigation, not just answer one-off questions. It suggests where to drill down next. It surfaces patterns you might not have thought to look for. It provides links to the exact locations in Dagster+ where you can take action.

A typical investigation might start broadly: "Give me a summary of my data platform's health." Compass responds with a breakdown of successful vs. failed runs across your common jobs, success rates by code location, and top and bottom performers over the last seven days.

From there, you can pull on any thread. Drill into failure time distribution, and Compass shows you rates by day and hour, highlighting spikes worth investigating. Ask what failed on a specific date, and it identifies the culprits, maybe one job that failed repeatedly, plus a few scattered others.

At each step, Compass suggests next moves: compare across days, examine dependencies, drill into specific runs. The conversation builds on itself, guiding you toward root causes rather than leaving you to figure out what to click next.

Our own data team has been using Compass to monitor the Dagster platform at a deeper level. Nick Roach, one of our data engineers, put it well: "I want a one-stop shop when I'm investigating what might be going wrong. Being able to interact conversationally—and have it suggest where to go next—is really exciting."

From Firefighting to Pattern-Driven Improvement

The obvious use case for Compass is incident triage. Something breaks, you ask Compass what happened, and you get an answer faster than you would have otherwise. That alone reduces time to resolution.

But the more interesting use case is what happens when you're not firefighting.

One of the hardest problems in operational analytics is what Monte Carlo calls "unknown unknowns" issues that don't trigger alerts because you didn't know to test for them. You can have comprehensive monitoring covering anticipated failure modes, but still miss the slow degradations, the cree