スキルの評価

Meet Compass — Dagster’s new AI data analyst for Slack. Turn questions into trusted insights, instantly. Try Compass now →Discover What assets do best, an animated, narrated story about how data assets work together. Watch now →

Try Dagster+Sign InWe built a light weight evaluation framework to quantiatively measure the effectiveness of the Dagster Skills, and these are our findings.

So you've written a skill. How do you know if the agent is actually using it? Is the correct context being referenced when you ask a question? Is it doing anything at all…?

We recently published our own collection of skills for working with the Dagster framework and the vast library of integrations that we support, and we had a similar question. We wanted to better understand what was going on under-the-hood when we asked Claude, "Put my dbt models on a schedule to run every 15 minutes". So we set out to create a light-weight evaluation framework to ensure users of our skills get the result they need.

Before we dig in to how we evaluated our skills, let's talk about the skills that we provide.

First, we have dagster-expert which is the cornerstone of context for ensuring your project adheres to the patterns and architectural decisions for working with the Dagster framework. This includes information like whether or not an asset should be partitioned, which automation strategy is best to use (declarative automation, schedules, sensors), the patterns for defining a resource, design patterns in data engineering, and how to organize your project(s). Additionally, this skill makes heavy use of the dg command-line utility which is how we provide deterministic actions to the agent. This means you can do things like scaffold a new project, launch assets to be materialized, view logs, and troubleshoot failures in both your local and deployed instances of Dagster.

Next up, we have the dagster-integrations skill, which has the largest surface area by far. These skills are broken down by category: AI & ML, ETL tools, storage options, validation frameworks, etc. As an example, if you are trying to solve a specific data engineering problem, this skill will be able to provide you with suggested tools, how to initialize the integration, and how you would configure it within you Dagster project. It's worth pointing out that an additional complexity of creating a skill for integrations is that many of these tools have their own context, or documentation, that is required to ensure the tool is being used correctly. For that reason, we are trying to encode as many of the “quirks” and “gotchas” that an engineer might run into while working with these tools.

And finally, we have dignified-python. While not specifically designed for data engineering, this skills was created to promote what we consider to be production-ready Python. It is the skill we use internally while building with Python, and it includes many of the important concepts we promote, such as: modern type annotations, exception handling patterns, API design principles, and more.

So now that you have an idea of what we're trying to validate, let's talk about how we've built a light weight evaluation framework for ensuring that our skills are providing the correct context, being invoked when we would expect, and are most importantly, producing correct outputs. Additionally, by having a light weight testing framework, it has enabled us to get quick and repeatable feedback on workflows with context on what does and doesn’t work. This allowed us to iterate quickly and drastically improve the Dagster skills

All of the source code discussed in this post can be found in the dagster-io/skills repository in the dagster-skills-evals directory.

The basic architecture of this framework is a simple utility that allows us to run Claude headlessly with prompts covering common use-cases, and a selection of plugins (skills) to use. From that we are able to create snapshots of tool and skill usage, token utilization, and of course the output that is produced. These snapshots are persisted in version control, which allows us to then monitor how they change over time as we modify the underlying skills.

Here is an example snapshot that is produced from a prompt requested to create an asset in an existing Dagster project. As expected, we can see that the dagster-expert skill was referenced, and we can see a narrative summary of what occurred during this request, and the tools that were used. We can see that thedg scaffold defs command was utilized, and that validation occurred by using the dg list defs command as we would expect!

{ "__class__": "ClaudeExecutionResultSummary", "execution_time_ms": 59191, "input_tokens": 56, "narrative_summary": [ "- User requested to add a new Dagster asset named 'dwh_asset' with an empty body", "- Invoked the dagster-skills:dagster-expert skill to understand project structure and how to add new assets", "- Discovered t