Amazon SageMaker Unified StudioとSageMaker Catalogを使用したオフライン機能ストアの構築

Building and managing machine learning (ML) features at scale is one of the most critical and complex challenges in modern data science workflows. Organizations often struggle with fragmented feature pipelines, inconsistent data definitions, and redundant engineering efforts across teams. Without a centralized system for storing and reusing features, models risk being trained on outdated or mismatched data, leading to poor generalization, lower model accuracy and governance issues. Furthermore, enabling collaboration across data engineering, data science, and ML operations teams becomes difficult when each group maintains its own isolated datasets and transformations.

Amazon SageMaker addresses these challenges through SageMaker Unified Studio and SageMaker Catalog, which organizations can use to build, manage, and share assets securely across projects and accounts. A key capability within this ecosystem is the implementation of an offline feature store—a structured repository designed for managing historical feature data used in model training and validation. Offline feature stores are designed for scalability, lineage tracking, and reproducibility, so that data scientists can train models on accurate, time-aligned datasets that prevent data leakage and maintain consistency across experiments.

This blog post provides step-by-step guidance on implementing an offline feature store using SageMaker Catalog within a SageMaker Unified Studio domain. By adopting a publish-subscribe pattern, data producers can use this solution to publish curated, versioned feature tables—while data consumers can securely discover, subscribe to, and reuse them for model development. The approach integrates Amazon S3 Tables with Apache Iceberg for transactional consistency, AWS Lake Formation for fine-grained access control, and Amazon SageMaker Studio for visual and code-based data engineering.

Through this unified solution, teams can achieve consistent feature governance, accelerate ML experimentation, and reduce operational overhead. in this post, we show you how to design a collaborative, governed, and production-ready offline feature store to unlock enterprise-wide reuse of trusted ML features.

Solution overview

This solution demonstrates how to implement an offline feature store using a SageMaker Unified Studio domain integrated with SageMaker Catalog to enable scalable, governed, and collaborative feature management across ML teams. The architecture establishes a unified environment, shown in the following figure, that streamlines how administrators, data engineers, and data scientists create, publish, and consume high-quality, reusable feature tables.

At its core, the solution uses a SageMaker Unified Studio domain as the governance and collaboration layer for managing projects, users, and data assets under centralized control. S3 Tables in the Apache Iceberg format serve as the foundation for storing and versioning feature data. SageMaker Catalog, which allows unified governance for datasets, acts as the central registry for publishing, discovering, and subscribing to feature tables.

The following describes how various personas interact in the end-to-end workflow:

• Admin uses AWS CloudFormation templates and the AWS Management Console to set up and configure the environment, including provisioning the SageMaker Unified Studio domain and onboarding users and groups.

• Admin creates the SageMaker Unified Studio data project and onboards Amazon Simple Storage Service (Amazon S3) datasets—such as airline_delay.csv and airline_features (an S3 table)—to the project catalog, then designates the data engineer as the project owner.

• Data engineer opens the data project, uses the Visual extract, transform, load (ETL) tool or a data processing job to build the feature pipeline, and creates features in the airline_features table within the project catalog.

• Data engineer accesses the data explorer tool to enrich the airline_features table with metadata for improved discoverability and governance.

• After validation and approval, the data engineer publishes the airline_features table into SageMaker Catalog for organization-wide access.

• Data scientist opens the ML project, uses AI powered search to find the airlines_features table in the catalog, and identifies the published feature table as suitable for their model development.

• Data scientist submits a subscription request to consume the published feature table. If auto-approval isn’t configured, the publisher must review and approve the access request.

• After approval, the data scientist gains access to the feature table through the project catalog, both through the data explorer and directly from Jupyter notebooks for model training and experimentation.

This structured workflow provides consistent data governance, promotes collaboration, and eliminates redundant feature engineering efforts by enabling enterprise-wide reuse of trusted, versioned ML features.

Core components

The offline feature store solution architecture is composed of several integrated components. Each component plays a distinct role in enabling secure data governance, scalable feature engineering, and seamless collaboration across ML personas. The key components include:

SageMaker Unified Studio domain: The SageMaker Unified Studio domain serves as the central control plane for managing ML projects, users, and data assets. It provides a unified interface for collaboration between data engineers, data scientists, and administrators. This domain enables enforcement of fine-grained access controls, integrates with AWS IAM Identity Center for single sign-on, and supports approval workflows to help ensure secure sharing of ML assets across teams and accounts.

S3 Tables with Apache Iceberg format: S3 Tables enables scalable, serverless storage for feature data using the Apache Iceberg table format. Apache Iceberg Open Table Format (OTF) enables ACID transactions, schema evolution, and time-travel capabilities, which teams can then take advantage of to query historical versions of feature data with full reproducibility. S3 Tables seamlessly integrate with Spark, Glue, and SageMaker for consistent data access across analytical and ML workloads.

Feature engineering pipeline: The feature engineering pipeline automates the transformation of raw datasets into curated, high-quality features. Built on Apache Spark, it provides distributed data processing at scale, enabling complex transformations such as delay-rate computation, categorical encoding, and feature aggregation. The pipeline directly writes outputs to S3 tables, helping to ensure traceability and consistency between raw/processed data and engineered features.

SageMaker Catalog: SageMaker Catalog acts as the organization-wide repository for registering, publishing, and discovering ML assets such as datasets, feature tables, and models. It integrates with Lake Formation for fine-grained access control and IAM Identity Center for user management. The catalog supports metadata enrichment, versioning, and approval workflows, so teams can securely share and reuse trusted assets across projects.

Together, these components create a cohesive ecosystem that simplifies ML feature lifecycle management—from creation and publication to discovery and consumption—while maintaining enterprise-grade governance and data lineage tracking.

Administrator workflow

The administrator workflow defines the initial setup required to establish a secure and collaborative environment for implementing the offline feature store. Administrators provision the SageMaker Unified Studio domain, enable IAM Identity Center for user authentication, and configure S3 tables with Lake Formation for governed data access. They also create dedicated producer and consumer projects, deploy the necessary infrastructure through environment blueprints (based on CloudFormation), and assign users and groups with appropriate permissions. This setup helps ensure a consistent, well-governed foundation that data engineers and data scientists can use to build, publish, and consume ML features seamlessly within SageMaker Unified Studio.

Prerequisites

The following prerequisites must be completed to help ensure that the required AWS services and permissions are properly set up for seamless integration and governance.

• Enable IAM Identity Center

Navigate to IAM Identity Center in the console.

• Choose Enable if not already activated.

• Choose Enable with AWS Organizations or standalone and complete the setup wizard.

• Enable S3 Tables

Go to Amazon S3 in the console.

• Select the S3 Tables section and choose Enable S3 Tables integration.

• Lake Formation administrator

Verify that the console role is added to Lake Formation administrators for proper resource creation and permissions management.

Set up the environment

After completing the prerequisites, the next step is to set up the environment by creating and configuring the SageMaker Unified Studio domain, which serves as the central workspace for managing users, projects, and data assets within the offline feature store architecture.

Set up a SageMaker Unified Studio domain:

• Navigate to the SageMaker Unified Studio console and choose Domains in the navigation pane. Choose Create domain and then choose Quick Setup.

• Choose to create a new VPC or select an existing VPC for the domain network configuration.

• Expand Quick setup settings and enter a domain name (for example, Corporate).

• For Domain Execution role and Domain Service role, select Create and use a new service role.

• Select a VPC (VPCs tagged with SageMaker Unified Studio should be correctly configured) and choose at least three private subnets, each in a different Availability Zone.

• Choose Continue to proceed with domain creation.

• Create an IAM Identity Center user account by providing an email address, first name, and last name (you will receive an email to activate your IAM Identity Center account after creation).

• Choose Create domain to start the domain creation process. Domain creation typically takes 2–5 minutes to complete.

Create a SageMaker Unified Studio project:

• Go to the SageMaker Unified Studio domain URL and sign in using the IAM Identity Center credentials created during domain setup and accept the invitation email if received.

• Create projects:

Go to Select Project and choose Create project from the domain homepage to create a producer project and name it airlines_core_features. Add descriptions, configure settings, and assign users and permissions.

• For Project profile, select All Capabilities.

• Repeat the two preceding steps to create a consumer project names airlines_ml_models. Note :- Project creation typically takes 5–10 minutes to complete.

• Copy the project role ARN

Navigate to airlines_core_features project details.

• Navigate to the project and copy the project role Amazon Resource Name (ARN) (arn:aws:iam::ACCOUNT:role/datazone_usr_role_*)

• Repeat the preceding two steps for airlines_ml_models

Infrastructure deployment

Deploy the CloudFormation stack only after completing the previous steps, including domain creation, project creation, IAM Identity Center setup, and S3 Tables enablement. The stack requires the following input parameters:

• IAM Identity Center ID: Navigate to the IAM Identity console to retrieve the first part of the AWS access portal URL. It will look like https://d-1234da5678.awsapps.com/start

• SMUSProducerProjectRoleName: Navigate to the Unified Studio project portal, open your producer project, and retrieve the project role name. For example, arn:aws:iam:::role/datazone_usr_role_xxxx_yyyy

• SMUSConsumerProjectRoleName: Repeat the preceding step to retrieve consumer project role name.

After the stack deploys successfully, the following AWS resources will be created:

• S3 bucket: For raw and curated data storage, such as amzn-s3-demo-blog-smus-featurestore-{account-id}

• S3 table bucket: For Iceberg table storage, such as amzn-s3-demo-airlines-s3tables-bucket

• S3 table namespace: Organized feature table namespace, such as airlines

• S3 table: for feature storage, such as fg_airline_features

• Glue Database: for source data catalog, such as airline_raw_db

• IAM Identity Center groups: Producer and consumer groups, such as FeatureStore-Producers, FeatureStore-Consumers

• Lake Formation permissions: Fine-grained access control, such as database and table permissions

• IAM roles and policies: For service integrations, such as – SageMaker execution roles.

###

Add SSO groups to the SageMaker domain

Assign IAM Identity Center groups for single sign on (SSO) to the SageMaker domain to enable user access and collaboration across projects.

• Navigate to the SageMaker Unified Studio console and choose domain from the navigation pane.

• Goto User management.

• Choose Add users and groups to manage domain access.

• Select SSO Groups.

• Choose the SSO groups that were created during infrastructure deployment:

feature-store-admin-group (for admin)

• feature-store-producer-group (for data engineering teams)

• feature-store-consumer-group (for data science teams)

• Choose Add to complete the assignment and enable group access

• The User Management section will show the SSO groups as Assigned.

###

Create IAM Identity Center users

Create individual users for data producers, consumers, and administrators to access the SageMaker domain.

Create the data producer user:

• Navigate to the IAM Identity Center console and choose Users from the navigation pane.

• Choose Add user enter the user details:

Username: Enter dataproducer.

• Email: Enter a valid address, such as dataproducer@example.com.

• First name: Enter Data.

• Last name: Enter Producer.

• Assign the user to feature-store-producer-group.

• Send an invitation email for account activation.

Create the data consumer and admin users:

Use the same steps to create the data consumer and admin users, with the following differences for step 2:

• Create the data consumer user:

Username: Enter dataconsumer.

• Email: Enter a valid address, such as dataconsumer@example.com.

• First name: Enter Data.

• Last name: Enter Consumer.

• Assign the user to feature-store-consumer-group.

• Create the admin user:

Username: Enter dataadmin.

• Email: Enter a valid address, such as dataadmin@example.com.

• First name: Enter Data.

• Last name: Enter Admin.

• Assign the user to feature-store-admin-group.

Add user groups to projects

Sign in to the SageMaker Unified Studio corporate domain as the admin user from UI Console and assign the user groups to their corresponding projects for proper access control.

Assign users to the Producer project:

Na