Amazon SageMaker AI エンドポイントの推論メタモニタリング

Without the ability to track machine learning (ML) model prediction quality, organizations only realize they have issues when their customers complain or when they conduct spot checks, which jeopardizes customer trust. This post introduces inference meta-monitoring for Amazon SageMaker AI endpoints. It provides a governance layer that sits above production ML inference pipelines to continuously track prediction and data quality metrics and visualize trends. You will learn how to design and implement an inference meta-monitoring system specifically for predictive models, using Amazon Quick and optionally with Amazon SageMaker AI MLflow App. The system includes drift detection, delayed ground truth data integration, and automated performance dashboards.

The inference monitoring gap

Developing predictive ML models is often a resource-intensive process. You invest months building training pipelines to drive strong validation accuracy for use cases such as fraud detection, credit scoring or demand forecasting. The performance of the models you deploy might silently degrade, and your teams might notice it only weeks later. Fraud case handlers start to see false positives spiking, and loan officers start to see more applications that should have been previously flagged. Corporate resource planners find themselves with excess inventory because of overestimated demand forecasts.

ML teams therefore need a system that provides them with continuous feedback on the performance of their models in production. The system should immediately alert them whenever model quality or data drift is detected. Teams can then take action early so that model performance remains consistent over time, and customer trust remains high.

Solution overview

This inference meta-monitoring solution combines AWS managed services (Amazon SageMaker AI, Amazon Athena, AWS Lambda, Amazon EventBridge, Amazon Quick) with open source ML tools (SageMaker AI MLflow Apps, Evidently AI).

Prerequisites

• AWS account.

• Use the CloudFormation template provided in the repo to create the virtual private cloud (VPC), subnets, SageMaker AI domain, user profile, and JupyterLab space. The template also clones the git repo and updates the .env with the actual values.

Setup

The CloudFormation template automates the entire setup. If you intend to use your existing domains, you can clone the repository, update the env values and run the notebooks in sequence.

Environment setup

Running the CloudFormation setup creates and initializes the variables.

If you intend to use your own SageMaker AI domain, copy over the .env.example to .env and update the file with your specific environment variables. The settings in .env override all other defaults and settings in config.yaml.

git clone --branch v2.0.0 https://github.com/aws-samples/sample-mlops-bestpractices.git
cd sagemaker-automated-drift-and-trend-monitoring
cp .env.example .env

Architecture

The system implements integrated training, inference, and monitoring pipelines, unified through a central Athena Iceberg table, Amazon Quick and SageMaker AI MLflow Apps.

Figure 1: End-to-end MLOps architecture built on AWS managed services

Training (1_training_pipeline.ipynb)

The training pipeline includes experiment tracking in MLflow. As shown in Figure 2, it sets up the end-to-end model training.

1. Download Kaggle dataset: Credit card fraud dataset is downloaded from Kaggle.

1. Data ingestion: S3 data is loaded into the training data. The architecture uses five Athena Iceberg tables to form the central data lake. The training_data table stores the 80% slice the model is fit on. The evaluation_data table is the frozen 20% held-out slice the model is scored on. evaluation_data is the drift-monitoring baseline, not training_data, because the held-out slice is what every registered model’s metrics are measured against. Both tables are populated from the same predictions CSV by a single seed step that applies a deterministic hash split on transaction_id, so the row partition is stable across pipeline runs.

The SeedAthenaTrainingData step (see Figure 2) idempotently loads the predictions CSV from Amazon S3 into the training_data (80%) and evaluation_data (20%) Iceberg tables using a deterministic hash split on transaction_id. Re-runs produce identical row partitions, so the evaluation slice is stable across model versions and usable as a frozen drift baseline. Alternatively, you can replace the training data with your own. For further details, refer to Bring your own dataset.

1. Training: The training and evaluation steps run and log metrics and artifacts to a SageMaker AI MLflow App.

Figure 2: Training pipeline execution

Model deployment (2_deployment.ipynb)

4. Deployment: A custom handler is deployed to a SageMaker AI inference endpoint.

The inference handler writes all the inferences to Amazon Simple Queue Service (Amazon SQS), which are then processed by an inference-logger Lambda function. The inference-logger Lambda function batches up to 10 predictions, or the number of predictions that arrive within 30 seconds, whichever comes first, and writes into Athena Iceberg tables. All the notebooks also include convenience scripts for role creation and infrastructure creation.

Inference monitoring (3_inference_monitoring.ipynb)

5. Ground truth simulation: Step 3.3 in 3_inference_monitoring.ipynb invokes the deployed model with drifted input features. This infuses data drift. In real environments, this would be your typical business applications that send inference requests with drifted inputs over time. The ground truth simulation is step 4 in the notebook.

Steps 3 and 4 of the solution are meant to be replaced by your business processes, which perform two steps before the data and model drift computations:

a. Step 3: Your applications invoke the endpoint for inference. This triggers the inference endpoint to push the inference request into SQS -> Lambda -> Athena (inference_responses).

b. Step 4: You generate ground truth data. This is the process of concluding if a prediction was right or wrong. In the example use case, it is determining if an inference request was a fraud or non-fraud. In this solution, the ground truth simulation randomly flips the actual field to be fraud/non-fraud, to infuse 15% inaccuracy in model performance. This induces model drift. In real-world scenarios, the actual fraud/non-fraud designation would be made when your fraud team investigates and confirms labels based on auto-approved transactions, customer-disputed chargebacks or a bank confirming fraud/non-fraud cases.

NOTE: Steps 2, 3, and 4 in notebook 3 (generate drifted data, send predictions, apply ground truth) are not scheduled operations. They exist only for dev/test purposes. In production you would replace these with your own business processes: applications hitting the endpoint for inference (steps 2-3) and fraud-investigation feeds writing to ground_truth_updates (step 4). To try this solution end-to-end, trigger these three steps via the notebook cells or the CLI scripts provided in the repo.

With steps 2 and 3, inference requests with data drift are already in inference_responses. Step 4 writes records into the ground_truth_updates table. Records with ground_truth = NULL are picked and new records are created in the ground_truth_updates table with actual labels. With this we can expect model drift upon computation.

At any point you can validate the records through a query editor of your choice:

Figure 3: Query editor showing the generated records

Figure 4: Records in the inference_responses table with ground truth updates and the ground_truth_updates table

Records in the inference_responses table with ground_truth=NULL correspond to the entries for which ground truth hasn’t been applied.

6. Merge ground truth with inference response: Predictions and inference requests captured in the inference_responses table can be joined with ground_truth_updates on inference_id. The inference_responses table captures every prediction along with its features, confidence scores, and timestamps. The ground_truth_updates table holds asynchronous label confirmations that are merged back into inference records through step 4 in the notebook.

Now that we have data to compute data and model drift, we can identify the baseline and current datasets for both categories of computations:

Metric

Component

Source

ID

Configuration

Example

Data Drift

Baseline

training_data

Snapshot_id in iceberg_table also recorded in baseline.json

Automatic during preprocessing step

If training data had transaction_amount with mean=$50, but recent predictions

show mean=$500, KS test would flag this distribution shift.

Current

Inference_responses for the lookback window

records where request_timestamp >= rolling time window

Drift lambda env var:

DATA_DRIFT_LOOKBACK_DAYS: 1 # Query last 1 days of inferences

DATA_DRIFT_THRESHOLD: 0.2 # Alert if ≥20% features drifted

Model drift

Baseline

Evaluation_data

Snapshot_id in iceberg_table and also recorded in baseline.json

Automatic during preprocessing step

Lambda queries predictions from last 30 days that have ground truth labels,

calculates

Lambda queries predictions from last 30 days that have ground truth labels, calculates current ROC-AUC. If baseline was 0.92 and current is 0.85, ROC-AUC degradation = 0.07 (exceeds 0.05 threshold) → Alert!

0.05 threshold) → Alert!

Current

Inference_responses with

records where request_timestamp >= rolling time window

Drift lambda env var:

MODEL_DRIFT_LOOKBACK_DAYS: 1 – set to 1 day in deploy_lambda_container.sh because the Lambda is on a daily schedule. Change this value based on how quickly ground truth arrives in your business processes. The config.yaml default is 30 days (for the notebook-driven flow, where a longer window is appropriate because ground truth is often delayed).

Note: If you intend to have a shorter window for drift computations, follow the steps detailed in Configuring Shorter Drift Windows (Minutes instead of days)

The monitoring_responses table stores outputs from the drift computations, including the ModelPackage ARN and Iceberg snapshot ID the run was computed against, so trends can be sliced per model version.

Every registered model carries a frozen baseline.json artifact that records the metrics it earned on evaluation_data, the Iceberg snapshot ID of that exact data slice, and the code commit SHA that produced it. The drift Lambda dereferences these pointers on every run, so the monitoring system always compares production against the precise version of the data and code the deployed model came from, ensuring there is never a stale or wrong-version reference. The lineage mechanism is described in detail in the deep dive section that follows.

7. Drift computation: The drift Lambda function resolves the model currently served by the endpoint and loads its registered baseline.json. Two independent checks run on each invocation, each against its own frozen baseline. For data drift, the frozen training_data slice (pinned through training_snapshot_id in baseline.json) is the reference distribution the model was trained on. Recent inference_responses.input_features from the rolling window is the current distribution. Evidently’s DataDriftPreset compares the two, auto-selecting a statistical test per column. It uses the Kolmogorov-Smirnov or Chi-square (p-values) for smaller samples, and Wasserstein or Jensen-Shannon distances for n ≥ 1000. For model drift, the frozen evaluation_data slice (pinned through evaluation_snapshot_id) supplies the baseline (target, prediction) pairs the model was scored on at training time. Current inference rows joined with ground_truth_updates are compared to that baseline via Evidently’s ClassificationPreset, which tracks ROC-AUC, precision, recall, and F1 degradation. Both baselines are immutable per model version, so re-seeding the source tables later cannot retroactively change historical drift comparisons. Because Evidently emits raw scores whose drift direction depends on the chosen test (p-values: lower = drift. Distances: higher = drift), the solution also computes a normalized companion field, drift_magnitude, so features can be compared and ranked test-agnostically. Follow the drift story for deeper insight into the metrics.

NOTE: Two additional tables are created but not currently used. The ground_truth table holds confirmed labels (fraud/not fraud) and is reserved as a placeholder to update the baseline data for cases where the model has been retrained and you would like to add additional data to the baseline. The drifted_data table is created as a placeholder to contain drifted datasets if needed.

8. Drift logging: The drift monitor Lambda function uses Evidently to generate interactive charts which get logged to SageMaker AI MLflow App. You can also choose to use only Amazon Quick for all your analytics. This solution provides both MLflow with Evidently and Amazon Quick support to cater to both data scientists and governance personas.

9. Monitoring automation: The drift monitor Lambda function checks drift thresholds and triggers alarms if the thresholds are exceeded.

10 – 11. The drift monitor Lambda function also pushes the computations to Amazon SQS and the Lambda writer reads the results and writes the output to the monitoring_responses table. Since the solution uses Apache Iceberg based Athena tables, updates are ACID-compliant.

Meta-monitoring (