Hugging Face Hubにストレージバケット機能を導入

Back to Articles Introducing Storage Buckets on the Hugging Face Hub

Upvote 53

Hugging Face Models and Datasets repos are great for publishing final artifacts. But production ML generates a constant stream of intermediate files (checkpoints, optimizer states, processed shards, logs, traces, etc.) that change often, arrive from many jobs at once, and rarely need version control.

Storage Buckets are built exactly for this: mutable, S3-like object storage you can browse on the Hub, script from Python, or manage with the hf

Why we built Buckets

Git starts to feel like the wrong abstraction pretty quickly when you're dealing with:

Training clusters writing checkpoints and optimizer states throughout a run

Data pipelines processing raw datasets iteratively

Agents storing traces, memory, and shared knowledge graphs

The storage need in all these cases is the same: write fast, overwrite when needed, sync directories, remove stale files, and keep things moving.

A Bucket is a non-versioned storage container on the Hub. It lives under a user or organization namespace, has standard Hugging Face permissions, can be private or public, has a page you can open in your browser, and can be addressed programmatically with a handle like hf://buckets/username/my-training-bucket

Why Xet matters

Buckets are built on Xet, Hugging Face’s chunk-based storage backend, and this matters more than it might seem.

Instead of treating files as monolithic blobs, Xet breaks content into chunks and deduplicates across them. Upload a processed dataset that’s mostly similar to the raw one? Many chunks already exist. Store successive checkpoints where large parts of the model are frozen? Same story. Buckets skip the bytes that are already there, which means less bandwidth, faster transfers, and more efficient storage.

This is a natural fit for ML workloads. Training pipelines constantly produce families of related artifacts — raw and processed data, successive checkpoints, Agent traces and derived summaries — and Xet is designed to take advantage of that overlap.

For Enterprise customers, billing is based on deduplicated storage, so shared chunks directly reduce the billed footprint. Deduplication helps with both speed and cost.

Pre-warming: bringing data close to compute

Buckets live on the Hub, which means global storage by default. But not every workload can afford to pull data from wherever it happens to live, for distributed training and large-scale pipelines, storage location directly affects throughput.

Pre-warming lets you bring hot data closer to the cloud provider and region where your compute runs. Instead of data traveling across regions on every read, you declare where you need it and Buckets make sure it's already there when your jobs start. This is especially useful for training clusters that need fast access to large datasets or checkpoints, and for multi-region setups where different parts of a pipeline run in different clouds.

We are partnering with AWS and GCP to start with, more more cloud providers coming in the future.

Getting started

You can get a bucket up and running in under 2 minutes with the hf

curl -LsSf https://hf.co/cli/install.sh | bash hf auth login

Create a bucket for your project:

hf buckets create my-training-bucket --private

Say your training job is writing checkpoints locally to ./checkpoints

hf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints

For large transfers, you might want to see what will happen before anything moves. --dry-run

hf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints --dry-run

You can also save the plan to a file for review and apply it later:

hf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints --plan sync-plan.jsonl hf buckets sync --apply sync-plan.jsonl

Once done, inspect the Bucket from the CLI:

hf buckets list username/my-training-bucket -h

or browse it directly on the Hub at https://huggingface.co/buckets/username/my-training-bucket

That is the whole loop. Create a bucket, sync your working data into it, check on it when you need to, and save the versioned repo for when something is worth publishing. For one-off operations, hf buckets cp

hf buckets remove

Using Buckets from Python

Everything above also works from Python via huggingface_hub

from huggingface_hub import create_bucket, list_bucket_tree, sync_bucket create_bucket("my-training-bucket", private=True, exist_ok=True) sync_bucket( "./checkpoints", "hf://buckets/username/my-training-bucket/checkpoints", ) for item in list_bucket_tree( "username/my-training-bucket", prefix="checkpoints", recursive=True, ): print(item.path, item.size)

This makes it straightforward to integrate Buckets into training scripts, data pipelines, or any service that manages artifacts programmatically. The Python client also supports batch uploads, selective downloads, deletes, and bucket moves for when you need finer control.

Bucket support is also available in JavaScript via @huggingface/hub

Filesystem integration

Buckets also work through HfFileSystem

huggingface_hub

from huggingface_hub import hffs # List files in a bucket directory hffs.ls("buckets/username/my-training-bucket/checkpoints", detail=False) # Glob for specific files hffs.glob("buckets/username/my-training-bucket/**/*.parquet") # Read a file directly with hffs.open("buckets/username/my-training-bucket/config.yaml", "r") as f: print(f.read())

Because fsspec is the standard Python interface for remote filesystems, libraries like pandas, Polars, and Dask can read from and write to Buckets using hf://

import pandas as pd # Read a CSV directly from a Bucket df = pd.read_csv("hf://buckets/username/my-training-bucket/results.csv") # Write results back df.to_csv("hf://buckets/username/my-training-bucket/summary.csv")

This makes it easy to plug Buckets into existing data workflows without changing how your code reads or writes files.

From Buckets to versioned repos

Buckets are the fast, mutable place where artifacts live while they are still in motion. Once something becomes a stable deliverable, it usually belongs to a versioned model or dataset repo.

On the roadmap, we plan to support direct transfers between Buckets and repos in both directions: promote final checkpoint weights into a model repo, or commit processed shards into a dataset repo once a pipeline completes. The working layer and the publishing layer stay separate, but fit into one continuous Hub-native workflow.

Trusted by launch partners

Before opening Buckets to everyone, we ran a private beta with a small group of launch partners.

A huge thank you to Jasper, Arcee, IBM, and PixAI for testing early versions, surfacing bugs, and sharing feedback that directly shaped this feature.

Conclusion and resources

Storage Buckets bring a missing storage layer to the Hub. They give you a Hub-native place for the mutable, high-throughput side of ML: checkpoints, processed data, Agent traces, logs, and everything else that is useful before it becomes final.

Because they are built on Xet, Buckets are not just easier to use than forcing everything through Git. They are also more efficient for the kinds of related artifacts AI systems produce all the time. That means faster transfers, better deduplication, and on Enterprise plans, billing that benefits from the deduplicated footprint.

If you already use the Hub, Buckets let you keep more of your workflow in one place. If you come from S3-style storage, they give you a familiar model with better alignment to AI artifacts and a clear path toward final publication on the Hub.

Buckets are included in existing Hub storage plans. Free accounts come with storage to get started, and PRO and Enterprise plans offer higher limits. See the storage page for details.

Read more and try it yourself:

Installation guide

CLI guide and reference

Example Bucket on the Hub

Storage pricing