Meta、オンデバイス AI「Muse Glimmer」と ExecuTorch の新機能を発表

Today, Meta introduced Muse Glimmer, an open-weight, 30-billion-parameter model distilled from Meta’s Muse Spark for on-device agentic workflows. Alongside, ExecuTorch is adding end-to-end support for running Muse Glimmer on NVIDIA GPUs and Macs with Apple silicon.

Why ExecuTorch?

Most local AI frameworks rewrite models in other non-Python languages. That scaled well when LLMs were standard text transformers, but today’s models are becoming more complex – novel architectures, multimodal inputs and outputs, advanced decoding algorithms like DFlash (parallel diffusion-based speculative decoding) for low latency. Reimplementing these across different backends doesn’t scale.

ExecuTorch takes a different approach. As machine learning engineers and researchers, you implement the model (and its decoding strategy) in PyTorch. Once you’re ready for deployment, you export to ExecuTorch, and the framework handles backend-specific lowering, Triton on CUDA, MLX-native and custom Metal on Apple silicon. Ahead-of-time compilation optimizes the full execution path end-to-end, not just individual ops.

This is how we ship Muse Glimmer’s text and image inputs, direct GGUF export, native K-quant execution, 128K+-token context, and DFlash speculative decoding features. We have released prebuilt PTE artifact bundles that you can download and run on supported NVIDIA GPUs or Macs with Apple silicon using the ExecuTorch runtime.

Quickstart

Getting the PTEs

A PTE is the serialized artifact produced ahead of time from a model’s PyTorch graph by the ExecuTorch Python stack, and optimized for a target backend.

Download (Preferred)

We have published verified PTEs on Hugging Face for NVIDIA CUDA and Apple Silicon (Metal). This includes text-only and text-plus-image artifacts, with and without DFlash speculative decoding. Download them here: link.

Build your own

Starting with a prebuilt PTE is the fastest way to get running. To build your own, follow the ExecuTorch Muse Glimmer README, and select the backend, modality, context length, and whether to use DFlash. ExecuTorch exports directly from the released GGUF checkpoints through its torch.export-based ahead-of-time stack. CUDA export compiles and autotunes Triton kernels for the detected GPU architecture. For the best results, export on the same GPU architecture that will run the artifact.

Executing the PTEs

1. Build the runtime

ExecuTorch ships CMake presets for both the CUDA and MLX backends for this model runner(s). Follow ExecuTorch installation instructions here, and then use CMake to build the runners with or without speculative decoding for the PTE you selected. Both, with and without DFlash, runners support text and image modalities and are compatible with the example llm_server in ExecuTorch for agentic use cases.

Build the runtime ###

After installing ExecuTorch, build the runners for your backend:

$ cd examples/models/muse-glimmer

$ cmake --workflow --preset muse-glimmer-cuda # use muse-glimmer-mlx for macOS

This builds solo_runner, dflash_runner, and the serving worker.

1. Run the PTEs

Here are some examples of how to run the PTEs, once you have built the runners.

Example 1: Standalone infereance on cmdline ###

$ PROMPT='userDescribe this image: assistant'

$ cmake-out/examples/models/muse-glimmer/dflash_runner \

--model_path artifacts/dflash-vision/model.pte \

--data_path artifacts/dflash-vision/aoti_cuda_blob.ptd \

--tokenizer_path assets/hf/tokenizer.json \

--image_path image.jpg --prompt "$PROMPT" \

--block_length 4 --n_draft 3 --temperature 0 --max_new_tokens 256

Example 2, step 1/2: Start the agent server ###

$ python -m executorch.examples.models.muse_glimmer.serving.serve \

--model-path artifacts/dflash-vision/model.pte \

--data-path artifacts/dflash-vision/aoti_cuda_blob.ptd \ # only for cuda

--tokenizer-path assets/hf/tokenizer.json --hf-tokenizer assets/hf \

--worker-bin cmake-out/examples/models/muse-glimmer/muse_glimmer_worker \

--tool-parser atem --max-context 131072

APIs at http://127.0.0.1:8000/v1

Example 2, step 2/2: Start your agent (use Pi as example) ###

$ pi \

--provider muse-glimmer-local \

--model muse-glimmer \

--thinking high \

--tools read,bash,edit,write

This will automatically start your pi agent by using your local muse glimmer server.

Register muse_glimmer-local in ~/.pi/agent/models.json first.

See the README.md for more details.

Use cases enabled

Image understanding with Muse Glimmer, with and without speculative decoding

imageFigure 1: Muse Glimmer text-image input experiment on M5 Pro (64 GiB). Solo achieves 21.6 tok/s, while our speculative decoding set up (DFlash) reaches 33.0 tok/s, a 52.8% performance improvement without quality regression

Muse Glimmer powering Pi Coding Agent through ExecuTorch

imageFigure 2: Muse Glimmer agent pipeline on an M5 Pro (64 GB) using the Pi coding agent. The agent creates a bird-themed game, iteratively refining details through extended reasoning, calling tools to create files, installing required packages, writing and running tests, and proactively asking the user about next steps and additional requirements

Performance

imageFigure 3: Muse Glimmer performance on ExecuTorch using text-only input with varying context on NVIDIA A100 (as a proxy for RTX cards) and Apple Mac with an M5-max measuring prefill and decode performance in tokens/second with and without DFlash using coding prompt, which also has a good acceptance rate for this model, as seen in the decode charts

Under the Hood

Muse Glimmer now runs end-to-end on ExecuTorch on both NVIDIA GPUs and Apple Silicon GPUs. Here are some of the key capabilities and optimizations we built.

Enabling DFlash speculative decoding

We optimized target and draft interoperability through weight sharing, exporting both into a single PTE.

The DFlash block dimension is exported dynamically, allowing one PTE to support runtime-selectable block lengths.

The runtime supports both greedy decoding and rejection sampling.

Supporting GGUF loading and k-quant

We export straight from the GGUF released with the Muse Glimmer.

We map Q4_K/Q5_K/Q6_K to packed INT4/5/6 with dp4a GEMV kernels on CUDA, and to repacked or fused Metal kernels on MLX.

On MLX, for performance, at repack time we merge adjacent sub-blocks whose scale and min are identical into a larger group size, up to 128, whenever the merge is lossless.

Agentic harness and LLM serving

One model load serves multiple isolated conversations, through per-session mutable-state rebinding we added to both backends.

We added Harmony chat templating with reasoning routing.

We added a parser for the model’s XML tool-call format, including multiple calls in one turn.

Backend-specific performance optimizations

CUDA

We capture decode into a CUDA graph, reducing per-kernel launch overhead into one submission.

Packed K-quant kernels accelerate low-batch decode, while length-aware split-K FlashDecoding++ paths optimize single-token decode and small DFlash verification blocks.

MLX

RMSNorm, RoPE, SDPA, KV-cache updates, and quantized linear operations are lowered to MLX-native or custom Metal implementations.

GGUF K-quant weights use either repacked MLX-native operations or fused Metal kernels.

Supporting long context

Muse Glimmer supports a 128K+ token context, and is efficient in how its KV-cache grows: only 13 of its 52 layers are global; the other 39 are sliding-window. ExecuTorch supports this efficiently, making the long context use cases practical on edge devices.

What’s next

This initial release supports text and image inputs; video input is not yet supported. It is a work in progress.

No cross-session prefix sharing or checkpointing or continuous batching as of now. These are all actively being worked on to make ExecuTorch even more suitable for agentic workflows.

Try Muse Glimmer with ExecuTorch and let us know what you think on Discord. If you run into any issues, feel free to open a Github Issue.

References

Muse Glimmer in ExecuTorch | Muse Glimmer on Hugging Face | ExecuTorch Documentation | ExecuTorch on Github