LLM 自ホストで数百万円節約する方法

Preface

imageIf you believe that cheap open weight models are the future then this blog is for you, we all know the why, but I will show you the how grounded in the economics behind it.

I wrote this because I can’t find this information in this form anywhere else. This is a single and arguably the only post that covers both the math of LLM inference and the dollars, grounded in production traffic and real numbers from inference provider negotiations.

If you find any errors in the math of this book, I will personally send you a printed copy of the book version and add you in credits section at the end.

Who is this for?

Software engineers/Managers: shipping open-weight models

AI research/Inference engineers: who want a working model of inference math + pricing

Management Execs and PMs: wondering if self-hosting is worth it. You can skim the math, and read the dollar numbers at the end to decide.

Students: chasing a job at a frontier lab

Prerequisites

All you need is high school algebra: every formula in this post is just multiplication, division, exponents etc. You don't need much ML background; transformers, MoE, KV cache, quantization, and everything else is either explained inline or linked to the best primer I could find. The blog is really dense, though, and the depth + understanding lives in the links. At a minimum budget a weekend if you want to actually internalize it.

What will we work with?

Our running example is Kimi K2.6, with Cline's real production traffic as the ground truth. The math works whether you rent your own GPUs or buy from an inference provider. We'll go in order: load the model, serve inference, batch it, and tune it for production.

Model Architecture

To self-host a model, you first load its weights onto a GPU. How much memory that takes is set by the model's architecture. Kimi K2.6 employs DeepSeek V3-inspired architecture with MoE parameters:

Total parameters: 1 trillion (sparse MoE)

Active Params: 32 billion

Hidden size: 7,168

Attention heads: 64

Routed experts: 384

Shared experts: 1

Experts per token: 8

MoE intermediate size: 2,048

MoE layer frequency: 1

Context window: 256K tokens

Want to derive these numbers from scratch? Read Kipply's legendary parameter counting post.

If you want some background on transformers architecture you must read: The Illustrated Transformer and if you are too tired to read another blog try LLM Visualizer which is the greatest LLM visualization I have ever seen.

Picking the GPU

We'll use the NVIDIA B200 (Blackwell). It's NVIDIA's flagship inference GPU and you can rent one from most neoclouds.

Here’s what you need to know about it:

192 GB HBM3e VRAM per GPU

The standard HGX rack ships with 8 GPUs, wired together by NVLink. NVLink is a fast interconnect between GPUs, so the 8 GPUs can act like one big GPU during inference.

Total GPU memory is 192 GB * 8 = 1,536 GB (~1.5 TB) per 8-GPU node

What does this node actually cost?

Let's lock in one number for the rest of the blog: $6 per GPU per hour. Real B200 pricing in 2026 is closer to $4 if you just rent the GPUS alone, but rounding up gives us a conservative ceiling, if the math works at $6, it will work in real life.

$6/GPU/hour × 8 GPUs = $48 / node / hour

$48 × 24 hours = $1,152 / node / day

$48 × 730 hours (avg month) = $35,040 / node / month

$48 × 8,760 hours = $420,480 / node / year

That $48/hour is your fixed cost. Serving 1 user or 1,000, the meter runs the same. The rest of this blog is about squeezing as many tokens as possible out of that $48/hour as that's what sets your real $/token.

Note:

Self-hosting only makes sense if you're already spending $35K+/month on API inference for this model. The model can't load on a smaller config, so $35K/month is the bare minimum entry ticket.

You can rent just GPUs alone for as low as 3.5-4$ but if you work with inference providers they will charge you 5.5-6$ because they handle the hosting of the model/speculative decoding and all the other model serving magic. As you will later learn in this blog post paying for the extra money to get the models hosted by someone else is a VERY solid deal and much cheaper than hiring an inference engineer if you do it right.

How many GPUs do we need to load the model?

Figure out how big the weights are

Kimi K2.6 has 1 trillion params. We'll run it in FP4 quantization (0.5 bytes per param). Quantization basically means storing each weight in the model with fewer bits, like rounding $4.7283910 up to $5. The model gets way smaller in memory, and if done carefully, it gives almost the same answers.

FP4 is the aggressive quantization tier, and modern models are trained quantization aware, so the quality hit is not that bad, the models are easier to host, and far less memory is needed to load them on GPU.

weight_bytes = 1,000,000,000,000 × 0.5 byte = 500 GB

Divide by your GPU memory

500 GB ÷ 192 GB per B200 = 2.60

So 3 B200s technically fit the weights of FP4 quantized model. That's the floor and its enough to load the model into VRAM and nothing more.

Quantization and Quality

Quantization is a trade off between price and quality. On the pricing side, every number downstream of this section is linear in bytes-per-param. FP4 puts the weights at 500 GB; FP8 doubles that to 1 TB (you'd need 6 B200s, not 3); BF16 doubles it again. FP4 is the cheapest tier that still ships acceptable quality on modern quantization-aware models, which is why we use it. If your evals say FP4 hurts your workload, redo the chain at FP8 as the math still works, it just costs more. Note: FP4 here means NVFP4 specifically, Blackwell's native 4-bit format. And we'll use "FP4" throughout the blog for brevity.

On the quality side, the trade-off isn't visible on a spec sheet. Some providers offer dirt-cheap, low-latency inference but quantize so aggressively that output collapses no matter how good their latency SLAs(Service Level agreements) look. The only way to know what you're shipping is to run evals (our setup is here).

How many GPUs do we need to serve the model?

Loading the model is one thing. Serving real traffic is another, and it costs more VRAM. Every concurrent user needs their own slice of KV cache (per-user memory that keeps attention from being quadratic so that new inference requests use caching). You also need bandwidth and headroom to batch their requests efficiently. We'll cover KV Cache, bandwidth and headroom in the next sections, for now, just trust that this is where the rest of the VRAM goes.

So we're taking the full 8-GPU B200 node: 1,536 GB of total VRAM. Here's how that breaks down:

What

Math

Memory used

Remaining

Total VRAM on the node

192 GB × 8 GPUs

—

1,536 GB

Model weights

1T params × 0.5 byte

500 GB

1,036 GB

Activations (working memory during a forward pass)

~80 GB

956 GB

Free for KV cache (i.e. concurrent users)

~956 GB

That ~956 GB is the budget for actually serving users. The next several sections are how we spend it.

imageServing the model

Two things happen on every request: the model reads the whole prompt (prefill), then writes tokens one by one (decode). We'll cover decode first because it's the dominant cost on long generations, then circle back to prefill.

Here's the TLDR of Decode. LLMs generate one token at a time. The model writes a token, looks at everything so far (your prompt + the tokens it just wrote), and writes the next one. It loops until its done.

[prompt] → token_1

[prompt + token_1] → token_2

[prompt + token_1 + token_2] → token_3

...

Each loop iteration is one forward pass: the input runs through dozens of layers, each layer reads its slice of weights from VRAM and hands its output to the next, and at the end you get a probability distribution over the next token. The model samples one and repeats.

So inference is just: read weights from VRAM → matrix math → emit a token → repeat.

Emitting a token (Decode)

For the 32B active params, if we have to generate an output token we need the following data flow(FP4 quantization is 0.5 bytes per param):

bytes_read_per_token = 32B active × 0.5 byte = 16 GB

16 GB of weight data flowing across the bandwidth pipe for every single token we generate. This is the main bottleneck number.

The B200 does 8 TB/s of memory bandwidth per GPU. Sounds enormous, but watch what happens when we run the math. Max output speed is just bandwidth ÷ bytes_per_token.

Single B200: 8 TB/s

8-GPU node (raw): 64 TB/s

8-GPU node (real): ~50 TB/s after sync overhead

50,000 GB/s ÷ 16 GB/token ≈ 3,125 tok/s theoretical

At 10–12% real-world MBU* ≈ 312–375 tok/s aggregate decode

So at small batch sizes, a full 8-GPU B200 node serving Kimi K2.6 in FP4 hits a theoretical ceiling of roughly 3,125 tokens per second of aggregate decode.

Decode is sequential so you can't generate token N+1 until token N is done. That's why decode is bandwidth-bound, and why TPS (tokens per second) is the metric that decides how fast the output streams. (Exception: speculative decoding.)

Caveats of theoretical max decode throughput vs actual max throughput

The 16 GB/token figure above assumes that every token reads its own personal slice of experts, which is true at batch=1, but it stops being true the moment you batch. Once many users flow through the same forward pass, tokens share expert reads inside MoE, the effective bytes-per-token shrinks, and aggregate decode climbs well past this number. There are also other aspects to consider. We'll work out exactly how this works in the Batching section.

That 3,125 tok/s number is theoretical: what you actually get is determined by your MBU (Memory Bandwidth Utilization), the fraction of nominal bandwidth your inference stack actually delivers to useful work. MBU varies widely: We measured ~11% MBU on vLLM + Kimi K2.6 + 8× B200 + 80K context, on the lower end.

Our practical takeaway is that you shouldn’t trust theoretical ceilings, and don't trust vendor claims either. You can do load tests and compute your achieved MBU as

(measured aggregate decode tok/s × bytes_per_token) ÷ nominal_node_bandwidth

Then cross-reference it with research papers to see if you are too far off, and if you are too far off then you must find out why.

Before going further, there's two ideas you need to lock in that will explain the real bottlenecks of GPU performance for inference and they are:

Memory-bound vs compute-bound

Arithmetic intensity

Memory Vs Compute bounds

This shows up constantly in transformer inference, and in deep learning optimization more broadly. The basic idea is that: to do any math on a GPU, you first have to pull the weights out of memory, which costs memory bandwidth. The good news (and this has been very well optimized) is that the GPU can start computing as soon as the first weights arrive memory reads and math run in parallel.

So at any given moment, one of two things is happening:

Compute-bound/Flop-bound: In this case, the GPU is choking on math. The math units are fully busy, and memory is sitting idle. A FLOP is one floating-point operation. One multiply, one add, one of those. When people say a GPU does "15 PFLOPS," they mean it can do 15 × 10¹⁵ floating-point operations per second. You can read more about compute/flop bounds in these lecture notes.

Memory-bound: the memory pipe is fully busy, and the math units are sitting idle.

Let me remind you once again that decode is memory-bandwidth-bound, and that is a huge bottleneck on how fast we can do our inference.

Arithmetic intensity

Since at any point the GPU is either memory or compute bound. Arithmetic intensity is the knob that flips a workload between the two: FLOPs done per byte read from memory.

arithmetic_intensity = FLOPs performed / bytes read from HBM

Every GPU has a break-even point (the "ridge" of the roofline model) where compute and bandwidth finish at the same time. Below the ridge you're memory-bound; above it you're compute-bound.

For the B200 at FP4:

ridge = peak_flops / peak_bandwidth

= 10 PFLOPS / 8 TB/s

≈ 1,250 FLOPs per byte

So on a B200, a kernel needs to do ~1,250 ops per byte of weight before compute becomes the bottleneck. Anything less and the math units are sitting idle waiting on HBM.

Now look at where decode and prefill land:

Decode (batch=1): ~2 FLOPs per byte read (one multiply-add per weight value). Deep in memory-bound territory. ~600× below the ridge. This is why decode tok/s is dictated by bandwidth, not flops(as i had mentioned before).

Prefill: You read the same weight once and reuse it across thousands of tokens (or batching). Arithmetic intensity scales roughly linearly with how many tokens share that read. Push enough tokens through one forward pass and you cross the ridge into compute-bound land.

This is the lens to keep in your head for the rest of the blog. Every optimization that follows is, underneath, a trick to raise arithmetic intensity so each byte of HBM traffic produces more useful tokens. As we do load testing later and put more weight on our GPUs, we'll see both of these bounds in practice.

Loading the prompt tokens (Prefill)

The next concept you need to learn is prefill. Prefill is when the whole prompt the user throws at you gets read in a large parallel mat mul operation so that the model can "process it" and give you a response (decode).

imagePhoto from the illustrated transformerWhy can you do prefill in parallel?

Because you already know the entire sequence, you can push it through the layers in parallel. In one weight read, thousands of tokens get processed simultaneously. The bandwidth bottleneck disappears because you're amortizing one read across thousands of tokens.

Prefill is compute-bound. In our case, the B200 GPU has way more compute than bandwidth: ~10 PFLOPS dense at FP4 per GPU, and we have 8 GPUs.

Realistic prefill speeds reach ~78,000 input tokens/sec aggregate on an 8-GPU B200 node, roughly 1 coding-agent-sized fresh prompt processed per second on a long Kimi K2.6 conversation. That's what we'd expect to measure at Cline running this workload.

Math to explain it

Flops per GPU = 10 PFLOPS

flops_per_token = 2 × 32B active = 64 GFLOPs

node_compute = 8 GPUs × 10 PFLOPS = 80 PFLOPS (dense FP4)

theoretical_max = 80 PFLOPS ÷ 64 GFLOPs ≈ 1.25M tok/s

realistic ≈ 6.4% MFU (attention quadratic + MoE all-to-all,

partially offset by B200 attention pipeline)

≈ 78,000 tok/s on long prompts (this was what we got in load testing)

MFU(Model FLOPS Utilization) = (achieved FLOPs/sec on model work) / (peak FLOPs/sec of the hardware)

MFU is the reality check on your prefill numbers. Compare your measured MFU against the ranges published in literature if you're too far off, either your stack has a real problem or another provider is squeezing far more out of the same hardware.

TTFT: Time To First Token

This is where the metric TTFT (Time To First Token) comes from, the wait between hitting enter on chat and seeing the first character stream back. Everything before the first token appearing on chat is prefill burning compute. Everything after is decode burning memory bandwidth.

If your node prefills at 80,000 tok/s and a user shows up with an 80,000-token prompt:

80,000 tokens ÷ 80,000 tokens/sec = 1 second TTFT

In practice, two things change this number by a lot in different directions:

1. Batching: That 80K tok/s is the whole node of the entire GPU. If 16 users are prefilling concurrently, each one only gets ~5,000 tok/s of effective throughput. An 80K prompt now takes ~16 seconds. Per-stream and aggregate are very different numbers, and you need load tests in different configurations to figure this out. We will explain batching more later in this document.

1. KV cache/Prompt reuse: In coding agents and chat, conversations build on top of each other. When a user fires off their 5th message in a long thread, you're not re-prefilling the entire 80K context the KV cache from earlier turns is already sitting in VRAM. You only prefill the delta: the new user message plus whatever fresh context got pulled in (file reads, tool outputs, etc).

At Cline, our system prompt is small, roughly 3–4K tokens, and most of the tokens come from file reads/searches over the back-and-forth conversation. Even when total context is 80K, the actual new tokens being prefilled on a given turn are usually 8–10K, mostly from file reads.

KV Cache: where the 956 GB actually goes

Remember that ~956 GB of free VRAM we left in the bank earlier? This is where we spend it.

Here's the core problem KV cache solves. Every time the model decodes a new token, the attention layer needs the K (key) and V (value) vectors of every previous token in the context. Without caching, you'd recompute every K and V from scratch on every single decode step, turning each new token into a fresh prefill of the entire history. That's quadratic work for nothing.

The fix is simple: K and V from past tokens are deterministic functions of those tokens, and the model weights don't change. So the K and V you compute at step 5 are literally the same numbers you'd compute again at step 6, step 7, step 10. Compute once, keep in VRAM, read back on every future step.

imageThe picture above is the main intuition.

Insight 2: the next token's attention only needs the new token's query vector and all past key/value vectors. New Q every step, but the Ks and Vs from prior steps are reusable forever.

imageSo the cache works like this: on token 5, you generate K_5 and V_5 fresh and append them to the K/V buffers. Tokens 1–4 get pulled from cache. On token 6, you append K_6 and V_6 and pull tokens 1–5 from cache. One new pair per step, never recomputed. That's the whole game.

How big is the cache, in bytes?

DeepSeek V3 and Kimi, which inherits the attention architecture, use MLA: Multi-head Latent Attention. Instead of caching K and V for every head, MLA compresses them into a single shared latent vector per layer plus a small RoPE component. The cache size becomes independent of head count.

bytes_per_token (MLA) = num_layers × (kv_lora_rank + qk_rope_head_dim) × bytes_per_value

= 61 × (512 + 64) × 1

≈ 35 KB / token / user

Caching in the 956 GB

KV per user = 35 KB × context_length. Free VRAM = 956 GB. So:

max_users ≈ 956 GB / (context_length × 35 KB)

Context length

KV per user

Theoretical max users

80K (Cline-shaped)

2.80 GB

~341

32K (typical chat)

1.12 GB

~853

This is the theoretical ceiling, but not the operational ceiling. All of this would only be true if we assume that the entire remaining VRAM was just used for KV cache but the truth is far more complex. We can't actually support 341 users at the same time(we can barely support 32). Now let’s figure the why out in the batching section up next.

Batching

The whole point of batching is that decode is memory-bandwidth-bound. At batch size=1, you pay the full cost of pulling weights from HBM just to generate a single token for a single user. That's wasteful, the weights are already on the wire, you may as well pipe more tokens.

So you stack many users' next-token computations into the same forward pass. Each pass reads the model weights once, and every user in the batch shares that read. If you can fit B users into a