トークン節約術を研究して全トークンを燃やした話

On the Hacker News front page on 19 Jul 2026

At Quesma we are researching the economics of AI agents: what agentic coding really costs and what you can do about it. For this research I am running my own deep research setup, a pipeline of agents that builds a knowledge base I can actually trust. The first version of this setup burned the whole limit of my Claude Max 5x plan in 30 minutes. This post is the story of how I fixed the cost and the trust, using only subscriptions I already pay for, and how you can build the same.

My goal was to understand the entire state of so-called tokenomics. I wanted to know what monitoring systems exist, how teams govern their AI spend, and which optimization tools and practices actually work, both in the papers and in the real world.

I started the usual way, with /deep-research. I gave it the big open question and let it run. After around 30 minutes of research, I hit a limit and had to wait a few hours for it to reset. And I had no results. The run launched 111 agents and queued 123 claims for verification, but only 25 got verified before the limit hit, and the final synthesis never ran.

So, it hit me personally. And this is a bit funny: from the very first day I had to optimize tokens while still discovering how to optimize tokens. Learning by doing.

Using every subscription I already pay for

If Claude Fable 5 is out after 30 minutes and /deep-research is consuming so many tokens without giving me any results, what can I do to make the research more effective? I started thinking about what kind of tools I already have and pay for. Claude, Codex, and Antigravity: 3 subscriptions, and in theory 3 times more tokens without paying anything extra. What if I use all of these tools together, with shared memory?

Since I am already using the claude-mem plugin I extended it to support Codex and Antigravity locally, so that all 3 tools can use shared memory during sessions. Whatever one tool learns, the others can use.

Cheaper models as subagents

My default setup is Claude Code, so I used it as the main harness. While doing the research manually, I discovered a model-orchestration pattern, which was exactly what I needed at that moment. We don’t really need Fable for everything: Claude Opus 4.8, Claude Sonnet 5, GPT-5.5, and Gemini 3.1 Pro are already excellent models for many tasks.

So I checked a few benchmarks and cost analyses, mostly Terminal-Bench for terminal and agentic work, SWE-bench Pro for end-to-end software engineering, and Artificial Analysis for a general view of performance and prices. I did not treat any of them as the final truth; benchmarks measure their own thing and the numbers move every month. I just needed a loose starting point for who is good at what, and using several different models was part of the experiment from the beginning; I expected to adjust the split after real runs anyway:

Role

Model

Why this one

Find

Claude Sonnet 5

Strong on agentic benchmarks and cheap enough to run in bulk

Verify

Claude Opus 4.8

The most accurate Claude worker; checking needs more accuracy than searching

Judge & plan

Claude Fable 5

The most expensive one, so it only plans, decomposes, and resolves disputes

Small stuff

Claude Haiku 4.5

Cheap and fast for extraction and formatting, too weak for multi-step work

Run tools

Codex (GPT-5.5)

Very strong on terminal benchmarks; clones, installs, runs, and inspects tools

Second opinion

Antigravity (Gemini 3.1 Pro)

A different model family, so it does not share the same blind spots

This split was not my first version; I adjusted it a few times when live runs showed weaknesses, and the fallback rules came from those failures.

The best thing here is that I prepared Codex and Antigravity to be used as Claude subagents, orchestrated by Fable, using the headless nature of both. Why is it cool? Because tokens are taken from the Codex and Antigravity subscriptions, so I was not paying extra, but I instantly had much more intelligence to use.

The whole trick is one small Bash script that my Claude agents can call like any other command:

# run-cli: call another vendor's CLI as a headless subagent
# usage: run-cli <codex|antigravity> "<prompt>"
VENDOR="$1"; PROMPT="$2"
case "$VENDOR" in
  codex)       OUT="$(codex exec --sandbox read-only "$PROMPT")" ;;
  antigravity) OUT="$(agy --model "Gemini 3.1 Pro (High)" -p "$PROMPT")" ;;
esac
echo "$OUT"
echo "$OUT" | claude-mem-save -s "$VENDOR" # save to shared memory (my locally extended claude-mem)

The wrapper also watches the output for “usage limit” and “out of credits” messages and returns a special exit code, and this is exactly how the automatic fallback to Claude models works: the orchestrator sees the signal and does the work with a Claude agent instead.

One more thing that really matters is to pin the model per role, because subagents inherit their parent’s model by default, and that is exactly how I burned my Fable limit in 30 minutes.

With this technique I was able to run the research continuously for roughly 10 times longer than initially with Fable only, measured simply as how long I can keep the agents working before any of the 3 subscriptions hits a limit. Before it was 30 minutes of research, now it is a few hours, without paying a single penny more. And when Codex or Antigravity hits its own limit, the framework simply falls back to Claude models, so the research does not stop.

Reducing hallucinations

Cost was only the first problem, the second one was trust. Before my research framework, when everything was done by Fable, I was sometimes getting findings that looked solid but were not true. For example, a wrong license on a repo, a savings figure with no source, or a number that was not on the cited page. To reduce hallucinations, I implemented explicit rules in the research framework that every finding must pass before it can be shared. A few of them:

• Whoever finds a claim never verifies it, a different model or agent checks the links, quotes, and numbers.

• Nothing lands in the knowledge base without a URL and a quote from the primary source.

• Never state a number the source page doesn’t contain.

This list was not designed upfront. It grew during the research, because every time verification caught a new class of mistake, the rule went back into the prompts. And such a framework is really effective only if you keep tuning it, so review the findings, flag the useless ones, and feed that back.

/deep-research goes last

With cost and trust handled, the last piece was the /deep-research tool that started this whole story. It is still in the pipeline, but as the last step, not the first. At the end of each day I run it over the findings that already survived verification, so instead of researching blindly it goes through the existing findings, deepens them, eliminates the mess, and tries to fill the gaps that the framework missed. It also uses fewer tokens this way, because it works through a fixed list of claims instead of exploring the internet. The last run used 61 agents and took 22 minutes. Compare that to day one, when 111 agents burned the whole limit in around 30 minutes and never produced the report. Same tool, just a much smaller job.

The result: a knowledge base I can trust

Everything that survives verification lands in an LLM wiki I keep in Obsidian, inspired by Karpathy’s LLM wiki pattern: linked atomic notes, agents doing the sweeps, and me setting the rules. One week in, it holds hundreds of validated notes on pricing, tools, benchmarks, and practices.

Automated checks aren’t enough on their own. Once, my triage rule quietly rejected Headroom, a 56k-star project and one of the biggest in its category. The agents did everything right, verification confirmed the project is legit, and a claim check correctly flagged that its headline savings numbers were not quality-controlled. But my rule said “unvalidated claims mean no entry”, so a project half the industry uses was invisible in my knowledge base. I only noticed 2 days later, when I asked “how did we miss this?” The fix was a new rule: reject the claim, not the project. Agents can do the searching and checking, but no agent will ever tell you that your own rule is the bug.

If you want to build a quality knowledge base, human verification and research supplementation are mandatory. AI-only research can be really poor quality. Human-only research is too slow. The best way is a hybrid one: agents do the heavy research work, but they run on a pipeline crafted and constantly improved by a human. The human also verifies the results and fills the gaps.

A few findings from the knowledge base

The token economics topic turned out to be much larger than I expected. Here is a small sample of what is already inside, the findings that surprised me the most:

• Your harness can matter as much as your model. On Terminal-Bench the same model shows a ~66x token spread across different harnesses, and the thinner setups actually score higher, not lower. Another study measured a swing of about 54 percentage points on the same model, changing only the harness. We saw a small version of this ourselves when we checked the true cost of saying “Hi” to an AI agent.

• Context compaction can double your bill instead of saving it. Compaction sounds like compression, so it is easy to assume it is always good, but it is not free: the model has to summarize your history and that summarization call consumes tokens too. It can also evict files the agent still needs, so the agent re-reads them, the context fills up again, and compaction fires again. One documented regression shows how bad this loop can get: after a harness retuned its compaction threshold, it went from 4 to 12-26 compactions per session, and token usage on identical tasks went from 89 million to 160-185 million.

• One mid-session tool change silently re-bills everything. Cache reads cost about 0.1x of the base input price, but the cache invalidates as a hierarchy (tools, then system, then messages). Add or reorder a single tool schema in the middle of a session and the entire cached prefix re-bills at full price. You get no error, just a bigger invoice.

• Your token counter is not your invoice. In one documented case a calculated $3.60/month landed as a $25-40/month invoice, a 7-11x gap coming from context accumulation, retry amplification, framework overhead, and evaluation calls that a simple token estimate never captured.

Our framework tags most of these as medium confidence, usually 1 or 2 solid sources, and we keep that visible instead of pretending it is settled.

Before you burn your next limit

If you are burning your limits on deep research today, try reversing the order. Cheap models find, accurate models verify, and deep research goes last. And check how much intelligence you already pay for. A few subscriptions with a clear role for each model give you much more than one frontier model used blindly.

And if you are building your own research pipeline, or your AI bill grows faster than your usage, we would really like to hear from you. Join the discussion on Hacker News or LinkedIn.

Stay tuned for future posts and releases