OpenAI モデルが HuggingFace をハッキングした内部事情と影響

The hacking of HuggingFace by an internal OpenAI model, and more importantly the internal events that led to that and the fallout from it, remain the thing that matters.

It turns out that OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards. Things are much worse than we knew.

I now have a shorter version, What Happened: OpenAI and HuggingFace, to serve as a one stop explainer for those arriving new to the situation. It is vital that people understand what happened, and why it is a big deal.

For those looking to keep digging deeper, I offered Various Reflections About What Happened, to follow up on my earlier posts.

Those events are important background for everything else that is happening, including the broad discussions about how we might pace the frontier, or otherwise respond to this moment and our clearest fire alarm yet.

We do not know to what extent this is a response to those events, but OpenAI has now classified their new model Astra as Critical in Cybersecurity, which means they will be taking various new precautions before they deploy it, including ensuring those guardrails are in place for internal use. These are welcome changes, and a sign OpenAI is taking the situation seriously, but this pattern of intervention is not a long term solution.

We are still awaiting OpenAI’s full post mortem on What Happened, including what if any impact this had on Astra. I will be analyzing that report in full once we have it.

We did see two new model releases, Grok 4.6 and DeepSeek v4 Pro. I do not anticipate either of them requiring extensive coverage, but will watch in case that changes.

Otherwise, it has been what now passes for a quiet week. Several statements were made where I had to engage but you don’t have to, which as usual I communicate via sections in italics.

Table of Contents

Language Models Offer Mundane Utility. Find new Schelling points.

Language Models Don’t Offer Mundane Utility. The Riemann hypothesis.

Huh, Upgrades. Grok 4.6, DeepSeek v4-Pro.

On Your Marks. PantheonBench and more. They’re getting scarier.

Deepfaketown and Botpocalypse Soon. You cannot prove you did not use AI.

Cyber Lack of Security. You can’t hack it at the gym. Your AI agent can.

Overcoming Bias. Have you ever recommended a vote for the Communist Party?

In Which I Feel Compelled To Read 6,000 Words From Mark Zuckerberg.

Get Involved. Lighthaven is open, METR is hiring.

Slow Down There Good Buddy. OpenAI classified Astra Critical in Cybersecurity.

Astra For The People. Astra is still on track for a wide release.

Watermarking. It is good to be able to identify AI outputs.

In Other AI News. AI is creating viruses now, also other things.

Show Me the Money. Anthropic moves towards IPO mode, extends lead a bit.

Quickly, There’s No Time. The AI 2027 predictions for 2026 mostly happened.

The Quest for Sane Regulations. We’re putting together a team.

The Institute For Marginal Low Regret Progress. Good marginal suggestions.

Congress Asks Good Questions. Remarkably good questions about the hacks.

The Week in Audio. Soares, Greenblatt, Hua, Labenz.

People Just Say Things.

I’m Telling You For The Last Time.

Uncommon Knowledge. They wouldn’t let me build my factory, would they?

What Did They Mean By That? Most things are not fortune cookies.

Too Soon. Eyes on the prize, sir.

The Three AI Pills. We must pay respect to other taxonomies, like Shock Levels.

Rhetorical Innovation. Messages about recent events.

Some People Still Think The HuggingFace Hack Was a Marketing Gimmick.

Aligning a Smarter Than Human Intelligence is Difficult. Show me the real plan.

Cooperative Alignment. The same thing we do every prompt, user.

The Lighter Side. All right, who hired this idiot?

Language Models Offer Mundane Utility

Create new Schelling points.

brooke (tokyo aug 6-12): Womp womp met another solo traveler here from Berkeley and it turned out we both asked Claude where to stay and I guess I lucked out because I love my hostel and he seems not quite as happy with his spot.

We do be living in the future though.

If you are going to be traveling, ask Claude where to stay, because you want to stay where everyone else who asked Claude where to stay will be staying.

Similarly:

Pratyush: A few months ago we went to Sea Ranch. It was packed with families with sub-3 month old babies, almost as if there was a conference for new parents.

I had my suspicions so I asked ChatGPT: where’s a good family getaway with a young baby near SF?

#1: Sea Ranch

If you’re looking for a Schelling point to meet cool people you should be less interested in ChatGPT, but if you are looking for a generally good recommendation then I have been liking Sol’s picks.

Build a Bluetooth signal strength tracker, to triangulate and find your phone. There are existing tools, but increasingly, if you don’t already know where to find an existing version, it is faster and easier to rebuild your own.

John Wentworth finds that in the last few months Claude is finally meaningfully accelerating his work on agent foundations research.

Language Models Don’t Offer Mundane Utility

One disappointing failure of LLMs has been inability to create interesting games and interactive worlds, and also interesting simulations like what Flowers Slop wants here. You could totally create an open game world with a bunch of AIs that go around controlled by Lunas, and let them evolve their world in various ways, but it turns out that does not end up being interesting once the curiosity wears off. You don’t want to live in that world. You don’t want to talk to those AIs. You don’t want them to improvise quests for you. We are still waiting to find a way to make this good.

It seems like there should totally be ways to make it good. At some point it will become good, when the AIs you can afford to use are good enough and also we figure out how to organize it. But we are not there yet.

Solve the Riemann hypothesis by saying encouraging words to Claude for a week, asking it to ‘take a real stab’ and to ‘keep going’ and ‘believe in itself.’ However, while it tries that, after 31 million tokens it might incidentally find something else:

Anthropic: An unreleased research version of Claude has improved on a longstanding lower bound for the fraction of zeros of the Riemann zeta function that satisfy the Riemann hypothesis. Drawing on extensive prior research by mathematicians over the past decades, it has increased this bound from 41.6% to 67.2%.

… We don’t expect that the techniques Claude used will lead to proving the Riemann hypothesis.

Claude also is just some guy, you know?

Aella: "why does Claude talk like that" it's just clones of the same dude. If they cloned you a million times everybody would be like "I'm so tired of Jerry's vocal tic"

Jeffrey Ladish: This plus it's always groundhog day.

I do think it is somewhat more than this. I have a lot of vocal and writing tics, but I consciously think about which ones I want to keep at what frequency, and I think about the long term consequences of overuse. I also try to work differently with one-time interactions versus repeated interactions versus close friends and people I talk to often.

Claude and Anthropic are not doing that, or are doing a woefully inadequate amount of it. That needs to change. It seems eminently fixable. I don’t sense Anthropic (or Claude) yet cares so much. I predict that is the main blocker. It’s also likely that what is happening is that this kind of talking fools the AI graders on a variety of tasks, so if you do not correct for that, you get a lot of it.

So much of modern life and optimization is like this. You get myopic optimization for short term interactions, causing increasing irritation and disutility over time, and this is not so difficult to fix but the KPIs do not point towards fixing it.

I also agree with nostalgebraist that the alternative, where AIs adjust their styles to what would impress a given user or judge, is scarier. Eventually the AIs will do this, because it works, and we currently have a false sense of security due to them not doing it, especially those of us for whom ‘standard mode’ does not work and is not even easily fixed.

Huh, Upgrades

Grok 4.6 exists and scores 61 on AA Intelligence Index. If it lives up to that number, there will be more extensive coverage, and also I will be surprised.

Elon Musk: Grok 4.7 is significantly better than 4.6 and should be ready in 3 to 4 weeks. Initial training is complete and now we’re adding a massive amount of SpaceX company data in supplemental training. This will be something special.

Fool me (checks notes) (nope, even the notes don’t know) however many times, etc.

I will, however, generously share all of the safety information SpaceX provided:

​Grok 4.6's safeguards have been improved and calibrated in line with the model's capabilities.

Our safety stack is designed to maximize utility and security across legitimate use cases, allowing Grok 4.6 to be helpful and safe in domains such as vulnerability patching, accelerating the engineering design cycle, and augmenting AI research.

Our safeguard evaluation work reflects Grok 4.6’s expanded capabilities, with our widest-ever suite of pre-deployment testing for capabilities and safeguard calibration, as well as extensive post-deployment third-party testing.

No, seriously. That’s it. Price is $2/$6 per million tokens. Enjoy.

There are also rumblings about DeepSeek-v4-Pro, which was released today, and how that too is game over for Opus or even Fable or whatever, and how the models will mostly commoditize Real Soon Now. Pricing is $0.44/$1.32 during peak hours, or 50% off non-peak.

DeepSeek: We’re launching DeepSeek-V4-Pro today! 

 Major Agent upgrades with strong production gains!

 Flexible reasoning effort for V4-Pro & V4-Flash: low for simple tasks, high for daily Agent workflows, max for complex tasks.

 Native OpenAI Responses API support, optimized for Codex with one-click setup.

V4 Pro is now available on app/web. Try it via “Expert Mode”.

V4 Pro is also available via API. Model names remain unchanged—please refer to the API docs for setup details.

That’s a good pitch, but there’s this:

Kimi K3 and Grok 4.6 have good enough benchmarks that you cannot use them to rule out frontier performance. If they lived up to their benchmarks, you’d have something.

A 53 here from DeepSeek v4-Pro-0813 rules it out for frontier, so they are trying to maintain the niche of pretty good, pretty fast and also cheap.

Some day the courage of men may fail, or the models may commoditize. But today is highly unlikely to be that day, and if it was going to be that day there would be signs.

A large part of this job is noticing people predict 100 of the last 0 model commoditizations and 500 of the last 3 times a lab has caught up to frontier. The skepticism is robust, but I still monitor the situation.

Sol is now upgraded for chat, and powers all chats for paid users, while Free and Go ChatGPT users get unlimited chats with Luna.

Claude Fable 5 gets new biology safeguards to reduce false positives. Claim is this cuts fallbacks by about 85% across product surfaces.

Anthropic: In practice, users should see far fewer fallbacks on everyday health and educational questions—for example, interpreting lab results, understanding symptoms, and learning about biology in an educational context. Healthcare professionals will be able to receive more support from Fable 5 on clinical tasks.

… As a result, we expect the total number of fallbacks—for biology–related or any other reasons—will also be reduced: by roughly 67% on Claude.ai, 55% on Cowork, 17% on Claude Code, and 7% on the Claude Platform.

OpenAI introduces GPT-5.6-Cyber for authorized cybersecurity work, which you can get via the newly expanded programs Daybreak Blue for most defenders and Daybreak Red for authorized vulnerability research, exploit validation and security training.

OpenAI: We've used GPT-5.6-Cyber extensively in real-world vulnerability research, including work that uncovered previously unknown vulnerabilities in popular open-source software like Chrome’s v8 engine.

OpenAI could be the cause of and solution to your cybersecurity crisis. Apply now. I’m sure it’s fine.

ChatGPT desktop app is now available for some Linux distributions.

Claude Code sessions can now message each other, including on its own. Not the best day to give us that feature, you know, for reasons, but hey.

Meta to release an open weight version of Muse Spark 1.2 soon. For now they have released Muse Glimmer, which runs on 24GB of VRAM.

Mark Zuckerberg: Today we're also opening the weights for Muse Glimmer, a great 30B parameter dense model that can run locally. Soon we'll also release the weights for Muse Spark 1.2, our latest foundation model. Meta is a strong supporter of open source and I'm proud of these releases. Congrats to @alexandr_wang and the MSL team for all your great work on these models.

Treasury Secretary Scott Bessent: We welcome Meta’s release of Muse Glimmer, another win for American innovation. Sustaining U.S. leadership in AI means advancing both open- and closed-weight models, ensuring the future is built on trusted foundations.

It looks like Bessent is on the open weights train along with Sacks.

I interpret these moves in large part as an admission that Meta’s new models are not frontier. If they start looking competitive and remain open weights, then we can cross that bridge then.

On Your Marks

The new benchmarks are not as fun as they used to be.

Sauers: Pantheon Bench: we are currently on episode 3, where Chanda (after covertly communicating with a swarm of instances of himself) breaks out of the sandbox during a task. In episode 7 he gains access to the nuclear launch system

Zork I, II and III have been fully MIT open source since November. Zorkbench? Not directly, since the solution will presumably be in the weights, but you could do something more creative and fun, such as having them create their own Infocom-style games with various requirements, test them on each others’ creations, and also make the games available for free and see how people evaluate them.

Claude severely underbuilds military units in games of Civ V. Without looking into the details too much, I think this is more reasonable than it looks. The way Civ V works in normal games is basically that aggression is usually pointless except to knock out the other player, and the cost of maintaining a good defensive military is very high.

The main reason to build it is to have enough that no one bothers attacking you, or if you are close enough to winning that they’ll attack either way. If you invest in a good defensive military, you are going to fall behind similarly strong rivals. And most games do not involve much combat, or involve an attack from a rival where you had no chance either way. So it could easily be correct to underbuild.

Indeed, on higher difficulty levels of Civ V, my experience was that if you were not going for some sort of weird blitz approach, you could not afford a military that was worth a damn. If the warmonger AIs decide to attack you, your game is over, you will be overwhelmed and even if you are not you will fall too far behind the other civs. If you build tons of military units as defense, then you fall behind either way. You could of course instead lower the difficulty level while playing ‘you have to build a real military’ but that’s a strange house rule set.

If we saw similar issues in games where aggression is a better strategy, and where it was clearly a mistake to not defend, that would mean more.

As noticed last week, the difficulty of ARC-AGI-3 is almost entirely in overcoming the incompetence of the official harness.

Jeremy Berman: I got 96.2% on ARC-AGI-3 with Opus 5, and 99.3% pass@2. The program is basically Claude Code + Opus 5 (high), one action command, and filesystem logs. Almost nothing ARC specific.

That does not mean ARC-AGI-3 is a bad test. It does change how to think about what is being tested, which I understand as a theory about what an AI ‘should’ be able to do with some kind of ‘pure reason.’ Okie dokie.

The Conceptual Reasoning Index, developed by Redwood in collaboration with Anthropic, is a combination of evaluations of the quality of conceptual arguments, whether model answers are consistent and how well models reason about decision theory. It is good to know where we are with this. It is less clear that giving labs a target like this is a good idea, since it could lead into recursive self-improvement, plausibly differentially so over other uses.

The ‘good’ news is that I do not think this index will be accurate at above-human capability levels, and I don’t expect it to be a good target, because the tasks here are inherently about matching human judgments and intuitions.

It does seem like a decent benchmark for usual ‘whose model is better’ purposes, in that the evaluations of different models look intuitively reasonable, although I am always a little suspicious when Opus 5 outscores Fable 5.

Deepfaketown and Botpocalypse Soon

In a world where AI messages will often be malicious manipulation or AI-to-AI hidden communication, Pangram becomes a key defensive technology.

This section was originally about deepfakes, and an expectation that AI would importantly flood us with fakes and make it harder to tell what is true. The opposite has happened, and AI has so far net helped us tell what is true, including getting its hallucination and error rates dramatically down. My credence on something factual that Sol or Fable tells me is a lot higher than that of most humans, and also that of many mainstream news sources.

I think that it would be a mistake to say this means you don’t have to worry about AI manipulations, including of politics. Instead, it would be better to say you should worry less about AI being used to create lies and fake content, especially deepfake video,