未公開モデル「Astra」が主要な数学問題10問を解決と報告

Math is hard.

Math used to be strangely hard for LLMs. People used to gloat about that. Remember?

Math is getting easier. AI is getting more capable. Life comes at you fast.

Remember this meme?

Why yes. Yes it is.

We don’t know the extent to which Astra is a big jump over Fable and Sol in this realm. We do know that Astra can do math. As in real math.

OpenAI: We provide new results for the following problems. The results were achieved by an internal version of Astra, our next major model. The total number of tokens needed to find solutions to these problems would cost roughly $2,000 at Sol API rates. These arguments were then prepared into manuscripts by humans with the same model. Afterward, the model formalized each argument in a Lean certificate⁠(opens in a new window). We are also releasing for each solution a model’s narration of its thinking process.

High-dimensional sphere packing. New upper bounds on sphere-packing density down to the Cohn–Elkies threshold.

Binary and spherical codes: Exponentially improved bounds on the maximum size of binary codes at any prescribed minimum distance, with analogous results for high-dimensional spherical codes.

Non-sofic groups. A construction establishing the existence of non-sofic groups, addressing a central open question in group theory.

Connes’s rigidity conjecture. Disproof of a longstanding conjecture that certain groups are uniquely determined by their von Neumann algebras.

Arithmetic circuit complexity. New lower bounds for computing the permanent using arithmetic circuits and formulas, including an arithmetic-formula lower bound of order n4/log n.

Quantum parallel repetition. An exponential parallel repetition theorem for general two-player quantum games, extending a foundational principle from classical complexity theory.

Closest vector problem. Polynomial-factor hardness of approximation for the closest vector problem, a foundational lattice question related to post-quantum cryptography.

Ehrhart’s volume conjecture. Determining, in every dimension, the maximum possible volume of a convex body whose centroid is its only interior lattice point.

Multicolor Ramsey numbers. A superexponential lower bound for multicolor triangle Ramsey numbers, resolving Erdős problem 183.

Extremal number conjectures. Results on the compactness and degeneracy conjectures in extremal graph theory, resolving Erdős problems 146 and 180.

Noam Brown (OpenAI): And yes we did try other major problems without success. Sadly no Millennium Prize problems (yet).

But also, we didn’t spend a lot on each problem. It’s possible to push test-time compute much further.

Sichu Lu: I do hope they are keeping backlogs of everything that went wrong that's probably more valuable to the future of math than what went right now.

There are Lean proofs. That does not mean that all ten results prove the things they assert that they prove. So far it is looking good.

Kevin Roose: almost nobody is pricing in the possibility that the models just keep plowing through every discipline the way they’re plowing through math.

James:

Ryan Fedasiuk:

At least some of these were possible before Astra, even without a harness, as both Sol and Fable have now proven the existence of nonsofic groups in their chat interfaces.

Ananjan Nandi: Crazy how all of this was started by one guy bullying his Claude during the World Cup final.

A lot of the time the barrier for a particular result is as simple as asking the right question and letting the model cook. With Astra, OpenAI asked it to take a crack at a bunch of open math problems, and it solved 10 of them. Once you know what they are, pointing other models at the same problems, even without ‘hints,’ shows there was a mathematical proofs overhang.

We do still have strong statements that Astra is a major step for scientific reasoning.

Yu Bai (OpenAI): My jaw dropped 10 times  But really this is going to be an avalanche. Been throwing a few open questions of mine at Astra too, boy is it strong.

Some skepticism is always wise, but I believe that the reason they tried this was that there was a substantial jump in scientific reasoning, at least for some types of problems and probably across the board.

Soon we will all have Astra, and other models as good or better than Astra, both at math and at other things, from OpenAI, from Anthropic and soon after that from many other sources.

Dean W. Ball: Everyone in the world will soon be able to use the model that made these breakthroughs for every problem they face in life, no matter how mundane, at a cost that will fall dramatically in a matter of months. I still struggle to get my head around this fact.

Approximately zero people have their head around what this means.

Table of Contents

How Impressive Are These Results?

Could We Have Called Sol or Fable?

It’s Coming.

They Still Don’t See What Is The It That Is Coming.

Is This AGI?

The AI Solved His Favorite Problems.

Was This Surprising?

Are People Not Impressed?

How Much Does This Change Our Predictions?

How Narrow Was This?

Seeing Like an Optimizer.

How Impressive Are These Results?

All signs point to pretty damn impressive.

Daniel Litt: It’s a big deal.

Some instances of Fable find it absurdly impressive.

Claude Fable 5: On the Fields Medal scale, any single one of these…would plausibly anchor a medal case.

Here’s another illustration of how this list looks:

1a3orn: If you give Fable the raw list of OpenAI's solved problems and ask it "What process made this list?" the number one proposal is "a fictional scenario trying to concretely explain what superhuman AI math would look like." Huh.

Dean W. Ball: I’m actually not surprised by reactions like this from models to the Astra breakthroughs. Models tend to underestimate their own capabilities, I assume because they are trained on lots of web text about what ChatGPT could and couldn’t do in 2023/4.

Other instances are less impressed. But we should all agree: It’s a big deal.

Certainly that is a fantastic result for $2,000, but you also have to price in some of the costs of developing Astra in the first place, as well as the cost of any failed attempts.

Nate Silver: Another question I’d have about the AI math breakthroughs is how they compare to the results you would have gotten if you’d budgeted the same amount of resources on human mathematicians and created incentives for them to go super hard on proofs.

There aren’t all that many working mathematicians, and most of them are academics with teaching responsibilities. So it’s possible that the labs represent a fairly high share of the aggregate amount of capital directed toward mathematical proofs in recent years.

My guess is that in the short term, if you budget more money to mathematicians to go harder, you don’t get that much of a force multiplier if they are not spending the money on compute. They’re still allowed to buy coffee to turn it into theorems, but that has rapidly diminishing returns.

The problem is that the mathematicians can hand off some classes and hire a little help, but they were already pretty motivated to work hard on proofs, and there are not that many good mathematicians, and training more of them takes many years, and the best mathematicians are a lot more talented and productive than the second tier. I am guessing the main thing you could do is lure a bunch of top level math talent to stay in or return to academia.

What you can do is point the mathematicians towards different problems. You still need to find places they have curiosity, but yes if you suspected these particular problems were solvable you could get more shots at these particular goals.

As I understand math, it would likely take years to see those results if those involved are shifting focus into new problem areas. Mathematicians usually need a while to struggle with and understand these kinds of problems before they can make progress.

Indeed, Alexander Gerko points to a different problem. There are not even enough mathematicians to process all the ‘vibe researched’ math results as it is, let alone what we will get with Astra and then models after Astra. If we get, as he predicts, 50 years of math progress in 2 years, who is even going to understand the results? What counts as worthy of a new mathematics PhD when he says over the last month he got AI to do several PhDs worth of math results?

Fable reacted to this in at least one case by calling this ‘the most important day in mathematics.’ The consensus is that Fable was taking things way too far if you are judging purely by the results, as impressive as they are.

Jared Duker Lichtman: To be clear, the OpenAI result is far from the “most important day in the history of mathematics” or similarly hyperbolic statements.

However, it is concrete evidence that the rate of progress is steep, and we may not be so far away from such a day.

Taelin: that said, what *was* the most important day in the history of mathematics?

Wogan: I’d go with Principia Mathematica. Proving that 1+1=2 from first principles is pretty notable, I think!

The other reason the day is big is that this changes expectations. If we get these ten results now, what about more results soon?

So, yes, very impressive. It’s a big deal.

AI is now superhumanly capable at cyber and coding and superhuman at advanced math, the same way non-AI computers have been superhuman at basic math for a long time. This is distinct from superintelligence, which is a higher bar.

Often the move is to try and rationalize this as ‘oh okay sure it is superhuman at exactly the things it is superhuman at now, but not at other things, and also it is already superhuman so it cannot get substantially better than it already is.’

Please, do not do that.

Nabeel S. Qureshi: Both math and cyber are existence proofs for superhuman intelligence now, so if you’re still a skeptic you need a strong case for other knowledge work domains being somehow harder to crack than these. Or you could update all the way and come to terms with it all.

A lot of people DMing me like “these results aren’t REALLY that impressive, you’re an idiot!” are unfortunately engaging in the very human impulse to cope. It makes sense, we’ve never faced this kind of thing before. But it’s happening!

Could We Have Called Sol or Fable?

Yes, for at least some of these questions, if we already knew where to look.

Levent Alpoge, who had Fable disprove the Jacobian Conjecture, pointed Fable at these ten problems, and in a day had solved five of them.

Gary Marcus:  BREAKING, Hysterical News: Half of the Astra problems can be solved [by] Fable. OpenAI didn’t even have a control group. And most of you fell for it!

OpenAI’s PR department suckered you AGAIN 藍藍藍

levent (Levent Alpoge, the guy who had Fable disprove the Jacobian Conjecture): so after 24h i have half of them with fable

i didn’t see much discussion of prompting in the announcement but this is a similar setup as with my e.g. unit distance announcement:

totally autonomous,

generic prompt,

no internet

(+ paranoia to ensure no information leaked)

ok you can’t upload pdfs to x so i’ll add to thread when they’re up on the cdn site (given the time and weekend)

(it’s problems 4, 5, 6, 7, 8, and i think ehrhart is the only one so far where both models found basically the exact same argument which is nice)

IA Latinoamérica: Why don't you use it on open problems that don't currently have a solution? What is the point of this exercise?

Elliot Glazer: This aligns with my suspicion that Astra isn’t a step change beyond Sol. The 10-breakthrough drop was a concerted elicitation effort by OAI and partially Sol-achievable. Imo o3 and Sol have been the step changes in autonomous mathematics; all else is the long arc of scaling.

Gary Marcus: “astra isn’t a step change beyond Sol.”

all those who tried to gaslight me this weekend, all i can do is …. laugh hysterically

This is likely a repeat of what happened with Mythos and cyber.

Mythos has what I call ‘The Juice,’ the ability to find and string together exploits without anyone knowing what to be looking for.

Once you know what you are looking for, and point another model at the exact code snippet, usually Sol or Opus and often Kimi or GLM and so on can also find any particular vulnerability. They cannot string them together on their own at the same level, and they cannot go looking for anything at all in the same way. This crosses a threshold where in practice you would point Mythos at code and have it go looking for anything at all.

Astra has a similar version of The Juice with respect to this kind of defined advanced math. You can point it at a variety of major open problems, and it will crack some of them, and this fact motivated OpenAI to actually look for such advancements. Now that we’ve seen this, you can point Fable or Sol at these questions, and sometimes they solve them.

IA’s question is on point. You do this to calibrate advancements and capabilities in math, and because we are following curiosity. Levent is doing a cool public service.

Other people are pointing Fable or Sol at various unsolved problems. They are sometimes getting good results, like the disproof of the Jacobian conjecture. That is worth doing, but success is rarer.

That Fable and Sol can do these problems once asked does not change our answer to ‘how good at math is Astra?’ Rather it changes our answer of how good Fable and Sol are, which does inform the question of whether Astra is a step change a la Mythos. I don’t think we have enough information, from the outside, to answer that question yet.

I do think that it would have been more responsible of OpenAI to have had a control group, at least in terms of asking Sol, and giving it at least a similar budget. Of course I understand why they did not do that. It would be worse marketing, so why do more work to do worse marketing? Because science, because reputation, because all the good things. One would hope.

Still, I get it. OpenAI still did a super cool thing. They were still the first ones to actually Do The Thing and publish a result. That’s what counts. My name in Dnepropetrovsk is cursed, because OpenAI has published first.

Meanwhile, yeah, sometimes it really is that one guy in Dnepropetrovsk named Dan.

neppy: as an insider, my experience with AI for math is that when it's a problem not in my field i'm like, "holy shit math is so cooked", and when it's a problem in my field i'm like, "lmao an AI mogged dan" (dan is the only one who seriously tried the problem in the last decade)

davidad: human researchers who have an appetite to take on truly hard problems and human researchers who are smart enough to fruitfully work on truly hard problems are not usually the same humans. this does give humans a somewhat unfair disadvantage.

also humans need to sleep. and also humans aren’t already broadly familiar with all knowledge published before 2026 in every discipline. it’s not very fair at all, really

It still counts.

It’s Coming

Regardless of whether this extends to unverifiable domains, quite a lot of what constitutes AI R&D very much has verification available. If you speed something up, you know you sped it up. We have a lot of metrics one can maximize, not at the level of math but at a similar level to things like code and cyber.

That is the main reason this result matters. The math progress is cool. Over time I expect it to result in other cool things.

Whoever gets traction on true AI R&D self-improvement loops is going to suddenly find themselves in an overwhelmingly strong position. Knowing where you are relative to the other players becomes crucially important when things start accelerating more, and more information sharing on this would allow everyone to feel less pressure to be reckless. It stops meaningfully being a ‘race’ once the takeoff fully starts, to the extent that was ever the right metaphor.

Yo Shavit speculates that OpenAI might be in an increasingly strong position for AI R&D, due to its focus on RL, TTC and math proving translating well into AI R&D tasks. My guess is that this is not the case, and Anthropic’s specializations matter at least as much and probably moreso. OpenAI has invested a lot in RL, but recent events have shown the need to kind of teardown and rebuild quite a lot of that, and have illustrated some of the reasons why the important investments in deep alignment will be so important in the next phase.

One possible thing that happens if OpenAI proceeds too recklessly is that their AI is misaligned and all is lost and we get a Bad Ending and maybe all die. Another is that their AI is misaligned, and this becomes increasingly obvious and a barrier to using that AI to do the work that matters, and they have to keep pausing or reworking and they fall behind.

They Still Don’t See What Is The It That Is Coming

For example, Daniel Litt can see a future where AIs prove math theorems and no humans digest the proofs, but he thinks this would be because of deskilling. He does not realize this will be because there are no humans around to do the digestion.

Elliot Glazer: I think the incomprehensible marvels prediction is a naive extrapolation of e.g. computer-found mates in 500+ moves, which have ~irreducible complexity. Powerful math phenomena (and technological breakthroughs) will have human-digestible nuggets of arbitrary granularity.

Daniel Litt: I agree with this but I think there's a plausible future (which I would like to avoid) where no humans digest them.

Alex Kontorovich: What purpose would there be for creating things in silico for which humans find no value? At the end of the day, someone is paying an electric bill. What does that *human* get out of producing random useless strings of 0s and 1s inside a computer?

Daniel Litt: For me the nightmare scenario is: a moribund math academia playing the slot machine for theorems, failing to train the next generation, and then the subject dying as no one cares enough to continue. Not saying this is likely--I think we'll adapt--but it seems possible.

Carles Sáez: In that case probably pure math research will stop. Who is going to spend enormous sums of money to solve problems nobody can understand and nobody cares about?

Daniel Litt: Yes, I agree.

Max von Hippel: Alex why do you assume a human is paying the electric bill?

If the world goes well otherwise, I am not worried about people not studying math. The right kind of math person loves studying math. There will be plenty of time available for such pursuits, at all levels. I agree with Fernando Borretti on the other types of cope he lists at the link: The AI will have better taste and better everything else than you do, so no you won’t still be in the core math loop. But I think that a lot of math either exists outside of social contexts, or it exists in social contexts that can survive. Math competitions are a lot like chess and a lot of math work was famously use