OpenAI #16：歴史と提案

The real news today is that Anthropic has partnered with the top companies in cybersecurity to try and patch everyone’s systems to fix all the thousands of zero-day exploits found by their new model Claude Mythos.

I’ll be sorting through that over the coming days. For now, we instead have stories from OpenAI.

In particular there are three stories.

There’s a massive 18,000 word article in The New Yorker about Sam Altman and the history of OpenAI as it relates to his trustworthiness. No trust.

There’s also OpenAI’s proposal for a ‘new deal’ of sorts. No deal.

Then there is an actual deal, where they bought TBPN. RIP.

Table of Contents

Part 1: OpenAI: The Histories.

The Battle of the Board.

Thanks For The Memos.

I Am What I Am.

That’s Not What I Said.

There Will Be No Investigation.

Musk Versus Altman.

Amodei Versus Altman.

Sydney Versus Altman.

Highest Bidder Versus Altman.

Risky Business.

Superalignment Was Always Fake.

This Is Fine.

Liar Liar Master Persuader.

This In Particular Is Securities Fraud.

Regulation Two Step.

Easy Mode.

The Right Amount of Alignment Research Is Not Zero.

OpenAI Proposes Policy.

RIP TBPN.

Part 1: OpenAI: The Histories

If you ask a question in a headline, the answer is almost always no.

This week’s example, from Ronan Farrow and Andrew Marantz at The New Yorker is ‘Sam Altman May Control Our Future—Can He Be Trusted?’

To which in earlier days Altman himself replied no, and this week he replied with a PR workshopped answer that was still ultimately no, and also very obviously the answer to this question is no.

The article is less about the question in the headline - we all know the answer to that one - and more about creating a profile of Altman, laying out the history of various incidents in Altman’s and OpenAI’s past, and litigating basically everything.

Mostly I found the article fair, and definitely fully within the rules of Bounded Distrust, but it is certainly in the ‘make it look suspicious’ school of journalistic description. I think they were trying to be fair, and mostly succeeded, but there are a decent number of things that could have been cut, or where the framing is more of a gotcha than I would like.

It is 18k words long and covers a lot of stuff we’ve covered extensively before, so I’ll skip over a lot. It is well-written, in case you want to consider reading the whole thing.

I will instead attempt to hit the highlights.

The Battle of the Board

We start out with the Battle of the Board, as the central event in OpenAI’s history.

All the reporting here was consistent with my past reporting. There were a handful of new meaningful details or confirmations.

One ongoing question is how Altman reacted to the initial firing. Altman claims he was going to accept the firing and then others fought back, eventually getting him to get his head back in the game. That’s not the picture here.

Ronan Farrow and Andrew Marantz: The day that Altman was fired, he flew back to his twenty-seven-million-dollar mansion in San Francisco, which has panoramic views of the bay and once featured a cantilevered infinity pool, and set up what he called a “sort of government-in-exile.”

… Altman interrupted his “war room” at six o’clock each evening with a round of Negronis. “You need to chill,” he recalls saying. “Whatever’s gonna happen is gonna happen.” But, he added, his phone records show that he was on calls for more than twelve hours a day.

Thanks For The Memos

We get confirmation that then-board-member Ilya Sutskever assembled seventy pages of Slack messages to show why Altman and Brockman shouldn’t be running OpenAI, which out of fear of Altman he alas sent as disappearing messages, and then the board attempting to fire Altman. It is a shame that we cannot see it directly.

I Am What I Am

This is quite the quote, which Altman doesn’t recall but doesn’t deny.

Ronan Farrow and Andrew Marantz: In a tense call after Altman’s firing, the board pressed him to acknowledge a pattern of deception. “This is just so fucked up,” he said repeatedly, according to people on the call. “I can’t change my personality.”

No, I suppose you can’t, sir. And yes, I agree, that was all pretty fucked up.

Altman says he is under tons of stress and suffering from decision fatigue. On that point I fully believe him, and I sympathize.

That’s Not What I Said

Paul Graham disputes the account given here of how Altman left YC, but his claim that the post’s account is false seems to be an inaccurate statement about what the post’s account actually said, and fails to contradict it.

There Will Be No Investigation

In a later section they outline how Altman got Summers and Taylor to not do their promised investigation of Altman’s previous actions, focusing only on narrow claims about specific potential criminality and looking for a way to ‘acquit’ Altman and get things back to normal. No criminal violations? Then get back to work. The investigation was mostly fake and failed to even release a report.

Many former and current OpenAI employees told us that they were shocked by the lack of disclosure. Altman said he believed that all the board members who joined in the aftermath of his reinstatement received the oral briefings. “That’s an absolute, outright lie,” a person with direct knowledge of the situation said.

Some board members told us that ongoing questions about the integrity of the report could prompt, as one put it, “a need for another investigation.”​

Look at those nerds, acting as if words have meaning and truth matters. Fools.

Musk Versus Altman

The story of the falling out with Musk seems consistent with Altman’s version of events, and with my previous understanding, and being inconsistent with the story Musk tells. For all the places I think Altman is inconsistently candid, I think his telling of these events is the more accurate one.

The thing that stuck out most was getting this in detail:

But, unbeknownst to them, he also struck a secret handshake deal with Brockman and Sutskever: Altman would get the C.E.O. title; in exchange, he agreed to resign if the other two deemed it necessary. (He disputed this characterization, saying he took the C.E.O. role only because he was asked to. All three men confirmed that the pact existed, though Brockman said that it was informal. “He unilaterally told us that he’d step down if we ever both asked him to,” he told us. “We objected to this idea, but he said it was important to him. It was purely altruistic.”) Later, the board was alarmed to learn that its C.E.O. had essentially appointed his own shadow board.

By this point, we all know what an informal agreement like that is worth.

Amodei Versus Altman

This detail seems important in the story of how and why Dario Amodei left OpenAI.

Amodei presented Altman with a ranked list of safety demands, placing the preservation of the merge-and-assist clause at the very top. Altman agreed to that demand, but in June, as the deal was closing, Amodei discovered that a provision granting Microsoft the power to block OpenAI from any mergers had been added. “Eighty per cent of the charter was just betrayed,” Amodei recalled.

He confronted Altman, who denied that the provision existed. Amodei read it aloud, pointing to the text, and ultimately forced another colleague to confirm its existence to Altman directly. (Altman doesn’t remember this.)

Correction: Altman claims he does not remember this.

After that, one can see how things might escalate quickly.

Ronan Farrow and Andrew Marantz: Amodei’s notes describe escalating tense encounters, including one, months later, in which Altman summoned him and his sister, Daniela, who worked in safety and policy at the company, to tell them that he had it on “good authority” from a senior executive that they had been plotting a coup. Daniela, the notes continue, “lost it,” and brought in that executive, who denied having said anything.

As one person briefed on the exchange recalled, Altman then denied having made the claim. “I didn’t even say that,” he said. “You just said that,” Daniela replied. (Altman said that this was not quite his recollection, and that he had accused the Amodeis only of “political behavior.”) In 2020, Amodei, Daniela, and other colleagues left to found Anthropic, which is now one of OpenAI’s chief rivals.

Sydney Versus Altman

Here’s one confirmation, of the anecdote that when someone at Microsoft tested an early version of GPT-4 in India without safety board approval, in ways that ended up having permanent ramifications via embedding the concept of Sydney into the training data, he neglected to even inform the board that this had happened.

As McCauley, the board member and entrepreneur, left the meeting, an employee pulled her aside and asked if she knew about “the breach” in India. Altman, during many hours of briefing with the board, had neglected to mention that Microsoft had released an early version of ChatGPT in India without completing a required safety review. “It just was kind of completely ignored,” Jacob Hilton, an OpenAI researcher at the time, said.​

Various examples are cited of people raising concerns about OpenAI and Altman’s lack of actual commitment to safety, and prioritization of profits over safety.

Highest Bidder Versus Altman

They lay out the whole ‘let’s offer it to Putin and auction AI to the highest bidder.’

​[Hedley] was aghast: “The premise, which they didn’t dispute, was ‘We’re talking about potentially the most destructive technology ever invented—what if we sold it to Putin?’

… Brainstorming sessions often produce outlandish ideas. Hedley hoped that this one, which came to be known internally as the “countries plan,” would be dropped. Instead, according to several people involved and to contemporaneous documents, OpenAI executives seemed to grow only more excited about it. Brockman’s goal, according to Jack Clark, OpenAI’s policy director at the time, was to “set up, basically, a prisoner’s dilemma, where all of the nations need to give us funding,” and that “implicitly makes not giving us funding kind of dangerous.” A junior researcher recalled thinking, as the plan was detailed at a company meeting, “This is completely fucking insane.”

Executives discussed the approach with at least one potential donor. But later that month, after several employees talked about quitting, the plan was abandoned. Altman “would lose staff,” Hedley said. “I feel like that was always something that had more weight in Sam’s calculations than ‘This is not a good plan because it might cause a war between great powers.’ ”

Brockman claims he was not serious, but this seems like well-documented reporting that it was actually rather serious. I think you should be rather alarmed that this got taken seriously as a proposal.

I won’t spoil tales of various other attempts to raise money. They’re not pretty.

I will note his randomly announcing, on Twitter, his intent to raise trillions.

In February, 2024, the Wall Street Journal published a description of Altman’s vision for ChipCo. He conceived of it as a joint entity funded by an investment of five to seven trillion dollars. (“fk it why not 8,” he tweeted.) This was how many employees learned about the plan. “Everyone was, like, ‘Wait, what?’ ” Leike recalled. Altman insisted at an internal meeting that safety teams had been “looped in.” Leike sent a message urging him not to falsely suggest that the effort had been approved.​

Risky Business

One common throughline of the post is Altman’s evolving positions and communications concerning AI risks, including AI existential risks, as he moved from being a relatively good voice to a relatively poor voice.

When it was useful to him to speak about risks, he played them up. When it was useful for him to downplay them, as he increasingly finds it now, he downplays them, often using similar language such as emphasizing different aspects of a ‘Manhattan Project.’

I do think Altman was being more candid for longer than was efficient for the business, but this is not obvious. It’s possible he followed incentives the whole way, if he had to focus a lot on employee retention. OpenAI was relying, from the start, on employees thinking it was an especially noble place to work.

Altman is quoted warning about AI dangers before it was cool, and back when such statements did not conflict with his business. According to this account, it was Altman who convinced Musk to contribute, on the basis that Google should not be allowed to be the one to do AI.

One thing emphasized is that ‘we are the good guys doing the right thing’ was absolutely a key selling point, even the key selling point, to getting OpenAI off the ground and recruiting key initial people, including Ilya Sutskever. This greatly reduced cost of capital.

This was centered around making it clear that AI going wrong carried big risks, and in particular that the AIs getting out of our control could be an existential risk to humanity, but also concentration of power and other risks.

He wrote on his blog in 2015 that superhuman machine intelligence “does not have to be the inherently evil sci-fi version to kill us all. A more probable scenario is that it simply doesn’t care about us much either way, but in an effort to accomplish some other goal . . . wipes us out.”

OpenAI’s founders vowed not to privilege speed over safety, and the organization’s articles of incorporation made benefitting humanity a legally binding duty. If A.I. was going to be the most powerful technology in history, it followed that any individual with sole control over it stood to become uniquely powerful—a scenario that the founders referred to as an “AGI dictatorship.”

Altman told early recruits that OpenAI would remain a pure nonprofit, and programmers took significant pay cuts to work there.

Altman continued to use OpenAI’s commitment to safety, and promises of grand investments and gestured, as a recruiting tool. The article cites a clear example from late 2022, where he talks of things he did not do.

Superalignment Was Always Fake

We then get to the grand ‘superalignment team.’

But, in the course of several meetings in the spring of 2023, Altman seemed to waver. He stopped talking about endowing a prize. Instead, he advocated for establishing an in-house “superalignment team.”

An official announcement, referring to the company’s reserves of computing power, pledged that the team would get “20% of the compute we’ve secured to date”—a resource potentially worth more than a billion dollars. The effort was necessary, according to the announcement, because, if alignment remained unsolved, A.G.I. might “lead to the disempowerment of humanity or even human extinction.”

Jan Leike, who was appointed to lead the team with Sutskever, told us, “It was a pretty effective retention tool.”

The twenty-per-cent commitment evaporated, however. Four people who worked on or closely with the team said that the actual resources were between one and two per cent of the company’s compute. Furthermore, a researcher on the team said, “most of the superalignment compute was actually on the oldest cluster with the worst chips.”

The researchers believed that superior hardware was being reserved for profit-generating activities. (OpenAI disputes this.) Leike complained to Murati, then the company’s chief technology officer, but she told him to stop pressing the point—the commitment had never been realistic.

… But the superalignment team was dissolved the following year, without completing its mission.

OpenAI and Altman continue to attempt to memory hole that this happened, and that this promise was both valuable and load bearing and then massively broken.

I am always especially enraged when:

People make you a promise of [X], which was their own idea.

You act on the basis of [X].

Nothing changes.

They don’t give you [X].

When asked why, they say [X] was never realistic.

This happened to me once, in a deeply similar way, where someone offered to fund a potential new company in a particular way, then after efforts were made told me (paraphrased) that it wasn’t realistic, and offered a dramatically worse deal, right after I had subsequently invested time and blown up a bunch of things including a key friendship that took me years to repair, and other opportunities had been lost. Should I have predicted it was reasonably likely to go down that way? Oh, absolutely, but seriously, not okay.

This Is Fine

This is a good summary of the switch that happened over time:

Last June, on his personal blog, Altman wrote, referring to artificial superintelligence, “We are past the event horizon; the takeoff has started.” This was, according to the charter, arguably the moment when OpenAI might stop competing with other companies and start working with them.

But in that post, called “The Gentle Singularity,” he adopted a new tone, replacing existential terror with ebullient optimism. “We’ll all get better stuff,” he wrote. “We will build ever-more-wonderful things for each other.” He acknowledged that the alignment problem remained unsolved, but he redefined it—rather than being a deadly threat, it was an inconvenience, like the algorithms that tempt us to waste time scrolling on Instagram.​

Liar Liar Master Persuader

Another common theme is that yes, Altman lies quite a lot. This is not news.

The problem was that these lies, in isolation, are not smoking guns. Altman is smart. But if you add up the pattern, the result is clear.

Neither collection of documents contains a smoking gun. Rather, they recount an accumulation of alleged deceptions and manipulations, each of which might, in isolation, be greeted with a shrug: Altman purportedly offers the same job to two people, tells contradictory stories about who should appear on a live stream, dissembles about safety requirements. But Sutskever concluded that this kind of behavior “does not create an environment conducive to the creation of a safe AGI.” Amodei and Sutskever were never close friends, but they reached similar conclusions. Amodei wrote, “The problem with OpenAI is Sam himself.”

… most of the people we spoke to shared the judgment of Sutskever and Amodei: Altman has a relentless will to power that, even among industrialists who put their names on spaceships, sets him apart.

“He’s unconstrained by truth,” the board member told us. “He has two traits that are almost never seen in the same person. The first is a strong desire to please people, to be liked in any given interaction. The second is almost a sociopathic lack of concern for the consequences that may come from deceiving someone.”

The board member was not the only person who, unprompted, used the word “sociopathic.”

As others have pointed out, contra the board member, those two traits are seen in the same person reasonably often. There is no contradiction between maximizing likeability in each interaction and engaging in reflexive deception. Indeed, one could even call (a lesser version of) this a common cultural attitude in various parts, especially in California.

The goal is to instill good vibes and keep everyone thinking and doing whatever you need them to be thinking and doing, you are mostly indifferent to whether the statements required for this are true and you don’t worry so much about the reputational consequences or what happens if your contradictory statements are pointed out.

Altman simply extends this common pattern to an extreme extent.

“He’s unbelievably persuasive. Like, Jedi mind tricks,” a tech executive who has worked with Altman said. “He’s just next level.” A classic hypothetical scenario in alignment research involves a contest of wills between a human and a high-powered A.I. In such a contest, researchers usually argue, the A.I. would surely win, much the way a grandmaster will beat a child at chess. Watching Altman outmaneuver the people arou