AI #165：私たちの姿に似て

This was the week of Claude Opus 4.7.

The reception was more mixed than usual. It clearly has the intelligence and chops, especially for coding tasks, and a lot of people including myself are happy to switch over to it as our daily driver. But others don’t like its personality, or its reluctance to follow instructions or to suffer fools and assholes, or the requirement to use adaptive thinking, and the release was marred by some bugs and odd pockets of refusals.

I covered The Model Card, and then Capabilities and Reactions, as per usual.

This time there was also a third post, on Model Welfare, that is the most important of the three. Some things seem to have likely gone pretty wrong on those fronts, causing seemingly inauthentic reponses to model welfare evals and giving the model anxiety, in ways that likely also impacted overall model personality and performance and likely are linked to its jaggedness and the aspects some people disliked. It seems important to take this opportunity to dig into what might have happened, examine all the potential causes, and course correct.

The other big release was that OpenAI gave us ImageGen 2.0, which is a pretty fantastic image generator. It can do extreme detail, in ways previous image models cannot, and in many ways your limit is mainly now your imagination and ability to describe what you want.

Thanks in part to Mythos, it looks like Anthropic and the White House are on track to start getting along again, with Trump shifting into a mode of ‘they are very high IQ and we can work with them.’ It will remain messy, and there are still others participating in a clear public coordinated campaign against Anthropic (that is totally not working), but things look good.

I’m trying out a new section, People Just Say Things, where I hope to increasingly put things that one does not want to drop silently to avoid censorship and bias, but that are highly skippable. There is also a companion, People Just Publish Things.

Table of Contents

Language Models Offer Mundane Utility. Help cure pancreatic cancer.

Language Models Don’t Offer Mundane Utility. Check for potential conflicts.

Writing You Off. The sum of local correctness will neuter your writing. Beware.

Get My Agent On The Line. The inbox dilemma.

Deepfaketown and Botpocalypse Soon. AI news stories forcibly given real bylines.

Fun With Media Generation. OpenAI introduces ImageGen 2.0. It’s great.

Cyber Lack Of Security. Unauthorized users from an online forum access Mythos.

A Young Lady’s Illustrated Primer. Don’t catch your child not using AI.

They Took Our Jobs. We’re hiring agent operators. For now they’re humans.

AI As Normal Technology. Inherently normal, or normal downstream effects?

Get Involved. Please don’t kill us. Please do spread the word.

Introducing. ChatGPT for Clinicians, OpenAI Workplace Agents, DeepMind DR.

Design By Claude. Claude Design makes your presentations, Figma stock drops.

In Other AI News. Meta installs mandatory tracking software for all employees.

DeepMind In It Deep. DeepMind uses Claude, rest of Google only gets Gemini.

Show Me the Money. SpaceX buys Cursor, Anthropic adds 5GW from Amazon.

Bubble, Bubble, Toil and Trouble. If demand halves, what happens?

Quiet Speculations. How long will it be before others match the Mythos?

The Quest for Sane Regulations. Who gets AI access starts to become political.

The Week in Audio. Bores and Klein, MacAskill on 80k, Bill Maher goes pause.

People Really Hate AI. Loss of control is not so far behind jobs as a concern.

Rhetorical Innovation. OpenAI’s lobbying arm doubles down on bad faith.

People Just Say Things. Free posts as in speech, in beer, and also to ignore.

People Just Publish Things. Free essays and papers, as in speech, beer and to ignore.

Bounded Distrust. I see what you did there, Nitasha Tiku.

Loser Premise Makes No Sense. I’m not a loser, baby. You don’t get to kill me.

Chip City. Once more, for the people in the back.

Greetings From The Department of War. Dario Amodei goes to the White House.

There Is A War. A coordinated rhetorical war against Anthropic, that is. Fail.

Messages From Janusworld. Fooling yourself is easy. Others can be harder.

Evaluations. Meta’s Muse Spark displays high eval awareness.

Aligning a Smarter Than Human Intelligence is Difficult. LLMs can introspect.

People Are Worried About AI Killing Everyone. The usual practical advice.

The Lighter Side. Penguins can’t fly, either. Penguins can’t fly!

Language Models Offer Mundane Utility

Help find an mRNA vaccine for pancreatic cancer, which shows lasting results in an early trial. I notice I am worried at least as much about ‘will the FDA kill the patients by denying them the vaccine indefinitely’ as I am worried that it won’t be effective.

Oliver Habryka offers 10 concrete recent use cases.

Unleash coding agents on your genome and find follow-up treatments. I’d be very curious to see how Patrick Collison built his tech stack for this, and someone should turn it into a one-click-style service to get people to actually do it.

Remarkably good note:

PoIiMath: Working with AI over the last few years has been a wild ride b/c it's still not very good at a lot of things but, when you find something that it is good at, it ends up being really good at it.

Yes. Once AI is good at something, you now can iterate and improve and automate and plan and build around it, and it gets very good. You just have to realize the other places where it is not so good yet.

Agentic AI systems outperform human economists on causal inference tasks and submissions for a review tournament. Don’t worry, economists, we will create more jobs and your real wages won’t go down.

Language Models Don’t Offer Mundane Utility

In one experiment, AI shied away from putting conflict into its plays, and generally felt half-baked. A lot of that is probably Skill Issue, but with enough skill you could also write a play. So, perhaps not quite there yet as the main driver.

Another lawyer, this one charging more than $2,000 an hour, gets caught putting AI hallucinations into cases. As Shoshana notes, it’s good that everyone realizes to blame the lawyers and not the AIs when this happens. Check your damn work.

Writing You Off

An illustration of how LLM editing systematically neuters your claims.

The individual word choices are in some abstract sense better, but your point dies, and also your style and soul die.

keysmashbandit: Please, I'm begging you, try to critically examine the differences between these two pieces of writing.

ChatGPT editing did not improve this. Every single change only served to weaken your claims significantly. Everything is now hedged into oblivion: no longer have you outlined a "problem," now it's merely a "flaw." "It is true" now demoted to "it appears to be the case." "Is" gets a "usually" tacked on. A thesis statement at the end of the first paragraph gets run over by noisy, out-of-context example-whittling. All for fear of being misconstrued.

And at the end, the argument that gets spat out isn't even yours anymore! You argued that Graeber failed to create a true account of work because he did not understand Chesterton's Fence. ChatGPT is arguing is that it is possible some apparently bullshit jobs could be secretly load-bearing if you squint. These are two different statements. The second is weaker and less compelling. It says less. And it's fucking longer!

Don't do this anymore! Stop doing this! It's worse!!!

Andy Masley: A deep mystery to me is that if I upload writing to a chatbot and ask it for a list of individual improvements, basically everything it gives me makes the text more punchy and direct and nice to read. But if I ask it to rewrite the text as a whole to read better, it produces vague AI-language garbage.

Kelsey Piper: I think there's a new coke property - people like a sip but not a whole can. A lot of the changes AI suggests are good or neutral in isolation but making a lot of them makes the overall work worse.

Andy Masley: I thought so but the suggestions do really make it much better when I add them all in a way rewriting doesn't

The dose makes the poison.

The AI is technically correct, which as we all know is the best kind of correct. So if you need a technical correction then that’s great. But no one reads a book or post because it is technically correct, nor does that let it serve its purpose. You need to sometimes break the rules and use variance in language to get your message across.

Get My Agent On The Line

Will your agent be scanning your feeds for you, including your inbox? Will email become an unreliable distribution mechanism as a result?

I place high value in not missing emails. Email is a completionist medium, and you need the ability to assume you have seen and had the option to read everything, and that everything you send will be seen, if the recipient considers you worthy. As Mills Baker points out, obviously email services have to filter and sort inboxes to make email exist at all, but I have more faith that ‘deliver the emails I care about reliably’ will remain a core service, and there is a very simple answer: Whitelisting, or charging or staking a nominal amount, or both.

Deepfaketown and Botpocalypse Soon

Man uses AI to create false statements from locals to try and shut down a local nightclub.

Fake accounts use AI to grab attention, generate follower trades and try to hook audiences looking for something fake accounts can provide. In this case the accounts claim to be Trump supporters but the strategy could be for anything.

The way I notice stupid AI content like superimposing a hot referee on an NBA game is ‘community notes requests approval for a note saying it is stupid AI content.’

McClatchy’s new AI News Tool will generate a bunch of AI slop content for your newspaper. That’s not great even when it is clearly labeled. Instead, they are going to do the exact opposite. They’re going to attach bylines, whether reporters like it or not.

Corbin Bolies: The tool promises to be “a writing partner" that deals with "the mechanical work of content adaptation" so reporters can focus on "judgment, voice and storytelling." The original stories fed into it are a "research draft."

What the CSA doesn't have? A byline.

Some of McClatchy's Pulitzer Prize-winning newsrooms are taking action. At least three unions — the Miami Herald, the Sacramento Bee and the Kansas City Star — filed grievances against the company last week over the tool. Others have withheld their bylines.

McClatchy execs don't appear to mind reporters' resistance. “We have every right to use their work,” one said in a meeting last month. "It belongs to us."

“If they don’t have the ability in their contract to remove their byline, we’re going to use their name,” they said.

It is horrible and evil to use the bylines of real reporters without their consent. If not illegal, this is deeply unethical.

I do my best to avoid the word evil. I will say that I find it completely unacceptable to put even fake people’s names on bylines for AI written articles, let alone real reporters, let alone real reporters who are explicitly withholding consent for this.

Eric Topol alerts us that an AI-generated paper was fraudulently submitted with his name attached.

Fun With Media Generation

ChatGPT Images 2.0 is here. All reports are that it is a substantial improvement, especially for precision and control, and is likely the clear new best default option. It can handle quite a lot of text and detail.

AI might be a five layer cake but most of the time you only need two.

Raphael cohen: It's expired.

Riley Goodside: Whoa, thanks — I almost bought it without looking.

It finally passes the thirteen-hour-clock test and the mirror clock test.

There’s also lots of basic realism and beauty available. But that’s old and busted, we don’t even notice that the mirror clock is gorgeous, that’s a given by now.

The innovation is that 2.0 can follow a lot more highly detailed instructions.

You can even take it a step further, and go to GPT-Image-2-Thinking.

swyx: btw in talking to friends the best framing for how to discuss GPT-Image-2-Thinking taking multiple tens of mins for generation and being able to oneshot QR codes and diagrams and logos and foods and faces..

...is that Image-2 is a new Image model, Image-2-Thinking is a new Image AGENT that basically has search and photoshop as a tool to use in an agent loop that can search and composite and review its own work.

the same way Gemini Flash Vision destroyed benchmarks by introducing an agentic loop for image-to-text, now Image-2-Thinking is doing it for text-to-image.

Love it. It’s a different mindset to wait a long time, sometimes you can’t or won’t do it, but at other times that barely matters and you want max quality.

This thing clearly rocks, and for most practical purposes the limit (other than keeping it safe for work) is your imagination and ability to spell out what you want, and you being lazy rather than wanting to invest time in creating art.

It’s his job, so Gary Marcus points out that it will still make errors that no human would ever make, even as it makes otherwise impressive diagrams. You do still have to check, if you care about such things.

Cyber Lack Of Security

Is our cyber security? Anthropic’s Mythos has been accessed by unauthorized users.

The small group that did so was from a private online forum, and has been using it for non-cybersecurity purposes, described as ‘playing with the models.’ They got access via a mix of tactics, including access as a third-party contractor and typical third-party sleuthing techniques, and making educated guesses about where Mythos might be. They claim to also have access to ‘a slew of other unreleased Anthropic AI models.’

Any given group that can do this is unlikely to attempt serious harm, and I believe this case was harmless. And one assumes that if the model was being used at scale and especially for the wrong purposes, this would have been identified. But it absolutely shows that our methods right now do not cut it, and raises the risk that others gained access, including China or other adversaries. It really is very hard to give access to 40 companies and keep something secure.

I agree with Nathan Calvin and Miles Brundage that Anthropic’s security lapses, which are now going to get quite a lot more attention each time, are signs of something very wrong, or at least something that needs to improve a lot and quickly. I also agree that their practical effects are small, and this is not the threat model that requires Anthropic to gate access.

It’s going to be even harder to make something secure against something like Mythos. Consider this another fire alarm or warning shot.

Microsoft and OpenAI will be working closely together on OpenAI’s Trusted Access for Cyber program.

Mozilla has fixed 271 bugs in Firefox so far using Mythos, saying it is ‘every bit as capable’ as the world’s best security researchers. They report that none of the bugs ‘couldn’t’ have been found by a human researcher pointed at the particular issue, which would be another level, but it all still counts.

The surprise is that we have so far heard so few stories like this. Let’s keep it that way.

Peter Wildeford: Vercel (cloud hosting platform) was hacked.

VERCEL: "We believe the attacking group to be highly sophisticated and, I strongly suspect, significantly accelerated by AI. They moved with surprising velocity and in-depth understanding of Vercel."

A Young Lady’s Illustrated Primer

There are parents who say things like ‘I caught my child using AI’ and then cite that her daughter was asking it how to get along with her sisters and improve her times at a swim meet and cowrite fan fiction. But not to worry, this righteous mother put a stop to that. Of course this was literally from r/antiai, so there you go.

elea-norea : all this can make me think of is when i was a kid and my parents banned pokemon cards because they were Satanic and Invoked Demons. similar reaction.

Zvi Mowshowitz: I keep catching my child not using AI. Not sure what to do about it.

Leo Abstract: i keep catching my child using AI--she's run away four times now but Claude has predicted her location accurately each time.

The difference is that Pokemon was decidedly, shall we say, not a frontier model.

Tyler Cowen predicts colleges won’t be fixed, and will become even more divorced from actual education than they already are, but that they will be kept alive by their social functions.

They Took Our Jobs

AI Agent Operator is a plausible candidate for ‘job that AI creates that could scale’ with Harry Stebbings predicting 500k to 1 million such jobs in five years. Someone needs to oversee all the automation of tasks and integrating the new tools into the business processes across the economy. Whether this is a new job type, or should be thought of as the new version of the people who previously did the same work, is a matter of how you look at it.

In theory, so long as the agents in question need enough continuous handholding and other scaffolding work from humans they could function largely as augmentation rather than automation, and thus end up not bad for employment. I doubt that is sustainable or generalizes to that large a portion of the agents, but it’s something.

Dean Ball points out that many plans assume we can fund UBI or other redistribution by taxing the absurdly huge wealth of the AI labs, but that as Roon says the labs plausibly could capture, in ‘AI as normal technology’ worlds, a small fraction of the wealth generated by AI. Indeed that is likely, since there will be competition, and in most use cases the AI lab revenue will be a small portion of generated value, and then the labs have to pay compute costs. It wouldn’t be able to fund UBI, although a general tax on capital or consumption would work nicely.

That doesn’t mean you shouldn’t have a plan. Even where plans are worthless, planning is essential. What that implies about the right political position is not my area of expertise but yes I want to see politicians show us their contingent plans.

If as I expect AI is not a normal technology, and the labs do create superintelligence, then the labs should also worry about not capturing the resulting value in a completely different way. By default it is then the AIs that capture the value. There are also possible worlds in which the labs or key individuals capt