Anthropic のテキスト透かし、AI 企業の文章への関心欠如を証明

imageEarlier this month, Anthropic announced that future versions of Claude will generate text that includes watermarks showing it was AI-generated. At the time, Anthropic did not explain how this would work, leaving us to speculate on the podcast: Would it somehow encode this into the text? Include invisible characters? Do something with the metadata? We now know, thanks to a blog post over the weekend, that Anthropic will do this by changing how its AI writes altogether.

“Nothing is added to the text and there are no hidden characters,” Anthropic wrote in that company blog post. “The difference between watermarked and un-watermarked text will not be distinguishable to readers.” The way it will work, the post explained, is that Anthropic will subtly alter the word choices in AI-generated text in a way that is only known to Anthropic and its algorithms. Anthropic will know the watermarking algorithm, which will change “the source of the randomness used to pick among words” and thus can write a tool to detect whether something has been AI-generated.

This research and approach is interesting in a data science kind of way, but Anthropic’s layperson explanation for how this will work shows how little the company thinks about the craft of writing or the subtle differences between words a human author might want to use to convey their thoughts.

Anthropic asks us to consider the difference between two sentences: “Take the sentence ‘The weather today was cold and…’. The next word is very unlikely to be ‘sugary.’ But it is quite likely to be ‘overcast’ or ‘grey.’ Under most circumstances, it doesn’t matter much to the reader which of these latter two words the model ultimately chooses—the meaning of the sentence is largely the same either way. In cases like this, the choice is settled by a random number,” Anthropic writes. “Watermarking uses low-stakes choices like these—which occur many times over a piece of generated text—to leave a pattern in Claude’s responses. That pattern is undetectable to the reader, but is detectable to anyone who has a key that encodes it. When watermarking is used, choices are still made at random, but the source of the randomness is different.”

Anyone who has written anything would, I hope, understand that the difference between the sentences “The weather today was cold and grey” and “The weather today was cold and overcast” are sometimes “low stakes,” as Anthropic describes, but not always. “Grey,” and “overcast” are different words, and there are any number of reasons why a human author might pick one over the other in a given context. In this example, however, Anthropic’s algorithm sees these words as totally interchangeable and thus its watermarking algorithm has decided that it can “nudge” the word choice one way or the other for the purposes of watermarking.

Anthropic continues: “Instead of using an arbitrary random number generator to pick the next word, watermarking uses the key and a few words that come before to settle what word the model should pick. That is, the words that Claude picks are still random, but now, one can check the sequence of words and see if it’s consistent with the choices Claude would make if it was using the key. If it is, one can assign a probability that the text was generated by Claude.”

Anthropic claims “Watermarking does not impact the quality of Claude’s output. To a reader, a watermarked response is indistinguishable from an unwatermarked one,” and that “in internal testing, we’ve seen no impact of watermarking on the content, level of creativity, or readability of Claude’s text.”

People are quite mad about Anthropic’s watermarking system, and understandably so. Synonyms are sometimes interchangeable, but not always, as is pointed out in this excellent essay by John Gruber of Daring Fireball, and by journalism academic Jeff Jarvis, in which he claims Anthropic “devalues writing.” In making this choice, “Anthropic declares words fungible, language random, choice meaningless,” Jarvis writes.

When I sat down to write this post, I was mad because it seems like Anthropic is  putting its thumb on the scale, messing with the outputs of its machine and saying that the resulting text is qualitatively just the same as the other AI text it was probably going to output. But as I began writing this, I realized that my problem is not necessarily with text watermarking but with AI-generated text altogether. It does not matter to me, necessarily, whether the output of Claude’s garbage AI text is one way or is a slightly different way. But it does matter to me that AI data scientists at huge tech companies think that word choice doesn’t matter, or that it is possible to statistically use synonyms wherever without fucking with the meaning of a sentence.

Throughout the blog post, Anthropic describes the act of writing as being akin to a probabilistic game of chance. In Anthropic’s own words, its writing is sometimes the result of an “arbitrary random number generator,” and “random” whenever its systems encounter a situation where its tool believes, based on pattern recognition, that the choice between several possible next words isn’t all that important. That may be true for LLM garbage, but is not true for the human experience of writing, which is why human writing almost always feels different than AI writing.

This watermarking approach, and Anthropic’s blog post about it, highlights something that should already be clear about a company that famously scanned and destroyed huge numbers of printed books and has trained its LLMs on stolen content: Anthropic does not care about the craft or effort of writing, and sees words as fungible and unimportant. Anthropic says it is making this change as part of the European Union’s new AI regulations, which are well-intentioned but problematic. While it can definitely be useful to have additional ways of detecting AI-generated content, the carelessness with which Anthropic has announced this decision highlights the broader problem with using LLMs to write: They are, as Anthropic notes, probabilistic tools that do not “write” in the way that humans do, rather, they mimic their training data which is, by definition, things that have already happened and been ingested.

Contrast this with how Anthropic sees code, something where it says an “exact output is required.” In writing, meanwhile, Anthropic suggests different words are often “equally good.” Over and over again, Anthropic and the researchers who work on this type of watermarking claim that text can be “nudged” in this way without being noticeable to humans or without impacting “quality.”

But it is worth noting that the people judging the “quality” of the AI-generated outputs are either data scientists or people asking AI tools to do their writing for them, not, say, people who care about reading or writing. The scientific paper that Anthropic cites was done by Google researchers on a Google watermarking tool called “SynthID,” which Anthropic’s watermarking is based on.

In the SynthID study, quality was assessed by randomly putting watermarking on some Gemini outputs, then asking Gemini users to either thumbs-up or thumbs-down the response: “A random fraction of queries were routed to a watermarked model and an equivalent number to the unwatermarked counterpart. The Gemini user interface allows users to provide feedback on model responses via a thumbs-up (good response) and a thumbs-down (bad response). We analysed approximately 20 million watermarked and unwatermarked responses and computed the thumbs-up and thumbs-down rates (both as a fraction of the total number of thumbs-up and thumbs-down feedback received). We found that the thumbs-up rate for the two models differed by 0.01%.”

I hope it is clear to anyone who has clicked on this article that asking someone who asked a chatbot something to thumbs up or thumbs down a response is not a very good way of assessing the “quality” of “writing.” The other human assessment that Google did was to ask people to assess side-by-side watermarked and unwatermarked text for quality. Here are examples given in an appendix of the study; apparently people did not really have a preference one way or the other:

imageOne could argue that these passages are two different ways of explaining something, yes. But they are definitively not the “same,” and it is unclear to any reader why one version is one way and the other version is another way. Why did the LLM write “respiratory failure” in one example and “cessation of breathing” in the other? The answer for both is an “arbitrary random number generator” and proprietary black box algorithmic weighting systems controlled by the AI company. In the watermarked version there’s been an additional “nudging” or messing with the machine that’s already just a pattern matcher.

The point is, there is no conscious thought or decision-making process happening here, so perhaps watermarked AI text is not all that much more offensive than regular AI text. But to see it laid out in such stark terms by the companies building these machines shows how little they actually care about writing. If you asked me, on the other hand, why I used one word instead of another, I might not be able to tell you exactly why, but I could probably explain to you what I was going for, the style of writing I do, my intended audience, my mood that day, whether my heart was racing or not, where I was, what I was doing, what I did earlier that morning and what I did later that day. Maybe it was a word my third grade teacher used all the time or which I read in an article last week or is an inside joke with my friends or which I have recently become obsessed with or tend to overuse. Why I wrote what I wrote or why I did anything at all is the result of my some mix of human experiences dating back to when I first acquired language as a baby and continuing on to this very moment that I may or may not be able to explain, but which result in a certain style of writing that is mine.

This is the case even when I’m working fast or carelessly dashing off text messages, when the thoughts just kind of flow from my brain to my fingers to my keyboard where I don’t know if what I’m saying is making sense at all but is probably legible because it’s coming from a human brain and not a random number generator.

This is why short passages of AI-generated text feel soulless and generic, as we have written about repeatedly. And there are many AI tools that use AI to make AI writing seem less generic (yo dawg, we heard you like AI so we put AI in your AI) by using synonyms that are supposed to make a passage sound more human — or less plagiarized — by picking words that are less commonly used. The text outputted by these tools, which are called “spinners” or “humanizers” are often just as uncanny and weird as AI writing itself. Or, when applied to things where, to use Anthropic’s own language, “an exact output is required” such as quotes in a news article, the output is often factually inaccurate, libelous, or just plain garbage.