X で非合意 AI 画像が急増、微妙な改変が拡散

imageI can clearly remember the first time I saw a deepfake. I was waiting for the train on my way to the Vice office in 2017 when I got a Twitter DM from someone who saw it posted to Reddit. It was a short .gif from the start of a porn video, showing a woman lying on a bed and talking to the camera, only the woman’s face had been edited to look like Gal Gadot.

The deepfake was shockingly realistic for the time, and as Sam’s reporting on it from that day on has shown, deepfakes have primarily been used to harass and degrade women. We were surprised by how advanced the technology seemed at the time and knew that the internet and the law wasn’t prepared to deal with it. The technology has advanced exponentially since then, and the internet and the law has still not caught up to it. But as concerning as deepfakes clearly were the moment we laid eyes on them, the problem was never that we thought Gal Gadot actually made an adult video. Deepfakes can look convincing, but they never fooled us that what we were seeing was real.

But for the past few months I’ve been looking at a new and strange type of AI generated nonconsensual image on X that I think will fool a lot of people. These images, which I’ve started calling subtlefakes, are not as clearly offensive, but they are a lot more convincing. The concerning implication is that it’s not just the technology that is advancing now, but that the people who wield it maliciously are getting more sophisticated. Much like “cheap fakes,” which in 2019 Data & Society described as an edited piece of media that manipulates the truth without requiring AI or any other advanced technology, subtlefakes are another form of synthetic media that has taken over the internet since the emergence of deepfakes in 2017.

These more convincing images are not AI generated nudes of celebrities or face-swapped identities onto porn videos, but AI generated images of celebrities that are subtly edited to make them more revealing or provocative. We’ve written countless stories about AI generated nonconsensual media of celebrities and non celebrities. It’s a form of harassment that causes psychological and material damage to people, and they don’t have to be believable to do that. Even if the image looks photorealistic, it is often clearly fake for the simple reason that it’s unlikely that the biggest actor in the world would suddenly produce hardcore pornography that’s posted by a random account to X.

The images I’ve been seeing lately are often only slightly altered. One type of image, for example, will edit a real photograph of a famous actor walking a red carpet, but edit it to make her dress more revealing or to make her butt or boobs bigger. Another type is entirely AI-generated post-workout selfies at the gym.

The only example I feel totally comfortable sharing is one that was called out and shared by the target of one of these fakes, actor Xochitl Gomez. As a side-by-side she shared on Instagram shows, someone had taken a real photograph of her at a parking lot, looking over her shoulder and smiling at the camera, and used AI to make it look like she was bending over in one image and putting her hand on her butt in the other. Similarly, someone had taken an image of her on a red carpet and used AI to make it seem like she’s turning her back to the camera and sticking her tongue out. These images are not just photorealistic, they also don’t have the usual context clues we’ve come to expect from nonconsensual images. I think I am very good at spotting AI generated images because it’s a big part of what we do here, and I don’t think I could tell these were AI generated if I was just scrolling on X.

These images are almost always posted by verified engagement farming accounts on X, which pays the people who operate them if their posts get enough impressions. X obviously doesn't care if people are doing this and a bunch of other stuff that’s as bad or worse. I’ve seen at least one account that shared a few of these subtle deepfakes, but when I scrolled down their post history I found nonconsensual fully nude and sexual images that they monetized via other platforms, so that’s another reason why X accounts might be doing this. There are many, many accounts like this and they are earning millions of views.

I think it’s interesting because it’s a new, and as far as I know, yet to be covered use of nonconsensual AI images. It’s also important and in the public’s interest to know it’s happening because people should know what kind of AI content is targeting them, especially if it’s at the expense of other people. It’s also possibly a warning sign for how people’s AI use is evolving from abusive but obviously fake, to more subtle, less offensive, but much harder to detect.

The other reason these types of images are going to be harder to moderate is that it’s easier to produce them. There are plenty of services that let people easily AI generate nonconsensual nudes of anyone, but they almost always involve some friction. People have to go to a sketchy site, install a sketchy app, or pay a small fee. As we’ve reported many times in the past, more easily accessible AI generators from tech giants can be manipulated to produce nonconsensual images as well, but generally speaking, most companies are trying to prevent people from making those images with their tools. While it’s not impossible, it does require some effort to bypass those guardrails.

It is not so hard to bypass those guardrails if the nonconsual images people are producing don’t include any nudity or sexual acts, assuming the AI generator is even design to prevent those type of images. Most mainstream AI generation tools at least try to prevent people from generating any type of nudity. Fewer of them try to prevent people from generating the likeness of a real person. I can’t say for certain where most of the subtlefakes I’ve seen come from, but I know that at least some of them were made with X’s own Grok because some images included the Grok watermark. X has repeatedly been caught enabling nonconsensual images and only attempting to do something about it after public outrage.

X did not respond to a request for comment.

“Large social media platforms that are really struggling under the weight of this for the first time,” Hany Farid, an image forensics expert and cofounder of deepfake detection firm GetReal, told me. You can see my full interview with Farid here. “They've always had a problem, but this seems to be the first time where they're like, ‘Oh man, this is really bad.’ And I think it's because the volume and the sophistication of the fakes is nothing we've seen before.”

The primary targets of deepfakes always were and continue to be women, but Farid said other nefarious uses of the technology are also becoming harder to deal with.

“If you go back 10 years, it was pretty clumsy,” Farid said. “It's getting more and more sophisticated. They're evolving. Cyber criminals evolve and that's going to make everybody's job much much harder as the years go on.”

Our reporting on deepfakes over the years has taught us that people make them because they feel entitled to women’s bodies, because they are intentionally trying to hurt them, or because it’s profitable. I’m sure all those reasons apply to subtlefakes, but it appears to me that the current state of X is pouring gasoline on that fire. People use subtlefakes as advertisements for accounts other platforms where they monetize nonconsensual images, but that is rare. It seems that the more logical scheme is to monetize pure engagement via X’s ads revenue sharing program, which pays X users for engagement. It’s not a lot of money, but it’s easy money. Since X is not interested in moderating this content, one person can also operate multiple accounts, posting dozens of times a day.

In general, my X feed is mostly different types of engagement farming. Another type of subtlefake I’ve seen a lot of doesn’t require AI at all. For example, this post about OnlyFans model Sophie Rain claims that Drake paid her $2,000,000 to take her virginity, and includes two random images of Drake and a completely unrelated TMZ video of Rain. People click on the video to see if she said what the post claims, and the account gets the engagement before they realize it’s a lie.

This kind of easy, stupid, but effective garbage is what X is rewarding, and these subtlefakes are just one concerning outcome.