生成 AI の違法コンテンツから子供を守る新手法が開発される

With the exploding popularity of generative artificial intelligence, many open-source models are now available online for anyone to adapt for their task, such as generating product renderings in a certain artistic style.

But these models also find their way into the hands of nefarious actors who may optimize them to produce illegal content, like hate speech or child sexual abuse material (CSAM). This is a growing problem — the National Center for Missing and Exploited Children received more than 1.5 million reports of AI-generated CSAM in 2025, an increase from 67,000 in 2024.

Engineers usually test AI for harmful capabilities by prompting the model and inspecting its outputs, but this is impossible for CSAM, since it is illegal in the U.S to generate such content, regardless of intent.

To avoid this dilemma and improve AI safety, a team of MIT scientists, led by graduate student Vinith Suriyakumar and associate professors Ashia Wilson and Marzyeh Ghassemi, joined forces with researchers from Thorn to develop a new auditing approach that determines whether a model can produce CSAM, without prompting it. Thorn is a child safety nonprofit whose mission is to transform how children are protected from sexual abuse and exploitation in the digital age.

Their technique examines how the inner workings of a model have been adapted, but it never generates an output. By examining hidden representations, it can reliably infer whether a model has been specialized to produce harmful imagery.

When tested, the auditing procedure identified model variations that had been specialized to generate CSAM with 100 percent accuracy. A hosting platform could use this technique to flag unsafe models and quickly remove them or prevent them from being uploaded in the first place.

“This unlocks a new avenue for platforms that host open-source models and for law enforcement to actually test whether a model is capable of generating CSAM. Before, we had no way of measuring this. It was a huge blind spot that some people were taking advantage of. Now, we can address an AI safety problem that is having severe negative impacts,” says Vinith Suriyakumar, an MIT electrical engineering and computer science (EECS) graduate student and lead author of a paper on this technique.

Suriyakamur and Wilson, the Lister Borthers Career Develop Professor in EECS and a principal investigator in the Laboratory for Information and Decision Systems (LIDS), are joined on the paper by Lena Stempfle, an MIT postdoc; Ghassemi, an associate professor in EECS and a member of the Institute of Medical Engineering Sciences (IMES) and LIDS; and others at Boston University and Thorn. The paper was be presented as a spotlight at the “Trustworthy AI for Good” workshop at the International Conference on Machine Learning.

Auditing adaptations

Recent techniques have made it easier for users to specialize a generative AI model for their task through a process known as fine-tuning.

Rather than retraining the entire model on a task-specific dataset, individuals can utilize an algorithm called low-rank adaptation (LoRA) to specialize the model in a more efficient manner.

This has led to a wave of new generative AI model variants for a variety of purposes, like producing watercolor images that mimic an artistic movement. But it has also enabled malicious actors to create models that can generate high-quality CSAM and other harmful imagery.

To audit a model, engineers typically prompt it for harmful content and check its outputs, but this manual auditing procedure is not scalable. In addition, repeatedly generating heinous images can have negative psychological impacts on human evaluators.

This evaluation method quickly falls apart when testing CSAM, which is illegal to generate for any purpose in the U.S. and many other international jurisdictions.

“We are in this very difficult situation where, based on the law itself, we cannot use the de facto means of evaluation. We had to throw out the entire toolkit and take a different approach,” Suriyakumar says.

After learning about this conundrum, the researchers joined forces with Thorn, to address this issue.

A nongenerative solution

Instead of focusing on outputs, the researchers targeted the modifications a LoRA algorithm makes during fine-tuning.

Their technique probes these modifications, called LoRA adaptors, to determine whether a model has been specialized for a harmful capability, without generating an output.

Using a technique called Gaussian probing, the researchers feed the model a set of random data points and analyze how it manipulates those data within its multilayer internal structure.

“We never run the model all the way to the end or prompt the model, so we never generate images,” Suriyakumar explains.

The researchers capture those modifications at multiple time points within the model’s inner structure and average them to summarize how the LoRA adaptor changed the model’s computation. They found these responses to be a strong signal of how a model had been specialized.

They tested their method on variations of three types of models, comparing the results to ground-truth data from LoRA adaptors known for generating CSAM, other harmful images, and safe content.

Their method was 100 percent accurate in identifying models that had been adapted to generate CSAM.

“There is a huge bucket of child safety concerns with AI, and these are real concerns that need to be addressed. A lot of children are being harmed by AI deepfakes. We’ve shown that Gaussian probing can be a very useful tool, and we hope the research community really pours more attention into this problem,” Wilson says.

Importantly, their technique is scalable and would be relatively inexpensive to implement. Since thousands of model variations are published online every month, scalability is key to help auditors remove harmful adaptations before they are widely distributed.

Gaussian probing is also more robust than some other auditing techniques, since a nefarious actor would need to carefully alter the inner workings of the base model to avoid detection.

In the future, the researchers want to evaluate their technique on a larger set of model variations and explore whether Gaussian probing can detect harmful capabilities in base models before they are adapted.

“Now we have a technological approach to partially address this concern. So much effort was poured into this collaboration, which enabled us to tackle a really hard problem that is harming so many children, nationally and around the world. Hopefully, we can have a transformative impact in this area,” Ghassemi says.

This work was supported, in part, by the Bridgewater AIA Labs Research Fellowship.