ミトス事件後のレッドチーム：Zico Kolter氏とMatt Fredrikson氏が語るグレー・スワン

AI Engineer World’s Fair regular bird tix will sell out ~today! Join us next week ahead of the Late Bird price hike and get >$40,000 in sponsor credits for attending!

Thanks to the US Government issuing an export control directive on Mythos and Fable, the risks of jailbreaks and (industry term) indirect prompt injection are suddenly the talk of the town, though we have been covering AI security for a few years now, from Hackaprompt to the enigmatic Pliny the Elder.

Zico Kolter, member of OpenAI’s board of directors on the Safety & Security Committee, and Matt Fredrikson, CMU professor and CEO of Gray Swan, co-authored the definitive paper on Indirect Prompt Injections, and Gray Swan were cited authorities on the Mythos model card, directly investigating the exact capabilities that are under scrutiny right now:

We seized the opportunity to ask them the state of AI Red Teaming, and Shade, the adversarial red teaming tool that Anthropic used to evaluate the robustness of their models against prompt injection attacks in coding environments. Shade is part of their overall toolkit covering Simon Willison’s Lethal Trifecta, including Cygnal, an AI guardrails product, and the world’s largest AI Red Teaming Arena, including AIRT celebrity Wyatt Walls.

All of this security tooling, and yet, we’re only staving off the inevitable.

The risks of extremely smart AI increasingly feel like gray swan events: an event that everyone can see coming.

In this episode, Gray Swan cofounders Zico Kolter and Matt Fredrikson join swyx to explain why AI security is not just “cybersecurity with AI,” why agents introduce a new class of vulnerabilities, and why the next major AI incident may be a gray swan: unlikely, but clearly visible before it happens.

We go deep on prompt injection, automated red teaming, model robustness, agent identity, computer-use agents, enterprise guardrails, and the emerging AI insurance/compliance stack. Zico and Matt also explain why frontier models are not automatically safer as they scale, why specialized red-teaming models can now beat humans at breaking AI systems, and why the future of AI security may depend on AI systems attacking, defending, and interpreting other AI systems.

We discuss:

Why AI systems need a different security mindset from traditional software

How prompt injection creates a new exploit class for agents like Codex and Claude Code

Gray Swan Arena and the rise of community red teaming

Shade: AI that can outperform humans at breaking models

Why LLMs are an alien form of intelligence that fail differently from humans

Human vs browser-agent robustness and why humans ranked fourth

Why eval awareness and capability elicitation matter

Cygnal: Gray Swan’s guardrail model for policy enforcement

Why bigger models do not automatically become more robust

The lethal trifecta: untrusted data, private data, and exfiltration

Why “just prompt it better” is not enough for enterprise AI security

OpenClaw, computer-use agents, and the agent security nightmare

Agent-native identity, permissions, and enterprise deployment

Why AI security may become part of insurance and compliance

Why the first major AI prompt-injection breach may be inevitable

Gray Swan

Website: https://www.grayswan.ai/

Zico Kolter

X: https://x.com/zicokolter

Website: https://zicokolter.com/

LinkedIn: https://www.linkedin.com/in/zico-kolter-560382a4/

Matt Fredrikson

Website: https://www.mattfredrikson.com/

LinkedIn: https://www.linkedin.com/in/matt-fredrikson-7596349/

Timestamps

00:00:00 Introduction

00:02:31 Why AI Security Is Different

00:06:38 Testing Claude, Codex, and Prompt Injection

00:07:47 Gray Swan Arena and Automated Red Teaming

00:11:14 AI That Breaks Models Better Than Humans

00:14:00 LLMs as Alien Intelligence

00:19:00 Humans vs AI Agents

00:24:35 Red Teaming, Jailbreaks, and Capability Elicitation

00:26:11 Cygnal: Guardrails for AI Agents

00:34:04 The Lethal Trifecta

00:39:31 Can AI Automate AI Research?

00:45:47 OpenClaw and the Computer-Use Security Problem

00:50:44 Agent Identity, Permissions, and Enterprise AI

00:54:24 The Future of AI Security

01:00:30 AI Insurance and Compliance

01:04:32 The Gray Swan Event Everyone Sees Coming

01:06:04 Closing Thoughts

Transcript

Introduction: Gray Swan, AI Security, and CMU

Swyx [00:00:00]: We’re here in the studio with Gray Swan, Matt and Zico. Welcome.

Zico [00:00:08]: Great to be here.

Matt [00:00:09]: Thanks for having us.

Swyx [00:00:10]: You’re visiting from Pittsburgh? The home of all good computer science. I don’t know if I’m overstating things. A very strong university.

Zico [00:00:18]: CMU has been the center of a lot of AI since really the dawn of the field.

Swyx [00:00:22]: Especially a lot of self-driving and some language learning. Congrats on your Series A. You’re here because you’re attending Snowflake Summit, and Snowflake is one of your investors. Let’s introduce crisply at the top: what is Gray Swan, and what have you chosen as your startup domain?

Matt [00:00:42]: At Gray Swan, our mission is to empower everyone to use AI safely and securely. Large language models are software, and if you want to deploy them or build applications on top of them, you need to understand the vulnerabilities and what can go wrong. That includes everyday mistakes, like an agent making the wrong tool call, but also worst-case scenarios where an attacker has an incentive to make your agent misbehave, leak data, or steal credentials. Gray Swan grew out of our research at Carnegie Mellon, where Zico and I have spent over a decade studying new vulnerabilities and attack surfaces in deep learning systems: how to test for them, understand their severity, and make inference more robust.

Adversarial Examples and Why AI Security Is Different

Swyx [00:02:05]: Honestly, a very fruitful area of study for any academic. Throwback, this is 10 years ago, which is basically the entirety of me. I got a lot of inspiration from Ian Goodfellow, a friend of the pod, and this is one of those initial adversarial settings.

Matt [00:02:23]: This paper was directly inspired by Ian’s work.

Swyx [00:02:29]: Zico, what about your side of the story?

Zico [00:02:31]: Like Matt, I have been faculty at Carnegie Mellon for a while. Fundamentally, we believe in the transformative power of AI. It has already transformed the software ecosystem, and it will transform many other ecosystems going forward. The issue is that these systems behave very differently from the software we are used to. I do not just mean that AI can find vulnerabilities in software, though it can. I mean that AI systems have inherent vulnerabilities of their own. They can be tricked in ways people can be tricked, so you need a different security mindset.

Zico [00:03:23]: This matters especially when there is the possibility of correlated failures. It is not just that there are many AI systems out there; it is that everyone is using a few models. If you find vulnerabilities in agents that everyone uses, like Codex and Claude Code, you have a new class of exploit. The labs are doing a lot of work here, but when a new platform emerges, a separate security system often emerges alongside it. That is where we are with AI: there is a need for specifically minded AI safety and security providers, and the demand is only going to grow.

Treating Models as Untrusted Systems

Swyx [00:04:55]: I want to highlight right at the top that this is not a cyber episode in the traditional sense. A lot of people looking at the title might think that, but you’re actually trying to treat these models inherently as untrusted entities?

Zico [00:05:11]: Exactly. This is a common conflation because AI is also good at cybersecurity problems, both solving them and causing them. But AI systems themselves introduce new vulnerabilities. Gray Swan is not about using AI to make your cyber infrastructure better; it is about understanding and mitigating the security risks you bring in when you adopt and deploy AI.

Matt [00:05:49]: A big part of that is how people are using artificial intelligence. Once you build entire autonomous systems on top of models and integrate them into your larger platform or network, you have a potential cybersecurity risk. The goal is to mitigate the risk posed by the AI as it relates to your broader cybersecurity goals.

Testing Claude, Codex, and Indirect Prompt Injection

Zico [00:06:17]: Part of this is red teaming. One reason we reached out to you was that you were involved in the Claude Mythos preview, where you were one of the authorities on IPI, or indirect prompt injection. When you receive a model, it does not have to be Mythos, but that is the most prominent one right now: what do you do with it?

Matt [00:06:38]: We do a range of things. In the Mythos case, the concern from Anthropic was how robust the model is to indirect prompt injection. If you operate a coding agent and use Mythos as the model, it will fetch untrusted content and read text you do not control. How robust will it be at staying true to its original objective and not getting hijacked? We also help frontier labs test their safeguards for issues like cyber misuse. Broadly, we provide adversarial safety and security evaluations so model builders can assess progress from one iteration to the next.

Zico [00:07:37]: They also do this in-house, and Anthropic is very ideologically inclined to do it. What do they choose to outsource versus keep in-house?

Gray Swan Arena and Automated Red Teaming

Matt [00:07:47]: So there are two things that I think, we stand out for. One is the Gray Swan Arena. So we operate a community of red teamers. We provide, prize challenges. a lot of these come from the needs of the lab sponsors. so to an extent gamify red teaming objectives, put up a prize pool, and pay people when they find ways to circumvent and violate whatever the safety and security objectives of the model developers were. So that’s, that’s one. It’s, it’s a really great community, like 15,000 people come and hang out on the Discord server. Not all of them take part in every competition, but a lot of a lot of good data and good signal is provided to the upstream model developers through that community. The second is the automated red teaming that we do. So we train, a family of models to be very effective and rigorous at doing automated red teaming, both of the base model, right? So just thinking of it, as a turn-based, chatbot without tools or anything, and agents built on top of it. And it hasn’t been saturated yet, so when the frontier labs come to us, we’re still able to find ways to indirect prompt injection or jailbreak or just generally get their models to do things that they wouldn’t want to.

Zico [00:09:11]: Did you say without tools?

Matt [00:09:12]: With and without tools.

Zico [00:09:13]: With and without tools.

Matt [00:09:13]: So we definitely operate on On agents as well.

Zico [00:09:16]: Obviously that would be more useful.

Matt [00:09:17]: Yep. that’s, that’s actually a fairly recent thing. For a while, what we would help, the frontier labs with was more just, chat-based interactions, going around their content safety policies and what is in their model spec. Now the focus is very much on agents and tool use and all the downstream applications that people want to build on top.

Shade: Automated Red Teaming Models

Zico [00:09:39]: This is a inspired topic. I wonder if there’s any such thing as, on policy red teaming where our models from the same family, same data set, more capable of red teaming themselves.

Matt [00:09:51]: That’s an interesting question. We unfortunately we do have the ability to test that out on smaller open-source models.

Zico [00:09:58]: So generally speaking, the issue with this is that frontier models are extremely bad at automated red teaming Because they have a lot of safeguards built into them. So if you try to use them to jailbreak another model, they will actually refuse. Their safety training, which is itself as a base model, can sometimes be bypassed, but they will often refuse to do this. Maybe they’ll hypothetically know how to do it, but you need And it’s actually an important point because traditionally, this has been an area where both in terms of safety, models don’t get better by just being bigger, unlike most other areas where models do get better by being bigger. Safety has not been like that traditionally. you have to train them explicitly to be safe or they won’t do that. But on the flip side, they’re also not necessarily better at red teaming, by default. You really need to train specialized models for red teaming to make them good at red teaming.

Matt [00:10:56]: That’s awesome for you guys.

Zico [00:10:58]: And so, and what do you need to do that? Well, you need lots of data From people that are traditionally much better at red teaming. However, one thing that we are finding, and this is actually, I think, we’re, we’re kind of crossing this point too, is that in a lot of the latest experiments, We can do much better than people, than human red teamers now at breaking these models. When I say we, our automated red teaming model. It’s a system called Shade. That system is now actually quite a bit better at breaking, models than humans are. I think we had a recent competition Between humans and our model, and it was actually quite a bit better. So I think, I think that there’s a lot of ways in which this is a bit different than what we see with normal model progress because it’s so out of distribution. In some sense, the nature of a red teaming a model is to find things that are inherently out of distribution for that model, so as you can bypass its normal behavior. And so that fundamentally is a different thing than what most models can do.

Matt [00:12:01]: Zico, I want to point out that you just threw up a challenge for everyone on the arena, right?

Zico [00:12:06]: Try to do better than Shade,

Matt [00:12:07]: It will, and I do want to caveat that a little bit. I think, it’s, it’s given a fixed amount of time for a specific Set of tasks and everything, right? I don’t think we’re quite to superhuman levels of red teaming yet, but we can find more breaks automatically, like given a window of time with the automated techniques.

Human Red Teamers, Alien Intelligence, and Model Weirdness

Swyx [00:12:26]: But just because we had the leaderboard up, and I always love to find out the human story behind some of these folks. Do you I assume some of them. Are they celebrities in their own right? what’s

Zico [00:12:35]: Wyatt’s a big person on Twitter. You should, you should follow him on Twitter If you’re not already. Yeah.

Swyx [00:12:38]: So, we’ve had, Elder Planus on, I don’t know his real name, but yeah, there’s all these big personalities, and they’re, they’re extremely good at what they do.

Matt [00:12:49]: They’re, they’re very good at what they do.

Swyx [00:12:51]: Oh, he’s an Aussie.

Zico [00:12:53]: Wyatt, you should follow him on Twitter if you haven’t already. He makes, he makes great He makes these really insightful posts. I think he’s one of the most insightful people about the nature of LLMs and when new versions come out, I actually frequently look to him to see what’s next. He’s a lawyer, I think, right?

Matt [00:13:09]: He’s an attorney.

Swyx [00:13:13]: There’s red lining, red teaming The other thing. Yep.

Zico [00:13:16]: Yes. Our top, competitors are often people that, Do this a lot.

Swyx [00:13:22]: What’s an example of a thing that you’ve learned from Wyatt? Oh.

Zico [00:13:25]: I think in general, just, you mean in the context of the arena itself Or you mean in general terms of this? I think he just has great insights in the nature of models as a whole. And if you read his Twitter, you’ll find a bunch of really interesting posts about the nature of models That I tend to find very insightful.

Swyx [00:13:42]: Riley’s like this as well, right? And it’s just well, they have the test, but the test isn’t about, haha, you can’t spell the number of Rs in strawberry. The test is, well, you’re actually not modeling intelligence inherently, and this shows it in a very

Zico [00:14:00]: I don’t know that it shows that you’re not modeling intelligence. I think these things are intelligent. I think LLMs absolutely are intelligent and maybe will be more intelligent

Swyx [00:14:07]: Conscious?

Zico [00:14:07]: At some point.

Swyx [00:14:07]: Are they conscious?

Zico [00:14:08]: Conscious is a weird word But I actually don’t, I don’t think so. I think, I think the way that we’re getting super philosophical now.

Swyx [00:14:16]: That’s, that’s the right answer.

Zico [00:14:16]: We’re getting very philosophical now. But I don’t think so. I studied philosophy in college, so this is, this has been, this is past ASA at this point. It is clearly a different form of intelligence than people. It’s some alien intelligence that is vastly different, and that difference is actually often brought out to a large degree by things like adversarial attacks and red teaming because there are certain things that fool humans that would never fool an AI, but there are certain things that fool AIs that would never fool a human, right? So it’s just, it’s just a different form of intelligence. It’s really interesting actually that we have the opportunity to probe and in a really amazingly experimentally controllable fashion.

Matt [00:14:59]: Like almost omniscient, right?

Zico [00:15:02]: I’m, I’ll, I’ll do the analogy to neuroscience here. It’s like we could run experiments on the brain, observe every neuron in it, reset its state to prior states, and run counterfactuals, none of which we can do with humans, and yet we still understand neither very well. Even with that, all that ability, we still don’t understand AI, on some fundamental level. So it’s, it’s definitely this different form of intelligence, but it’s clearly

Swyx [00:15:30]: We’ve done a number of mech interp pods, and you can see honestly the scaling in mech interp is two, three orders of magnitude less than capability scaling. so we’re hopelessly behind is what I’m saying.

Mechanistic Interpretability and Automating AI Research

Zico [00:15:44]: So I have, I could go off. It’s a little off tangent here. We’re getting, we’re getting, we’re getting, we’re getting a bit, but yeah.

Matt [00:15:48]: Well, no, I think it actually, it does relate, right? Go ahead. Do your tangent.

Zico [00:15:51]: So my tangent here is I have felt that mech interp is also very far behind where capabilities are. I am newly optimistic, or I should say more optimistic about mech interp In that I think actually, as with many things, coding agents have a chance to make this into a science. So the problem with mech interp, and I’m Okay, so I shouldn’t say the problem. I don’t want to call it a field. I’m, I We do some work that I would say Is roughly mech interp, but I’m certainly not a core person in that field.

Swyx [00:16:19]: For folks to see.

Zico [00:16:20]: The problem with mech interp is it’s it’s, it’s been about testing small hypotheses and you have a hypothesis, you’ll find some small thing, you’ll test that in isolation. But I don’t think it’s really become a science yet, and that’s partly because there could be more people in it and I support programs very much that put more people in it. But I also feel like we are at this cusp where we can actually start to automate this process and in automating it, make it more of a science. And that’s actually one of the most fascinating things about coding agents actually, is they can, they can do a lot of experimentation In an in an automate