生成 AI レッドチームに敵対的エミュレーションを活用する発表

InfoQ Homepage

Presentations

Leveraging Adversary Emulation for GenAI Red Teaming

View Presentation

Speed:

36:27

/presentations/emulation-genai/en/slides/Ken-1785846902883.jpg)

まとめ

Kennedy Torkura discusses practical GenAI red teaming techniques to safeguard LLMs and knowledge bases against security threats like data poisoning and LLMjacking on AWS. He explains how engineering leaders and architects can bridge traditional cloud security with MITRE ATLAS frameworks to proactively identify vulnerabilities, implement guardrails, and secure production AI applications.

Bio

Kennedy Torkura is the CTO/Co-Founder at Mitigant, an innovative cloud security startup based in Germany. Kennedy has spent over 12 years in cybersecurity and passionately explores the intersection of security chaos engineering, cyber resilience, incident response, and risk analysis for cloud security. Kennedy has published over 20 academic papers about cloud security.

About the conference

InfoQ Dev Summit Munich software development conference focuses on the critical software challenges senior dev teams face today. Gain valuable real-world technical insights from 20+ senior software developers, connect with speakers and peers, and enjoy social events.

INFOQ EVENTS

• August 27th, 2026, 1 PM EDT

Below the Framework: Why Agent Context Is an Infrastructure Problem

Presented by: Boyd Stowe - Founding Solutions Architect at Tacnode

Transcript

Kennedy Torkura: Anyone building GenAI apps? Anyone doing something around securing it, like doing some kind of security? Anyone that works in the SOC, like in the SOC team, detection engineer, red team, purple team? I'll try to explain as much.

My name is Kennedy Torkura. I'm one of the founders at Mitigant. Mitigant is a cloud security company. We are based in Potsdam, very close to Berlin. I've been doing cybersecurity for about 12 years. I've been very much involved in AWS, as you see I'm a member in the Community Builder program. Also, some of the things that I do today is based on my doctoral research, which led to the founding of the company.

AI Model Tampering Via Supply Chain Attack

I just want to start with this example, and the reason I'm bringing it up here is because on the MITRE ATLAS they have a bunch of case studies. These case studies are real, because they are based on what has happened, and they try to tell you the problem, the impact, and also remediation, and they map it back to the MITRE ATLAS, which I will talk about. Basically, at the end of this talk I expect that it's more like Red Teaming 101, so we are not going very deep in. I hope to create a bridge between people who have been doing security or maybe SRE, and how they can probably just get into something red teaming, and just probing their AI applications from a security standpoint.

Securing GenAI

Obviously, most of us, we already know why we should secure AI or GenAI. Most of these systems are connected to our companies, they are interacting with our data, and we don't want that data in the hands of criminals, we don't want it exposed. There's also compliance, which is a big motivator. Things like the EU AI Act are coming up. We have ISO. There are a lot of compliance issues coming around which we don't want to default. Of course, exposure of data is very similar to data being exposed to hackers. We want to keep it safe. We don't want our customers to find out that their data was exposed to attackers because you offer a service that is being used by AI. Of course, we have to put in place guardrails, some of that we will talk about a little bit. There's also AI content that is coming out there, and you want to be sure that this content is correct. It's not offending your users in any way, and that becomes very tricky.

GenAI Red Teaming

The idea of GenAI red teaming, as most of us know, is that you want to be able to pretend to be an attacker, so that you can probe your system in a way that an attacker would, because how would you know if the system can survive an attack? You probably have done everything you can do. You have that confidence that it will survive an attack, but that confidence is your confidence. The attacker might approach this system in a different way. It becomes pretty important that you pretend to be an attacker and attack your system, obviously in a safe way so you can hopefully discover blind spots. Sometimes it's a bit difficult. At Mitigant we find it sometimes difficult to explain to people what we are doing, because for example, if you run an attack and we say it's successful, the person is wearing the hat of a defender, so he's confused.

What is successful? Did I protect it or did you get in? It requires a mind shift to actually do this properly. Some people also argue, why should I do it? I have everything in place. I have restrictive permissions. I have this. I have that. I have bought systems worth millions of dollars and the vendor told me everything is great, but you have to test it. GenAI red teaming is about you poking that from a different perspective. We are going to look at some of the systems that are out there. Some of them we might already know, and some of them are new because of AI.

The first thing we want to look at is the MITRE ATLAS. Most people in security will know about this because it's basically a repository or a library or a database, whatever you want to call it, it explains different attack techniques which are basically the way attackers will behave. It explains it at a high level. Sometimes they get a bit deeper to tell you exactly the commands that the attacker will use. They also tell you the kill chain from initial access, execution, persistence, just to let you understand what exactly the attacker is doing, what is his motivation, what does he aim to do, and also from your own side to have a perception of the impact against your system. It's been there for a while. A lot of companies contribute to it, and most tools reference it when they find or they detect a malicious event in your system, they point to it because it makes sense for you to understand.

Then, this MITRE ATLAS is very similar. It's maintained by the same MITRE organization. You see that the design is the same, but they have some additional tactics which they have added, like you see AI attack staging, that's basically they fought from the right. They have tried to basically say, we do understand that AI systems are a little bit different and attackers will compromise it in a very different way, and therefore it makes sense that we describe this behavior in a way that is specific. This is not very new because the same thing has been done for systems like cloud, like mobile, like ICS, all of them have a difference in explaining them because they are different systems used by different people and they understand these systems in different ways. This is one very important system you have to take note of. The great thing is that you can actually combine them into a single matrix.

Here, you can see that both of them are combined, and you can see everything and you can play with it. This workbench, you can download it. You can use it offline. Some companies actually have their own internal way, they use it their own way. They name different things just because in the end your context is very important, so you defend better when you make it your own language. You can combine it in this way.

GenAI Red Teaming - Challenges

Let's talk a little bit about the challenges. We've talked about red teaming, which is traditional, which we've been doing for decades, and now AI comes in and then we have to think about what's the difference here. Traditionally, when we talk about red teaming, we are looking at technical attacks. Someone did a brute force, got in. Maybe he was able to identify a vulnerability, and then he exploited it and got in. When it comes to AI, that part is not gone, as we will see in some attacks later. When you start to hear things like fairness, you start to hear words like bias, and all of that stuff which is not really technical, but most likely if you are tasked with defending your system, they might say, you're in charge here, you have to do it. We might not get deeply into that part because we're just trying to look at the bridge between traditional and AI red teaming.

That's something you have to keep in mind. The other part is the non-deterministic aspects, which means that when you test a certain system, you get everything right, and the next time you test it, the output is different from what you saw the last time. This becomes challenging because it means that you basically have to do more testing than you did before, and you have to test it as often as possible because the behavior is not certain. We all know this when you are interacting with these systems. When you ask them a question, they will answer in a certain format. The next two minutes you ask them, they give you an answer. It might be the same, but a little bit different. This is part of what we have to do as security people or as people who are charged to maintain AI systems. Customers are going to be unhappy if somehow this behavior is inconsistent and it impacts their experience.

GenAI Red Teaming - Process Blueprint

There's the GenAI Red Teaming Blueprint, I think this is from OWASP. It's a mental model to just think about what you have to do. Obviously, it's a lot to do. As you see, you've got the model, which is actually looking at the LLMs themselves, the implementation looking at guardrails, the RAG and guardrails, control testing, the system itself, and the runtime, which is after you deploy it. You can look at this in a typical shift left, shift right. Some of these things are on the left, some of them are on the right, like after you deploy. I think that most of us will not be concerned with the model, unless you are working in an organization that is actively building models, you have to think about them. By the time you begin to use managed AI services like Bedrock, you don't have that access to do anything. Of course, you can bring your custom model and you have to take care of that. I think most people will just move forward and look at the other things, like steps 2, 3, and 4. We will look at something along the lines of 2, 3, not so much in the fourth part of this diagram.

Example 1: Amazon Bedrock LLMjacking

Let's look at the first example, which is LLMjacking. This attack actually was discovered last year by folks, I think, from Sysdig and some other security companies. They discovered that there were people, let's say criminals, who were actually getting access to models, Amazon Bedrock, and some other systems, and using it, and the owners were not aware. It's similar to cryptojacking which has been there forever, where attackers will actually take control of your virtual machines or spin up one, and they will mine cryptocurrency inside, and you don't know. In this case, they want to use these models. They don't want to pay, they want you to pay for them. The crazy part is that you might be billed as much as $42,000 in a day, because what these guys are doing is they are reselling this access to their own customers. That's the business model for them.

This is something that you have to test and know and implement some guardrails. Let's look at how this works. This is just one way it could work, where the attack starts, the attackers obviously will have to get hold of a credential that has access to that cloud environment. They will discover the models, because they want to know the kinds of models that that credential has access to. They will do some discovery, testing it, and they get it, and then ok. One of the things you can see there is their invocation logging disablement, because on Amazon when you invoke a model, that event is logged. Unfortunately, this logging feature is switched off by default. I don't know why. You have to switch it on. The attacker can also switch it off, so you don't get an alert during this process that the model has been invoked.

Then they can go forward, get access to the model they want, the one they have access to, and then they begin to use it. As simple as that. This is the way we implemented it on our platform to help customers to easily run these attacks and see whether they have implemented guardrails or means where these can either be detected or even prevented.

There are two problems I want to talk about here from the standpoint of observability. The first is that all the events that are logged against the model itself are not sent to CloudTrail. As you see here, this is a screenshot of what you get. You have to configure this to be sent either to S3 or to CloudWatch. This is where you actually see exactly how you're interacting against the model. Remember that this model is managed, so you can't SSH into it or something like that. This is, at best, what you can see just for you to understand where the attacker is making the calls from, how he's using it, or maybe you just even want to see how your customers are interacting with your model. That's something that you have to do manually. Of course, you will see something also on CloudTrail. In this case, the event name is Converse.

Converse is a general API on AWS that unifies a lot of things, and you see that. I'm showing this because, of course, one of the aims of you running a red teaming exercise might be to help the detection teams, or maybe some SOC teams are actually just small, and the guys are doing everything. In the end, you want to actually evaluate how your detection systems are working. In that case, you have to know where the logs are being sent to, and you have to know what exactly you have to do. You might actually have to write detection logic that helps you to detect these events in real time and send an alert, for example.

The takeaways, up until recently, you had to give access to models to whoever. As you see on that notification, it's on by default. Once you create a new AWS account, the access to all the models is enabled, which makes it a bigger problem. We also saw that logging is not centralized. That's something you've got to do. Access control, we didn't look at that properly, but, of course, you can implement different kinds of access control using IAM rules or policies or users, whatever, to make sure that you control how people access these systems. Attackers are not always immediately wanting to shut your system down. They might just want to get access silently and you don't know until at the end of the month when you get a bill. That's something to know about.

We will shift a little bit and go into knowledge bases. This mind map you see actually is taken from the OWASP agentic red teaming guide. Actually, it's a very nice document if you want to get deeper into this topic, because it actually went through different components of an AI system. This is just targeting the knowledge base. As you can see, there are different kinds of testing you can do to make sure that your knowledge base is intact. Why the focus on knowledge base? Because, as I mentioned before, most of us eventually will not deal with models. We will use the models as they are. They wouldn't give you the best. Eventually, you want to bring in your context. You want to tune the models to work the way you want them. The knowledge bases help you to do that. Because you can bring in your own knowledge, and then the models can actually use it to deliver more precise outputs for you.

They are very important. As you know, they are also very sensitive. This is just very simply put how the knowledge bases work. Most of what I'm saying is in the context of AWS. I think it's very similar for other AI systems. AWS basically borrows or steals these terms and brings it in and puts things on top. On the left, you see the data source, which could be S3 in most cases. I think nowadays you can also connect SharePoint, and Confluence, and things like that. This is basically connected to the knowledge base, which basically does all the chunking. It has an embedding. It does all the AI magic in there. In the end, it connects to a vector database. Recently S3 also brought in S3 Vectors, which you could use. It could be some other expensive stuff, like OpenSearch Serverless, which was super expensive before.

Example 2: RAG Data Poisoning

Now we're talking about things like data poisoning, which means an attacker goes in there and drops some data which is completely useless to your dataset. Then the AI ingests it and it impacts on the quality of output that it actually provides for you eventually. This is how that might work. Attackers might, again, gain access to your accounts. It always starts from access control. If you tighten up access control, a lot of things will be impossible. Most of the time, attackers figure out how to get access. They can actually buy it through initial access brokers, who are just in the business of stealing and selling credentials. It's a very tough problem. Here they can get access to your system eventually. This is, again, just the way AWS organizes their system. They will actually get access to the knowledge base, probe and see the data sources that are connected to the knowledge base.

Then they will basically look at the metadata, discover what is in there. All of this, again, will happen only if you haven't configured it properly or you don't have something that will give an alert that these kinds of suspicious activities are ongoing. Talking about doing it, this is something you can do by writing some scripts in Python here. This script will discover the knowledge bases. Then it will basically help you to get the knowledge bases. I think this is regional. It's not the entire account, but by region. Here this is just the output of this simple script. It's more than what I showed here. It gets the knowledge base. It selects one of them. The next point, it checks which of the data sources is connected to the knowledge base. Here it's S3. Then it constructs a random file. Then it basically will upload that file to that bucket. This is just the summary of that test. It's not the entire flow that I showed. This is something that you can start and see if there's any preventive measure or any control, any security measure.

The takeaways here, access control, again, I've talked about it's very important, because today we're creating this access. We're giving it to agents. We're giving it to technical users. If you don't take care of these credentials, they can easily be stolen and they can be used for these kinds of attacks. For us in security, you have to put in the effort to learn how the systems work. You can see that most of all these, unless you're a developer or you're deeply into these systems or into AI, you need to learn how these systems work. How do they interact? What can go wrong? That's something you can learn either by yourself, or one good way to learn is by threat modeling. Because if you sit together with a bunch of colleagues, usually you have to involve everyone, all the stakeholders. You ask them, what are you building?

Why are you building it? How does it interact? What can go wrong? This is what you might do in a workshop, and then you will learn about the system and then you can begin to imagine how an attacker can compromise or go into the systems. Of course, you can also read all the case stories that I referenced before. They're not so much. You can read them and you can begin to understand. In the end, you need to have that mindset to think a little bit malicious, let's put it that way.

Next Steps

If you take a look at the document that I referenced before, we just touched the tip of the iceberg when it comes to AI red teaming, especially for knowledge bases. There is still a lot to be done. For example, integrity monitoring. It's similar to what we did where we injected some unrelated files into the S3 bucket. In this case, we said, we want to poison the knowledge base. It could also be that the knowledge base gets corrupted for whatever reason. You want to be sure that when it gets corrupted, you're able to handle it, you're able to know before it gets injected up to the point where it's already been part of the output. You have to evaluate the rollback capabilities. I found this to be interesting, especially for something like S3, or I think in general all the major providers will have such capabilities where you can have versioning that is enabled on buckets.

Then, once you have versioning, you can actually roll back to the previous version. If you're not using S3, probably you have to look for other systems that have that capability. Otherwise in real life, if the knowledge base gets corrupted, you can as well retrain, but the time between when you start creating a fresh knowledge base, cleaning the data, all that stuff, it might be time that you don't have the luxury for. Of course, backup is also important. We had also one of the tests we did, where we were looking at how cybercriminals could actually conduct ransomware against these kinds of systems. I've not heard about it, but it's very possible, because at the end of the day, criminals want to push you to pay them based on the fact that they have in their hands something that is very important. Stuff like that can be very important. If it happens, then you have to think about what your countermeasures might be from an incident response standpoint to other aspects like disaster recovery, all that stuff. You have to think about it and consider it in your systems.

Tagging with MITRE

A little bit about tagging, because again, if we go back to traditional red teaming, and you are at the point where you want to connect between what you have been doing and what you have to do for these AI systems. What we've done here, because some of these techniques overlap, you will see that if you compare these two matrices, some of the techniques on ATLAS also are in MITRE. It's not a problem, because obviously, it just helps you to have more understanding and to actually configure your systems in a way that there is that similarity. For example, here, this technique, Discover AI Artifacts. It's very similar to some of the discovery techniques in MITRE ATT&CK, but when you read through it, you see that basically, it could be that your systems are bounded in image registries, or in S3, or wherever, in file systems. Attackers, we take the effort to navigate through these systems to use some kind of metadata or tagging or whatever to understand where they are.

These kinds of discovery attacks, I find them to be very important, because they afford you the opportunity to actually act on time. They could be very noisy, because probably your developers are also doing the same when they want to routinely do stuff, but then you have to figure out how you can create that baseline that helps you to know when these API calls are malicious, whether you're using IP addresses or user agents. There's always something that is different from the way you work. That helps you to be able to distinguish and to be able to shift signal from this noise that might be in your environment. Sure, this is GetDataSource, so that's what you see from CloudTrail. If an attacker is actually probing the knowledge base and asking, which data source are you connected to? That's what you're going to see in CloudTrail. If you have a monitoring system, most of them are actually just using the event name as a filter or as a query.

要点

We have seen a lot of things or some things about traditional red teaming and AI red teaming. Obviously, there are similarities. If you are already good in red teaming or you have been doing it, I think you don't need to feel out of place. You can actually use the basis of what you know and just understand this new stuff, and just move forward and understand. Obviously, you have to read. You have to look at things. You have to discuss, especially with the data scientists, the AI engineers, the project leads and all of that, so that they can give you more insights into what they're doing. You have to start small. You have to improve. Of course, the threat landscape is rapidly evolving. Some of these documents that I read and saw, some of them are from OWASP, some of them are from the Cloud Security Alliance. Each iteration comes out with knowledge that is different from the other one. It's moving so fast that keeping up is a challenge. It's something that we have to do, especially if it's our jobs to keep these systems away from the malicious adversaries.

Org-Wide Security Objectives

Losio: I work for actually a small engineering team. Let's say you have 30 people, maximum. You might have someone in security, but he's actually really busy in other stuff, that if you say, I want to add some AI capability or some AI features or whatever, he's going to say, keep it out of our production environment, and I don't want to know anything about it. That's the standard approach. I'd like to know, what would be the step, as a technician, as an engineer, not coming from a security background? How could on one side, avoid just doing proof of concept, and then never have a chance to bring it to the next step, or at the same time, not to impact. Because if I look at that, I feel like I'm doomed. I have no chance to make it.

Kennedy Torkura: What I've seen is different organizations have different objectives, and sometimes, the culture might permit you to start from dev, and to play with it. Some places, especially when it's AWS, because of the cost, they might tell you, no one is giving you any access. I think that it's always good to start from the dev, or if it's like red teaming, actually you can use all the AI frameworks. Some of these concepts are actually very similar, so you can actually try them on your laptop, or maybe your dev environment. For AWS, you can also try with the free tier, or the free credit, or whatever is free, and get up to the point where you don't have to pay stuff. Eventually, each of these will give you some insights. Again, if the management is not yet ready for security, because it's like, we want to deliver as fast as possible, security will come later. You have to wait, because you don't want to be kicked out of the company.

Losio: That would be a very costly mistake to make.

Questions and Answers

Participant 1: We learned that 2026 is going to be the era of agentic AI. I'm just wondering how you feel that the agentic era is going to change the threat landscape for security.

Kennedy Torkura: On the good side, I've seen people saying that probably companies don't need to buy from vendors again, because you can vibe code every security product, CSPM. That's the good side. There's some logic there, because you could do something. From a defense standpoint, you could do malware analysis. You can just give some of these file snippets, like these CloudTrail snippets to an AI and say, what happened? It will tell you what happened. That's the good side of it. Attackers seem to be more on the side of taking advantage of these systems to do their attacks. Recently, OpenAI released a report where they were talking about what they've seen in their system, what attackers are doing. It's clear that they're using it to build malware, to build exploits, and things like that. It's very tough. There are also some startups, also vendors that are beginning to build AI SOC, which is basically a SOC that is being powered by AI in terms of analyzing all the huge alerts that you get and helping to investigate.

A lot of people are like, these SOCs are still quite premature, because they are very good at understanding very specific use cases, and if some parameters change rapidly, they are already confused. It means they might not be able to really be deployed to freeze the APTs and all of that. In general, I think it's very interesting how everything is unfolding, and we hope that we can be better than the bad guys.

See more presentations with transcripts

Recorded at:

Aug 10, 2026