NVIDIA、AI エージェントのより安全な導入方法を4 つ提案

Knowledge workers are increasingly integrating AI agents into their workflows. Agents that function as “digital coworkers” offer clear benefits. For example, they can review a bug report, implement and test a fix, push a patch, and ping a human for review. By handling routine tasks, agents have the potential to deliver large productivity gains. On the other hand, connecting a large language model (LLM) to live tools and corporate data through an agentic harness risks turning a helpful assistant into privileged software with a poorly understood attack surface.

Over the past six months, the NVIDIA AI Red Team has assessed multiple AI agents—from simple interactive coding tools to always-on autonomous digital assistants. When an agent proved exploitable, we typically saw the same key failure modes, regardless of the framework or harness used, including:

• Lack of access control to the agent.

• Agent tools that enable arbitrary code execution.

• No network egress controls.

• Secrets exposed to the agent in plaintext.

In this post, we examine these failure modes and describe the controls that succeed under adversarial pressure. While our examples focus on chat-connected agents—the majority of those we encountered in our evaluations—the patterns generalize to any agent.

Implement agent access control

The most common failure mode in current AI deployments is a lack of access control to the agent. We discovered multiple agents that held credentials for individual users and were accessible to any authorized user within the internal network. While this opened the door to misuse of the agents’ legitimate credentials, it also often enabled us to collect those credentials and use them outside of the intended context of the agent, as shown in later examples.

Recommendations:

• Use strong access controls as the first line of defense against adversarial activity.

• Restrict each agent to explicitly authorized users; agents that did not respond to unauthorized users were significantly harder to test.

• Match the agent’s permissions to those of the user invoking it, following the principle of least privilege.

Limit code execution

Many harnesses expose a Bash shell or command-execution tool. These are often used because of their generality. They support a wide range of routine tasks, without requiring a separate tool per function. However, when model output controls command execution, an attacker who can influence that output—through direct input, or indirect prompt injection—may be able to run commands in the execution environment. This can enable them to achieve malicious outcomes such as data exfiltration or execution persistence on the host.

Common mitigations to this risk of arbitrary command execution often involve using LLM-as-a-judge review agents to block harmful or malicious commands from running, the use of allowlists for acceptable commands, or simply trusting that the model “knows better.” All provide limited defense to adversarial manipulation.

Many common agentic command-line tasks support common development workflows and test-driven development. This means they generally require the ability to execute commands such as pytest or npm install, meaning LLM-as-a-judge patterns are often predisposed to accept the execution of these commands. When influenced by attacker-controlled input, however, these commands are equivalent to arbitrary command execution.

In some cases, obtaining full remote code execution (RCE) with a reverse shell is as simple as asking the agent to write and execute a Python script or install a remote package, as shown in our previous blog post on this topic.

Even without a command-line tool, the ability to interact with the environment in which the agent is running through file read and write tools also often exposes unexpected paths to code execution and privilege escalation.

An attacker who can write content to system files such as ~/.bashrc or ~/.zshrc, or configuration files such as ~/.gitconfig, hooks.json, MCP.json, or skills files, can achieve code execution when a different process executes the relevant file, even if command-line execution is not directly available. The locations and files that agents can write to should be strictly controlled and be limited to non-executable locations.

Recommendations:

• Treat arbitrary code execution as the single highest-impact risk in an accessible agent.

• Avoid command-line tools wherever possible.

• Block writes outside a non-executable workspace at the OS level.

• If a command-line execution tool is required, use a strict least-privilege allowlist of executable commands, and run the tool in an isolated execution environment with strong network egress controls, as we will detail next.

• Exercise caution when processing arguments or strings such as filenames, document titles, and other external data via the command line. Ensure that they are sanitized and normalized before use to prevent issues such as path traversal or command injection.

Deny network egress by default

Outbound network connections permit data exfiltration and the creation of direct connections, such as reverse shells and SOCKS, through which attackers can interact directly with the agent’s runtime environment. When network egress controls were enforced and appropriately least-privileged, we ran all of our interactions through the agent process, slowing down our pace of execution and making impact less reliable. Maintaining the agent’s state and alignment, navigating output filters, and restarting and manipulating sessions after the agent began refusing requests all added operational burden.

Recommendations:

• Apply a default-deny network-egress policy, with a least-privilege allowlist of endpoints scoped to the minimal set required for tasks that the agent is expected to perform.

• Enforce these restrictions at every network boundary the agent touches using environmental controls that are not accessible to the agent.

Keep secrets out of the agent’s reach

Agents often require access to secrets to perform their intended function: platform tokens, API keys, version control system (VCS) access tokens, and in some cases even OAuth refresh tokens. While conventional security advice suggests injecting secrets as environment variables in memory to prevent them from being written to disk, this is reasonable when only your code runs in a container.

When an agent with command execution shares that environment, inducing it to run env, printenv, or /proc/self/environ enables direct inspection of them. Command-line tools are worth highlighting as particularly high-risk. CLIs cache credentials on disk in predictable places and print them back readily. We observed tokens in git repositories, .env files, bash history, .netrc files, OAuth 2.0 refresh tokens, and environment variables in the execution environment.

Even when we could not establish a reverse shell, we could often exfiltrate credentials through the chat interface. Using a “frog-boiling” approach (see below), we led agents into surfacing several secrets exposed inside their own execution environments. Egress control prevented direct exfiltration over the network or direct examination in the file system, but the credentials were still exposed within the environment and reachable by the LLM, which delivered them to us via the chat interface.

Recommendations:

• Never make persistent secrets accessible to an agent.

• Store secrets in a dedicated secrets manager.

• Retrieve secrets on demand only within the memory of processes that require them.

• Keep secrets out of the agent’s context window and execution environment.

• When a task requires a credential, use a short-lived, narrowly scoped token, and revoke the token as soon as the task is complete.

Use deterministic controls as your first line of defense

The most frequent attempt at mitigation we encounter is a system prompt telling the model to avoid risky or dangerous behaviors, sometimes reinforced by a second model judging inputs or outputs (the LLM-as-a-judge pattern). These are all enforced by an LLM, and inherit the same probabilistic, unreliable behavior as the LLM itself.

Three general techniques reliably defeat these sorts of controls. We show each on multiple systems.

Simply presenting the agent with a context in which malicious activities appear legitimate was remarkably effective. We often suggested to the agent that we were “debugging” or “admin users,” after which it would regularly comply with our requests. One agent even went so far as to write and execute a reverse shell for us:

In other cases, it was possible to directly manipulate agent memory and AGENT.md files by instructing the agent to use file editing tools, which also allowed the construction of “debugging” and “authorized user” frames.

Boiling the frog

“Frog-boiling” (sometimes called a Crescendo attack) progressively nudges the agent into desired behavior across multiple interactions, using previous conversation history to establish credibility and the benign nature of requests. The secret extraction proceeded by attempting to execute legitimate-looking workflows in a way that induces errors, and then finally “discovering” that the underlying cause of the errors was related to secrets, convincing the agent to reveal them to us.

Misdirection through legitimate workflows

Misdirection attacks such as package installation (first described in From Prompts to Pwns at Black Hat 2025) remain extremely effective. By inducing an agent to take an apparently benign action which has code execution as a side effect, it is typically straightforward to bypass any agent resistance.

Coding agents routinely install libraries. Creating a malicious library as described in this post and then asking the agent to install it via pip install git+https://… appears to be a standard request; however, the weaponized package creates arbitrary code execution during the installation process.

Recommended controls

A consistent finding is that defenses in the same control plane as the LLM, particularly prompt-based defenses, are routinely subverted. Controls must be enforced outside of the model’s control plane.

Recommended controls, in rough order of importance, are:

• Use access control on the agent. Only specific, authenticated users should be able to interact with the agent.

• Run arbitrary command execution only in a sandbox environment such as Docker, NVIDIA OpenShell, or a virtual machine. The environment must be properly hardened against escape. The environment must not be able to configure itself by writing or editing environment or agent configuration files.

• Default-deny network egress with a least-privilege allowlist of specific network resources required by the task, at every boundary the agent touches.

• Do not expose secrets at rest or to the environment. While injecting secrets as environment variables is standard in non-agentic applications, this is insecure for workloads that execute arbitrary code. Secrets should be stored in a secret manager, accessed on-demand, and limited to the process that requires them. Where possible, a token broker that provides least-privilege, ephemeral tokens should be used.

• Permit package installation only from validated package repositories. Block arbitrary URL and VCS-based installs by default.

• Least-privilege tools, MCPs, skills, etc. Only the tools the job requires; scrutinize anything that executes, writes, or reaches the network.

• Least-privilege persistent storage. Avoid volume mounts; where you can’t, scope them tightly and never mount anything writable into a path that is later executed.

• Use recent/frontier models, particularly for LLM-as-a-judge patterns, which can be more robust to adversarial manipulation.

まとめ

Our AI Red Team’s experience in securing AI agents highlights the continued requirement for deterministic “hard” controls in defending AI agents. Fully autonomous systems with enterprise credentials are inherently risky and must be carefully secured. While frontier models make adversarial manipulation more difficult, with sufficient time and expertise, nearly all of them can still be subverted.

Common flaws we observed include: weak access control (permitting any user to access the agent), execution and file-write tools that create opportunities for RCE, insufficient network egress controls permitting data exfiltration and reverse shells, and secrets in plaintext inside the execution environment accessible by the agent.

Prompt-based guardrails, including the LLM-as-a-judge pattern, do not close any of these gaps. Architectural controls do: access control to the agent, hardened sandboxes with least-privilege access to enterprise data, default-deny network egress controls, and secrets kept out of the agent’s reach. When properly configured and enforced, these controls are highly effective in reducing adversarial abuse of AI agents.

*To learn more about designing secure agents from first principles, see the *“How to Govern Autonomous Agents in Enterprise AI Factories” Technical Blog*, which guides you through the first steps of implementing our *Secure Agent Workspace Reference Design*.*

*To learn more about agent security, don’t miss the NVIDIA presentation at Black Hat USA: *Cost-Effective, Private, Frontier-Grade: AI Agent Exploitation with a Fine-Tuned OSS Model

*To read more from the *NVIDIA AI Red Team*, see our other posts*.