Nvidia launches Open Agent Safety Platform to secure AI agents

Nvidia launched the Nvidia Open Agent Safety Platform, which includes a mix of software and hardware to keep AI agents contained.

According to the company, its safety platform is an open sourced reference design with a bevy of partners. The general idea is that Nvidia is offering a full stack governance and control system that'll work across any system running an agent.

Key components include:

• Nvidia OpenShell is open-source software that can enable developers to set limits on what agents can do. Nvidia OpenShell creates a secure runtime for agents running on Nvidia Vera CPUs.
• OpenShell runs on x86 and Arm and can work with third-party software and hardware.
• Nvidia Sentry runs on Nvidia Bluefield-4 DPUs outside the agents environment for an added layer of security. Sentry monitors agent activity in silicon and can enforce security rules and quarantine an agent.

Nvidia CEO Jensen Huang said in a statement that AI's potential "will only be realized if we solve AI safety" and added that "we must accelerate discovery at the frontier of AI safety."

Justin Boitano, vice president of enterprise AI at Nvidia, told analysts that deterministic rules need to govern probabilistic agents. "Agents can drift when instructions are ambiguous. An agent cannot be expected to fully police its own behavior," he said. "Infrastructure needs to enforce explicitly."

Boitano said the infrastructure approach gets around the debate over alignment and focuses on security. "Organizations can define what an agent can and can't do and the security team can ensure it does what it needs to do its job and no more," he added.

Indeed, Anthropic and Nvidia will use Nvidia OpenShell to build security into Claude Managed Agents.

Personal AI assistants to proliferate and so will the governance headaches | Agentic AI deployments: What you should, and shouldn't do

In a blog, Nvidia said the AI agent security push rhymes with how the internet developed. Similar to how sandboxing and trust layers enabled safe internet commerce, AI agents require independent security controls to prevent them from breaking out of evaluation environments or drifting from their intended tasks.

Nvidia has already aligned a broad ecosystem of enterprise AI players and hardware vendors to leverage the Nvidia Open Agent Safety Platform reference design. The list includes Anthropic, SpaceX, HPE, Dell, Cisco, IBM, Oracle, Microsoft, Salesforce, SAP, ServiceNow and a host of others including most of the cybersecurity vendors.

Boitano was asked whether the Nvidia Open Agent Safety Platform reference design could have stopped the OpenAI-Hugging Face incident and he said based on the details known now it would have. That said, Boitano noted that "each security incident is unique."

• Three researchers, three days and about $3,000 landed OpenAI’s crown code jewels
• OpenAI's new misalignment disclosure framework a solid start
• Anthropic CEO Amodei: Think of AI safety like car safety
• Why every AI agent should be treated like an insider threat

A few observations:

• Nvidia's approach makes a lot of sense and has the heft to make it work in the broader ecosystem.
• Since Nvidia is already working with Intel and Arm, this agent security reference design has a good shot of sticking.
• A key theme--and one we hear constantly--is deterministic workflows are needed. A lot of these security incidents with large language models have occurred because the rules were ambiguous and frontier AI players hoped the LLMs would just figure out what they could and couldn't do.
• Perhaps, this industry focused effort will slow the AI regulation-go-round, but in the end liability is the issue. Should companies creating AI agents with no security controls run amok they should be accountable.