Anthropic CEO Amodei: Think of AI safety like car safety

Anthropic CEO Dario Amodei likened AI safety to car safety and said there's a mix of corporate responsibility as well as government standards and regulation.

During Salesforce's Dreamforce keynote, CEO Marc Benioff briefly interviewed Anthropic CEO Dario Amodei, who touched on his weekend essay that exploded into an ongoing debate on AI regulation. There's an opinion a minute about Amodei's proposals.

Amodei posted a personal blog arguing that AI labs should better pace their releases when there are big leaps, use embedded evaluators that would have editorial freedom and access, coordinate to have common safety standards in democratic nations and put limits on unchecked AI progress. Amodei also said democratic governments should also attempt to coordinate with authoritarian governments if possible.

This missive, which apparently has the buy-in from SpaceX CEO Elon Musk and OpenAI CEO Sam Altman, is the latest chapter of the big American AI freakout. That freakout has only ramped since Jacob Coxon, an Anthropic researcher, quit over safety concerns that AI labs are building systems they won't be able to control. In a Wall Street Journal story, Coxon posted a long thread on X about his decision. President Trump has said the AI pushback about regulation is bunk.

Microsoft penned a Code of Conduct for its Microsoft AI models. It covers scenarios, behavior and even writing tone.

Amodei said:

"Maybe it's easiest to explain it by an analogy. So let's say you're running a car company, and another car company, not yours, they have some kind of safety incident, right? Something goes wrong with the brakes, you know, something with the manufacturing. Now you believe you have the best safety record in the industry. What do you do? How do you approach it? Obviously, it's very tempting to attack your competitor and say these guys are unsafe or safe. But I think the more responsible way to respond to it is to say, "Hey, first of all, let's look. Let's look at our own record, right? We may not have had this big, high-profile incident, but I'm sure we're not perfect. Let's look at everything, and we can always be better. Let's make our practices better. Let's recommit to transparency. Let's invest more in safety. Then let's organize the rest of the industry and say, what can we do to set standards for everyone? What can we do to make everyone's standards better? And then finally, there should be an international component to it, and that's essentially the three-step plan that I laid out that I laid out in the essay. The first of which we committed to, and the others of which we're going to have a dialog with the rest of the industry. And I think I think that's the way to lead the industry forward, to set an example, to say that everyone can always be better."

The rest of Amodei's talk was a plug for Salesforce's Agentforce being integrated with Anthropic Claude and create Claudeforce, but the analogy sheds a bit more light on the issue.

Amodei's comment was notable since OpenAI and its Hugging Face hack is the other carmaker in the analogy.

Constellation Research Chief Distiller Esteban Kolsky said the car analogy doesn't hold up with AI agents.

"This is what Dario is missing: you are not building analog systems like brakes. Those are easy to monitor and don't need (but will often have) regulation. You are building non-repeatable digital solutions. This is different, and where standards like SOC and ANSI matter. You don't want laws? Fine then invest in standards and compliance."

Nvidia CEO Jensen Huang, who has been commenting on the AI regulation issue, also talked about his take at the Dreamforce keynote. He said:

• “Safety is paramount in a lot of ways. It's job one. However, safety is an engineering problem. We're developing software after all. We're developing computing systems after all. It's complicated computing system, but it's ultimately a computing system. And so, so first thing is to make sure that we create the test environments for testing these complex systems, and those test environments have to be built in a safe way, and then the second thing is we have to test these products.”
• “If we're if we're not confident about the safety of the products, like all companies, then don't release it. And so that's a very obvious thing to do. You pace yourself until you are confident you're releasing something that the market would appreciate.”
• “The market forces are already there. We don't need any new laws. We don't need new regulations. I think innovation and safe products are a false choice. You could definitely have both at the same time. Run as fast as you can, but if you feel at any given point the product's not going to be safe, take a pause and make sure you get it right.”

I’ve dropped a bunch of thoughts in my notebook about this AI regulation back and forth, but here’s the recap.

• Amodei is trying to look like the AI grown-up and get ahead of the regulation backlash to AI (while poking OpenAI in the eye).
• Amodei’s essay spends a good chunk of time on recursive self-improvement, which is what makes AI agents dangerous. Well, you could just stop doing recursive self-improvement.
• And finally, Amodei forgot something. L-I-A-B-I-L-I-T-Y. If your crazy AI agents jump your sandbox and safeguards and hack me or my company (think OpenAI-Hugging Face incident), I should be able to sue you for damages. It's no different than a rabid dog that hops the fence. These AI agents are your creations and they jumped your safeguards. Liability would end this mess in a hurry or at least lead to better safeguards.
• Cybersecurity is surging. You thought Mythos drove returns for cybersecurity companies just wait what the end of the world will do. Those gains are reflected in cybersecurity shares today.
• SaaS is back. Frontier models aren't replacing SaaS anytime soon. Wall Street seems to agree. Think of all the risk you can offload to your SaaS providers as frontier LLMs plot the end of the world.
• All of this performative regulation talk makes me wonder about ulterior motives, notably cementing the current frontier model duopoly.
• It's a bull market for regulation, but Congress can't figure out its Wi-Fi password.
• Career tip go to school to be an AI auditor.