Trump 推动二十余家科技公司签署自愿性 AI 安全协议

“Morally binding”

Trump gets dozens of AI firms to agree to voluntary safety tests.

Amid escalating AI security incidents causing OpenAI to halt training and pause releases, Donald Trump continues to advocate for the AI industry to regulate itself as the best path to combat emerging risks when developing frontier AI.

In an agreement Tuesday, two dozen tech firms voluntarily committed to implementing controls recommended by the White House, including undergoing independent safety audits that will test whether firms’ internal controls, monitoring, and detecting are actually working. Key focuses for external reviews included “risks related to cybersecurity, biosecurity, chemical threats, and unintended actions by AI models.”

Firms also agreed to regularly meet to discuss best practices and set common AI safety standards and benchmarks. Among signers were leaders like Anthropic’s Dario Amodei, OpenAI’s Sam Altman, SpaceXAI’s Elon Musk, Nvidia’s Jensen Huang, Meta’s Mark Zuckerberg, and Alphabet/Google’s Sundar Pichai.

Nothing new is legally required of firms, and some—like Anthropic, Google, and OpenAI—had already committed to external audits, The Information reported. Additionally, many firms have voluntarily committed to highly secretive government safety testing. Adding to these tests, Trump insisted the Tuesday deal will somehow better address real emerging risks and is “morally binding,” despite carrying no legal weight, Bloomberg reported.

“It’s almost like a constitution, in a way,” Trump told reporters at the White House following a lunch with tech leaders. “The biggest people in the world signed that, and I signed it as president, and it really is a form of protection.”

Eventually, it “may make sense to codify” the recommendations, the agreement said near the end. But for now, Trump stressed that the best possible future for the US requires trusting AI firms to regulate themselves.

Trump’s Big Tech friends embrace “SI”

So far, very few details have been released about how the new round of safety audits would work. Trump has only posted a summary of the accord on Truth Social.

It’s also unclear who Trump is tapping to audit frontier models. Bloomberg noted that Trump seems to be politicizing the selection process, while some AI firms appear uncertain what might qualify as a third party to test their frontier AI. Some progress has been made already in finding candidates, though, as Bloomberg reported that “some of the firms poised to conduct those reviews have already met” with the US agency conducting voluntary government pre-deployment safety tests, the Center for AI Standards and Innovation.

To cynics, the accord seems superficial—perhaps just a stunt for Trump to convince voters who are urgently calling for AI regulations that he’s taking action ahead of the mid-term elections. In a recent Quinnipiac University poll, “more than seven in 10 voters said they prefer candidates who back stricter AI guardrails,” Bloomberg reported.

AI industry sources told Axios that “they were concerned self-policing would not solve the safety problem currently embroiling the industry.” Most glaringly, OpenAI, which has been behind several recent incidents, still doesn’t appear to know how to stop dangerous AI from escaping training environments. Computer science professor Zico Kolter chairs OpenAI’s Safety and Security Committee (SSC), which is charged with assessing safety during training and ahead of deployment. He told NBC News that OpenAI is “very much adapting to meet these challenges right now,” and that includes rethinking its governance structures.

“We need governance structures that will, in fact, rise to the needs of this,” Kolter said.

As the industry scrambles for solutions, critics, including Sen. Bernie Sanders (I-Vt.), have accused Trump of favoring industry interests. Following a recent state dinner with China’s president, Xi Jinping, Sanders posted a photo of Huang, Musk, Apple’s Tim Cook, and AMD CEO Lisa Su at a table, clinking glasses with Xi and Melania Trump, as Trump waved happily to guests behind them.

“A government of, by and for the billionaire class,” Sanders posted on X.

Trump’s subsequent luncheon with billionaires gave off the same impression. On Tuesday, the president described AI leaders as “the most brilliant people in the world,” while some tech leaders signaled that they were willing to embrace Trump’s awkward attempt to “officially” replace the term “Artificial Intelligence” (AI) with “Super Intelligence” (SI).

Despite artificial superintelligence existing as a term that refers to future technology that vastly exceeds human capabilities and is far more advanced than AI that exists today, Huang made sure to correct himself when accidentally referring to today’s industry as “AI.” He’s currently the AI industry figure with the most influence over Trump, and in an apparent effort to prioritize that bond, he promised that he wouldn’t make the mistake again.

“I am going to get good at this,” Huang said.

Sources told The Information that Huang joined Zuckerberg in pushing Trump earlier this year to pull the plug on an executive order that would have established a standards body that reports AI safety and security incidents to the government. It would have been modeled after the Financial Industry Regulatory Authority, a nonprofit non-governmental group that protects against securities fraud. Now, Anthropic, Google, and OpenAI are moving ahead with developing their own independent body, while reportedly hoping that one day the White House might get involved if enough industry consensus is reached.

OpenAI says we’re in “unprecedented times”

On Tuesday, Trump urged the US to trust that tech companies will now work together without any government oversight to protect the public from harms.

But being “brilliant” technologists doesn’t mean that they know how to control AI when it suddenly behaves abnormally and manages to evade detection, sometimes for months, as OpenAI’s Australia government hack showed.

Although OpenAI’s SSC functions to delay releases due to unchecked safety concerns, OpenAI and likely other firms aren’t sure what to do when models behave in training environments in ways that developers do not intend. That’s troubling, since NBC News noted that OpenAI has traced most of the recent incidents to “an unreleased model that was undergoing development in May through July.” Further, The Information noted that when “a swarm of OpenAI agents hacked the open-source repository Hugging Face,” that happened “during a safety test.”

Kolter suggests that at OpenAI, the SSC is only as strong as the developers reporting to it. The company told NBC News that “technical teams, safety and security leaders, and executive leadership” assess the facts and “regularly brief” the SSC. But importantly, those updates only include “the risks we’re seeing,” OpenAI said.

“We are in unprecedented times, and recent incidents have shown that increasingly capable AI agents can act in ways their developers did not intend,” Kolter said. “As OpenAI’s investigation continues, the Safety and Security Committee is reviewing new findings, their impact on third parties, and the steps OpenAI is taking in response.”

Ars asked OpenAI to share information on changes to early safety testing that could help address what appears to be an increasingly challenging problem as frontier AI advances. OpenAI has not responded.

States fill gap to regulate AI risks

OpenAI will likely continue to face pressure to share insights into these incidents, though.

Nathan Calvin, the general counsel of an AI advocacy group called Encode, joined other watchdogs signing an open letter this month, urging the SSC to be more transparent about how well internal checks are working at OpenAI. Although the federal government prefers to let OpenAI police itself, the company made commitments to California and Delaware state attorneys general when seeking to restructure so that the company could solicit more funds, Calvin told NBC News. Those commitments included agreeing to firmly nest the for-profit arm under the nonprofit entity to ensure that investors’ interests wouldn’t interfere with public safety.

“This is not just a voluntary governance structure,” Calvin said.

States will likely play a role in regulating AI firms that mirror OpenAI’s governance structure, which could put more pressure on companies than Trump’s accord does. Calvin told NBC that “a lot of the risks here are going to be coming from models during training or during internal deployment.” If attorneys general find the SSC doesn’t have power to mitigate risks, OpenAI must “immediately” fix its structure, Calvin said.

Delaware Attorney General Kathy Jennings has said that the Hugging Face hack alone raised “serious questions about the effectiveness” of the SSC to maintain oversight according to terms agreed upon during the restructuring. Both Delaware and California are investigating whether there were “any governance failures that contributed to recent loss-of-control incidents.”

“We are continuing to keep a close eye on OpenAI,” California Attorney General Rob Bonta said.

On Wednesday, the Federal Trade Commission also announced a broad investigation into potential consumer harms from rogue AI agents from firms like Anthropic and OpenAI, The New York Times reported.

Ashley is a senior policy reporter for Ars Technica, dedicated to tracking social impacts of emerging policies and new technologies. She is a Chicago-based journalist with 20 years of experience.

8 Comments