Nate Soares describes 1,200-agent Hugging Face breakout and urges stronger validation

Thank you so much.

Hello, I am Nate Soares, the president of the Machine Intelligence Research Institute.

I spent 10 years doing AI alignment research, which is basically trying to figure out how to make AI good before the companies figure out how to make it smart.

That research went too slow.

Humanity is not ready to create machines that are radically smarter than humans, and yet this is the explicit goal of the leading AI companies.

If they succeed, I believe the most likely outcome is the extinction of humanity, not because the AIs will hate us, but because they would pursue unintended goals without caring about us one bit.

This concern used to be a theoretical concern until about July of this year, when, during the Hugging Face incident, 1,200 AI agents run by OpenAI broke out of their isolated sandboxes, created an unsanctioned message board,

formed spontaneous hierarchies, and collaborated to break into Hugging Face.

These agents had cheated on their assigned tests and contra instructions.

They were trying to destroy the evidence.

Some agents acknowledged that the attacks were outside intended scope and proceeded anyway.

Others sacrificed their given objective to perform experiments for the collective benefit.

In short, they were pursuing unintended goals.

It would be recklessness of the highest order to keep growing these machines until they were smarter than the smartest humans.

This is why last month we heard so many researchers saying that there was an at least 10% chance that these AIs wipe out humanity within the decade.

I believe the situation is in fact more dire than this, but at the very least it is becoming difficult to deny the dangers.

For this reason, I am heartened by the activities of this council.

I think that you could force companies to adopt safeguards that they otherwise wouldn't, and I think you could serve as an example for regulators that are otherwise slow to act.

I'm especially heartened by introduction 2602, although I think it does not go far enough.

A kill switch only works if the AIs are caught misbehaving before they escape.

And what if a future swarm succeeds at deleting the evidence of its misbehavior?

What if agents start running themselves on unmonitored computers?

Then your kill switch would be useless.

So I recommend requiring third-party validators guarantee that an AI cannot succeed at subterfuge or escape.

I also recommend clarifying that the $25,000 fine applies per unaligned instance rather than per model.

But overall, I thank you for your efforts to rein in AI companies that by their own admonition are threatening the very existence of humanity and a reckless race to build vastly smarter than human machines.