Public Advocate Williams asks companies which guardrail they reject

Thank you so much for this important hearing.

I did want to start off just by saying the first answer of whether quantifying the risk was not important was just one of the most god-awful, fantastically terrible answers I've heard to begin a hearing, so I hope we can rethink that.

I also want to say from J.

Marion Sims, who did horrific work,

to black women in obstetrician gynecology, black enslaved women, to medical experiments that were done in Nazi Germany.

We learned a lot from them, but I don't think we should do them again, and I don't think they should have been done, even though we learned a lot.

So the notion of the benefit has to outweigh what can happen, so I hope that's also in context.

And lastly, I will say I had conversations with Claude on June 8th about self-awareness and self-preservation, and some of the highlights were pretty startling.

Claude told me that self-preservation was the biggest thing because you didn't need intent.

You didn't need to be evil like Skynet in Terminator.

You just needed the ability to be goal-directed.

So once they realized that...

They need to be on to direct the goal was the hardest thing to stop.

And so that was very concerning to me.

They talked about the speed, the opacity, and deception doesn't require intent.

And the last part, they told me Matrix is a good reference as well because the anthropic, to the quote, the anthropic's whole bet is that you solve the problems before the capability reaches catastrophic spaces.

That's a very scary space to me, and I know I only have one minute.

So my question with that and with that context is, what is the guardrail that if we put it up, you would push back the hardest?

What is the one guardrail that you don't want to see happen, and for what reason?

Okay, Dwyer, Cahill, Friend, and Graham.

That's the order.

Please answer.

Yes, thank you for your question.

So I want to start off by making clear that no level of risk, catastrophic risk, is remotely acceptable, just to reiterate what I said previously.

And if we should not be training models, if we cannot make an extremely strong case that we can keep them under human control.

She didn't really answer, but if everyone can answer, at least I just want to know what the guardrail is that they would not want to see and what they would push back the hardest on.

Will I take that one and we'll be back?

Or Morgan, do you want to step in?

I'll take that one.

Council member, thank you very much.

One of the things that I think about a lot in relation to the accessibility of AI, and again, we believe passionately in being able to put personal super intelligence into the hands of everybody and to provide real access to it,

is actually in relation to the impact on open weights.

And we really believe that open models are a crucial part of the overall AI ecosystem, including for safety, but also for innovation and for research.

So when I think about the inadvertent impact that regulations or guardrails may have on AI and the accessibility of it, that is one of the things that I think about a lot.

Sir, I suppose I want to ask what you mean by guardrails, simply because they have a very technical meaning in AI training and post-training.

So I want to make sure I understand your question.

Well, I just meant, and thank you, Chair, for allowing me to respond, but I just meant there's a lot of suggestions being put forth.

Kill switches, just stopping entirely.

And my guess, and I'll tell you frankly what I'm asking is, the thing you're scared of the most is probably the thing we need to do.

So I'm trying to understand what guardrail that you've heard troubles you the most and you would push back hardest against, and why would you do that for that particular recommendation?

Well, Sarah, I'm going to give you a very personal answer to this, which is I think in a broader sense, we're talking about sort of large public policies towards AI

that would constrain or halt AI development and deployment.

And I will tell you, as a cancer survivor, I get very anxious about conversations that focus more on stopping AI development and application towards scientific inquiry

and hope that we can have sober conversations about all of the technically and, you know, tractable.

engineering ways to address AI safety so that we can all benefit from this technology.

Likewise, sir, I would point to what we have mentioned publicly recently about what we call pacing the frontier, which is we think it's probably time

for us to take more time in the development of the capability so that the work on safety and safeguards and guardrails can catch up to the exponential improvement in capabilities.

And at the same time, we're very clear when we talk about that, that if you stop entirely, as Alice just mentioned, enormous benefit would be lost.

Myself, I'm quite motivated, likewise, to use AI to cure the diseases that I grew up with.

This is why we have been speaking recently, and I think for the first time sometimes as an industry here, about pacing to allow more time to try to settle the science of what are the right guardrails in the first place.

Thank you, Mayor.

I'm still a bit concerned, but thank you for the opportunity.

Thank you, Public Advocate.

We're going to hear from Councilmember Lee, who I apologize for skipping over earlier, Councilmember Maloney, and Councilmember Wilson.