Schulman questions companies on bioweapons and DNA synthesis safeguards

Councilmember Shulman, followed by Brooks Powers, followed by Brewer.

Thank you.

Good afternoon.

I am Councilmember Lynn Shulman.

I'm chair of the New York City Council Health Committee.

As a breast cancer survivor, I am looking forward to the day AI can help find cures for that and other diseases.

That said, there is a dark side to AI intervention in health care.

Most viruses that are found in nature can be contained fairly quickly.

But an artificial pathogen has no constraints.

In August, a team from Stanford used an AI model to design an entirely new virus from scratch.

In addition, Anthropic has its own AI-powered biology lab.

What measures are being put into place such as placing, I'm not going to say guardrails, I had that originally, safeguards on DNA synthesis to prevent rogue viruses from being created and replicated at will to jeopardize the public?

Thank you for your question.

The question about how AI could enhance or make worse biological risks is central to OpenAI's preparedness framework.

That's the framework that we use to guide our decisions about model risk and whether we have the right safeguards in place.

So we evaluate for biological risks as part of that framework.

Again, we use third-party evaluators and red teamers.

We also work with the federal government as it pertains to national security risks.

And we are transparent on the results of those evaluations.

So with the launch of every major model, we release what's called our system card.

And that card details the evaluations that we performed as well as the results.

And those evaluations, as you mentioned, do include biological risks.

Council member, this is such an important area in question.

So thank you for raising it.

Biological risks is one of the core areas, one of the key risk domains that's part of our scaling framework at Meta.

We have, as I mentioned earlier, have a multi-layered approach to risk assessment across training,

evaluation and release.

And that includes adversarial testing with external experts.

We work with governments, other stakeholders and experts in relation to that.

And as I mentioned earlier, as part of our commitment to partnering with independent third party evaluators and assessors also.

Yes, ma'am.

Similar to our colleagues, we also have bio or biological capabilities as part of our frontier safety framework.

As we look at our critical capability levels that I referred to earlier or the levels at which we would trigger additional mitigations in our models, we look for what's called uplift.

That is to say, the model will be able to assist users in creating information or knowledge that's not otherwise available through widely accessible sources.

And we think very carefully about how to measure for bio CCLs and are constantly upgrading and updating our ability to do that.

When it comes to those mitigations or those safeguards, as you so aptly put it, we implement those at both the model and the application levels to ensure that

casual users cannot access critical bio.

It is very important to consider the dual use nature of biological capabilities, of course, because these are the kinds of capabilities that scientists, particularly

medical investigators, need in order to make advances in our own health care as well.

And so we think very carefully about how to release models to scientists that can use the technology responsibly.

We did that for the first time with our AlphaFold model, which, of course, is how millions of investigators around the world are able to.

Look at the folding of proteins and to use that research to find cures for diseases.

So there's a real careful approach that we take to ensure that science can continue, but that dangerous uses of biological knowledge cannot.

Likewise, we take this extremely seriously.

In fact, if I'm not mistaken, it was my team and myself that developed the first call for action on these risks, the first evaluations, the first tests, the first industry-government partnerships on biological risks and evaluating models and their safeguards

several years ago at this point.

Not only that, I personally think we need to go further here.

You mentioned DNA synthesis screening.

I think this is personally a no-nonsense approach.

There's much else we can do to work on biological defense.

And I will, you know, alongside the other companies, we maintain the same and, in fact, try to compete for the best safety practices here.

And you can see some of what we found, I think, disclosed for the first time ever in a recent threat intelligence report, where we have observed actors attempting to do biologically risky activities with models.

The threat is, I think, clear to us now.

We think we need urgent action.

But I will close by saying that, you know, as somebody that grew up with a number of chronic diseases, going blind and couldn't walk, I very intimately feel that the upside of the same capabilities is enormous.

But increasingly in my work, I feel optimistic that we can defend against the downside sufficiently.

If I could just say just one in closing, the upside is very, has a great potential.

The downside is extraordinarily detrimental to humanity, but you should consider about screening orders for synthetic nucleic acids, which is DNA, and the equipment that's needed to make them, because right now AI

is now outperforming PhD-level virologists.

So an AI can take this and go somewhere else with it.

So that's something that's really important to have those safeguards.

Thank you.

Thank you.

Councilmember Brooks Powers, followed by Brewer, followed by Narcisse.