De La Rosa questions IRMA bias review and website discrepancies

Thank you, Council Member.

I want to personally thank you because I think you know how important it is to support our students with disabilities, and I just appreciate your legislation on this.

I will remind everyone, according to testimony during our executive budget hearings, that I think we're all in agreement that we do need to open up more specialized programs for students with IEPs, and there's a little roadblock called OMB.

That was the testimony.

From the last hearing.

I want to go back to the audit, but before that I just want to go to the tool review, the AI tool review.

I'm a little confused by the guidance that currently exists.

It's on your website, so if you could walk us through, that would be very helpful.

The current guidance says the Enterprise Request Management Application, IRMA, is DOE's privacy and security review process.

The guidance also says that AI tools can't be used with any student or staff data unless they've passed the IRMA process, correct?

Okay.

For the record, they nodded their heads yes.

Oh, sorry, yes.

No, it's...

Yes.

That just means you're good listeners.

That's all it means.

But the guidance says IRMA does not evaluate algorithmic bias, equity impact, or instructional effectiveness, correct?

Correct.

But you also say on another part of the website that the legal team reviews bias.

Does the legal team review bias?

No.

That may be a misprint on the website.

I'm not sure what part of the website.

You should have had AI fact check your website.

Possibly.

I'm just kidding.

Relax, everyone.

I just want to read it to you so we're on the same page.

Legal and compliance review.

New York City public school teams review legal terms, privacy protection, security measures, instructional value, and AI-specific issues like bias and transparency.

So you're testifying that that's a typo?

Well, I would say transparency is something that gets conducted as part of a legal review, but that's not...

The same as bias.

I mean, transparency.

The word bias is there.

So you're testing that that is a typo.

The legal team does not review for bias.

Okay, so it's on the website, so I'm sure your webmaster will go and delete that right now.

But this is the guidance, by the way, that's been up for however long, for weeks or since, was it March it was published?

It's been up for months, so please correct that.

And I also want to add, so IRMA does not evaluate algorithmic bias.

We've determined the legal team does not review bias either, and you'll fix the website.

And New York City Public School is actively building the capacity to review for algorithmic bias.

That is correct that that is something the public school team is working on.

Say the question one more time.

The question is, is the website accurate that New York City public schools are actively building the capacity to review for algorithmic bias, which, yeah.

Go ahead, Scott.

I guess I'll start.

For tools that otherwise look like they have a great educational purpose or maybe even operational support purpose,

I think we want to extend our review of that to bias.

We can't do all tools all the time, but for the ones that we want to use and encourage our people to use, we want to make sure that the bias is also evaluated.

There's a variety of ways to do that.

One of the best ways is to make sure the tool uses a curated data set to do its work.

And we have several tools that follow that kind of processing pattern.

But you don't have a policy that all tools, you don't have a requirement that all tools use a curated data set or just the data that you allow it to use, correct?

There are tools that will use a large language model in their operation, yes.

With a large language model, you know, takes from anywhere.

I'm saying that.