De La Rosa presses on algorithmic bias evaluation commitment

Is the New York City Public School currently building the capacity to review for algorithmic bias in all of their approved tools and all the tools they plan to approve?

I think I just located the section that you're talking about in the guidance.

And I just wanted to ask Dr.

Pete if she can come up.

I actually just quickly sent, I think, the thing that you see.

And I just wanted to know if Dr.

Pete can speak to it.

Yeah, I'll read it to you.

Bias and equity review.

Irma currently evaluates data privacy.

New York City public school system is actively building the capacity to also review for algorithmic bias, equity impact, and instructional effectiveness.

But I'm focused on the bias and equity impact piece.

Sure.

So we started with the instructional effectiveness with the tool evaluation framework.

The equity impact that we're looking at, particularly in...

Particularly in tools, we started with exploring that through our equity and ethics and equity fellowship.

I want to make sure I get that naming right.

A teacher training fellowship over the past year.

In terms of bias and equity impact in tools,

given the amount of tools and even across the private sector, there are not bias evaluation of tools readily available.

And so as the New York City public school system, we're looking at how we can actually work with our existing big partners on the enterprise level.

And I'm looking at our partners at DIT.

As well with the directionality of OTI and what the city is going to do with bias evaluation and equity impact because it's a new area, just like the research within AI.

But algorithmic bias is nowhere near a new area.

I mean, it's in AI now.

You're asking.

But is...

It sounds like you're going to be having conversations.

Your website says you're building the capacity, but it sounds like you're kind of talking about going to be working with enterprise partners to figure out what maybe, how to figure out what bias looks like.

We're not.

That's kind of what I'm hearing.

I understand that's what you're hearing.

I respect that as well, and I hear that.

So let me try and clarify.

Thank you very much.

Sure.

In terms of tool evaluation at our size and scale, algorithmic bias is something that we can evaluate for.

Doing it at the scale and across the number of tools we have in New York City public schools is something that we have to do in development.

It's not something that we have for you right at this moment.

It is something that we are thinking through, we're working through, and looking at what's the availability of us in terms of what tools are available to actually evaluate that, the algorithmic bias within all of our tools at the various levels.

We have enterprise tools, we have smaller ed tech tools, we have curriculum supplemental tools, so it's across a wide number of tools.

So whose decision in the DOE is it to approve tools in the

first place without discussing or figuring out what the bias or the equity impact is?

The first step of any tool approval is for a principal, a superintendent, or a central leader to demonstrate its educational value and impact.

That's step number one within the IRMA process.

So it has to be an educationally purpose-built tool.

That's number one.

I'm going to pass it on the bias part, not because I'm passing the question.

I'm passing it to Dennis in terms of what our current expectations and our current...

The current IRMA process has in terms of what vendors can share to us, it's not a requirement within our process at the moment, but it is something that we're looking to build into it.

So I'm going to pass that to Dennis.

Yeah, thanks, Tara.

Right now, again, the legal team at least does not evaluate for bias.

I'm not exactly sure which part of the website you're referring to, but if you could get it to us, we could try to correct that because I know there's other parts of the guidance and website that...

Are clear that the IRMA process does not yet evaluate for bias.

And I think when we're talking about tool approval also, we're talking about approval just from a data privacy and security compliance standpoint, which is really just

the minimum floor for a tool to be used with student data.

I think in order for an AI tool to be approved,

Like it has to go through additional layers of approval beyond that.

IRMA is just evaluating for the data privacy and security portion.

I understand.

But given the explosion of AI and the challenges we face, can the DEA commit to ensuring that bias and equity impact are part of the tools that you use to evaluate these tools, which

you've already released into the classroom and already released to our schools, but can that be one of the things you commit to doing?

So let me get to the answer first.

So I just, I have a printout of the guidance.

Me too.

Yes.

So the section that you're referring to is under gaining clarity together, concerns and considerations.

Earlier in the...

Document there is a section three how tools are evaluated and we clearly say there that it uh the irma process currently reviews tools for data privacy and security it does not yet evaluate for algorithmic

today's not my day with this word uh bias it is that one word it is it really is

um it's a terrible word i just wanted to to show the nuance of this is how our tools are evaluated in section three and that that is under the considerations and sort of next steps for that.

And in terms of like bias and tools, you know,

You know, bias lives well beyond our tools as well.

It's part of our commitment around ensuring a culturally responsive and sustaining curriculum.

It's been part of our...

of our investment even in the hidden voices, the work behind the integration in our curriculum, because we also know that just our curriculum, you know, we're always checking, is it representative, is it biased, what other,

you know, primary source documents can we bring into a curriculum?

So in terms of the ask on the AI tools, I'm just gonna take that as under consideration.

I don't, I,

Can I commit to even the capability to do that as part of what we hope to consider?

I just don't want to lock in on the feasibility of how we would be doing that yet.

Okay, I mean, I think you should.

I mean, I think there are certain baselines, and I think what was testified to earlier, it seems as if there's a very minimum standard for approving tools that...

You know, that can pose a danger cognitively, emotionally, academically to our students, and that there aren't certain, that we're not taking this a little more seriously in terms of like, you know,

the minimum standards is a little shocking, honestly.

You know, that we are putting the car before the horse by releasing these tools with the, I don't know, the same guidance, it seems, that we're using for any other vendor.

But this is a new world, and we have to really take seriously all of the elements that have been brought up today, including bias and equity.

And I do want to quote, because Mr.

Doyle asked where it was in the

Armor process, step six.

According to the website, and I understand it's a typo, but I did want to cite it for you.

Step six, legal and compliance review.

The teams review legal terms, privacy protection, security measures, instructional values, and AI-specific issues like bias and transparency.

Number six.