Hanif questions AI tool evaluation and beta testing process
Dr. Pate describes a tool scorecard being developed to assess educational value and student outcomes, and beta testing with 50 teachers across grades 6-12 in core content areas. Tara Carozza adds that DOE is currently only evaluating teacher-facing tools, not direct student AI use, due to concerns from the feedback window.
And then can you share, how is the necessity of an AI tool going to be evaluated before introduction?
Oh, thank you.
So as these new tools are coming, the question is how do we, you're asking how we're going to evaluate them.
I'm going to ask Tara or Mia to jump in on some of the work that's underway.
So both saying yes to specific tool or company and then evaluating them year one, year two?
So what we are working to develop is a tool scorecard that will give us a more in-depth assessment of not only the educational value add, but also impact, meaning outcomes
for students.
We are also currently doing some beta testing.
Could you share just in more kind of clearer language what the testing includes, how that is being conducted?
It's being conducted through professional learning experiences where
some teachers and school leaders, 50 teachers actually, and school leaders are engaged in programs that lend itself to AI-empowered tools.
And we are assessing whether or not these tools have instructional value add
and then collecting surveys, talking to teachers, figuring out the different creative ways that they are utilizing those programs so that we can start to better determine how we can more effectively assess
the value add.
of a tool that may be utilized within New York City public schools.
And these 50 teachers, what's the representation?
So they span from sixth to 12th grade.
In addition to that, they span our core content areas, so math, science,
ELA as well.
And will there be another set of teachers?
The goal is that we, as we develop this scorecard, we want to enlist the partnership of our educators.
It is really about like how we trust them.
To inform us more about the value add of a particular tool.
It is not about taking the word of the tool creator to say, hey, this is great and go use it.
No, what are our educators saying?
How are they utilizing the tool?
What are their thoughts around the value add of the tool?
And what are the academic impact outcomes that inform us that this is, in fact,
a tool that we should invest more time and energy into as a whole, not leaning towards what the vendor is saying is good.
I would imagine they would all say that they're great and good.
So we can't take their word for law as it relates to the impact.