Charlotte Pope of NYCLU testifies on AI vendor evasiveness and pilot concerns

Hi, my name is Charlotte Pope.

I'm with the New York Civil Liberties Union.

Thank you so much.

Since the June oversight, we've seen a lot of scary shifts, like AI ed tech vendors providing coaching to school leaders to help them rebrand AI and sell screen time to families.

Educators tell us that their DOE inboxes are full of vendors assuring them that their AI products comply with city policy.

The chancellor said publicly that DOE shouldn't be responsible for combing through every product, but right now that burden falls on teachers, students, and families to scour vendor materials.

But these products are slippery.

After policy prohibited AI grading, vendors that once sold AI grading systems now instead call it AI feedback or AI insights or AI scoring.

An electronic hall pass vendor recently changed their branding from, quote, AI powered to, quote, a rules-based algorithm.

AI vendors are simply shifting their language to squeeze into really imprecise city policy.

When the mayor endorsed five pilot products for high schools, those vendors instantly celebrated through press releases, emphasizing how safe their products are and saying these pilots are for collecting evidence about AI.

We're concerned about how students will be monitored to collect this evidence on top of their inability to opt out of these classrooms.

Thank you, Chair, for talking about the issue of consent.

Thank you also for calling.

Alling students in these pilots test subjects, DOE's answers about data collection are worrying and we'll watch for this so-called pilot framework.

The great progress we've made makes AI governance even more important and thank you for the continued close oversight and we appreciate your partnership.

We have a lot more in our submitted testimony.

Thank you.

Thank you.