Charlotte Pope of NYCLU on student surveillance industry
Pope from the NYCLU focuses on the student surveillance industry, noting that FOILed data processing agreements are heavily redacted under trade secrets exemptions. She describes AI products that monitor devices, track physical movements and social connections, and assign mental health wellness scores, calling them a digitalized school-to-prison pipeline and questioning DOE's enforcement of prohibitions.
Thank you so much for your testimony.
Hello, my name is Charlotte Pope.
I'm with the New York Civil Liberties Union.
Thank you for convening this hearing.
My comments focus on the student surveillance industry, where AI represents an imprecise blanket marketing term with no set meeting.
A note up top that we foiled for nondisclosure in data processing agreements, and they are full of redactions under the trade secrets exemption, and the refusal of ed tech companies to explain how their AI systems were built and what they intend.
tend to do long term, protects these vendors from public scrutiny, and has enabled big data systems that claim to do one thing to accomplish another thing, which is the accumulation of student information and public money.
Even where we can explain how a system works, there's still the question of whether certain systems should exist in schools at all.
Schools are testing grounds for AI products that monitor school-issued devices, even simply typing and drafting of ideas, tracking physical movements and social connections in friend groups, and synthesizing moment-to-moment personal data spanning across years.
Some companies claim their AI surveillance systems can measure and track student emotions like aggression or generate and assign mental health wellness scores.
All of this produces floods of sensitive student data for private harvesting and opens the door to a digitalized school-to-prison pipeline.
It was confusing to hear today that DOE is not sure what surveillance software is in use other than GoGuardian,
deferring that those purchases are made school by school, but also somehow very confident that certain GoGuardian products aren't in use and haven't been purchased.
As you asked, in the AI guidance, surveillance and behavioral monitoring products are both named as beyond the scope of the guidance, but also never allowed.
But what is their definition of behavioral monitoring, and what is the mechanism for getting these prohibited products out?
When students complained about AI-enhanced bathroom tracking apps, which is surveillance, DOE deferred.
to principal discretion.
The guidance claims, and the DOE said again today, that these apps must serve a clearly defined educational purpose, but we doubt that schools are convincingly articulating what they're asking these private surveillance sector companies to solve for them.
Thank you.
Thank you so much for your testimony.