AI Now's Ashna Agarwal urges friction and vendor liability in city AI contracts

Good evening, Speaker Menon and members of the Council.

My name is Ashna Agarwal, and I'm a program associate at the AI Now Institute.

The AI industry has never been louder about abstract and existential safety.

But for New Yorkers, safety is about how AI is embedded in our hospitals, utilities, city services, and workplaces.

Facing estimated trillions in projected revenue shortfalls, AI companies are racing to lock in public customers like New York.

My testimony addresses how these financial interests conflict with the city's safety and how the council can protect New Yorkers.

The city must maximize friction if and when it integrates AI into our systems.

No AI system should enter city operations without an affirmative decision by the city.

That includes free and discounted offers and AI upgrades that are slipped in through software updates and contract renewals.

Second, the city must address the harms New Yorkers already face.

The council should ban the use of facial recognition technology in public accommodations and in policing, requiring notice when AI shapes decisions about New Yorkers.

and surveillance pricing and wage setting.

Finally, we cannot lose sight that the concentration of power in AI and the single point of failure it engenders is a safety risk for New York City.

City contracts should require vendors to meet rigorous security standards, disclose breaches and outages promptly, and assume financial liability when their systems fail.

New York City still holds the leverage to decide which systems enter its agencies and who bears the cost of failure.

I urge the council to use it now.

Thank you.

Thank you so much for your testimony.

Up next, we have Tony Tice-Durette.

If you're online, please unmute yourself and you may begin.