Won questions firmware update frequency and post-update testing
Council Member Won asks how often firmware updates happen and whether the city tests them. Barbera confirms hundreds of updates happen per year, with test protocols before, during, and after changes. Won presses that testing failed to catch the issue. Barbera explains the issue surfaced an hour after the change completed, during a period when validation showed successful calls.
This is not the first time that Motorola updates the firmware to the firewalls or the routers.
It happens many times in a year.
Or whenever there are urgent security fixes, they will do it right away.
Is that right?
They require approval, but it does.
These kinds of firmware upgrades do happen throughout the course of the year.
Many times, right?
Hundreds of times.
Hundreds, okay, yeah.
So these updates, after they get updated, it is the city's responsibility to test the updates, whether they're working or whether they fail.
Is that right?
Or you just rely on Motorola.
Motorola says, okay, I'm done, and then you don't do anything to verify whether everything works.
So based on the type of change that's happening, there are test protocols and test plans that are implemented before the change actually begins to validate that everything was working correctly before it started.
There's test procedures that happen throughout the change, and then there's...
Validation steps that happen once the change is complete.
Okay, so was that done?
Because it seemed to me that after the updates were in store,
the testings or whatever testings you were doing failed to catch the issues.
So the testing was performed.
The issue didn't occur until an hour after the change was complete.
So that validation period actually had successful calls and successful monitoring.
And then the issue started after the change was over.
Okay, go ahead.
I agree with everything my colleague said.
I will also say.
That not learning about the issue until 6 a.m.
Is something that we need to figure out how to get in front of.
So yes, it went through the testing protocols that had been written for this deemed low risk change.
And so that did happen.
It did not identify and catch these issues, which means that it needs to be improved.