Feliz questions history of past 911 outages—2021 FireCAD and 2022 EPO button

A few more questions on other outages that we've had in the past.

Obviously, as we know, this is not a new issue.

We've had many outages in the past few years.

So a few questions on the history of the issue and also lessons that we've learned.

The August 16th outage is not an isolated incident.

In 2021,

FDNY's FireCAD dispatch system had about three outages when it was first launched, again in 2021.

Also during the same period, the EMS computer-aided dispatch system experienced three outages that same period for a total of about four and a half hours.

In 2022, a private contractor

triggered a shutdown of the FDNY dispatch system when they mistakenly pressed a shut off button.

Also in September 2024, there was a seven hour outage in the Bronx due to a configuration error.

So, you know, this is not the first time that it happens, but just curious about the lessons that we've learned.

Can you go through each of those?

Outages and just talk about the steps, what caused the outage and also measures that were put in place to make sure they don't happen again.

I guess I'll repeat them again.

I know I went through a whole list.

The first one, the fire CAD dispatch system having three outages when it was first launched.

I will defer to my FDNY colleagues.

Good morning and thank you for your question.

I just want to separate the 911 call system from the dispatch systems because they're different, right?

So from the 911 system perspective, the F, oh, I'm sorry.

Press this button.

Can you hear me?

Okay.

So this was a 911 system outage, so it was a phone call outage, right?

So it's just separate from the dispatch systems.

In the 911 system, we're second chair, really.

We don't answer the phone.

You know, the calls come in, a PD call taker takes them, they say police, medical, or fire, and based on that response, they forward them to us.

Regarding your question on the dispatch systems, yes, in 2021, we implemented a new system.

We were replacing a fire dispatch system that had been in place for about 30 years prior.

There were some hiccups in the beginning, but things have been stable now for years on that system.

Regarding EMS CAD, that is another antiquated system that we are in the process of replacing now with a targeted go-live date of November 2nd, 2027.

In the interim, the system is stable.

It is being cared for.

And we haven't had an issue for some time now.

All right, very good.

And of course, you know, new systems, there's always glitches.

Same question.

I'll skip the second.

Out is the third one.

In 2022, a private contractor triggered a shutdown when they pressed the button.

Yeah, so that was there's a button called EPO in all data centers.

And I run.

Technically, the button is there for emergency services.

And what it does is it's supposed to be if there's a fire in a data center, the last person out is supposed to press that button.

And when they press that button, everything in the data center goes off.

A contractor that was doing some cabling work was leaving the facility, and they thought the button on the door was the release button.

They had pressed it.

Half the data center went out.

Then they pressed the other one, thinking they pressed the wrong one to get out of the door, and they pressed it, and then the other half of the data center went out.

During that disruption, though, the fire CAD dispatch system did not go out because it was in a data center that was not on that floor.

The EMS CAD system did go out for about 15 minutes because it's co-located between PSAC 1 and PSAC 2.

It was live in PSAC 1 at the moment.

It took us about 15 minutes to transfer things over because we didn't have power.

So we were able to recover pretty quickly.

Okay, thank you for that information.

Any measures put in place to make sure that something like that never happens again?

I mean, if I'm correct, there was a 90-minute outage due to one button.

For certain things, yes.

So for the countermeasures put in place, there's like a lid over the button now.

There's all signs.

Everybody's clearly trained.

Vendors are just scored out.

Nobody's allowed near the button.

We take that very seriously.

Okay, what about the September 2024 outage, a seven-hour outage in the Bronx as a result of a configuration error?