Public Safety Advocate: Hurricane Aftermath

Three major hurricanes in only a few weeks. First, Harvey hit Houston with a lot of wind and an unbelievable amount of rain. Then came Irma, which hit the islands before working its way up to Florida, and finally Maria, the biggest one of all, which devastated the Virgin Islands and Puerto Rico. Puerto Rico had been side-swiped by Irma, but Maria hit it head on. The FCC’s Disaster Information Reporting System (DIRS) was activated for all three of these storms and received mandated reports covering cellular companies, television, and radio broadcast stations. These reports do not include any information on existing Land Mobile Radio systems (LMR) used by first responders in the affected areas but some information regarding how they fared is available.

The other part of this reporting by the FCC DIRS is that it sheds light on exactly how fast the cell companies, tower companies, and others are putting their sites back online. The chart below depicts the official FCC DIRS reports for cell sites out of service and the total number of cell sites. (This number is not the actual number of cell sites in a given area but rather the total number of sites reported on a given day.) I have also included the number of customers with cable or phone service where reported to the FCC.

If you read the chart, a number of things should stand out. First is the fact that in the Harvey hurricane most cell sites stayed up and operational with only 365 of the 7,805 that were reported to the FCC out of operation in the 55 counties the FCC says made up the majority of the hurricane area. Within six days (September 5) all but 70 of the sites were back in operation. Further review of the area shows that most damaged or out of service sites were close to the gulf and were in the direct path of Harvey as it came ashore. You will also notice that the day the storm ended there were 283,593 customers without cable and/or phone service and again six days later, the number was down to 153,850.

Early reports from those in the Houston area are that most of the public safety land mobile radio and Harris County Band 14 LTE public safety broadband networks remained in operation and were heavily used during and after the storm All in all, damage to the Houston area in comparison to Florida and the islands could be characterized as bad, with Irma coming in as worse, while Puerto Rico and the Virgin Islands’ damage was the worst of the worst.

Turning back to the chart and looking at Irma’s communications system damage (which I have limited to the state of Florida) shows that during the storm, of the 14,562 cell sites reporting, 3,973 were out of service and more than 7 million customers were without cable or telephone. Looking ahead to the seventh day we see only 437 sites of the 14,730 reporting still out of service and cable and phone service had been restored to all except 893,409 customers.

Both sets of figures show the impact of preplanning and staging repair crews and materials can have on repairs to the communications infrastructure coming much faster. In areas such as the Puerto Rico it was simply not possible to pre-stage people and equipment because of dire warnings posted about Maria coming ashore as a Category 5 storm. In the case of the Virgin Islands, they started out after the storm with 82 of the 107 sites reporting out of service and by the ninth day, 73 sites were out of service BUT more sites were reported, 146 on October 1, 2017.

FCC DIRS Information by date:

Storm	Date of Storm Area Impacted	Outage Report from FCC	Out of Service Cell Sites	Total Reporting Cell Sites	Cable/Wired No Service (POP)
Harvey	8/25 to 8/29/2017
	55 Counties	8/29/17	365	7,805	283,593
	55 Counties	8/30/17	329	7,804	267,426
	13 Counties	9/5/17	70	4,200	153,850
Irma	8/30 to 9/16/2017
	Florida Only	9/11/17	3,973	14,562	7,597,945
	Florida Only	9/13/17	2,661	14,730	8,190,407
	Florida Only	9/15/17	1,254	14,729	1,691,484
	Florida Only	9/18/17	437	14,730	893,409
Maria	9/20 to 9/22/17
	Virgin Islands	9/21/17	82	107	n/a
	Virgin Islands	9/25/17	64	106	n/a
	Virgin Islands	9/28/17	71	106	n/a
	Virgin Islands	10/1/207	73	146	n/a
	Virgin Islands	10/2/17	73	106	n/a
	Puerto Rico Only	9/21/17	1,703	1,789	n/a
	Puerto Rico Only	9/25/17	2,427	2,671	n/a
	Puerto Rico Only	9/28/17	2,471	2,671	n/a
	Puerto Rico Only	10/1/17	2,371	2,671	n/a
	Puerto Rico Only	10/2/17	2,359	2,671	n/a
Note:	Cell site totals represent total reported that day, not necessarily total number of sites

As we all know, Puerto Rico was hardest hit. The day after the storm passed, the island had no power, no water, and 1,703 of 1,789 cell sites reported were out of service. We have all seen stories of people finding a lone working site and parking along the road to use the site to call their loved ones, mostly back on the United States mainland. During the first days after the storm it was reported that most UHF and VHF public safety land mobile radio systems were up, or at least usable on a unit-to-unit or simplex basis. On October 2, 2017, totals for Puerto Rico show 88.3 percent or 2,359 of 2,671 sites were still out across the entire island and 15 of the island’s counties still show 100 percent of their cells sites out of operation.

According to an article in Daily Tower News, reports from the three largest tower providers on the island (American Tower, Crown Castle, and SBA) stated through a spokesperson for American Tower that, “Of the sites we’ve been able to gain access to and inspect, it appears that the structural integrity of our towers has held.” He goes on to say, “However, a substantial amount of the carrier customer equipment on the towers and at the sites is badly damaged.” American Tower has 118 towers on the island and SBA has 100, half of which they have surveyed and reported that these towers also came through the storm with no damage. This is certainly good news for the cell providers and both American Tower and SBA are coordinating with the FCC, FEMA, and other agencies and relying on the cell companies to help determine their priority for equipment repairs at the sites.

According to another article in Daily Tower News, AT&T, Sprint, and other network operators are flying in Cells On Wheels (COWs), generators, additional satellite phones, and the other parts and personnel needed to put the sites back into operation. AT&T has a ship bound for Puerto Rico with additional COWs, parts, and manpower. Part of the issue, of course, is that as the cell sites come up they will have to be run on generators for what appears to be months until the power grid can be restored. The power in Puerto Rico was all above ground. All it takes to see how much has to be done to bring power back to the island is to view videos coming out of the area that were mostly taken during drive tours of parts of the island that can be reached by vehicles.

There is a difference in the type of generator needed for ordinary backup power and those that will have to run 24/7 for months at a time. As Houston found out, the generators at its sites came on and worked and kept their sites alive but then they started to burn oil. Trips had to be made to the sites not necessarily to refuel the generators but to replenish their oil supply. The generators being-off loaded in Puerto Rico are heavy duty and designed for heavy duty cycles that will be needed.

Folks arriving on the island to provide support and supplies are generally self-contained when it comes to communications. Certainly, FEMA and the military are, the Red Cross, as I noted last week, asked for and received 50 Red Cross certified amateur radio operators to move health and welfare messages off the island and to help with logistics at the various shelters that have been set up. The lack of communications for the most part affects the citizens and businesses on the island. This also means reporting an emergency to anyone outside a village is nearly impossible until the satellite phones being brought in are distributed to those without any communications. This issue becomes more difficult since you also need to make sure there is at least some AC for battery chargers or supply DC chargers for use in vehicles.

The Rebuilding and FirstNet’s Network

Houston faired pretty well with its Band 14 system (FirstNet). Now that Texas has opted in to FirstNet I would expect that the existing Harris County system would somehow be incorporated into the AT&T system. Further, I have to believe AT&T and the other carriers will learn a lot of valuable lessons from their networks’ operations during and after all three hurricanes. As mentioned, each storm ended up being different and worse than the one before it. The amount of water dropped in the Houston area from the rain was, in places, more than 50 inches. You have to wonder how generators in low-lying areas were able to survive since generators are not designed to run under water.

Irma was worse. More cell sites were down but then Florida covers a larger area than Houston. Still, the pre-staging of power and cell companies’ personnel and equipment proved valuable and even more help was driven in right after the storm. The cell companies probably brought some sites back up before there was AC power although we have no way of knowing how many site failures were due to site issues (power outages, antennas being blown around, or radio vaults being flooded) or because backhaul to the tower was out. You might remember during Sandy a number of fiber circuits were out because they are below ground in New York City and parts of these areas were flooded. Still, all the networks put most of their tower sites in the Houston area back online within the first few days.

The final and greatest challenge is just getting underway in Puerto Rico while the Virgin Islands have already seen progress. However, because of the devastation, lack of access, power, water, and food, the job for both those handling the power and the cell sites is most daunting. As it rebuilds its cell sites, I am sure AT&T is spending extra time and money making sure more sites are hardened to a point where they can withstand some of nature’s fury. I say “some” because I am not certain how you could ensure your hardened sites are, in fact, hardened to withstand what Maria brought in the way of wind and rain. I think the lessons learned in Puerto Rico are to harden high-priority sites better and to add more battery and generator backup, but more than anything to have backup capabilities in the form of drones (UAVs), flying and stationary COWs, and satellite backhaul sooner than this time around.

Again, in all fairness to FirstNet/AT&T, communities have not yet learned that there are people who can come in and help re-establish communications even before the area is safe enough for the general public. You would think we would have learned our lesson during Katrina when land mobile radio vendors and cell providers were staged outside the city of New Orleans but were not permitted to enter. Or we might have learned during Sandy when NYPD escorted fuel trucks in to refill generators but would not let cell folks in because the areas were deemed too dangerous. There has to be a way to let communications people and supplies in to where they can re-establish communications more quickly. I believe bringing communications back in an area should be one of the most important priorities of the people responding to the disaster.

If FirstNet was up and running in Puerto Rico, would it have survived what Maria did to the island? To answer that question, you have to ask another question: Just how much money should be spent on ensuring a network is reliable even during a Maria? And more importantly, if you do spend all that money, will it really make a difference? I think you should do the best you can and be prepared to provide backup services quickly even to an island in the Atlantic because its 3.5 million citizens are U.S. citizens. Coming right on the heels of two other major storms, Maria was as unwelcome as the grim reaper. Unfortunately, people who study the weather say this won’t be the last Harvey, Irma, or Maria this country sees in the next decade.

We need to learn from these storms and to make sure that if there are things that can be done better they are done better. I am reminded of people who say we need true mission-critical networks. Others say we need public safety grade networks everywhere, while others say we need 5-9’s reliability in a 4-9’s world. However, when a Maria moves over land it may not matter whether the network is mission-critical, public safety grade, or 4-9’s because after the hurricane blows through, chances are the sites will be none of these. Then the issue will not be how fast we can return to 4-9s service but how soon we can return to any service at all.

Andrew M. Seybold
© 2017 Andrew Seybold, Inc.