Public Safety Advocate: Public Safety Grade

AT&T recently stated at a congressional hearing that there was no “real” definition for the term, “Public Safety Grade.” The public safety community responded with disbelief, led by the National Public Safety Telecommunications Council (NPSTC) which had published a paper addressing all of the issues needed to qualify as a Public Safety Grade Network. The document, “Defining Public Safety Grade Systems and Facilities,” was published in May of 2014 after much work by many people. Starting on page 108 of the document is a list of individuals and organizations that contributed to this report and the list is three pages long.

In all fairness to AT&T, at last week’s NPTSC meeting in Washington, DC, the AT&T senior vice president in charge of FirstNet stood up and apologized to NPSTC for his comments and then spoke about how AT&T is moving toward public safety grade status. His apology and comments were well received by NPSTC and the clarification was timely and well-articulated. What he did not say and what I hope to show in this week’s Advocate is that there are differences between public safety grade for Land Mobile Radio (LMR) sites and an LTE network. It is important for LMR communications professionals to understand these differences and not hold AT&T to public safety grade for every cell site in the network. The methodology for hardening an LTE network is different but effective and it is these differences that need to be understood.

The Report

During the preparation of the NTSTC paper I was vice-chairman of the APCO broadband committee under the leadership of Bill Schrier, who is now at FirstNet. We worked out an agreement with NPSTC that APCO’s committee would undertake the work for what was required to make a radio site (tower site) public safety grade. NPSTC worked on the rest of the criteria and incorporated our input with some changes into the final document that was approved by the NPSTC board and then sent to the FirstNet Public Safety Advisory Committee (PSAC) where it was reviewed, accepted, and forwarded to FirstNet for reference.

The first paragraphs of the executive summary set the tone for the balance of the paper and reads, in part:

“The term “Public Safety Grade” is a conceptual term that refers to the expectation of emergency response providers and practioners that their equipment and systems will remain operational during and immediately following a major natural or manmade disaster on a local, regional, and nationwide basis. The term Public Safety Grade (PSG) in this document is used to refer to network hardening or network sustainability. This document is intended to provide guidance for the First Responder Network Authority (FirstNet) as it constructs and implements the Nationwide Public Safety Broadband Network (NPSBN). In developing this document, the Public Safety Grade Task Group, operating under the National Public Safety Telecommunications Council (NPSTC’s) Broadband Working Group (BBWG), also considered Public Safety Grade requirements and recommendations for Land Mobile Radio (LMR) communications sites. These recommendations and requirements should be applied to both existing and new LMR sites.”

Most of us working on this report realized when it was finished that there are, in reality, few if any existing land mobile radio systems in use by the public safety community that meet all of the requirements set forth in the document. However, we also knew the report should provide both FirstNet and the land mobile radio community with what is required in order to meet as many of the public safety grade parameters as possible. We also knew LMR systems were better hardened and had fail-over and fallback capabilities commercial broadband networks did not have. Commercial networks are considered “best effort.” Sometimes people try to define a network’s capability to stay on the air using the “9’s” term, which refers to the percentage of time a network is up and running and shows the amount of downtime associated with the various percentages.

Therefore, many people wrongly think 5-9’s, which I do not believe any network has achieved, means the network would have to be fully operational 99.999% of the time. Translating this to outage time for a given year, the network would only be out of service for a total of 5.26 minutes, 4-9’s would mean a yearly total outage of 52.56 minutes a year, and 3-9’s has a yearly outage time of 8 hours and 76 minutes. The goal, of course, would be to build all public safety networks to meet the 5-9’s criteria. In reality, this is not practical nor is it affordable. However, when you look at users’ ability to communicate, not the network itself, the percentage of operational times changes, at least for LMR systems today. If an LMR system is a P25 trunked radio system and there is a failure, the individual sites that are still operational revert to standalone repeater systems, usually on a designated channel. If the entire network fails, first responders are still able to communicate unit-to-unit or one-to-many over fairly large distances depending on the portion of the spectrum and the terrain.

Today, when an LTE cell site fails there is no fallback for its operation. Therefore, you might make the case that providing public safety grade sites for FirstNet is even more important than providing them for LMR sites, at least today. Standards are being developed that will permit a single cell site to remain in operation if it is cut off from the network, but so far there is no way for broadband devices to talk to each other off-network or when a network is not operational. Providing public safety grade networks for LMR and public safety grade networks for LTE is different in many respects. When one LMR site is out of operation its coverage is not available and units in that coverage area have to revert to unit-to-unit communications. Usually there is no backup site that can fill in or take over for the site that has failed.

LTE Is Different

LTE broadband sites have more flexibility and coverage operations than LMR sites. For example, LTE sites are divided into sectors (usually 3) and can change the down-tilt of the antennas. This feature is used every day to load balance networks. For example, during normal hours in Times Square, cell sites are set for optimum coverage over a wide area but then the shows let out and the networks begin to fill up. The nearest cell sites can lower their antennas’ patterns to cover less area so a cell sector serves fewer customers while cell sites further away can enlarge their patterns to take up some of the traffic in that area. There are a number of other tricks that can be used either automatically or by remote control to change coverage patterns on these networks and, therefore, capacity and speed capabilities.

Translating this to a real-world example, I will discuss use of a site I know well since I spent ten years working on systems at the site and had my own ham radio repeater at the site. The site is known as Santa Ynez Peak and it is owned by the U.S. Forest Service, thus radio users on the site are there as a result of agreements with the Forest Service. There are about six major towers on this site, which is, by the way, up the hill about ten miles above the Reagan Ranch. There is a building and two towers that house county fire, police, paramedic, and other radio systems. They also house the Civil Air Patrol, Search and Rescue, a federal agency or two, and some ham radio and Red Cross radio systems. It is, in fact, the major hub for the county’s emergency communications systems. It was nearly burned over by a recent fire but was saved by a drop of some water and retardant in the nick of time.

Also at this site there is a state tower for the California Highway Patrol and other state systems, and then several commercial sites that lease space to customers. One of these is home to AT&T, Verizon, T-Mobile, and Sprint as well as a host of paging and other systems. The elevation of the site is 4,298 feet above sea level and a long expanse of California Highway 101 can be seen from there, as can the Pacific coastline and the ocean while the other side looks down on Solvang and other smaller cities. Santa Ynez Peak provides wide-area coverage for all of the systems located there. The reason commercial network operators are on the site is because when they were first building out their analog systems they used it as a high-level site to assist in meeting their coverage requirements. Today, some cellular operators don’t like using high-level sites but as you will see, there is a real advantage to having some. The site is accessible only via 4-wheel drive vehicles on a rutted, dirt path that passes for a road. However, it would probably be considered 60% public safety grade since the towers have been designed to withstand high winds and icing (yes, even in Santa Barbara). There are batteries and generators in each of the buildings and there is both microwave and fiber, but only one underground power feed coming up the backside of the hill. Still, in the ten years I was there, the site withstood everything nature could throw at it. Below this site along 101 and elsewhere multiple cell sites have been added to increase coverage in the area and to provide better coverage where the area is shadowed from the towers.

Now suppose there is a major disaster in the area, a storm, tsunami, or earthquake. The small sites below are knocked out, the county portion of the site is damaged but the building housing the cell systems is okay and the systems are still operating. This site will continue to provide coverage down below it, and even if the smaller cells are offline the highway and other areas will remain covered. Meanwhile, the county will have a major hole in its public safety coverage until it can reach the site or set up an emergency system. Yes, public safety can still communicate in the area via off-network on its LMR systems but it will also have FirstNet available from this site until the smaller cells can be put back into service or Cells On Wheels (COWs) can be brought in. In this case, the design of the AT&T/FirstNet system with both high and low cell sites offers a level of redundancy the public safety network does not.

Even if the LMR site is still operational, the FirstNet broadband system will be up and running and accessible. Perhaps data speeds will be slower, or perhaps coverage below will be spotty in some places, but FirstNet can be used until the situation is corrected. If the Santa Ynez Peak site goes out, the smaller cells below can pick up and handle the load. The point here is that I don’t believe every LTE site that is part of FirstNet needs to be hardened to public safety grade standards and that is probably true for some of the LMR sites as well. Commercial networks have been deploying batteries and generators for years and they have caught flack for not providing every cell site with the same power backup capabilities.

What they have done is to weigh the importance of a site to the community it serves and then add backup capabilities as warranted. In my model above the, the Santa Ynez Peak site would have full battery backup and a generator with multiple days of fuel in the tank (which it does). The small cells below would have batteries and some would have generators, depending on the degree of overlapping coverage. There are some sites that cover small areas that do not have any battery backup capabilities but with most of those sites there is the ability to drive up to the site and set up a generator.

The County of Santa Barbara has six major (soon to be seven) LMR sites to cover the county. The systems are simulcast and will revert to repeaters during a system failure. But the cost to the county to upgrade each and every one of these sites to meet all of the public safety grade standards is beyond reach of the county, as, I am sure, it is beyond the reach of most counties and cities. As a comparison, the average number of cell sites in the county is about 70-80 depending on the network. The cost to upgrade each and every site to meet public safety grade would be high. However, I believe the same type of redundancy can be designed into the system by evaluating the importance of each site and what it covers and then upgrading it based on the need to keep it operating.

Pre-Planned versus No Warning

There are, of course, differences between pre-planned incidents such as a hurricane and events that happen with no warning such as an earthquake or tornado. When pre-planning time is available, generator fuel tanks are topped off, network operators stage personnel and equipment in order to respond when needed, and checklists are reviewed and double-checked. This pays off big time for all concerned. During Harvey, commercial networks faired pretty well except in the area of the direct hit from the Gulf. During the NPSTC meetings in DC this past week, Tom Sorley, chairman of the FirstNet Public Safety Advisory Committee, called in and gave us an update. He had been in the Houston EOC for what he felt like was a month. When he reported on the LMR network his only lesson learned was that because generators at the sites had to run 24/7 during the storm, they burned oil. Trips to the sites with water-capable craft were necessary to fill the oil reservoirs back to acceptable levels. As an after-action, they plan to investigate this and see if there is a solution.

When there is no warning, hopefully there is a pre-plan in place for people to respond and to start cells on wheels and other emergency communications vehicles toward the scene. The issues for this type of incident will depend on the expected duration. If it is a major warehouse fire where evacuations are required, that is usually a fairly short-term incident and the strategy for communications backup will be different from the response to an earthquake or a tornado when the incident will go through multiple stages and communications are needed for a longer period of time. In these instances, it is possible to move COWs into the area, set up some LMR portable repeaters, and put the needed resources on the scene.

Public Safety Grade

The operational status of the network or networks in the incident area is the critical factor. If the incident is in Miami for example, an LTE cell site being out of service in Nevada will not pose an issue but an outage in the area of the incident certainly will. The good news in all this is that AT&T, as well as the rest of the commercial network operators, has been working with the FCC for a while to look at and try to resolve issues that could impact the network’s operation. As AT&T builds out FirstNet and adds hardening to certain sites, the network should be more resilient than it is today, but no network is immune from forces of nature or man-made issues.

In Northern California and Arizona within the past few years, fiber cables have been cut, either because some thief thought they contained copper wire or because they were intentionally targeted. All across the country radio sites have been vandalized by people after the copper wire in the antenna cables. In one instance, a truck drove up to a site, put a rope around a bundle of cables, and pulled them off the tower. During hunting season, how many “hunters” take aim at antennas or buildings for fun not realizing the damage they could be doing?

Public safety grade networks are important. However, they will never be 5-9’s capable. As mentioned earlier, there need to be fallback modes of operation for public safety so that when things fail they still have ways of communicating. Today LMR has that type of fallback but LTE does not. Hopefully, in the near future there will be fallback solutions for FirstNet as well. In the meantime, I think it is important to understand that public safety grade is a concept that in reality is not 100% achievable, but that does not mean sites, backhaul, and other aspects of a network should not be hardened as needed.

The subject of public safety grade can be simplified to the ability for public safety personnel to communicate with their dispatch center and with each other in the field. If the network is up and running, great. If not, there needs to be another way to achieve communications, which is one more reason I am a strong advocate for keeping LMR systems up and operational in addition to the FirstNet/AT&T solution. Having two networks is better than one and having fallback is a necessity regardless of how close to public safety grade a network may be.

Andrew M Seybold
©2017 Andrew Seybold. Inc.