Best Practices for the Continuity of Operations Planning Process
By Richard Gaston
Tuesday, February 02, 2021 | Comments
Mission-critical communications are called that for a very good reason, which is that they cannot fail regardless of the circumstances. But, fail they do, for all sorts of reasons. Below are just a few examples.

On Sept. 29, 2020, a 9-1-1 system outage that affected 14 states occurred. It still is unclear what caused the outage, but initial speculation was a software glitch at one or more commercial providers of 9-1-1 services. In August 2018, a mistake made by a technician who was configuring a commercial 9-1-1 service provider’s routing network resulted in emergency calls failing to route to the appropriate center. The issue lasted for 65 minutes and affected 9-1-1 centers in multiple states.

In April 2014, an outage that lasted for about six hours and affected 81 emergency communications centers (ECCs) occurred. Again, the culprit reportedly was a software glitch at a commercial 9-1-1 service provider. According to the Washington Post, the problem affected the automatic assignment of unique identification codes, which is necessary to send emergency calls to the appropriate 9-1-1 center. The number of incoming calls reportedly exceeded preset limits established by the software. “As a result, the routing system stopped accepting new calls, leading to a bottleneck and a series of cascading failures elsewhere in the 911 infrastructure,” according to the Washington Post’s story.

In June 2012, a derecho — a powerful storm that features hurricane-force winds — roared across the eastern third of the United States, from the Midwest to the East Coast. The winds, which peaked at 97 miles per hour (mph) were powerful enough to knock out commercial wireless operations and electrical power. In many areas, backup power either failed or proved inadequate. The result was that millions of people were unable to connect with 9-1-1 systems, in some cases for days.

Numerous ransomware attacks have compromised public-safety and emergency response communications networks and systems in recent years, in particular shutting down 9-1-1 operations in some places. Ransomware is a specific type of malware deployed by cyberattackers to exploit a system’s vulnerability and then launch a program that encrypts the organization’s data files, essentially locking them and rendering them unusable. Then, the cyberattacker demands a ranson, hence the name, to provide the digital key that unlocks the files. The FBI says that it rarely sees anything other than ransomware attacks these days, and the reason is that it has become big business for the hacker community.

Mission-critical communications can be threatened by circumstances other than network and system failures. This year’s COVID-19 pandemic provides an example. Some 9-1-1 centers struggled to maintain operations in the early stages of the pandemic because the number of available telecommunicators shrank due to personnel contracting the virus and being unable to work and/or staff members were working remotely due to government-mandated shelter-in-place orders.

Indeed, major incidents such as tornados, hurricanes, wildfires, earthquakes, hazardous-materials spills and terrorist attacks all have the potential to disrupt mission-critical communications by rendering facilities inoperable, inaccessible or uninhabitable.

The key takeaway from all of this is that public-safety and emergency management officials need to assume that sooner or later their organization’s ability to provide mission-critical communications to protect the citizenry and to enable field responders to perform their jobs effectively and as safely as possible may be inhibited. Once that realization is made, the next step is to ensure that a well-conceived and well-practiced continuity of operations plan (COOP) is in place.

Such plans should contain the following:

  • Overarching strategies and tactics that ensure the organization’s mission-essential functions can be sustained throughout a disruptive event
  • A disaster recovery plan that addresses the organization’s information technology (IT) assets and is intended to keep them secure and operational
  • A crisis communications plan that identifies how to keep internal and external stakeholders informed during the event, as well as mainstream and social media.

The rest of this story details the key things that the COOP planning process should focus on.

Mission-essential functions. It is critical to define the functions that together enable the agency to perform its overarching mission(s) that cannot be compromised for any reason, for any length of time. For an ECC, it would be handling 9-1-1 calls and dispatching the appropriate emergency response resources. For a law enforcement agency, it would be apprehending criminals and preventing crimes from occurring. For a fire department, it would be fire suppression, fire prevention and conducting rescue operations. For an EMS agency, it would be basic and advanced life support and transport of victims to hospitals and trauma centers.

Mission-essential personnel. It is equally critical to define the everyday roles and responsibilities of all agency positions, from the bottom of the organizational chart to the top. The people in these positions are responsible for performing the agency’s mission-essential functions. Important questions must be asked about each position. For example, what do they do that supports each function(s)? What tasks are essential to service delivery? What relationships exist between the position and other positions in the agency? If their position goes unfilled, what would be the short-term and long-term impacts on the organization’s mission?

Staff succession and delegation of authority. Things move fast during a crisis, and roles often need to change out of necessity. Responsibilities will need to be assigned to specific individuals filling essential positions. For example, who is going to speak to the media? Who is going to liaison with third-party contractors to restore vital communications that were rendered inoperable? How will payroll be processed and emergency supplies procured? Who will decide when it is safe to re-enter a facility that was uninhabitable in the immediate aftermath of the crisis? These responsibilities might need to be fulfilled by people other than those who fulfill them every day. Further, thought must be given to how a person who is incapacitated by the crisis will be replaced or their responsibilities delegated, at least for the short term. These are not decisions that should be made on the fly.

Continuity strategies. Every agency has intergovernmental agreements and memoranda of understanding with neighboring agencies and vendors. A common agreement defines the mutual-aid assistance that would be provided by agencies to each other during a major emergency incident or disaster. Other agreements define devolution procedures, such as when operations have to be moved to another facility. It is important too to find out whether the agency’s vendors have their own business continuity plans. The agency must know that its vendors can be counted upon during a crisis. Strategies must be developed concerning the agency’s mission-critical data, such as the data used to locate 9-1-1 callers. Is the data secure? Where is it stored and is it stored in a geodiverse system or in the cloud? Can it be accessed easily yet securely and by whom? Other strategies need to be developed for restoring critical infrastructure quickly and effectively. All of this and much more needs to be memorialized in a COOP plan.

Internal/external communications. Strategies must be developed for communicating with internal and external stakeholders, senior government officials, the media and the public during a crisis. Also, warning and alerting procedures need to be identified. Mitigation of a crisis hinges on effective communications. The following are the essential steps:

  • Identify the team. One of the most critical elements is identifying the crisis communications team. This group will support the development and delivery of outgoing messages to stakeholders and manage incoming messages and the appropriate responses accordingly. To do so effectively, the team should consist of staff members from different levels and departments within the agency including, but not limited to, agency directors and managers, public information officers (PIOs), human resources personnel and frontline staff such as telecommunicators and field responders. As part of the planning process, each team member should be assigned a role that leverages his or her skillsets and has clear, defined responsibilities on which to execute.
  • Identify the stakeholders. The stakeholder group that needs to be communicated with will vary by crisis. It is important to have a clear understanding of who makes up each stakeholder group and where they fit within the communications hierarchy. For example, a single employee testing positive for COVID-19 affects mainly internal stakeholder groups. Communications can be limited to staff members, specifically those who have encountered the positive employee. On the other hand, an outbreak of COVID-19 cases among staff members would expand communications to include external stakeholders, such as community members, members of local government and the media, to inform them of the situation and relay any messages regarding operational changes.
  • Develop the messages. Message development should be a substantial element of the crisis communications planning process. A set of prepared, or “canned,” messages for different crises helps expedite the message development process during an active crisis by providing a message template that can be customized to the specific scenario. During this process, it is important to remember that it is nearly impossible to plan for every single scenario. Consequently, the crisis communications team should establish a message review and approval process for all outgoing messages. This helps to ensure that messages are accurate, concise, appropriate for the audience, fit the agency’s values and mission, and are error-free.

Training and testing. Once the COOP plan is created, it must be tested to identify any gaps that might exist, and all staff members should be trained on use of the plan. Such plans do absolutely no good if no one knows how to act on their elements, or worse, have no idea what is in them. Discussion-based exercises, such as a tabletop exercise, are excellent for this purpose and should be conducted at least annually. Operational exercises are also beneficial. For instance, if you’re going to relocate a 9-1-1 center’s operations to a backup facility, it’s an excellent idea to try it before needing to do it for real.

Update, update, update. Again, a COOP plan is a living document or should be. During every major emergency, disaster and crisis, critical lessons that should be used to evolve the plan are learned. If that doesn’t happen, it is inevitable that the COOP plan one day will be ineffective just when it is needed most. After-action reports are a time-tested effective way to identify the lessons. The COOP plan should be updated after every activation and should be reviewed at least annually, and biannually is even better. If the plan changes appreciably, it should be tested, and staff members should be trained on any new elements.

Disaster recovery. It is imperative that every agency initiates a regular, ongoing network monitoring program to identify vulnerabilities that could be exploited by cyberattackers. But it is equally imperative that agencies have a plan in place, as an element of its overarching COOP plan to implement when an attack occurs, or for situations where its primary facility has been rendered inoperable, inaccessible or uninhabitable. Such a plan should enable the agency to move its operations to a backup facility that is geographically separated from the primary facility by a sufficient distance — perhaps in an adjoining county, or even state — or to backup networks and systems stored in the cloud.

Because no two jurisdictions are the same in terms of resources, capabilities, socioeconomics, geography, topology and other differentiating factors, a cookie-cutter approach to developing a COOP plan is not advised. Rather, it should be customized. One thing that is advised is to include all departments in COOP plan development. Another is to train all staff on the plans once they are developed.

A COOP plan represents a significant investment of time and resources. But, they are essential to maintaining operations that provide critical services, which is especially important during crises. No public-safety official wants to stand at a podium and answer the following question from the media or the public: why weren’t we ready? Having a COOP plan at your fingertips is the best way to avoid that question.


Richard Gaston is a senior consultant for Mission Critical Partners (MCP), a mission-critical communications and information technology consulting firm headquartered in State College, Pennsylvania.



 
 
Post a comment
Name: *
Email: *
Title: *
Comment: *
 

Comments
On 2/3/21, Eric Abdullateef said:
This piece nailed it.


Education







Events
September 2021

27 - 30
International Wireless Communications Expo (IWCE) 2021
Las Vegas
https://www.iwceexpo.com/

October 2021

18 - 20
Colorado NENA/APCO 2021 Conference
Denver
https://www.conenaapco.org

19 - 21
Comms Connect Melbourne
Melbourne, Australia
https://melbourne.comms-connect.com.au/

November 2021

2 - 3
Wireless Leadership Summit (WLS)
Austin, Texas
https://www.enterprisewireless.org/wls2021

More Events >

Site Navigation

Close