Considering Power and Telecom Codependency Risks in California
By F. H. Rick Smith
Friday, October 02, 2020 | Comments
Power and telecommunication codependency risks are nothing new. Yet in California, the landscape has evolved so as to make this an acute concern.

Recall that codependency is a situation where two things feed on each other. For example, telecom will not work without power, and power will not work without telecom. So, you really can’t afford to lose either, and if you do, it is hard to “get back on track.” In today’s complex distributed generation power grid architecture, electric systems depend on telecommunications for three critical functions:
1. Real-time machine-to-machine supervisory control and data acquisition (SCADA) communications, real-time internet information feeds to people, and people to people communication needed to balance generation and load in a timely manner
2. Machine-to-machine and people-to-people communications necessary to localize problems in real time, or near real time, to prevent small issues from cascading into larger ones
3. Once serious failures have occurred, either on the power, or telecom side, there must be a way for people to talk to each other to expedite fixing the situation.

The immediate concern over a potentially unhealthy telecom/power codependency in California is the result of a number of changes to the environment that have evolved slowly over time, and changes that have occurred recently.

No one may be connecting the dots relative to the associated risks. There are a variety of risk factors.

California’s generation fleet is unique in that it is made up of a vast number (thousands) of independent generators that all have to be coordinated in real time to maintain generation/load balance while managing transmission resources within limits at the same time. This alone created the need for a vast number of telecom circuits. Having the large number of independent generators has made it logical that many/most of these circuits are supplied by the telecommunications providers. Also, the sheer number of different generation entities involved has made it challenging to maintain private independent communications channels for humans. Accordingly, most human-bound communications take place via public landline or cell networks lacking prioritization for this function.

This wouldn’t be much of a problem except that the telecommunications providers have recently been on a path of indifference towards backup power integrity out on the customer end of the line, and they have been lacking concern for a careful transition from TDM to IP, as it relates to supporting power infrastructure. My experience — I hope others’ is different — is that our nation’s telecom infrastructure, while getting better in many respects, has drifted away from traditional best practices in an effort to rush to meet consumer desires for web bandwidth and entertainment. And in the midst of a fast-paced competitive environment, many corners have been cut, either out of competitive necessity, or neglect. I also believe there is a vast under appreciation for the role the old TDM networks still play “in keeping the wheels on this nation.”

Power generation in California has a growing dependence upon telecommunications circuits, and telecommunications in general as a result of California’s desire to integrate as much variable renewal energy into the grid as possible. It is now not just a matter of ramping resources up and down throughout the day as demand changes; the modern regimen now involves swapping generation resources in and out during the day as necessity requires, such as the sun going down and natural gas having to pick up the slack, and as desired such as cutting back on natural gas when the wind is blowing well. The net result of this is more change and more risk. The silver lining is that grid operators and artificial intelligence (AI) systems have a lot of real-world experience to pull from.

The idea of the Western U.S. power grid, from a reliability perspective, is having a number of smaller systems, each being whole in and of themselves, joining together, such that in an emergency, they can help each other out. To do that, more than 40 control areas have been established wherein generation and load within each of those areas is held in balance, and each of those areas is responsible for having their own reserves. This offers a lot of security but does not provide an opportunity for excess power from one area to be utilized by another, either for cost efficiency or to minimize greenhouse gas emission.

California has been on a mission to lower its carbon footprint and has pioneered pushing the limits of how far primary grid operator CAISO will go in terms of operating “out of balance” to further the green objective. This doubles down on the already challenging objective of keeping telecom in check across the already very large CAISO control area, but extends the telecom codependency to other states.

Experience gained from the recent fires in California suggests the best practice in the future will be to turn power off during periods of extreme fire risk. Given the inattention to backup power infrastructure at the edge and the fact that many modern telecom implementations have more points in the system where power is required, there is a hidden and heightened risk of rapid telecommunications circuit failure during public-safety power shut-offs (PSPS). Worse yet, with the relationship of which power sources underpin which telco circuits being unknown, what may seem to be a safe power down from a telecommunications perspective may result in an unanticipated and uncomfortable loss of connectivity/control.

Risk implications
The odds of PSPS outages being long enough so as to interrupt telecommunications are high. Fortunately, the odds of these outages being in places with strategic power grid significance is low. Nevertheless, this new development is a good reason to re-evaluate the full topic of power-telecom codependency given the evolution of the power-telecom landscape as described above.

Keep in mind that while the core of our nation’s carrier telecom networks have good redundancy and adequate back up power, most of the power generation assets live off the core, on the branches of the telecom carrier’s network, where less redundancy exists, and more often than we would like, inadequate, or inadequately maintained backup power resources are the reality. My guess is that while the historic minimum standard for power-related telecom power backup was eight hours, I believe a current audit would reveal backup times much shorter than that in many cases. The eight-hour standard came from a time in the past when power grid networks were smaller and involved fewer players that needed to be coordinated with. Now, not only is compliance with the eight-hour rule a question in my mind, but with the sheer size, complexity and number of independent players involved, is it reasonable to assume grid issues can be sorted out in eight hours?

Once the power grid has been down for more than eight hours, regardless of whether the telecom broke the power, or the power broke the telecom, modern restoration effectiveness may be something akin to the “Three Stooges” movie. Here is why. California would have a very disjointed power-related communications situation when carrier network branches go down. Much of the SCADA/data needed to manage the power network wouldn’t work. The many separate organizations that make up the California power generation scene are particularly vulnerable because of their huge dependence upon the public telephone carriers who are so confident about their own wireless networks they principally rely upon their own wireless networks to communicate over to fix both their wired and wireless networks.

While some satellite phones may be in the mix for commercial carriers, the rank-and-file repair folks generally use their cellphones, which would be at risk of not working because of non-working branches of their network needed to complete connectivity back to the evolved packet core (EPC). To confirm this cellphone reliance, I recently asked Mark Crosby, president and CEO of the Enterprise Wireless Alliance (EWA), an FCC-certified frequency coordinator, to check FCC radio licensing records. The record reveals that commercial carriers have Part 90 licenses, but to what extent and for what purpose they may use Part 90 devices is unknown.

The transmission operators —utilities — have their own networks that provide SCADA and two-way capability. To a certain extent these networks work towards meeting connectivity requirements but likely fall short of being able to keep/restore the grid to whole absent the carriers, both because the transmission operator networks don’t touch many of the independent generation locations and because the transmission operator’s networks are themselves dependent upon telecommunications provider circuits in many cases. All of this leads me to believe that grid failure and subsequent carrier network failure will result in a slow ineffective recovery of the power grid.

The dilemma
Let’s reflect on the foundations of our nation’s infrastructure for a moment. We ask ourselves, what is the innermost core of your nation’s infrastructure? Is it power, telecom, oil and gas, water, transportation or agriculture? Arguably, the innermost core is the indivisible combination of power and telecom because these two components weave into everything else.

Ironically, the success of these two components, coupled with all organizations’ drive for efficiency, have fostered a pervasive trend to regionalize and nationalize everything around this “time honored invincible utility electricity-carrier provided telecom core.” The good side of this is that efficiency has been driven into virtually every business in every sector of the economy. There are countless examples of how the scale/synergies of technology implementations, big data and more have revolutionized the way we do things today. For example, we often set out for a destination without a map, knowing GPS will get us there.

The bad side is we have built a world where “everything works as long as everything works." Unfortunately, this also means if something does not work, many other things may not. Take GPS as an example.

We are so proud of our sophisticated organizations that are nearly 100% dependent upon assumed unbreakable regional and national telecommunications. We have lost consideration of how we could communicate and function effectively locally if this infrastructure we take for granted were to become unavailable.

We have allowed the presumption of successful wide-area connectivity to affect the evolution of communications technologies themselves. Think about it, most automated meter reading systems (AMS), most internet of things (IoT) development including Sigfox and LoRa WAN, certainly Wi-Fi, Citizens Broadband Radio Service (CBRS), and 5G millimeter wave are not really designed to be the backbone but are rather designed to ride on top of a backbone.

Another manifestation of this mega trend that everything should ride a carrier backbone is the momentum of FCC spectrum policy that has greatly favored the big carriers. Private microwave spectrum very suitable for power-related telecom infrastructure has been taken out of play to support growth of public cellphone networks. Worse yet, the workhorse 6 GHz microwave band is now being subjected to inadequate protection from a next-generation of consumer wireless devices. In general, quality licensed spectrum for both SCADA and voice communications is in scarce supply. Fortunately, one good thing has happened recently with the FCC’s decision to rework the 900 MHz band to enable private LTE systems.

Reducing the power-telecom codependency risks will invariably involve doing something different with connectivity. Unfortunately most IT organizations, even in electric power utilities, have sort of moved on past telecom connectivity issues to chase the shiny objects. These shiny objects are things that carry the promise of better efficiency and cost reduction — things such as network automation, virtualization of computing and network functions, rewriting applications around restful application programming interfaces (APIs) for more efficient use of code, expanding the use of IoT, and leveraging big data.

Ultimately, doing anything positive in this space will require a realization that something needs to be done. We often pridefully think that since things have never been a problem, how could they be? If you have had the good fortune of either watching the episode of Saturday Night Live where they explained what happened at Three Mile Island, or if you have read the 1st Corinthians 1:27 Bible verse, you will understand the theme of this discussion.

Mitigating these risks
Experts have thought about the “chicken and egg” aspects of our wonderful national infrastructures for some time now. In “Nuclear EMP Attack Scenarios and Combined-Arm Cyber Warfare,” a recent work by Dr. Peter Pry, he references the following paragraph from a 2004 Congressional Electromagnetic Pulse (EMP) Commission Executive Report:

“EMP is one of a small number of threats that can hold our society at risk of catastrophic consequences … It has the capability to produce significant damage to critical infrastructures and thus the very fabric of U.S. society, as well as the ability of United States and Western nations to protect and influence and military power … The recovery of any one of the key national infrastructures is dependent upon the recovery of the others. The longer the outage, the more problematic and uncertain recovery will be. It is possible for the functional outages to become mutually reinforcing until at some point the degradation of infrastructure could have irreversible effects on the country’s ability to support its population.”

An EMP is about the worst thing that could befall us. It’s something we all hope will never happen. There are other things that are not a matter of if, but rather a matter of when including the big California earthquake; the upset of the western power grid by a solar flare like the Carrington event; a widespread power outage caused by a combination of grid-load stress, combined with either equipment failures on either the power or telecom side, extraordinary circumstances like wildfires or human error; well-coordinated terrorist physical or cyberattack to either power or the telecom infrastructure or both.

The common thread between an EMP, and all of these issues listed above, is they all involve things that other things depend upon breaking, creating an exponential challenge to get things back to normal.

The key opportunity is to recognize the important role telecommunications design can have in changing the current paradigm. We need to go from if the grid goes down, we are done for to where we have confidence knowing that no matter what happens, not everything is going to break, and being able to communicate, we will quickly bring pieces of our infrastructure up, and shortly thereafter, put the pieces back together.

The main thing to improving our response to big issues is doing whatever it takes to get out of the rut of having our plan A and our plan B both subject to a world where nothing works until everything works. It certainly wouldn’t hurt to harden our telecom infrastructure by being more concerned about the integrity of backup power for telecom. This alone will not be transformational.

The big improvement, in my belief, is having the agility to have power and telecom function as one big network on a normal day but have the ability to rapidly break up into smaller logical power and telecom-coordinated balancing areas, with self-contained people, procedures, SCADA connectivity, and people-to-people communication when needed.

Say we woke up tomorrow morning with the burden on our hearts to implement a better coordinated power/telecom environment capable of being whole on a local basis, there would be one elephant in the room. That elephant being the fact that SCADA telecom circuits today often have distant dependencies, particularly when opposite ends of circuit involve different carrier territories, cross LATA boundaries, and/or involve carrier hub handoffs. Telecom plumbing and the electric plumbing usually do not follow the same pathways. More often than not, telecommunications circuits take circuitous routes to get where they are going without regard to physical distance or even whether they have crossed the San Andreas fault or not.

A huge solar flare is a good example of where this agility would be helpful. A solar flare becomes a problem for the power grid by inducing low frequency AC currents into the earth, effectively creating a DC potential between different points on the earth’s surface. The further apart the points are on earth, the larger the potential difference is. Power lines provide a pathway for the big transformers on the power grid to receive this DC energy. Even a small percentage of this DC power mixed with the usual 60 hertz (Hz) AC power becomes a problem for the big transformers, causing their cores to magnetically saturate, which causes large 60 Hz AC currents to flow, overheating the transformers and potentially damaging them forever. By breaking the power grid up into smaller pieces, the transmission lines are effectively shorter and the induced voltages now fall below the thresholds where this damaging effect takes place. A fortunate thing about a solar flare is that we will know it is coming in eight minutes and have hours to prepare before the damaging plasma arrives.

Regardless of what the stress is to our power/telecom core, having a better hardened telecom/power coordinated strategy will pay big dividends when the stresses come. The easy path is to take telecom connectivity for granted and ignore the need to perhaps add some telecom pathways that are suitably local, separate and mission focused to underpin the electric core. This is not to suggest that our nation’s fantastic commercial networks shouldn’t be leveraged for many things, this is simply to suggest reflecting on their strategic fit for those things known to be core.

Would you like to comment on this story? Find our comments system below.


F. H. Rick Smith, an IT infrastructure architect, recently completed a successful 44 year career in the energy industry having had the pleasure of working for Getty Oil, Texaco and most recently Chevron. During his career, Smith was fortunate to have worked across the disciplines of automation, telecom, IT and electric power. Smith has had a long-standing interest in FCC telecom policy, serving on the Enterprise Wireless Alliance (EWA) board of directors since 1998, and having been the chair of the board for 2004 and 2005. Smith also served as part of the API Telecommunications Subcommittee from 1986 onward focusing on spectrum and wireline issues needed for the safe and efficient operations of U.S. oil and gas infrastructure.



 
 
Post a comment
Name: *
Email: *
Title: *
Comment: *
 

Comments
On 10/14/20, Ian Duffy said:
I cannot agree more with most of the comments in this article. My thoughts on the thankfully very slow in UK replacement of the PSTN by VoIP systems are one of Eggs in one basket making the internet do everything.
In the UK BT and OLO s have battery back-up in their broadband MUX cabinets but it cannot be more than an hour or two... and only then if the batteries have been thought about and looked-after.
In the good-old-days our phone exchange-lines were copper pairs all the way back to the telephone-exchange where there was a huge battery back-up supplying the exchange equipment with diesel generator back-up to that. The exchange where I mainly worked had the ability to run continuously for well over a week that I knew-of on the generator with the fuel held on-site.
Apart from one unfortunate major power-related telecomms isolation in the early 2000 s in Southampton I have not known of any other major power-related isolations.
IRD. 15-10-2020.

Site Navigation

Close