Datacenter Disaster Preparedness: Lessons Learned from the Texas Outages

Datacenter Disaster Preparedness: Lessons Learned from the Texas Outages

On February 13, 2021, the North American winter storm, also called Winter Storm Uri, started making its way through Texas, causing temperatures to drop to as low as the single digits, with subzero wind chills. Besides disrupting Texans’ lives and leading to even more stringent stay-at-home guidelines, the cold weather also made power-generation resources partially inoperable. Multiple Texas datacenters were affected by the cold, and had to adapt to rarely seen circumstances in the region.

The ERCOT Dilemma

Amidst a series of logistic challenges faced by the state, the Electric Reliability Council of Texas (ERCOT), which manages the flow of electric power to about 90% of the state’s electric load, found itself unable to meet the higher demand on the Texas power grid. ERCOT eventually began asking operators to carry out rolling blackouts, some of which lasted multiple days.

More than 52,000MW in power went offline due to Uri, while demand for power simultaneously surged to 69,000MW (about 15,000MW higher than ERCOT’s estimate for peak demand). As the power grid began to fail, ERCOT looked to reduce the state’s electric usage by focusing on the large users, including datacenters. Even partial power reductions to any of the 200 facilities in state could be used to power thousands of households instead.

How providers put their disaster preparedness to the test

Winter storm Uri, which brought record-low temperatures in some Texas cities, put datacenter providers’ emergency preparedness to the test.

Most facilities have certain disaster recovery measures in place that are usually deployed to address severe thunderstorms or hot weather, which is more common in the region. Fortunately, these disaster recovery plans can be altered to address other types of natural disasters.

Even though the impact the Uri had on facilities varied across the region, those that were not significantly impacted appeared to have had adequate personnel and infrastructure at their disposal.

Datacenter operators with national fuel contracts were also impacted significantly less than those who obtained their fuel from local providers. Likewise, operators with fixed-rate electric plans avoided certain large costs they could have incurred in their electric bill, due to the impact of the winter storm.

The course-of-action plan that was taken

Due to previous experience with extreme heat during the summer, providers were already familiar with ERCOT’s approach to surges in demand to the power grid. For example, most datacenters had diesel generators and fuel tanks that would allow them to operate for several days before needing new fuel. Providers, however, were asked to drop from the power grid voluntarily by ERCOT. While some opted in, relying on their generators due to the incentives put in place, others continued operations as normal.

Providers took additional precautions, such as keeping food and water stocks to serve their on-site staff, with some even offering to help identify temporary housing for employees and families that were significantly impacted by the storm.

Lessons to remember

Despite having their generators to fall back on, most datacenter providers worried they would run out of fuel before they got access to their usual refueling source. In this case, having a long-term or larger fueling contract would pay off, as facilities that had a well-established relationship with fuel providers could refuel more easily. Also, facilities that fully owned their infrastructure and equipment, instead of leasing them, were not as significantly impacted.

Finally, providers with variable-rate electric contracts would certainly incur higher costs than those with fixed-rate; consequently, they must decide whether to absorb that cost or pass it on to customers.

Although the extreme cold temperatures were rare, Uri could have a long-lasting impact on the Texas datacenter market. A state had been seen as inexpensive in power pricing will now likely see hikes in prices, as providers attempt to recover from the losses they incurred due to the natural disaster.

Disaster Recovery & Data Management Push Storage to the Breaking Point