Steve Miller's Blog

A digital workshop for systems-minded tech commentary.

Google Cloud Traced Its London Outage to a Data Center PLC Misconfiguration: The Building Controls Under Your Cloud Are Part of Your Uptime Too

Three anonymous facilities and operations engineers in business-casual clothes talk through a plan at a table with a closed laptop and blank notepads in a bright, modern office.

Written by

in

We like to treat a cloud region outage as a software problem with a software fix: a bad deploy, a config push, a control plane that fell over. Google’s incident report for the October 20 europe-west2 (London) event is a data center PLC misconfiguration outage in the plainest sense, and it is a useful reminder that the layer under your availability zone is a building full of programmable logic controllers (PLCs) that nobody on your team will ever patch or see. Google’s report names a PLC misconfiguration as the root cause, and that changes what a resilience plan should cover.

Anatomy of a Data Center PLC Misconfiguration Outage

According to Google Cloud’s incident report for europe-west2, posted in final form on November 4, 2026, at 22:53 Pacific on October 19 (06:53 BST on October 20), the utility’s primary feed A to the data center hosting zone europe-west2-b failed because of an external underground cable fault. Google says the site was already in a “temporary single-feed (1N) bridged maintenance posture”: the feed B breaker had been opened for repairs after issues found in a quarterly generator load test on October 14.

Then the controls took over. Google says the investigation “identified the root cause as a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders.” The undervoltage sensing inputs were wired on the line side of the open feed B breaker, and feed B was still energized upstream. The two master PLCs “sensed live voltage” and concluded a healthy utility source was available. They never sent the generators a start command.

The UPS batteries carried the load “as intended” (Google puts their runtime at approximately five minutes) until they were exhausted, and the data halls lost power. Impact began at 23:00 Pacific and lasted 6 hours and 8 minutes, across 40 Google Cloud services that the report lists as affected in the region. Google says it found no impact outside europe-west2.

What It Was Not

This was not a cooling failure, and the report does not point to a cyberattack. The trigger was a cable fault outside the building. The PLC logic and sensor wiring, plus a maintenance posture with one feed open, turned a survivable utility loss into a zone outage. The report does not say how long the misconfiguration had existed, and I will not guess.

Redundancy That Shared the Same Mistake

One detail stands out to me as an operator: the facility had a dual-redundant PLC pair, and Google says the active and standby masters ran identical control logic. Two copies of the same wrong rule give you availability against a failed controller and no protection against a wrong rule.

The second detail is the change. A site that has dropped from 2N to 1N for repairs is running a different system than the one that was commissioned, and the controls logic has to be correct for that posture too. NIST’s SP 800-82 Rev. 3 covers operational technology, a category it says includes building automation systems, and says configuration management helps “reduce risks from outages due to configuration issues.” That is the failure mode here.

What Google Is Changing

The remediation list is a good template for questions to put to any provider or colo vendor. According to Google’s report, the PLC was reconfigured and validated on October 20 and again on October 26. Google ran full generator load-transfer tests with utility cutover, and rewrote its emergency procedures to include manual generator restart and transfer within the UPS runtime, practiced twice per shift. It froze planned maintenance at sites with similar topologies and added elevated approvals for configuration changes outside standard designs. It also plans PLC code reviews at facility commissioning, with a second Google team validating within a quarter after a facility is declared ready (estimated completion December 18).

Those fixes suggest good questions for your own vendors. When did you last test automatic generator start through the controls with the utility actually cut? How does your battery runtime compare with how long the manual procedure takes?

Why a Single-Zone Fault Still Hurt Multi-Zone Customers

Google’s geography and regions documentation says “Zones should be considered a single failure domain within a region” and tells you to deploy across multiple zones for fault tolerance. Yet Google says some customers in zones a and c, or on regional architectures, were affected anyway, and names two causes: demand surges on neighboring zones and control-plane dependencies across zonal boundaries.

  • Capacity: Google says some Compute Engine failover attempts “were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism.”
  • Placement drift: Apigee X is designed to survive a single-zone loss, but Google says a scheduling misconfiguration put two of three datastore replicas in the zone that lost power. The datastore lost quorum and the runtime returned errors for 7 hours and 19 minutes.
  • Slow failover: Regional AlloyDB primaries saw up to a 95-minute delay in starting automated failover because failover notifications were delayed.
  • Configuration that worked: Bigtable multi-cluster instances with HA routing failed over, while single-cluster instances in europe-west2-b had the expected zonal unavailability.

A Resilience Checklist for the Layer You Cannot See

These steps are my opinion, not Google’s guidance.

  1. Ask for the facility story. Ask your colo or cloud account team about generator auto-start testing, controls change review, and what changes during maintenance postures.
  2. Audit replica placement. Verify that quorum-based services are spread across three zones today, and alert on drift. The Apigee case is a placement bug, not a power bug.
  3. Run a zone-loss game day. Drain a zone in staging and watch how the survivors handle the surge. Price reserved capacity in the surviving zones if it matters.
  4. Remove single-zone and non-HA tiers from critical paths. Google’s report treats loss of zonal resources in a failed zone as working within design.
  5. Decide what needs a second region. The incident stayed inside europe-west2, so workloads already running in a second region would have been outside the blast radius, at real cost and operational effort. Be selective.
  6. Update your risk register. Uptime Institute’s 2026 annual outage analysis (industry survey data, not a Google measurement) found power was the cause of 45% of respondents’ most recent impactful data center outages in 2025, down from 54% in 2024. Among respondents with power-related IT outages in the past three years, the most common causes were UPS failure (39%), transfer switch failure (34%) and generator failure (28%); controls failure was cited by 8%. They are survey results, but they point at the switchover path.

The Takeaway

Google’s report is unusually detailed, a rare chance to study a facility-level failure from the customer’s side of the wall. A change on a building control system carries the same kind of risk as one on a production network, and testing has to match the posture the site actually runs in. Your uptime plan should name the layers you do not control and say how you will find out when they fail.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *