Steve Miller's Blog

A Power Event in One Google Cloud London Zone Disrupted Banks and Public Services: Why Your Single-Region Answer Should Be in Writing Before Regulators Ask

Three professionals in a London office meeting room review a failover diagram sketched on a whiteboard

Everyone’s architecture diagram says “highly available.” On Tuesday, October 20, a lot of UK organizations found out what that means when one zone in Google’s only London region loses power. The Google Cloud London outage, and the single region risk behind it, is now on the desks of lawmakers and financial regulators. Nobody has asked your company for a written single-region answer yet, but the questions being asked of regulators point that way. Here is what happened, what is confirmed, and what I would write down now.

What Google Says Happened

Google’s preliminary incident report, posted October 22, gives the timeline in US/Pacific time. The incident ran from 23:00 on October 19 to 05:10 on October 20, a duration of “6 hours, 10 minutes.” That is about 7:00 a.m. to 1:10 p.m. in London on Tuesday, or 2:00 to 8:10 a.m. ET. Google says the disruption was “triggered by an electrical power interruption at the datacenter facility hosting availability zone europe-west2-b.”

The root cause, in Google’s words, is a chain of failures. A grid disruption hit the facility. Load moved to UPS batteries while diesel generators started up. But “the Programmable Logic Controller (PLC) automation system (both primary and backup PLCs) failed to initiate the logical automatic start/transfer operations of the generators.” When the batteries ran out, hosts, network switching fabric, and storage in europe-west2-b “lost power and went offline.” Google calls this preliminary and says a final report with preventative actions will follow.

The part operators should study is what happened next. Google says workloads failed over to the other two zones, europe-west2-a and europe-west2-c, as designed. Then “the sudden influx of redirected query executions, high-availability database failovers, and compute workloads saturated available compute slots and processing capacity in other availability zones.” The report says customers in the unimpacted zones “also experienced unexpected impact,” because of “instantaneous demand surges on unimpacted compute capacity” and “synchronous control-plane validation dependencies across zonal boundaries.” It names BigQuery, Looker, Compute Engine, GKE, Regional Persistent Disk, and Cloud SQL among the affected products. So the zone failed, but the region’s multi-zone design did not fully absorb it.

Who Was Hit, and Who Says So

The names come from press reporting, not from Google, which says only that a “very small number” of customers remained affected late in the day. Computer Weekly’s account lists banks and fintechs including Revolut, Lloyds Bank, HSBC, Barclays, Bank of Scotland, NatWest, Halifax, and Nationwide. On the public side it lists NHS England, HM Courts & Tribunals Service, and Gov.uk. Much of that list is based on user reports to DownDetector, and Computer Weekly says the public-sector sites “appeared to be affected.” The clearest confirmations are these:

So “froze banks” overstates it. Some banking services at several banks were disrupted for hours. Computer Weekly says most services recovered by late afternoon.

What Regulators Have Actually Done

This is where the sheet’s headline went furthest, so I am being careful.

What is already in force: Reuters says Google Cloud falls under both the EU’s critical third-party framework and the UK regime, which designated four cloud providers. Under the UK regime, designated providers “must report serious outages to the FCA and the BoE.” Reuters adds that under UK rules, “banks ultimately are responsible for the services they provide to customers, including when they rely on third-party providers.” What I did not find: any regulator asking firms for a written single-region or concentration-risk answer about this incident. That may come. It has not been reported yet.

Single Region Risk, in Plain Terms

Computer Weekly notes that europe-west2 is Google’s only UK region, so customers “that had not built multi-region resilience had no second UK location to failover to.” Google’s own report adds a second lesson: three zones in one region share more than you think. That includes the control plane and the spare capacity everyone fails over into at once.

My Recommendations: Write the Answer Before Someone Asks

This section is my opinion, not regulatory guidance.

  1. List every workload pinned to one region. Include the ones that are “multi-zone” but single-region, and the managed services you never configured yourself.
  2. Write down what customers experience if that region degrades for six hours. I mean what they see, not just what your architecture diagram shows. That is the answer a regulator, auditor, or board is most likely to ask for.
  3. Plan for capacity, not just failover. Google’s report shows failover into peer zones can saturate them. Reservations or a second region are how you buy room ahead of time.
  4. Test it. A failover runbook that has never been run is a hypothesis.
  5. Check data residency before you pick a second region. If the only UK region fails, your fallback may be outside the UK. Know whether your contracts and regulators allow that.

Here is a hypothetical. Your payments API runs regional Cloud SQL across three London zones and your dashboard says “HA.” One zone drops, failover begins, and the surviving zones are already full of everyone else’s failovers. Your status page goes yellow for hours anyway. That is my illustration, built on the failure mode Google describes, not a reported case.

The Bottom Line

One London zone lost power because generator automation failed, and the rest of the region struggled to absorb the load. MPs and regulators are now asking questions in writing, mostly of each other and of Google. Your turn may come, so write your single-region answer now, while it is still your own exercise.

Exit mobile version