Picture this runbook. Step one: if the on-prem side of the VPN drops, fail over to the backup tunnel. Step two: if ExpressRoute degrades, shift traffic to the VPN. Step three: check the Azure portal to confirm the gateway is healthy. It would read fine right up until a night when the gateways themselves failed to load in the portal and management operations started timing out. Every step would have assumed the cloud side stays up. That’s the uncomfortable lesson of Wednesday’s Azure maintenance hybrid cloud VPN outage, and it’s why I’ve spent the morning reading Microsoft’s incident report instead of doing anything fun.
What Microsoft Says Happened in the Azure Maintenance Hybrid Cloud VPN Outage
Microsoft’s Azure status history entry for tracking ID 7Q30-010 is a preliminary review, marked “Mitigated.” It says that between 20:30 UTC on September 30 and 02:15 UTC on October 1, which is 4:30 PM to 10:15 PM Eastern on Wednesday, “a subset of customers using gateway services in multiple regions experienced degraded or interrupted network connectivity.” That’s five hours and 45 minutes from start to mitigation.
The services Microsoft lists are the hybrid networking stack:
- Azure ExpressRoute Gateway
- Azure Firewall
- Azure Application Gateway and Web Application Firewall
- Azure VPN Gateway
- Azure VMware Solution
Customers may have seen gateways failing to load in the Azure Portal, along with failures or delays in network management operations. If you run Azure VMware Solution or connect your data center over ExpressRoute or site-to-site VPN, that’s your whole bridge between two worlds.
The Timeline, in Eastern Time
- 4:30 PM (20:30 UTC): customer impact began.
- 5:29 PM (21:29 UTC): Microsoft began investigating ExpressRoute Gateway connectivity issues in UK South.
- 6:27 PM (22:27 UTC): Microsoft identified that multiple regions were affected.
- 7:05 PM (23:05 UTC): Microsoft found a correlation with operating system servicing activity and paused further servicing.
- 9:36 PM (01:36 UTC, October 1): recovery progressed across most regions, while configuration changes were applied to France Central, North Europe, Southeast Asia, UK South and UK West.
- 10:15 PM (02:15 UTC): Microsoft confirmed mitigation.
On regions, Microsoft’s review names only the five late-recovering regions, plus UK South as the place the investigation started. It also carries a note that impacted regions in a previous communication were corrected. The Register, reporting during the incident, counted 18 affected regions at the time of writing, across the US, Europe, Asia, Africa, the Middle East and Mexico. It said Azure VMware Solution wasn’t impacted in four of them. Treat 18 as a mid-incident snapshot, not a final count.
Was It Really “Routine Maintenance” That Took Things Down?
Partly, and the nuance matters. According to Microsoft, “a recent change to a regional gateway management service triggered a higher-than-expected load when an unrelated operating system servicing maintenance proceeded gradually through multiple regions.” Normally the gateway management service would autoscale. This time, demand on dependent services kept those regional services from scaling as expected. Microsoft paused the OS servicing and reverted the gateway manager change, which let services recover. It says it’s still investigating the scaling behavior and the safeguards needed to prevent a repeat.
So the maintenance didn’t break things on its own. A change and a maintenance rollout collided, and the safety net, autoscaling, didn’t catch it. That’s also why I wouldn’t say Azure “took down” hybrid clouds. Microsoft’s own words are “degraded or interrupted” for “a subset of customers,” and The Register relayed a Microsoft update saying some affected VPN gateways saw reduced redundancy rather than a complete loss of connectivity. For some shops it was an outage. For others it was a night of losing half their safety margin without knowing it.
A full Post Incident Review is still coming. Microsoft says it generally publishes one to impacted customers within 14 days, so some of this may get sharper.
When the Change Window Becomes the Incident
What follows is my opinion, not Microsoft’s. Here’s what bugs me as an operator. We write change plans assuming our change is the risky one. We schedule our maintenance, tell the business, prep rollback and lean on the cloud provider as the stable half of the equation. This incident flips that. The provider’s maintenance, combined with the provider’s own change, was the event. Your change calendar would have shown nothing at all.
The detail that worries me most is the management plane. When gateways fail to load in the portal and network management operations fail or stall, every runbook step that starts with “go into the portal and…” is suddenly a coin flip. That includes resetting a gateway, updating a connection or flipping a route. The portal is the cloud side of your runbook, and it’s exactly what was impaired.
What I’d Put on the List This Week
- Turn on Service Health alerts properly. Microsoft’s guide to Service Health alerts covers alerting on service issues and planned maintenance by subscription, service and region, through email, SMS, push, webhooks or ITSM. It recommends selecting all services and all regions, since alerts only fire for regions where you actually run things. Route them to the on-call phone, not a shared mailbox.
- Check your gateway redundancy, not just your tunnel count. Microsoft’s VPN Gateway high-availability guidance notes that every gateway is active-standby by default, and it describes active-active gateways with multiple on-prem devices for full-mesh tunnels. If “reduced redundancy” happened to you Wednesday, would you have known?
- Build at least one runbook path that doesn’t need the portal. Pre-staged CLI scripts, on-prem routing changes, and BGP preferences you control from your side of the wire.
- Ask your account team about maintenance communications. What notice do you get for infrastructure OS servicing in your regions, and how fast does Microsoft pause a rollout when health signals drop? Here, impact began at 4:30 PM ET and servicing was paused at 7:05 PM ET.
- Write the post-incident review for your side. Check your own logs against the 4:30 to 10:15 PM ET window. Microsoft notes its times cover the whole incident, and actual impact varies by customer and resource.
None of this means hybrid is a bad idea. It means the cloud side of hybrid is a dependency with its own change windows, and it doesn’t always tell you when one starts. Plan for it the way you’d plan for a carrier circuit: assume it’ll have a bad night, and make sure you can still steer from your end when it does.