Steve Miller's Blog

A digital workshop for systems-minded tech commentary.

Category: Cloud

  • Google Cloud Traced Its London Outage to a Data Center PLC Misconfiguration: The Building Controls Under Your Cloud Are Part of Your Uptime Too

    Google Cloud Traced Its London Outage to a Data Center PLC Misconfiguration: The Building Controls Under Your Cloud Are Part of Your Uptime Too

    We like to treat a cloud region outage as a software problem with a software fix: a bad deploy, a config push, a control plane that fell over. Google’s incident report for the October 20 europe-west2 (London) event is a data center PLC misconfiguration outage in the plainest sense, and it is a useful reminder that the layer under your availability zone is a building full of programmable logic controllers (PLCs) that nobody on your team will ever patch or see. Google’s report names a PLC misconfiguration as the root cause, and that changes what a resilience plan should cover.

    Anatomy of a Data Center PLC Misconfiguration Outage

    According to Google Cloud’s incident report for europe-west2, posted in final form on November 4, 2026, at 22:53 Pacific on October 19 (06:53 BST on October 20), the utility’s primary feed A to the data center hosting zone europe-west2-b failed because of an external underground cable fault. Google says the site was already in a “temporary single-feed (1N) bridged maintenance posture”: the feed B breaker had been opened for repairs after issues found in a quarterly generator load test on October 14.

    Then the controls took over. Google says the investigation “identified the root cause as a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders.” The undervoltage sensing inputs were wired on the line side of the open feed B breaker, and feed B was still energized upstream. The two master PLCs “sensed live voltage” and concluded a healthy utility source was available. They never sent the generators a start command.

    The UPS batteries carried the load “as intended” (Google puts their runtime at approximately five minutes) until they were exhausted, and the data halls lost power. Impact began at 23:00 Pacific and lasted 6 hours and 8 minutes, across 40 Google Cloud services that the report lists as affected in the region. Google says it found no impact outside europe-west2.

    What It Was Not

    This was not a cooling failure, and the report does not point to a cyberattack. The trigger was a cable fault outside the building. The PLC logic and sensor wiring, plus a maintenance posture with one feed open, turned a survivable utility loss into a zone outage. The report does not say how long the misconfiguration had existed, and I will not guess.

    Redundancy That Shared the Same Mistake

    One detail stands out to me as an operator: the facility had a dual-redundant PLC pair, and Google says the active and standby masters ran identical control logic. Two copies of the same wrong rule give you availability against a failed controller and no protection against a wrong rule.

    The second detail is the change. A site that has dropped from 2N to 1N for repairs is running a different system than the one that was commissioned, and the controls logic has to be correct for that posture too. NIST’s SP 800-82 Rev. 3 covers operational technology, a category it says includes building automation systems, and says configuration management helps “reduce risks from outages due to configuration issues.” That is the failure mode here.

    What Google Is Changing

    The remediation list is a good template for questions to put to any provider or colo vendor. According to Google’s report, the PLC was reconfigured and validated on October 20 and again on October 26. Google ran full generator load-transfer tests with utility cutover, and rewrote its emergency procedures to include manual generator restart and transfer within the UPS runtime, practiced twice per shift. It froze planned maintenance at sites with similar topologies and added elevated approvals for configuration changes outside standard designs. It also plans PLC code reviews at facility commissioning, with a second Google team validating within a quarter after a facility is declared ready (estimated completion December 18).

    Those fixes suggest good questions for your own vendors. When did you last test automatic generator start through the controls with the utility actually cut? How does your battery runtime compare with how long the manual procedure takes?

    Why a Single-Zone Fault Still Hurt Multi-Zone Customers

    Google’s geography and regions documentation says “Zones should be considered a single failure domain within a region” and tells you to deploy across multiple zones for fault tolerance. Yet Google says some customers in zones a and c, or on regional architectures, were affected anyway, and names two causes: demand surges on neighboring zones and control-plane dependencies across zonal boundaries.

    • Capacity: Google says some Compute Engine failover attempts “were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism.”
    • Placement drift: Apigee X is designed to survive a single-zone loss, but Google says a scheduling misconfiguration put two of three datastore replicas in the zone that lost power. The datastore lost quorum and the runtime returned errors for 7 hours and 19 minutes.
    • Slow failover: Regional AlloyDB primaries saw up to a 95-minute delay in starting automated failover because failover notifications were delayed.
    • Configuration that worked: Bigtable multi-cluster instances with HA routing failed over, while single-cluster instances in europe-west2-b had the expected zonal unavailability.

    A Resilience Checklist for the Layer You Cannot See

    These steps are my opinion, not Google’s guidance.

    1. Ask for the facility story. Ask your colo or cloud account team about generator auto-start testing, controls change review, and what changes during maintenance postures.
    2. Audit replica placement. Verify that quorum-based services are spread across three zones today, and alert on drift. The Apigee case is a placement bug, not a power bug.
    3. Run a zone-loss game day. Drain a zone in staging and watch how the survivors handle the surge. Price reserved capacity in the surviving zones if it matters.
    4. Remove single-zone and non-HA tiers from critical paths. Google’s report treats loss of zonal resources in a failed zone as working within design.
    5. Decide what needs a second region. The incident stayed inside europe-west2, so workloads already running in a second region would have been outside the blast radius, at real cost and operational effort. Be selective.
    6. Update your risk register. Uptime Institute’s 2026 annual outage analysis (industry survey data, not a Google measurement) found power was the cause of 45% of respondents’ most recent impactful data center outages in 2025, down from 54% in 2024. Among respondents with power-related IT outages in the past three years, the most common causes were UPS failure (39%), transfer switch failure (34%) and generator failure (28%); controls failure was cited by 8%. They are survey results, but they point at the switchover path.

    The Takeaway

    Google’s report is unusually detailed, a rare chance to study a facility-level failure from the customer’s side of the wall. A change on a building control system carries the same kind of risk as one on a production network, and testing has to match the posture the site actually runs in. Your uptime plan should name the layers you do not control and say how you will find out when they fail.

  • Leaseweb’s Cloud Exit Survey: Test Your Cloud Provider Exit Plan Before You Need It

    Leaseweb’s Cloud Exit Survey: Test Your Cloud Provider Exit Plan Before You Need It

    On Nov. 4, 2026, Leaseweb published survey numbers that should make any ops lead uncomfortable: among the C-suite leaders it polled, nearly nine in ten said they could not move business-critical workloads off their largest cloud or infrastructure provider within three weeks. That is a different problem from getting through a bad day in one region. Outage response is about restoring service within agreed time and data-loss limits, sometimes by failing over inside the same provider and sometimes outside it. A cloud provider exit plan is about replacing your dependencies on that provider altogether. Mix up those two clocks and you end up running the wrong drill.

    What the Survey Actually Measured

    Start with the publisher and the sample, not the headline. In its Digital Independence Survey 2026 press release, Leaseweb, an IaaS and cloud provider with a commercial interest in alternatives to hyperscale lock-in, reported that 89% of respondents could not migrate business-critical workloads away from their largest cloud or infrastructure provider within three weeks. Only 11% said they could make that move in that window. More than half (55%) said it would take longer than three months or was not currently possible.

    The same release says 76% reported complete or high control over their ability to move data to another provider or environment. Leaseweb presents that next to the migration figure as a gap. I read it as two different tasks. Copying data out is one job. Moving a whole workload means rebuilding the compute, identity, networking, managed services, deployment pipelines, and monitoring around that data. A team can be confident about the first and still realistic about how long the second takes.

    Censuswide ran the fieldwork for Leaseweb among 800 C-suite leaders at organizations with 500 to 5,000 employees, 200 each in the UK, US, Germany, and the Netherlands, between Sept. 22 and 30, 2026. The release summarizes the findings but does not publish the questionnaire wording. Read “most companies” as most of this vendor-sponsored, self-reported sample of organizations in four countries, estimating how long an exit would take. It is not a census of every cloud customer.

    The October 2025 Outages Illustrate a Different Problem

    Provider exit is a planned migration measured in weeks or months. Incident recovery is measured against your recovery time objective (RTO), meaning how long until the service is usable again, and your recovery point objective (RPO), meaning how much data loss you can accept. Bringing DynamoDB clients back in us-east-1 tells you nothing about whether you could re-platform the estate onto another vendor by Friday.

    Amazon’s own summary of the Amazon DynamoDB service disruption in Northern Virginia (us-east-1) says impact began at 2:48 AM EDT on Oct. 20, 2025. AWS says the broader event ended at 5:20 PM EDT that day. In between there were separate periods of DynamoDB API errors, Network Load Balancer connection errors, and EC2 launch and connectivity problems. AWS attributes the DynamoDB portion to a latent defect in automated DNS management that caused endpoint resolution failures. That postmortem is about dependencies inside a single region: DNS, control planes, and shared building blocks. It is not a case study in leaving a provider.

    Customers also published their own accounts. Coinbase’s retrospective says it saw severe disruption from the us-east-1 outage on Oct. 20, 2025. Customer impact ranged from login failures to being unable to trade, with about 3 hours 17 minutes of degraded performance for users. Engineers were paged at 3:08 AM EDT, and Coinbase says all services were restored at 6:45 PM EDT after AWS fixed the systems it depends on. Coinbase also notes it could not reach AWS Support because the Support Console was offline. That is incident response and dependency triage during a provider outage, not a three-week migration to someone else.

    Microsoft’s Azure Networking team offers another recovery example from the edge side. Its December 2025 post on Azure Front Door lessons from the October outages covers customer-impacting incidents on Oct. 9 and Oct. 29, 2025, both caused by incompatible configuration metadata spreading through the global edge. The fixes Microsoft describes are configuration safety, tenant isolation, and faster automated recovery, including a proposed practical recovery target of about ten minutes for Front Door—not a guarantee or proof of achieved recovery times. That is RTO work, not advice to move off Azure this quarter.

    What a Cloud Provider Exit Plan Drill Should Actually Test

    The rest of this post is my own analysis as an operator. I’m not claiming Leaseweb’s 89% applies to your shop, and I’m not claiming lock-in caused any of those October outages.

    1. Write both clocks down. For each critical user journey, record the RTO and RPO next to an honest estimate of how long a provider exit would take. If the exit column is blank or says “call the vendor,” you have found a documentation gap.
    2. Inventory the sticky parts first. That means data gravity (object stores, databases, logs), identity (IAM roles, workload identity, SSO apps), DNS and TLS (registrars, certificate automation, traffic managers), and managed services with no clean equivalent elsewhere (queues, ML endpoints, proprietary APIs). Exit drills tend to fail on that glue, not on the VM list.
    3. Keep recovery tests and exit tests separate. Failing over to another region or another account with the same provider exercises RTO and RPO. It does not test an exit, because you are still on the same provider’s identity, control plane, and managed services. Imagine a tabletop where the team announces “we cut over to another provider by Monday” without export paths, IAM remapping, or named DNS cutover owners. That exercise is fiction, and the gaps it skips are your real findings.
    4. Run a bounded exit drill on one workload. In a hypothetical drill, you pick a non-critical service and time a restore onto a different provider or an independently operated on-premises environment. Moving to a different account or region does not count here. Write down what breaks: secrets, service endpoints, observability agents, payment webhooks. That stopwatch reading feeds your exit estimate. It does not prove you could move your most critical systems in three weeks.
    5. Budget for people and change windows. Leaseweb’s sample already says full workload migration is slow, even among leaders who feel confident about moving data. Your real constraint may be contract terms, data-residency rules, or the one engineer who understands the Terraform state.

    Use the survey as a reason to write the exit estimate down. Use the October postmortems as a reason to harden the recovery dependencies inside the provider you already pay. Keep those two drills on separate checklists, so the next outage doesn’t get answered with a three-month migration plan.

  • A Power Event in One Google Cloud London Zone Disrupted Banks and Public Services: Why Your Single-Region Answer Should Be in Writing Before Regulators Ask

    A Power Event in One Google Cloud London Zone Disrupted Banks and Public Services: Why Your Single-Region Answer Should Be in Writing Before Regulators Ask

    Everyone’s architecture diagram says “highly available.” On Tuesday, October 20, a lot of UK organizations found out what that means when one zone in Google’s only London region loses power. The Google Cloud London outage, and the single region risk behind it, is now on the desks of lawmakers and financial regulators. Nobody has asked your company for a written single-region answer yet, but the questions being asked of regulators point that way. Here is what happened, what is confirmed, and what I would write down now.

    What Google Says Happened

    Google’s preliminary incident report, posted October 22, gives the timeline in US/Pacific time. The incident ran from 23:00 on October 19 to 05:10 on October 20, a duration of “6 hours, 10 minutes.” That is about 7:00 a.m. to 1:10 p.m. in London on Tuesday, or 2:00 to 8:10 a.m. ET. Google says the disruption was “triggered by an electrical power interruption at the datacenter facility hosting availability zone europe-west2-b.”

    The root cause, in Google’s words, is a chain of failures. A grid disruption hit the facility. Load moved to UPS batteries while diesel generators started up. But “the Programmable Logic Controller (PLC) automation system (both primary and backup PLCs) failed to initiate the logical automatic start/transfer operations of the generators.” When the batteries ran out, hosts, network switching fabric, and storage in europe-west2-b “lost power and went offline.” Google calls this preliminary and says a final report with preventative actions will follow.

    The part operators should study is what happened next. Google says workloads failed over to the other two zones, europe-west2-a and europe-west2-c, as designed. Then “the sudden influx of redirected query executions, high-availability database failovers, and compute workloads saturated available compute slots and processing capacity in other availability zones.” The report says customers in the unimpacted zones “also experienced unexpected impact,” because of “instantaneous demand surges on unimpacted compute capacity” and “synchronous control-plane validation dependencies across zonal boundaries.” It names BigQuery, Looker, Compute Engine, GKE, Regional Persistent Disk, and Cloud SQL among the affected products. So the zone failed, but the region’s multi-zone design did not fully absorb it.

    Who Was Hit, and Who Says So

    The names come from press reporting, not from Google, which says only that a “very small number” of customers remained affected late in the day. Computer Weekly’s account lists banks and fintechs including Revolut, Lloyds Bank, HSBC, Barclays, Bank of Scotland, NatWest, Halifax, and Nationwide. On the public side it lists NHS England, HM Courts & Tribunals Service, and Gov.uk. Much of that list is based on user reports to DownDetector, and Computer Weekly says the public-sector sites “appeared to be affected.” The clearest confirmations are these:

    • Revolut told customers that “all our customers’ money and data are safe,” Computer Weekly reports. Reuters reports that logging in, card payments, and cash withdrawals were affected, according to Revolut’s website.
    • The Ministry of Justice confirmed that the courts’ Cloud Video Platform had not fully recovered by mid-afternoon, according to Computer Weekly.

    So “froze banks” overstates it. Some banking services at several banks were disrupted for hours. Computer Weekly says most services recovered by late afternoon.

    What Regulators Have Actually Done

    This is where the sheet’s headline went furthest, so I am being careful.

    • UK Parliament, not a regulator, put questions in writing. Computer Weekly reports that Treasury Committee chair Meg Hillier wrote to the Bank of England governor and the FCA’s chief executive. She asked nine questions, with an initial response due by November 4 and full details by January 20, 2027. One question asks: “What lessons should regulated financial firms (rather than Critical Third Parties) take from this incident when thinking about their own operational resilience?”
    • The FCA said it was engaging with Google, according to Reuters.
    • The Bank of England declined to comment to Reuters.
    • In the EU, the European Banking Authority told Reuters it had contacted Google Cloud “to obtain further information on the incident, understand its implications and assess its potential impact on financial entities in the EU.”

    What is already in force: Reuters says Google Cloud falls under both the EU’s critical third-party framework and the UK regime, which designated four cloud providers. Under the UK regime, designated providers “must report serious outages to the FCA and the BoE.” Reuters adds that under UK rules, “banks ultimately are responsible for the services they provide to customers, including when they rely on third-party providers.” What I did not find: any regulator asking firms for a written single-region or concentration-risk answer about this incident. That may come. It has not been reported yet.

    Single Region Risk, in Plain Terms

    Computer Weekly notes that europe-west2 is Google’s only UK region, so customers “that had not built multi-region resilience had no second UK location to failover to.” Google’s own report adds a second lesson: three zones in one region share more than you think. That includes the control plane and the spare capacity everyone fails over into at once.

    My Recommendations: Write the Answer Before Someone Asks

    This section is my opinion, not regulatory guidance.

    1. List every workload pinned to one region. Include the ones that are “multi-zone” but single-region, and the managed services you never configured yourself.
    2. Write down what customers experience if that region degrades for six hours. I mean what they see, not just what your architecture diagram shows. That is the answer a regulator, auditor, or board is most likely to ask for.
    3. Plan for capacity, not just failover. Google’s report shows failover into peer zones can saturate them. Reservations or a second region are how you buy room ahead of time.
    4. Test it. A failover runbook that has never been run is a hypothesis.
    5. Check data residency before you pick a second region. If the only UK region fails, your fallback may be outside the UK. Know whether your contracts and regulators allow that.

    Here is a hypothetical. Your payments API runs regional Cloud SQL across three London zones and your dashboard says “HA.” One zone drops, failover begins, and the surviving zones are already full of everyone else’s failovers. Your status page goes yellow for hours anyway. That is my illustration, built on the failure mode Google describes, not a reported case.

    The Bottom Line

    One London zone lost power because generator automation failed, and the rest of the region struggled to absorb the load. MPs and regulators are now asking questions in writing, mostly of each other and of Google. Your turn may come, so write your single-region answer now, while it is still your own exercise.

  • When the Zero Trust Proxy Goes Down, Some Microsoft 365 Users Go Down With It: Planning for Your SSE Provider’s Bad Day

    When the Zero Trust Proxy Goes Down, Some Microsoft 365 Users Go Down With It: Planning for Your SSE Provider’s Bad Day

    On October 13, some Microsoft 365 users could not reach their mail, files and meetings, and the cause was not Outlook or Teams themselves. Microsoft’s own incident notes point to its Global Secure Access service, the cloud network layer many organizations use to route traffic to Microsoft 365. Search for “Microsoft Global Secure Access outage Microsoft 365” and you will find coverage of exactly this event. That shorthand is useful as long as you keep the narrow scope in mind. This post lays out what Microsoft’s notices actually say, then what I would change in a plan that depends on a security service edge (SSE) provider.

    What Microsoft’s Notices Say

    Microsoft tracked the event as incident MO1505952. The notice text, as republished by 365Fanatic from a West Europe tenant and in the University of Pennsylvania’s mirror of Microsoft’s incident reports, reads like this:

    • Title: “Users may be unable to access multiple Microsoft 365 services using Global Secure Access (GSA).”
    • Services listed: “Impacted services may have included, but not been limited to: Outlook, Outlook on the web, OneDrive, SharePoint Online, Microsoft Teams, Windows 365 Cloud PCs.”
    • Root cause: “A section of GSA infrastructure was performing below acceptable performance thresholds, affecting connectivity to Microsoft 365 services and resulting in impact.”
    • Start and end: “Tuesday, October 13, 2026, at 7:40 AM UTC” to “Tuesday, October 13, 2026, at 6:31 PM UTC.” That is 3:40 AM to 2:31 PM ET, or about 10 hours 51 minutes.

    The cause Microsoft names is in Global Secure Access itself. Nothing in these notices attributes the problem to Microsoft Entra ID or Exchange Online, and I am not claiming either way beyond what Microsoft says. I am also not going further on root cause. Microsoft has published a post-incident report, according to the status entries, but I could not read its contents.

    Microsoft Global Secure Access Outage, Microsoft 365 Impact: Narrower Than the Headlines

    This matters for anyone deciding how worried to be. The impact was scoped to users “using GSA,” not to everyone on Microsoft 365. IncidentHub, which recorded Microsoft’s updates, writes that its updates “do not say Microsoft 365 was unavailable for users who were not using GSA.”

    The geography changed during the incident. Early updates said “Some users in Germany and the United Kingdom attempting to access Microsoft 365 services listed in the More info section using GSA may be impacted.” Microsoft’s final notice narrows that to “Some users in Germany.” IncidentHub’s timeline says the scope was narrowed to Germany only in the 17:26 UTC update. So “UK and Germany” is accurate for most of the incident, and “Germany” is what Microsoft’s final notice states.

    The start time also moved. IncidentHub notes that updates before 17:26 UTC stated an 08:00 UTC start, which Microsoft later revised to 07:40 UTC. Aggregators therefore disagree on duration. I use Microsoft’s final start and end times.

    One more useful detail. Microsoft’s updates show it first tried “performance and traffic optimizations,” then “redirecting traffic from the underperforming infrastructure,” then “targeted restarts.” A status entry also says “As a workaround, users may disable GSA to access the affected Microsoft 365 services until the resolution of this incident.” IncidentHub says that workaround first appeared at 18:20 UTC, about 11 minutes before the stated end. Other trackers time that entry slightly earlier, so treat the exact minute as approximate. Either way, it appeared late in a window of almost 11 hours.

    Planning for Your SSE Provider’s Bad Day

    Everything from here down is my own opinion and recommendation. Microsoft and the sources above do not say any of it.

    Here is my view of the lesson. Moving to zero trust and a cloud security edge removes some old single points of failure, such as one VPN concentrator. It can also add a new dependency. If your users reach Microsoft 365 through a cloud service edge, then that service’s health is part of your Microsoft 365 availability. This incident does not show that every organization has that exposure. It shows that organizations that do route through GSA were exposed to it.

    Picture this hypothetical. Your help desk gets a wave of “can’t open Outlook or Teams” tickets from one region. Your tenant dashboard shows no Exchange or Teams problem, and half the team spends the first hour troubleshooting endpoints. Nobody thinks to check whether the security edge is the common factor.

    What I would put in place:

    1. Know which traffic depends on the edge. Document which user groups, regions and Microsoft 365 workloads are forwarded through your SSE service, and which are not.
    2. Write the bypass decision down before you need it. Microsoft’s own workaround in this incident was to disable GSA. Decide who may approve that, what security controls you give up while it is off, and how you turn it back on.
    3. Keep a break-glass path. Have a tested way to reach critical services if the edge is degraded, such as a small group of pre-approved devices or a documented exception policy, with logging and a time limit.
    4. Test fallback policies in pilot rings. Try switching a small group off the edge during a planned test, so the first time you do it is not mid-incident.
    5. Monitor from the user’s side. Synthetic checks that sign in and open mail or files from the regions you care about will catch a path problem that the provider’s dashboard may not describe yet.
    6. Add the SSE service to your incident triage. When several Microsoft 365 apps fail at once for one region, check service health for the edge service early.
    7. Ask for the post-incident report. Read it, and ask your account team what changes are planned.

    The Bottom Line

    This was a real incident of nearly eleven hours, with a named cause in Microsoft’s Global Secure Access infrastructure, and a scope that narrowed from Germany and the UK to Germany. It was not a global Microsoft 365 failure. The planning lesson, in my view, is still worth taking: if a security service sits in front of your productivity suite, write the runbook for its bad day before it has one.

  • IDC Frontier Unauthorized Access Causes a Partial Cloud Outage for 495 Customers: Why You Need an Exit Plan for Your Regional Host

    IDC Frontier Unauthorized Access Causes a Partial Cloud Outage for 495 Customers: Why You Need an Exit Plan for Your Regional Host

    Imagine you run IT for a mid-sized town. Your disaster recovery plan has a tidy section on what happens if a big hyperscaler region goes dark. It probably has nothing on what happens if the quiet regional host that runs your website and a couple of line-of-business apps gets breached, and then has to pull the plug on its own platform to contain it. That second scenario is roughly what the IDC Frontier unauthorized access incident looks like from the customer side, and it turned into a partial cloud outage that operators everywhere should study.

    Here is what the provider has actually said, what it has not, and what I would do with it.

    What IDC Frontier Says Happened

    IDC Frontier runs IDCF Cloud, and its own notices describe the parent company as SoftBank Corp. In its second notice on October 7, 2026, the company said that since about 3:40 a.m. Japan time a fault had been occurring in East Japan Region 1. It said its investigation had found the fault was caused by “a ransomware attack by a third party.” It listed the scope as “495 companies and local governments that have contracted IDCF Cloud,” and said it was contacting affected customers individually.

    That 495 figure is one combined count of contracted customers, companies and local governments together. IDC Frontier has not published a split between the two, so I am not going to invent one.

    The response is the part operators should read twice. IDC Frontier said that “to prevent secondary damage and data leakage,” it cut East Japan Region 1 off from the network and then stopped the systems. As a precaution, it also stopped the customer-facing management consoles for its other regions while it checked whether they were safe, with plans to restore them once that was confirmed.

    How Unauthorized Access Became a Partial Cloud Outage

    The third notice, on October 8, narrowed the damage. The affected area is part of East Japan Region 1, specifically the tesla, henry, pascal and joule zones. There, virtual servers have stopped and cannot be restarted. IDC Frontier said customer data in those four zones is expected to be “difficult to extract or restore.” In its current view, recovery is possible only from backups customers hold themselves, and it is advising those customers to prepare a separate environment and rebuild.

    For the remaining zones of East Japan Region 1 (radian and newton), East Japan Regions 2 and 3, and West Japan Region 1, the company said no unauthorized access has been confirmed so far. It is still asking those customers to take their own backups. Because the consoles are down, IDC Frontier staff are starting and stopping customers’ virtual servers on request. The notices also say IDCF Cloud TypeS and IDCF Private Cloud are outside the scope of the incident.

    So the outage was partial, and it has two layers. IDC Frontier attributes the stopped, unrestartable servers in four zones to the ransomware attack itself. Its own containment then cut the region off the network and took away the self-service consoles everywhere else. Customers in zones with no confirmed intrusion still lost their control panel. I think that distinction matters more than the headline number.

    The downstream effects were visible fast. The Asahi Shimbun reported that the websites of the Ibaraki Prefecture government and prefectural police became inaccessible. It also reported that a logistics subsidiary of seafood company Nissui, which uses an IDC Frontier data center, had its logistics system shut down.

    Who Did It, and What We Still Do Not Know

    IDC Frontier’s notices describe the attacker only as “a third party.” They do not name a group, and neither will I. The company says it is working with an outside security firm on the intrusion route, has reported the incident to supervisory authorities and the Tokyo Metropolitan Police Department, and is still investigating the scope of the impact. Its fourth notice, on October 9, says it set up an emergency task force at 9 a.m. on October 7, working closely with SoftBank. SoftBank’s corporate division is handling customer contact, backup guidance and migration proposals.

    Still open: how the attacker got in, whether any data left the building, and when, or whether, the four zones come back.

    Opinion: The Regional Host Is a Concentration Risk You Can Measure

    This section is my opinion, not a finding from any source. We spend a lot of tabletop time on “us-east-1 is down.” That is a fine exercise, but it trains us to think of provider failure as an availability event that the provider fixes and we wait out. This incident is a different shape. A provider that is breached may decide, correctly, that the safest move is to isolate its own platform and lock customers out of the control plane. When that happens, your recovery time is set by your own backups and your own rebuild capability, not by the provider’s status page.

    Calling the regional provider “your single point of failure” is my framing. Whether it is true for you depends on how many of your services sit there and what you can stand up somewhere else.

    What I Would Check About My Own Hosting Provider

    • Count what lives there. List every public site, app and integration on each hosting provider, including the ones a vendor runs for you.
    • Own a backup the provider cannot touch. IDC Frontier’s own guidance is that recovery for the four zones depends on customer-held backups. Make sure yours are off-platform and tested.
    • Plan for losing the console, not just the servers. If you could not log in to your provider’s portal tomorrow, could you still fail over DNS, certificates and traffic?
    • Write the exit runbook now. Pick a second environment, document the rebuild steps, and time a restore. “Prepare a separate environment and rebuild” is a bad instruction to receive for the first time in the middle of an incident.
    • Ask the provider the hard questions. How are management plane and customer workloads separated? What would it isolate in a breach, and how would you be told?

    None of this requires distrusting a regional host. Regional providers do a lot of essential work well. It just requires admitting that “the provider will handle it” is a dependency, and dependencies belong on the risk register.

  • AI Spending Is Hard to Budget: Why IT Needs Token Quotas Before Finance Sees the Bill

    AI Spending Is Hard to Budget: Why IT Needs Token Quotas Before Finance Sees the Bill

    Picture this: a department rolls out a chat assistant on a Friday. By Monday a few power users have pasted whole contracts into it, an overnight agent loop is retrying failed calls, and a seat add-on nobody budgeted for has lit up on the invoice. That is not a horror story from a vendor deck. It is the shape of the 2014 cloud-bill surprise all over again, except now any employee or agent can create spend with a single prompt. IT leaders already know how to track seats. Usage-based token costs are a different animal, and the AI spending budget is the place that pain shows up first. The job is to forecast, cap, and charge them back before finance has to reverse-engineer the bill.

    I am deliberately not calling that “impossible.” That word is editorial heat, not a finding. What the sources do say is that AI costs are harder to forecast and allocate than the flat software licenses most of us grew up on, and that quotas, budgets and hard spend ceilings are the controls practitioners and vendors are actually recommending.

    Why AI Spending Budgets Struggle With Token Costs

    The FinOps Foundation’s FinOps for AI guidance puts the problem in operator language. Forecasting for new AI work is “much more challenging,” with more forecast variance and a need for shorter forecasting windows. On budgeting, it says top-down dollar budgets are limited by heterogeneous vendor pricing, bottom-up budgets are overloaded with per-component estimates, and for pilots “consumption volumes are virtually impossible to predict,” so budgets need more frequent revision. Tokens are one of the primary cost meters, but the guidance also flags non-token costs and the harder problem of tying spend to an outcome.

    The Wall Street Journal reported in June 2026 that CFOs are finding usage-based, token-priced AI “more unpredictable and harder to model,” and that only 26% of companies in a survey the paper cited said they had a comprehensive view of their AI costs. The Journal’s free summary does not name the survey firm or sample size, so treat that number as directional rather than definitive. The Journal’s CIO coverage the same season made a similar point from the other side: AI is putting pressure on corporate IT budgets, and leaders are already worrying about rising token costs past the pilot phase.

    Token Quotas and Spend Ceilings Are Real Controls

    This is the part of the sheet headline that holds up. FinOps for AI calls for “much more wide implementation of multiple limits such as quotas, use of reserved capacity, throttles, etc.” It also notes you may relax some rules for pilots, which is not the same as having no ceiling at all.

    The vendors have shipped matching knobs. OpenAI’s own spend-limits documentation describes monthly organization and project spend limits, spend alerts that notify without stopping traffic, and hard spend limits that return HTTP 429 errors with organization_spend_limit_exceeded or project_spend_limit_exceeded once tracked spend hits the cap. Alerts and hard limits are meant to be used together: warn first, then stop. Microsoft’s Azure guidance for Foundry and Azure OpenAI costs is blunter about the gap: create budgets and alerts in Cost Management, but “While OpenAI has an option for hard limits that prevent you from going over your budget, Azure OpenAI doesn’t currently provide this functionality.” Enforcement there needs custom automation off action groups. So “set a quota” is not one checkbox across every provider.

    What I Would Put in Place This Quarter

    Opinion: treat AI usage the way we eventually learned to treat cloud: owner, meter, ceiling, showback. In practice that means:

    • Name an owner. Someone in IT or FinOps owns the AI spend number the way someone owns the AWS bill. FinOps explicitly calls for clearer persona coverage because non-traditional “engineers” can now spin up AI projects.
    • Meter by use case, not just by vendor. FinOps argues for unit economics such as cost per inference or cost per outcome, not only total tokens. A chat pilot and a production agent will not share a sensible average.
    • Set alerts and, where the platform allows, hard caps. Use OpenAI’s project and org spend limits where you are on that API. On Azure, set Cost Management budgets and decide in advance what the action group does at 80% and 100%.
    • Charge back or at least show back. Tag or project-map usage to a cost center before the invoice lands. If you cannot allocate it, you cannot explain it.
    • Revisit the forecast on a short cycle. FinOps’s own advice is more frequent budget revision while the work is still experimental. Annual AI line items written in January will be fiction by April.

    None of that needs a new platform purchase. It needs the same discipline we built for cloud, applied to a meter that moves every time someone hits Enter.

    The CFO Invoice Is a Symptom

    If the first time IT hears about token burn is a forwarded invoice, the control plane failed earlier. Quotas and spend ceilings will not make AI cheap. They will make the surprise smaller, the owner clearer, and the conversation with finance about tradeoffs instead of forensics. That is the bar.

  • Azure Maintenance Hybrid Cloud VPN Outage: A Gateway Change, OS Servicing and Nearly Six Hours of Degraded Connectivity

    Azure Maintenance Hybrid Cloud VPN Outage: A Gateway Change, OS Servicing and Nearly Six Hours of Degraded Connectivity

    Picture this runbook. Step one: if the on-prem side of the VPN drops, fail over to the backup tunnel. Step two: if ExpressRoute degrades, shift traffic to the VPN. Step three: check the Azure portal to confirm the gateway is healthy. It would read fine right up until a night when the gateways themselves failed to load in the portal and management operations started timing out. Every step would have assumed the cloud side stays up. That’s the uncomfortable lesson of Wednesday’s Azure maintenance hybrid cloud VPN outage, and it’s why I’ve spent the morning reading Microsoft’s incident report instead of doing anything fun.

    What Microsoft Says Happened in the Azure Maintenance Hybrid Cloud VPN Outage

    Microsoft’s Azure status history entry for tracking ID 7Q30-010 is a preliminary review, marked “Mitigated.” It says that between 20:30 UTC on September 30 and 02:15 UTC on October 1, which is 4:30 PM to 10:15 PM Eastern on Wednesday, “a subset of customers using gateway services in multiple regions experienced degraded or interrupted network connectivity.” That’s five hours and 45 minutes from start to mitigation.

    The services Microsoft lists are the hybrid networking stack:

    • Azure ExpressRoute Gateway
    • Azure Firewall
    • Azure Application Gateway and Web Application Firewall
    • Azure VPN Gateway
    • Azure VMware Solution

    Customers may have seen gateways failing to load in the Azure Portal, along with failures or delays in network management operations. If you run Azure VMware Solution or connect your data center over ExpressRoute or site-to-site VPN, that’s your whole bridge between two worlds.

    The Timeline, in Eastern Time

    • 4:30 PM (20:30 UTC): customer impact began.
    • 5:29 PM (21:29 UTC): Microsoft began investigating ExpressRoute Gateway connectivity issues in UK South.
    • 6:27 PM (22:27 UTC): Microsoft identified that multiple regions were affected.
    • 7:05 PM (23:05 UTC): Microsoft found a correlation with operating system servicing activity and paused further servicing.
    • 9:36 PM (01:36 UTC, October 1): recovery progressed across most regions, while configuration changes were applied to France Central, North Europe, Southeast Asia, UK South and UK West.
    • 10:15 PM (02:15 UTC): Microsoft confirmed mitigation.

    On regions, Microsoft’s review names only the five late-recovering regions, plus UK South as the place the investigation started. It also carries a note that impacted regions in a previous communication were corrected. The Register, reporting during the incident, counted 18 affected regions at the time of writing, across the US, Europe, Asia, Africa, the Middle East and Mexico. It said Azure VMware Solution wasn’t impacted in four of them. Treat 18 as a mid-incident snapshot, not a final count.

    Was It Really “Routine Maintenance” That Took Things Down?

    Partly, and the nuance matters. According to Microsoft, “a recent change to a regional gateway management service triggered a higher-than-expected load when an unrelated operating system servicing maintenance proceeded gradually through multiple regions.” Normally the gateway management service would autoscale. This time, demand on dependent services kept those regional services from scaling as expected. Microsoft paused the OS servicing and reverted the gateway manager change, which let services recover. It says it’s still investigating the scaling behavior and the safeguards needed to prevent a repeat.

    So the maintenance didn’t break things on its own. A change and a maintenance rollout collided, and the safety net, autoscaling, didn’t catch it. That’s also why I wouldn’t say Azure “took down” hybrid clouds. Microsoft’s own words are “degraded or interrupted” for “a subset of customers,” and The Register relayed a Microsoft update saying some affected VPN gateways saw reduced redundancy rather than a complete loss of connectivity. For some shops it was an outage. For others it was a night of losing half their safety margin without knowing it.

    A full Post Incident Review is still coming. Microsoft says it generally publishes one to impacted customers within 14 days, so some of this may get sharper.

    When the Change Window Becomes the Incident

    What follows is my opinion, not Microsoft’s. Here’s what bugs me as an operator. We write change plans assuming our change is the risky one. We schedule our maintenance, tell the business, prep rollback and lean on the cloud provider as the stable half of the equation. This incident flips that. The provider’s maintenance, combined with the provider’s own change, was the event. Your change calendar would have shown nothing at all.

    The detail that worries me most is the management plane. When gateways fail to load in the portal and network management operations fail or stall, every runbook step that starts with “go into the portal and…” is suddenly a coin flip. That includes resetting a gateway, updating a connection or flipping a route. The portal is the cloud side of your runbook, and it’s exactly what was impaired.

    What I’d Put on the List This Week

    1. Turn on Service Health alerts properly. Microsoft’s guide to Service Health alerts covers alerting on service issues and planned maintenance by subscription, service and region, through email, SMS, push, webhooks or ITSM. It recommends selecting all services and all regions, since alerts only fire for regions where you actually run things. Route them to the on-call phone, not a shared mailbox.
    2. Check your gateway redundancy, not just your tunnel count. Microsoft’s VPN Gateway high-availability guidance notes that every gateway is active-standby by default, and it describes active-active gateways with multiple on-prem devices for full-mesh tunnels. If “reduced redundancy” happened to you Wednesday, would you have known?
    3. Build at least one runbook path that doesn’t need the portal. Pre-staged CLI scripts, on-prem routing changes, and BGP preferences you control from your side of the wire.
    4. Ask your account team about maintenance communications. What notice do you get for infrastructure OS servicing in your regions, and how fast does Microsoft pause a rollout when health signals drop? Here, impact began at 4:30 PM ET and servicing was paused at 7:05 PM ET.
    5. Write the post-incident review for your side. Check your own logs against the 4:30 to 10:15 PM ET window. Microsoft notes its times cover the whole incident, and actual impact varies by customer and resource.

    None of this means hybrid is a bad idea. It means the cloud side of hybrid is a dependency with its own change windows, and it doesn’t always tell you when one starts. Plan for it the way you’d plan for a carrier circuit: assume it’ll have a bad night, and make sure you can still steer from your end when it does.

  • Anthropic Akamai Cloud Deal: $11.6B of AI Infrastructure, and It’s CPUs, Not a CDN

    Anthropic Akamai Cloud Deal: $11.6B of AI Infrastructure, and It’s CPUs, Not a CDN

    Back when I was racking servers for a living, “the CDN” was the box you checked so the marketing site stayed up on launch day. Nobody asked what was inside it. That’s why the Anthropic Akamai cloud deal caught my eye this week: $11.6 billion of AI infrastructure committed to the company most of us still think of as the people who cache our JPEGs. The tempting headline is that AI labs just became the biggest CDN customers on Earth. The paperwork says something more specific, and frankly more interesting.

    What the Anthropic Akamai Cloud Deal Actually Covers

    Start with the primary source. In its September 24 press release, Akamai announced “$11.6 billion of contractual commitment over seven years” from Anthropic, to “support Anthropic’s accelerating CPU workload demands by leveraging Akamai Cloud’s distributed AI infrastructure and software.”

    Note what’s missing from that sentence: content delivery. Akamai’s Form 8-K filing goes further. It says two new project plans, signed September 18 under a master services agreement dated May 5, cover “dedicated cloud computing capacity and related managed support services.” Each plan runs for an initial seven years from its own service start date.

    The key numbers:

    • $11.6 billion committed over seven years. TechCrunch, citing Bloomberg, reports it’s the largest contract in Akamai’s history.
    • Up to $9 billion more if the relationship expands, for a potential total of about $20 billion.
    • About $5.5 billion in capital spending to build the capacity, plus roughly $1.7 billion added to 2026 capex to pre-buy components including memory.
    • A warrant for up to about 5% of Akamai’s stock, at $111.33 per common-share equivalent. About 2% is tied to the initial commitment, and roughly 1% more vests with each additional $3 billion Anthropic commits.

    So the $11.6 billion is real, but it’s a compute contract, not a CDN bill.

    The Fine Print Operators Will Recognize

    The filing has a line any of us who’ve signed a hosting contract will appreciate. Anthropic’s commitment is “subject to any termination described below and satisfaction of certain delivery and service availability requirements.” TechCrunch put it plainly: the commitment “isn’t ironclad.” Akamai has to deliver the capacity and keep it available, and either side can end the agreement under certain conditions. The 8-K also lets Anthropic terminate a project plan after a material outage, subject to conditions.

    That’s an SLA with real teeth. If you’ve ever explained to a CFO why a 99.9% target isn’t the same as 99.99%, you know the difference between “committed” and “paid” lives in clauses like that.

    Timing matters too. TechCrunch reports Akamai won’t see revenue from the deal this year. Executives told investors they expect $150 million to $300 million in 2027, starting in the second half, and an annual pace of about $1.7 billion by the end of 2028. That’s a long runway of building before billing.

    Why CPUs, and Why Akamai?

    Here’s the part I think gets lost. The AI infrastructure conversation is usually about GPUs, giant training clusters and power plants with a data center attached. This deal is about CPUs, the plain general-purpose chips. TechCrunch notes CPU demand has grown as AI agents take on more tasks like running code and browsing the web. It also notes Akamai didn’t say what Anthropic will use them for.

    That gap is where the CDN framing creeps in. In The Boston Globe’s take, IDC’s Ashish Nadkarni thinks Anthropic will use Akamai’s network to get closer to its customers: “It’s kind of similar to what you do with streaming music today, where the edge network essentially speeds up the delivery of the data.” The Globe also reports that Akamai expects the new equipment to draw at most an additional 110 megawatts, spread across dozens of data centers where it already leases space.

    That’s an analyst’s informed guess, not a disclosed use case. But it lines up with Akamai’s pitch about “a continuum of compute from core to edge” spread across thousands of points of presence. It also lines up with the hook I can’t shake: model traffic is starting to look like a content-delivery problem with an inference bill attached.

    So Is “Biggest CDN Customer on Earth” True?

    No, or at least nobody has shown it. None of the sources I read rank Anthropic, or AI labs generally, as the largest CDN customers anywhere. The deal isn’t described as CDN services at all. What the sources support is narrower:

    • It’s the largest contract in Akamai’s history, per Bloomberg as reported by TechCrunch, and the biggest single contract Akamai has ever won, per The Boston Globe.
    • It’s for dedicated cloud computing capacity serving CPU workloads, not content delivery.
    • How Anthropic will use it, edge inference or something else, hasn’t been disclosed.

    That’s why this post isn’t wearing the original headline.

    What This Means If You Run Infrastructure

    Picture a team that would have tried to rent a few hundred CPU nodes for an internal agent platform next spring. In that hypothetical, they might find the friendly regional cloud quoting longer lead times. They wouldn’t have done anything wrong. A buyer with $11.6 billion and a seven-year horizon would simply have gotten in line first. Akamai pre-buying memory with $1.7 billion of this year’s capex is a hint about where supply chains are tight.

    A few practical takeaways:

    1. Don’t assume AI capacity means GPUs. Agent workloads spend a lot of time on ordinary compute: calling tools, parsing pages, running code. Plan and budget for both.
    2. Read the availability clauses. If Anthropic negotiated an outage termination right, you can ask your providers what happens to your commitment when they miss theirs.
    3. Watch vendor concentration in both directions. Akamai now has one customer with a potential equity stake and a contract that’s big relative to the company. That’s a dependency worth understanding if you rely on Akamai.
    4. Treat edge inference as a design option, not a given. Running smaller models closer to users is plausible. Until someone discloses it, it’s still a hypothesis.

    The CDN didn’t become an AI company overnight. A company that spent decades putting servers close to users just sold a lot of that closeness, and a lot of new CPUs, to a customer that needs both. I’d call that the most sysadmin story of the AI boom so far: the winning move was boring infrastructure, in the right places, with an SLA.

  • Google Cloud API Gateway MCP Tools for REST APIs: A Handy Preview That Turns Opted-In Endpoints Into Agent Doors

    Google Cloud API Gateway MCP Tools for REST APIs: A Handy Preview That Turns Opted-In Endpoints Into Agent Doors

    Picture this ticket. Somebody’s shiny new assistant “finds a way” to reset a user’s MFA, and if you traced it back, you might find it went through something like /internal/v2/admin/resetFactor, an endpoint documented in one wiki page from 2021 and a Slack thread nobody can find. In that scenario, the assistant wouldn’t have hacked anything. It would have read a description, decided the endpoint was a skill it had, and used it. That’s the scenario I kept imagining while reading about the new Google Cloud API Gateway MCP tools for REST APIs, which let a gateway you already run hand your API operations to AI agents as callable tools.

    The feature is useful, and the fine print matters more than the headline. Let’s go through what Google actually shipped and which doors it opens.

    What Google Actually Announced

    On September 24, Google’s developer blog published Turn your REST APIs into MCP tools with Google Cloud API Gateway. The short version: in Public Preview, API Gateway can now act as a remote Model Context Protocol (MCP) server. You annotate the OpenAPI spec you already deploy, redeploy, and the gateway starts serving MCP on a /mcp path. Agents built with frameworks like Google’s Agent Development Kit can then discover your operations and call them. You don’t have to build or host a separate MCP server.

    A few specifics worth pinning down:

    • It’s a preview, not GA. The MCP overview in Google’s documentation marks the feature as subject to the Pre-GA Offerings Terms, available “as is” with possibly limited support. The API Gateway release notes list it as a Public Preview feature dated September 11, about two weeks before the blog post.
    • It’s opt-in. Nothing happens to your APIs until you add the x-google-api-management MCP setting to your spec. Google didn’t flip a switch on anybody’s endpoints.
    • OpenAPI 3.x only. Specs still on OpenAPI 2.0 aren’t supported, so some shops will have a migration to do first.
    • Tools only. The gateway supports initialize, tools/list and tools/call. MCP resources and prompts aren’t supported, and neither are stdio transport or streaming and long-running calls. Google says resources, prompts, response streaming and Model Armor payload inspection are on the roadmap.
    • Some limits. Operations that return empty bodies, like HTTP 204, aren’t exposed. A gateway serves up to 1,000 tools. MCP and model routing can’t be enabled in the same API config.

    So the accurate version of the hot take is narrower: if you run Google Cloud API Gateway and choose to turn this on, your REST operations can become agent tools with a few lines of YAML. That’s still a big deal. It just isn’t every endpoint on the internet.

    How Google Cloud API Gateway MCP Tools for REST APIs Handle Auth

    This is the part I actually care about, and Google’s design is mostly sensible. According to the blog, the gateway transcodes each tool call into the matching REST request, and it’s “indistinguishable from a normal REST call.” That means the JWT or API key checks, quotas and logging you already configured still apply, and MCP and REST traffic share one policy path.

    The configuration guide lays out the authentication model by method:

    • initialize and the handshake acknowledgment are unauthenticated.
    • tools/call reuses whatever authentication the underlying operation requires. If the REST endpoint wants a JWT, so does the tool.
    • tools/list, the method that tells a client what tools exist, is unauthenticated by default. Google strongly recommends locking it down, and if you do, it has to be a JWT scheme. API keys aren’t supported for discovery in the preview.

    Google’s own blog says the quiet part plainly: an open tools/list “publishes your tool names and input schemas to anyone who asks.” Tool names and schemas are a map. Even if every call is authenticated, you’ve handed out a labeled floor plan of the building.

    The Global Toggle Is the Real Door

    Here’s where the “every endpoint” worry has some truth to it. You can enable MCP two ways. Per operation, you mark the specific paths you want exposed. Globally, with mcp: true, the docs say all eligible operations are exposed as tools. Eligible means GET, POST, PUT, PATCH and DELETE. By default the tool name is the operationId and the description comes from the operation’s description or summary.

    One gotcha deserves bold type. Configuring the mcp setting as an object, which is how you secure tools/list, also enables MCP globally. Google’s blog flags it right in a code comment. So the admin who does the responsible thing and requires a JWT for discovery has also, without trying, published every eligible operation in the spec. You then opt individual operations back out with x-google-mcp-tool: false. That’s default-open behavior hiding inside a security setting, and it’s exactly how the half-documented admin endpoint ends up in an agent’s toolbox.

    The docs’ own example is a DELETE operation described as “Permanently delete a shelf and every book on it.” Sure, it’s a bookstore demo. But swap “shelf” for “tenant” and you have the ticket I opened with.

    Descriptions Are Now Security Controls

    Google notes that a tool’s description “is the primary signal an LLM uses to decide when to call it.” That changes the job of your OpenAPI summaries. For years, “Deletes stuff (internal, do not use)” was a note for humans. An agent may read that as a capability with a mild warning. If an operation shouldn’t be called by an agent, don’t rely on the description to say so. Keep it out of the tool list.

    What I Would Do Before Turning It On

    1. Audit the spec first. Every operation in that OpenAPI file is a candidate tool. Find the leftovers: admin routes, debug endpoints, anything “temporary.”
    2. Prefer per-operation opt-in over global. If you must use the object form to secure discovery, add explicit opt-outs and review the resulting tools/list output.
    3. Require a JWT on tools/list in anything that isn’t a sandbox. Unauthenticated discovery is fine for a demo and a gift for recon.
    4. Separate agent credentials from human ones. Since the backend can’t tell MCP calls from REST calls, give agents their own identities so the logs can.
    5. Treat destructive methods as their own review. PUT, PATCH and DELETE exposed to an agent deserve the same scrutiny as handing a junior admin the keys.

    I like this feature. Rebuilding auth and quota logic in a separate MCP server was always going to produce worse security than the gateway you already hardened. But convenience moves the risk to your spec file. The gateway will faithfully publish what you describe, so make sure you’re describing only the doors you meant to install.

  • Oracle Cloud Layoffs Meet AI Data Center Spending: 546 Infrastructure Jobs Out, Up to $95B In

    Oracle Cloud Layoffs Meet AI Data Center Spending: 546 Infrastructure Jobs Out, Up to $95B In

    Every operator knows the budget meeting where somebody says the quiet part out loud: hardware is a capital expense, people are an operating expense, and only one of them shows up nicely on the investor slide. Oracle is running that meeting at a scale most of us will never see. The combination of Oracle cloud layoffs and AI data center spending this month reads like a change request that says “remove the people who keep the cloud upright, add more GPUs, power, and cooling.”

    What the Numbers Actually Say

    The number making the rounds is 546, and it needs a little context before anyone repeats it. According to Business Insider’s reporting on a leaked Oracle document, published September 22, the document lists the job titles and ages of 546 employees in Oracle’s America Cloud Infrastructure organization who were laid off, about 7.6% of the 7,185 employees listed. Oracle said the information was provided to comply with federal age discrimination laws. It is not a company-wide total. Business Insider says Oracle has not disclosed how many people it laid off in total in this round, and the publication could not determine whether the document covered every cloud infrastructure employee who was cut.

    It is also not 546 engineers. The mix is broader than that, and more interesting:

    • Software developer III was the single hardest-hit title, with 57 people cut, followed by program manager IV and principal core infrastructure engineer.
    • Software developers made up around 17% of the roles cut.
    • Roles with “manager” in the title accounted for 128 cuts, roughly 23% of the total.
    • Oracle’s data center support services, the group that maintains and supports its data centers, lost 41 people, including the division’s vice president and two senior directors.
    • Most of the affected employees were over 40, and around 16% were 60 or older.

    Separately, a WARN notice covered by FOX 13 Seattle shows Oracle America told Washington state regulators on September 14 that it will lay off 359 employees working from its two Seattle facilities or remotely in Washington, with separations beginning November 13. That filing is one state, and it should not be added to or confused with the 546 figure. They are different slices of the same restructuring.

    On the spending side, the $95 billion figure checks out, with an asterisk. In its June earnings call, Oracle said fiscal 2027 capital expenditures would land between $90 billion and $95 billion, up from $55.6 billion in fiscal 2026, as DatacenterDynamics reported. So $95 billion is the top of a range, not a single committed check. Oracle also began emphasizing a “project net cash outlay” of about $70 billion, with customer prepayments and timing effects accounting for the gap. Business Insider’s September piece says the company still expects to spend $90 to $95 billion this year building new data centers.

    Why Oracle Cloud Layoffs and AI Data Center Spending Collide

    From a pure finance view, this is not mysterious. Oracle has signed enormous AI infrastructure contracts, and building data centers eats cash long before the revenue arrives. DatacenterDynamics notes the company expects to raise about $40 billion in debt and equity in fiscal 2027. At that scale, headcount is the biggest lever that does not require a supplier negotiation.

    Business Insider also reports that Oracle’s workforce declined by 21,000 employees, or 13%, in the fiscal year that ended May 31, and that the company had 141,000 employees before this latest round. So this is not a one-time trim. It is a sustained reallocation from people to plant.

    What makes the operator in me wince is where the cuts landed. The cloud infrastructure division is the part of Oracle that is growing fastest. Business Insider notes Oracle reported quarterly revenue from the division up 121% year over year. Cutting inside your fastest-growing unit while you pour concrete for more capacity is a bet that the humans were the inefficient part of the system.

    The Operator Comedy of Fewer Hands and More Racks

    Here is the thing about data centers that never makes the capex slide: they do not run themselves. Every megawatt you bring online comes with a tail of work. Firmware updates, failed drives, cooling loops that develop opinions at 3 a.m. Change windows that somebody has to plan, execute, and roll back when the vendor’s “non-disruptive” patch turns out to be disruptive after all.

    DatacenterDynamics quotes CEO Clay Magouyrk saying Oracle was seeing GPU utilization of 97.5% globally. That is a spectacular number for revenue. It is also a spectacular number for risk. At 97.5% utilization, you have very little slack for failover, maintenance, or the one rack that decides to cook itself. Slack in a system is not waste. It is the thing that keeps an incident from becoming an outage.

    Now pair that with cutting the data center support services team, including its vice president and two senior directors. Oracle knows its operations better than I do. But from where I sit, removing senior operational leadership and experienced engineers while you expand capacity is the organizational equivalent of running a cluster with no spare nodes and calling it efficient. It works right up until correlated failure shows up, and correlated failure always shows up on a holiday weekend.

    Institutional Knowledge Does Not Live in the Runbook

    The age data deserves a mention for a practical reason, not just an HR one. Most of those affected were over 40, and about one in six was 60 or older. In infrastructure work, tenure often maps to the kind of knowledge that never made it into documentation: why a particular site’s power feed behaves oddly, which vendor escalation path actually works, what broke the last time someone tried that migration. You cannot buy that with GPUs, and you usually find out it is gone during the first bad incident after it walks out the door.

    What This Means for Everyone Else

    If you run workloads on Oracle Cloud, none of this means your tenancy is about to fall over. It does mean it is worth paying attention to support responsiveness, incident communication, and maintenance notices over the next few quarters. Those are the early signals of operational strain, long before anything shows up on a status page.

    If you run infrastructure anywhere, the broader lesson is familiar. The industry is in a phase where capital spending is the story and operating staff are the cost to be managed. That math always looks great in the quarter it happens. The bill for thin operational coverage tends to arrive later, as longer outages, slower recoveries, and a lot of expensive consultants rediscovering what the laid-off engineers already knew.

    Oracle may be right that AI demand justifies every dollar of that $90 to $95 billion. But racks do not patch themselves, and the people who keep the cloud upright were never the line item that made it fall down.