Steve Miller's Blog

A digital workshop for systems-minded tech commentary.

Author: steve

  • AI Spending Is Hard to Budget: Why IT Needs Token Quotas Before Finance Sees the Bill

    AI Spending Is Hard to Budget: Why IT Needs Token Quotas Before Finance Sees the Bill

    Picture this: a department rolls out a chat assistant on a Friday. By Monday a few power users have pasted whole contracts into it, an overnight agent loop is retrying failed calls, and a seat add-on nobody budgeted for has lit up on the invoice. That is not a horror story from a vendor deck. It is the shape of the 2014 cloud-bill surprise all over again, except now any employee or agent can create spend with a single prompt. IT leaders already know how to track seats. Usage-based token costs are a different animal, and the AI spending budget is the place that pain shows up first. The job is to forecast, cap, and charge them back before finance has to reverse-engineer the bill.

    I am deliberately not calling that “impossible.” That word is editorial heat, not a finding. What the sources do say is that AI costs are harder to forecast and allocate than the flat software licenses most of us grew up on, and that quotas, budgets and hard spend ceilings are the controls practitioners and vendors are actually recommending.

    Why AI Spending Budgets Struggle With Token Costs

    The FinOps Foundation’s FinOps for AI guidance puts the problem in operator language. Forecasting for new AI work is “much more challenging,” with more forecast variance and a need for shorter forecasting windows. On budgeting, it says top-down dollar budgets are limited by heterogeneous vendor pricing, bottom-up budgets are overloaded with per-component estimates, and for pilots “consumption volumes are virtually impossible to predict,” so budgets need more frequent revision. Tokens are one of the primary cost meters, but the guidance also flags non-token costs and the harder problem of tying spend to an outcome.

    The Wall Street Journal reported in June 2026 that CFOs are finding usage-based, token-priced AI “more unpredictable and harder to model,” and that only 26% of companies in a survey the paper cited said they had a comprehensive view of their AI costs. The Journal’s free summary does not name the survey firm or sample size, so treat that number as directional rather than definitive. The Journal’s CIO coverage the same season made a similar point from the other side: AI is putting pressure on corporate IT budgets, and leaders are already worrying about rising token costs past the pilot phase.

    Token Quotas and Spend Ceilings Are Real Controls

    This is the part of the sheet headline that holds up. FinOps for AI calls for “much more wide implementation of multiple limits such as quotas, use of reserved capacity, throttles, etc.” It also notes you may relax some rules for pilots, which is not the same as having no ceiling at all.

    The vendors have shipped matching knobs. OpenAI’s own spend-limits documentation describes monthly organization and project spend limits, spend alerts that notify without stopping traffic, and hard spend limits that return HTTP 429 errors with organization_spend_limit_exceeded or project_spend_limit_exceeded once tracked spend hits the cap. Alerts and hard limits are meant to be used together: warn first, then stop. Microsoft’s Azure guidance for Foundry and Azure OpenAI costs is blunter about the gap: create budgets and alerts in Cost Management, but “While OpenAI has an option for hard limits that prevent you from going over your budget, Azure OpenAI doesn’t currently provide this functionality.” Enforcement there needs custom automation off action groups. So “set a quota” is not one checkbox across every provider.

    What I Would Put in Place This Quarter

    Opinion: treat AI usage the way we eventually learned to treat cloud: owner, meter, ceiling, showback. In practice that means:

    • Name an owner. Someone in IT or FinOps owns the AI spend number the way someone owns the AWS bill. FinOps explicitly calls for clearer persona coverage because non-traditional “engineers” can now spin up AI projects.
    • Meter by use case, not just by vendor. FinOps argues for unit economics such as cost per inference or cost per outcome, not only total tokens. A chat pilot and a production agent will not share a sensible average.
    • Set alerts and, where the platform allows, hard caps. Use OpenAI’s project and org spend limits where you are on that API. On Azure, set Cost Management budgets and decide in advance what the action group does at 80% and 100%.
    • Charge back or at least show back. Tag or project-map usage to a cost center before the invoice lands. If you cannot allocate it, you cannot explain it.
    • Revisit the forecast on a short cycle. FinOps’s own advice is more frequent budget revision while the work is still experimental. Annual AI line items written in January will be fiction by April.

    None of that needs a new platform purchase. It needs the same discipline we built for cloud, applied to a meter that moves every time someone hits Enter.

    The CFO Invoice Is a Symptom

    If the first time IT hears about token burn is a forwarded invoice, the control plane failed earlier. Quotas and spend ceilings will not make AI cheap. They will make the surprise smaller, the owner clearer, and the conversation with finance about tradeoffs instead of forensics. That is the bar.

  • Warlock Ransomware Is Still Exploiting Old and New SharePoint Vulnerabilities: Why Patch Debt Matters for Water and Telecom

    Warlock Ransomware Is Still Exploiting Old and New SharePoint Vulnerabilities: Why Patch Debt Matters for Water and Telecom

    Here is a hypothetical that will sound familiar to anyone who has sat through a cloud migration kickoff. The on-prem SharePoint farm is “being retired.” It has been being retired for two years. Patching it slips to “after the cutover,” monitoring quietly lapses, and the box keeps answering on port 443 the whole time. That server is exactly the kind of thing the Warlock ransomware crew is still breaking into, according to new research. It is getting in through SharePoint vulnerabilities, some of which have had a patch available for more than a year.

    The source here is the Symantec and Carbon Black Threat Hunter Team’s October 1 report on Warlock. It describes a group Symantec calls Longlegs, which Microsoft tracks as Storm-2603, hitting critical infrastructure through on-premises SharePoint. Here is what the report actually says, where the sheet headline went a little too far, and what I would check this week.

    Who Got Hit, and What Symantec Actually Says

    Symantec says that in the past two months Longlegs attacked at least four organizations. Two were critical infrastructure operators, a water utility and a telecommunications provider. The other two were a regional government body and a university. All four were in Portuguese- and Spanish-speaking countries across Europe, Africa and Latin America. Symantec does not name the victims, and the report does not describe any disruption to water treatment or network operations. I am not going to imply either. The report calls the group China-nexus. It notes that Warlock has previously turned up in a wider range of countries, including the United States, but none of these four were US organizations.

    This is not the group’s first SharePoint campaign. Warlock emerged in June 2025 and became notorious weeks later, when it was tied to the “ToolShell” zero-days in on-prem SharePoint Server. In its July 2025 write-up, Microsoft said it had observed Storm-2603 exploiting those flaws to deploy ransomware. Microsoft released security updates for all supported SharePoint Server versions that month.

    Which SharePoint Vulnerabilities, and Which Patch

    The sheet headline said “year-old flaws.” That is only partly right, and the distinction matters for what you patch:

    • The 2025 ToolShell chain (CVE-2025-49704, CVE-2025-49706, CVE-2025-53770 and CVE-2025-53771). Symantec says these “likely remain in the group’s arsenal.” Note the word likely.
    • Newer SharePoint flaws. Symantec points to the ones covered in a July 2026 CISA advisory. CISA’s SharePoint hardening alert lists six actively exploited 2026 CVEs affecting all supported on-prem versions: Subscription Edition, 2019 and 2016. CISA added them to its Known Exploited Vulnerabilities catalog between April and August 2026.

    Symantec does not pin each intrusion to a single CVE. For the attack it walks through in detail, it says initial access was “likely” through SharePoint Server vulnerabilities. So the honest framing is “old and newer SharePoint flaws on servers that were not patched or otherwise mitigated,” not “a year-old bug did it.”

    Inside One Intrusion: Nine Days From Webshell to Ransomware

    Symantec’s detailed case is one intrusion at a critical infrastructure operator. The report does not say whether it was the water utility or the telecom. The timeline is worth reading slowly:

    • July 22, 2026: A webshell landed in the SharePoint LAYOUTS directory. Symantec says the group drops it for several SharePoint versions at once, and uses it to harvest the farm’s ASP.NET machine keys. With those keys it can forge validly signed payloads and run code inside the SharePoint application pool.
    • July 24 to 29: Reconnaissance, DLL sideloading and payloads pulled from public file-hosting services. The attackers added a domain account to local Administrators on three more hosts. Symantec says its name was likely chosen to pass for a SharePoint setup account. They installed a Visual Studio Code tunnel as a service for covert remote access and ran NetExec for Active Directory enumeration and credential spraying.
    • July 31: An AV/EDR killer was pushed to at least 40 hosts within about two hours. Warlock showed up on at least 33 hosts almost as soon as protection dropped on each one. The payload was staged in the domain’s SYSVOL share, and on three hosts telemetry showed ordinary SYSVOL replication delivering it.

    That is nine days from first observed activity to ransomware. It is not a smash-and-grab either. In my view, that is plenty of time for someone watching the right logs to notice.

    One correction to a common shorthand. Symantec says Longlegs abuses a signed but vulnerable K7RKScan driver (CVE-2025-1055) to kill security tools, and BleepingComputer’s coverage repeats that. But in this specific intrusion, Symantec says the identity of the vulnerable driver “remains unknown.” K7RKScan is what the group used in other recent attacks.

    Opinion: Patch Debt Is the Real Playbook

    This section is my opinion, not Symantec’s finding. Nothing in this report needed a fresh zero-day to be scary. What made it work, as far as the public details go, is the stuff every ops team recognizes: an internet-facing on-prem server that probably was not first in line for patches, a farm whose machine keys a webshell could simply read, and a domain where a new executable in SYSVOL does not set off alarms. I call that patch debt. Like financial debt, the interest compounds quietly until someone else collects it.

    Water utilities and telecoms are not uniquely careless. They tend to run long-lived systems with tight change windows, which is precisely the environment where “we’ll patch it after the migration” turns into a two-year habit.

    What I Would Check This Week

    • Find every on-prem SharePoint server, including the one “being retired,” and confirm the latest Microsoft updates are actually installed, not just approved.
    • Get it off the open internet. CISA’s first recommendation is to avoid exposing SharePoint directly. If you must expose it, put it behind an application-layer proxy that enforces authentication.
    • Hunt before you rotate. CISA says to look for machine-key harvesters and other intrusion artifacts before rotating IIS machine keys. Otherwise you are just handing the attacker fresh keys.
    • Turn on AMSI integration for each SharePoint web application, as CISA recommends.
    • Watch SYSVOL like it is production code. New executables under the scripts folder deserve an alert, and so does a VS Code tunnel service showing up on a server.

    None of that is exotic. It is the backlog. Warlock is betting you will not get to it.

  • Zammad Zero-Day DIVD Breach: How AI Agents Chained Two Flaws to Root, and What It Means for Help Desk Security

    Zammad Zero-Day DIVD Breach: How AI Agents Chained Two Flaws to Root, and What It Means for Help Desk Security

    A hypothetical: suppose your help desk box gets popped on a Sunday. Nobody notices until Monday, and by then the attacker has gone from the ticketing app to root and is poking at other services on the network. Now suppose your entire job is telling other people about their vulnerabilities. That is roughly what the Dutch Institute for Vulnerability Disclosure says happened to it, and the Zammad zero-day DIVD breach is the best argument I have seen in a while for taking help desk security as seriously as the domain controller.

    DIVD is a mostly volunteer group that finds flaws, reports them to vendors, and warns the owners of exposed systems. On its DIVD-2026-00014 case page it puts it bluntly: “DIVD got hacked through AI agents.” Here is what the group has said, what it has not, and what I would do about it.

    The Zammad Zero-Day DIVD Breach: What DIVD Says Happened

    According to DIVD’s timeline, the first malicious access to its systems came on September 21, 2026. DIVD spotted the activity on September 22, blocked access to everything in its datacenter, and started a forensic investigation with Merlon Security.

    DIVD’s September 30 statement says the attackers got in through two zero-days in Zammad, the open-source ticketing system. Together the flaws “allowed session hijacking, remote code execution and privilege escalation from the Zammad user to root, in seconds due to the agentic part of this hack.” From there, DIVD says, the attackers could reach other services and exfiltrate data. DIVD credits network segmentation and fast work by its IT and incident response team for stopping them from going deeper. It also says it is assuming breach until it can prove otherwise.

    The AI part rests on DIVD’s own reading of the logs. DIVD says the agent decided each next step itself, fast and on sloppy logic. Its scripts carried comments justifying its own actions, which DIVD notes a human attacker would not bother writing. The attack was, in DIVD’s words, “loud and very messy.”

    The Two Flaws, Without the Hype

    DIVD opened a second case, DIVD-2026-00015, to track the bugs and notify owners of vulnerable instances. The details matter, because several write-ups flattened them:

    • CVE-2026-102489 is a session hijack that leads to remote code execution as the zammad user. DIVD lists Zammad 6.3.0 through 6.5.4 as exploitable. The code is also present in 7.0.0 through 7.1.3 but, per DIVD, is not exploitable there because of the runtime environment. DIVD scores it 8.7 on its own.
    • CVE-2026-102490 is a local privilege escalation from the zammad user to root. DIVD lists versions 1.5.0 up to the 7.1.0 alpha and describes it as affecting all versions, including the latest alpha. DIVD scores it 8.5 on its own.

    DIVD gives both a 9.4 critical score when they are chained, and that number is the honest headline. Neither bug alone hands a stranger root. Together they did. CISA added both CVEs to its Known Exploited Vulnerabilities catalog on October 2.

    Patch Status as of October 6, 2026

    This is still moving, so I am pinning it to a date. As of the afternoon of October 6, 2026, here is the status per Zammad’s October 5 security advisory.

    For CVE-2026-102489, Zammad says 7.0 and later are not affected in practice. It says exploitation only works on 6.5 and earlier, which are out of support and no longer get security updates. It also says it first received a report on the issue in August 2026, and that a hardening change to the affected code shipped in 7.2.0.

    For CVE-2026-102490, Zammad calls it a local privilege escalation that cannot be exploited remotely on its own, and ties it to a confirmed vulnerability in packager.io. Its wording is that the team “is working on a solution.” I found no fix announced. The Zammad 7.2.1 release notes, published October 6, do not mention this CVE. Zammad also says 102490 went public before it had the details to reproduce it, a disclosure dispute I will leave to the two parties.

    What Got Out, and What Nobody Knows Yet

    Early coverage, including Help Net Security’s October 1 report, said DIVD had not yet shared which data was accessed. DIVD’s own statement from the same day goes a bit further. Volunteer data got out, “such as DIVD email addresses and possibly contact details.” DIVD says whose data and exactly which data are still under investigation, and warns that it is now easier for someone to pose as a DIVD volunteer. That is all DIVD has confirmed about the data.

    The same goes for who did it. DIVD says it sees no link to any known public threat actor. It also says no facts so far point to a deliberate, targeted attack on its most sensitive assets, though it cannot rule that out.

    Why Help Desk Security Belongs on the Critical List

    Opinion: I would call the help desk crown-jewel infrastructure. That is my label, not DIVD’s. The case is simple. Think about what a typical ticket queue holds in most shops, not DIVD’s specifically: password reset threads, screenshots of admin consoles, VPN error messages with hostnames in them, and the email address of every user who ever asked for help. It usually holds a mail integration, often an LDAP or SSO hookup, and API tokens for other systems. It is internet-facing by design, because customers and staff need to reach it. And in a lot of shops it is the app nobody owns after the install weekend.

    DIVD’s story has the shape of every bad night I remember: an old version, a half-known bug, and a service account one step from root. The difference is speed. If an agent can chain those steps in seconds, the gap between “first access” and “it reached the next box” is shorter than your alert routing.

    What I Would Do This Week

    • Check your version today. If you are on Zammad 6.5 or older, you are on unsupported code that is exploitable for 102489. Upgrade, or follow DIVD’s advice and take the instance offline until you can.
    • Copy your logs before you upgrade. Help Net reports that the Dutch NCSC recommends keeping application and network logs, which will help if more details on the second flaw emerge. DIVD also publishes an indicator-of-compromise log check script on its case page.
    • Lock down the server, not just the app. Zammad’s own guidance is to restrict access to the underlying server to trusted administrators. Since 102490 has no announced fix, assume anything running as the zammad user could become root.
    • Segment it like it matters. DIVD credits segmentation with limiting the damage. The help desk host should not be able to reach everything else just because it is internal.
    • Watch the vendor channel. Keep an eye on Zammad’s security advisories for the 102490 fix.

    DIVD deserves credit for publishing this in near real time, embarrassing parts included. The least we can do is look at our own ticket queues.

  • AI Researcher Phishing Impersonation: China-Aligned TA419 Spoofed a Former US Official

    AI Researcher Phishing Impersonation: China-Aligned TA419 Spoofed a Former US Official

    Picture this message landing in a researcher’s inbox. It would look like it came from someone whose name everyone in the AI policy world already knows, inviting them onto a shiny new advisory committee. The subject would feel urgent, not spammy. If the mail gate treated that famous name as VIP traffic instead of a lookalike, the reply would open the door to the real payload. That’s the AI researcher phishing impersonation pattern Proofpoint just documented, and it’s exactly the kind of email security problem operators keep underestimating.

    The attack didn’t need a zero-day. It needed a recognizable name, a believable pretext, and a mailbox that still trusts famous people.

    What Proofpoint Actually Reported

    On October 1, 2026, Proofpoint published research on a group it tracks as TA419. Proofpoint calls TA419 a China-aligned, espionage-motivated threat actor. That is the vendor’s attribution language. The report does not publish a numeric confidence score, does not name a Chinese government agency or contractor, and does not claim a courtroom-grade linkage. I’m sticking to that wording rather than the broader “Chinese hackers” label some headlines used.

    Proofpoint says it has watched TA419 run targeted credential phishing against people at US- and Japan-based think tanks, defense contractors, universities, and law firms since at least April 2025, and that the group’s activity had not been reported publicly before.

    The campaigns that match this post’s topic are narrower than “AI experts” in general:

    • July 2026, starting July 8: TA419 impersonated Lynne Edwards Parker, the former Principal Deputy Director of the White House Office of Science and Technology Policy, and then Heidi Crebo-Rediker, an economist and foreign policy expert. The targets were AI policy experts at US think tanks, universities, and law firms.
    • February 2026: the same group impersonated a senior Anthropic employee, whom Proofpoint does not name, to reach an AI policy analyst at a US think tank. The subject line referenced military use of Claude.

    Proofpoint names Parker and Crebo-Rediker in the primary report, so those names appear here. It does not name the Anthropic employee, and it declines to name the targets. Reuters later identified one target independently; I’m not repeating that name. Stick to what the vendor published.

    How the AI Researcher Phishing Impersonation and Email Security Chain Worked

    The sequence is worth walking for anyone who runs mail or identity:

    1. A benign opener. Invitations to join a fictitious “AI Policy Advisory Committee,” or to contribute to a supposed Senate Foreign Relations Committee report on AI export controls and supply chains. No malware. Just rapport.
    2. If the target replies, a shortened URL that claims to share more information.
    3. A multi-stage redirect through actor-controlled domains, including a Cloudflare Turnstile check behind a fake OneDrive loading screen.
    4. An adversary-in-the-middle (AitM) page built on a customized Frameless BitB kit. The victim sees what looks like a real Microsoft sign-in. The proxy relays the login to genuine Microsoft infrastructure, so password, MFA, and conditional access can all succeed while the attacker captures the session cookies.

    Proofpoint says both July campaigns used the same first-stage domain and the same second-stage AitM domain. Sender addresses were lookalike webmail accounts, not the real people’s corporate mailboxes. Nextgov/FCW’s write-up of the same report is consistent on the names, dates, and Microsoft-focused AitM chain.

    Reuters, summarizing Proofpoint, said the recent campaign involved fewer than ten individuals at a handful of organizations, and that Proofpoint’s view of the targeting suggested an intelligence interest in US policymaking rather than technology theft alone. That “fewer than ten” figure is Reuters citing Proofpoint; it is not in the public Proofpoint blog post itself, so treat it carefully. Reuters’ own page returned a bot block to my link check, so I am citing it by name only and linking the sources that did return HTTP 200.

    One more attribution note, because the headline asked for it. Proofpoint told Reuters it based the China alignment on malware types, infrastructure, and targeting priorities that line up with Chinese intelligence collection. The published Proofpoint blog leans harder on targeting history, Cloudflare-fronted file-share domains, NameSilo registrations, and a recurring self-signed TLS certificate cluster than on a named malware family. Either way, the confidence language available to the public is “China-aligned” plus “likely supports wider Chinese intelligence objectives,” not a scored attribution to a specific PLA or MSS unit.

    What This Means for Email Security Operators

    Everything below is my ops opinion, not Proofpoint’s checklist.

    Spam filters are not the control that failed here. The lure was a conversation that looked like work. The payload was a session-stealing login that can survive ordinary MFA. If your mail gate still boosts messages that look like they come from famous people, or if your SOC treats “VIP sender” as a reason to lower scrutiny, you are playing into the same habit this campaign abused.

    • Treat VIP lookalikes as hostile until proven. Display-name matches on free webmail, or near-miss addresses that echo a public figure, should raise priority, not lower it.
    • Verify unexpected outreach out-of-band. Proofpoint’s own recommendation is the right one: confirm unsolicited subject-matter invitations through another independent channel before you reply, click, or join anything.
    • Assume AitM against Microsoft 365. Passkeys and other phishing-resistant, origin-bound authentication matter here more than another banner warning. If cookie theft is the goal, push MFA alone is not enough.
    • Watch the reply, not just the first message. The dangerous link arrived after the target engaged. Thread-aware detections and reply-path URL checks beat first-message-only rules.
    • Inventory who your researchers talk to. Policy shops, universities, and outside counsel in the AI regulation lane are soft targets for this pretext. Know which mailboxes sit at that intersection.

    Proofpoint assesses that TA419 will likely keep targeting think tanks and policy experts, and will keep spoofing real subject-matter experts. I take that as a vendor forecast, not a guarantee. The durable lesson for operators is smaller and meaner: famous names are not authentication, and a mailbox that still treats them as urgent is doing the attacker’s first hop for free.

  • California AI Workplace Laws Are an IT Compliance Problem Too: No Logging Mandate, but a Real Burden of Proof

    California AI Workplace Laws Are an IT Compliance Problem Too: No Logging Mandate, but a Real Burden of Proof

    Picture this email landing in the IT queue in late 2027. Legal would want to know which tool flagged an employee for termination, what data it used, and who signed off on the result, and they’d want it by Friday. If the only answer were “the vendor dashboard, I think,” that would be a bad week. That’s the angle I care about with the new California AI workplace laws IT compliance teams are about to inherit. The laws are written for employers and HR. The evidence that an employer followed them, though, is going to live in systems ops runs.

    Before anyone panics, let’s look at what Governor Newsom actually signed, and what the text does and doesn’t require. Quick disclaimer: I’m a sysadmin, not a lawyer, and this isn’t legal advice. Talk to counsel about your obligations.

    What Got Signed, and What Got Vetoed

    On September 30, the Governor’s office announced a batch of AI bills. Four of them deal with AI in the workplace:

    • SB 947 (McNerney), automated decision systems in employment, Chapter 859, operative July 1, 2027.
    • SB 951 (Reyes), AI-driven layoff notices under the state’s Cal/WARN Act, Chapter 860.
    • AB 1331 (Elhawary), workplace surveillance in bathrooms, Chapter 851.
    • AB 1883 (Bryan), AI-powered workplace surveillance tools, Chapter 853.

    The same day, Newsom vetoed two related bills, which are now marked vetoed on the Legislature’s site. KQED’s rundown describes them. AB 2656 would have required notice to public employees when AI performs work within the scope of their jobs. AB 2575 would have protected health care workers who override AI recommendations they judge unsafe.

    Only SB 947 carries its own start date in the text. The other three don’t set one, so the usual January 1 start for new California statutes would apply, but confirm that with counsel.

    What the Text Actually Says About California AI Workplace Laws and IT Compliance

    I read the chaptered text on the Legislature’s site so you don’t have to. The meatiest bill for us is SB 947, which adds Labor Code sections 1520 through 1526.7. The highlights:

    • No robo-firing. Section 1522(b) says an employer “shall not rely solely on an ADS when making a disciplinary or termination decision.” If an employer primarily relies on ADS output, a human must corroborate the decision using the data behind the output or other supporting information, such as supervisor evaluations, personnel files, work product, peer reviews or witness interviews.
    • Bad output gets tossed. If the output can’t be corroborated, or the reviewer finds it “inaccurate, incomplete, or misleading,” the employer can’t use it for the decision.
    • A data description on request. Under 1522(d), employees can request “a meaningful, objective description of the employee’s own data used by the ADS,” with other people’s personal information anonymized.
    • A post-use notice. Section 1524 requires a written, plain-language, stand-alone notice at the time of the decision. It must say an ADS was primarily relied on, that a human reviewed and corroborated the output, who to contact, and that retaliation is prohibited.
    • The burden of proof shifts. This is the one ops should circle. Under 1526.1(c), once it’s shown an ADS was used, “the employer must demonstrate” that it didn’t primarily rely on the ADS or that it complied with the corroboration and notice rules. Penalties are $500 per violation.

    One definitional detail made me smile. SB 947’s definition of an ADS explicitly excludes spam filters, firewalls, antivirus software, and “identity and access management tools.” Your IAM stack isn’t the regulated thing. It just might be where the evidence lives.

    The other three bills are narrower:

    • AB 1883 bars using a workplace surveillance tool “that uses artificial intelligence” to recognize or infer an employee’s emotional state, or to collect neural data. Using surveillance tools for safety is still allowed.
    • AB 1331 bans surveillance tools in workplace bathrooms unless a court orders it, and lets employees leave tools behind when they go in. Badges required for ID or safety, or to open a locked bathroom, are allowed if they don’t record audio or video and have no embedded AI.
    • SB 951 requires layoff notices caused “in whole or in substantial part” by AI to include the number and occupations affected, the job functions being automated, and “the specific category or type” of AI system involved. The notice must be labeled “This notice is for a technology displacement.”

    What the Text Doesn’t Say

    Here’s where I have to push back on the original framing of this post. None of these four bills contains an explicit logging mandate, an access-log requirement, an audit-trail standard or a record-retention period. “Ops will own the logs” isn’t in the statute. What the statute does is make the employer prove corroboration and notice, and hand over a description of employee data on request. So the logging duty comes from those obligations, not from any line in the text.

    What I’d Do Anyway (My Opinion, Not the Law)

    Everything below is my own ops recommendation. None of it is required by the text. If SB 947’s burden of proof lands on your employer, I’d rather have the evidence ready than reconstruct it from vendor exports in a hurry.

    1. Inventory anything that scores people. Performance analytics, productivity tools, ticket-quality scoring, call-center QA. If it issues a score, classification or recommendation about employees, flag it for HR and legal review as a possible ADS.
    2. Keep the inputs. If a human has to corroborate using “data that was collected or used to produce the ADS output,” that data needs to still exist. Work with counsel on retention, and don’t let the vendor’s default 30-day purge decide for you.
    3. Log the human step. Record who reviewed an ADS-driven decision, when, and what they looked at. A ticket or workflow record beats an email thread.
    4. Audit your surveillance stack for AB 1883 features. Some monitoring and meeting tools advertise “sentiment” or emotion detection. Find out what’s turned on, and what’s on by default.
    5. Check badge and sensor coverage against AB 1331. Know which devices record audio or video and which have embedded AI, and where they work.
    6. Plan for the data-description request. Figure out now how you’d produce one employee’s data from an ADS while anonymizing everyone else’s.

    Is this HR theater? No. These laws carry real penalties and a real burden of proof. But they’re employment laws first. Ops doesn’t own compliance here. Ops owns the systems that will either prove compliance or quietly fail to. I’d pick the first one, and start the inventory well before July 2027.

    Not legal advice. Summaries are based on the chaptered bill text and official announcements as of publication.

  • Azure Maintenance Hybrid Cloud VPN Outage: A Gateway Change, OS Servicing and Nearly Six Hours of Degraded Connectivity

    Azure Maintenance Hybrid Cloud VPN Outage: A Gateway Change, OS Servicing and Nearly Six Hours of Degraded Connectivity

    Picture this runbook. Step one: if the on-prem side of the VPN drops, fail over to the backup tunnel. Step two: if ExpressRoute degrades, shift traffic to the VPN. Step three: check the Azure portal to confirm the gateway is healthy. It would read fine right up until a night when the gateways themselves failed to load in the portal and management operations started timing out. Every step would have assumed the cloud side stays up. That’s the uncomfortable lesson of Wednesday’s Azure maintenance hybrid cloud VPN outage, and it’s why I’ve spent the morning reading Microsoft’s incident report instead of doing anything fun.

    What Microsoft Says Happened in the Azure Maintenance Hybrid Cloud VPN Outage

    Microsoft’s Azure status history entry for tracking ID 7Q30-010 is a preliminary review, marked “Mitigated.” It says that between 20:30 UTC on September 30 and 02:15 UTC on October 1, which is 4:30 PM to 10:15 PM Eastern on Wednesday, “a subset of customers using gateway services in multiple regions experienced degraded or interrupted network connectivity.” That’s five hours and 45 minutes from start to mitigation.

    The services Microsoft lists are the hybrid networking stack:

    • Azure ExpressRoute Gateway
    • Azure Firewall
    • Azure Application Gateway and Web Application Firewall
    • Azure VPN Gateway
    • Azure VMware Solution

    Customers may have seen gateways failing to load in the Azure Portal, along with failures or delays in network management operations. If you run Azure VMware Solution or connect your data center over ExpressRoute or site-to-site VPN, that’s your whole bridge between two worlds.

    The Timeline, in Eastern Time

    • 4:30 PM (20:30 UTC): customer impact began.
    • 5:29 PM (21:29 UTC): Microsoft began investigating ExpressRoute Gateway connectivity issues in UK South.
    • 6:27 PM (22:27 UTC): Microsoft identified that multiple regions were affected.
    • 7:05 PM (23:05 UTC): Microsoft found a correlation with operating system servicing activity and paused further servicing.
    • 9:36 PM (01:36 UTC, October 1): recovery progressed across most regions, while configuration changes were applied to France Central, North Europe, Southeast Asia, UK South and UK West.
    • 10:15 PM (02:15 UTC): Microsoft confirmed mitigation.

    On regions, Microsoft’s review names only the five late-recovering regions, plus UK South as the place the investigation started. It also carries a note that impacted regions in a previous communication were corrected. The Register, reporting during the incident, counted 18 affected regions at the time of writing, across the US, Europe, Asia, Africa, the Middle East and Mexico. It said Azure VMware Solution wasn’t impacted in four of them. Treat 18 as a mid-incident snapshot, not a final count.

    Was It Really “Routine Maintenance” That Took Things Down?

    Partly, and the nuance matters. According to Microsoft, “a recent change to a regional gateway management service triggered a higher-than-expected load when an unrelated operating system servicing maintenance proceeded gradually through multiple regions.” Normally the gateway management service would autoscale. This time, demand on dependent services kept those regional services from scaling as expected. Microsoft paused the OS servicing and reverted the gateway manager change, which let services recover. It says it’s still investigating the scaling behavior and the safeguards needed to prevent a repeat.

    So the maintenance didn’t break things on its own. A change and a maintenance rollout collided, and the safety net, autoscaling, didn’t catch it. That’s also why I wouldn’t say Azure “took down” hybrid clouds. Microsoft’s own words are “degraded or interrupted” for “a subset of customers,” and The Register relayed a Microsoft update saying some affected VPN gateways saw reduced redundancy rather than a complete loss of connectivity. For some shops it was an outage. For others it was a night of losing half their safety margin without knowing it.

    A full Post Incident Review is still coming. Microsoft says it generally publishes one to impacted customers within 14 days, so some of this may get sharper.

    When the Change Window Becomes the Incident

    What follows is my opinion, not Microsoft’s. Here’s what bugs me as an operator. We write change plans assuming our change is the risky one. We schedule our maintenance, tell the business, prep rollback and lean on the cloud provider as the stable half of the equation. This incident flips that. The provider’s maintenance, combined with the provider’s own change, was the event. Your change calendar would have shown nothing at all.

    The detail that worries me most is the management plane. When gateways fail to load in the portal and network management operations fail or stall, every runbook step that starts with “go into the portal and…” is suddenly a coin flip. That includes resetting a gateway, updating a connection or flipping a route. The portal is the cloud side of your runbook, and it’s exactly what was impaired.

    What I’d Put on the List This Week

    1. Turn on Service Health alerts properly. Microsoft’s guide to Service Health alerts covers alerting on service issues and planned maintenance by subscription, service and region, through email, SMS, push, webhooks or ITSM. It recommends selecting all services and all regions, since alerts only fire for regions where you actually run things. Route them to the on-call phone, not a shared mailbox.
    2. Check your gateway redundancy, not just your tunnel count. Microsoft’s VPN Gateway high-availability guidance notes that every gateway is active-standby by default, and it describes active-active gateways with multiple on-prem devices for full-mesh tunnels. If “reduced redundancy” happened to you Wednesday, would you have known?
    3. Build at least one runbook path that doesn’t need the portal. Pre-staged CLI scripts, on-prem routing changes, and BGP preferences you control from your side of the wire.
    4. Ask your account team about maintenance communications. What notice do you get for infrastructure OS servicing in your regions, and how fast does Microsoft pause a rollout when health signals drop? Here, impact began at 4:30 PM ET and servicing was paused at 7:05 PM ET.
    5. Write the post-incident review for your side. Check your own logs against the 4:30 to 10:15 PM ET window. Microsoft notes its times cover the whole incident, and actual impact varies by customer and resource.

    None of this means hybrid is a bad idea. It means the cloud side of hybrid is a dependency with its own change windows, and it doesn’t always tell you when one starts. Plan for it the way you’d plan for a carrier circuit: assume it’ll have a bad night, and make sure you can still steer from your end when it does.

  • Niche AI Tools Are the New Shadow IT: The Cybersecurity Risk Infrastructure Operators Can’t Fully Inventory

    Niche AI Tools Are the New Shadow IT: The Cybersecurity Risk Infrastructure Operators Can’t Fully Inventory

    Picture this ticket landing at 4:45 on a Friday. An API key nobody recognizes would show up in a production config. It wouldn’t be from the cloud provider or any vendor in the asset register. After an hour of digging, someone might work out that it belongs to a little AI transcription service a project lead signed up for with a corporate card three months earlier, one that had been wired into a workflow and quietly fed meeting notes ever since. That’s a hypothetical, but it captures the niche AI tools cybersecurity risk infrastructure operators keep underestimating. The problem isn’t the big-name chatbot everyone argued about in the policy meeting. It’s the five hundred little ones nobody argued about at all.

    I wanted actual numbers before writing this, not vibes. Three 2026 reports line up uncomfortably well.

    The Long Tail: Why Niche AI Tools Are a Cybersecurity Risk for Infrastructure Operators

    The most useful of the three is the 2026 AI Agent Risk & Action Report from Nightfall AI, released this spring. It’s built on anonymized, aggregated data from Nightfall’s own customers, so the numbers are a vendor’s telemetry, not a random sample. But the numbers are hard to ignore:

    • More than 35,000 distinct applications discovered across its customer base, including 519 GenAI tools and 103 AI agent tools.
    • The median organization runs 182 applications and 11 AI tools. The largest runs over 8,000 applications with more than 170 AI tools.
    • 81% of GenAI tool adoption happens outside the top three providers, OpenAI, Anthropic and Google.
    • 98% of organizations use GenAI, and 49% are using or experimenting with AI agents.

    The report’s own description of the long tail could be the subtitle of this post: “For every OpenAI or Anthropic deployment that security teams are aware of, there are dozens of niche AI tools — translation, content generation, code assistance, design, research — operating with limited visibility.” Most of those tools are used by only a handful of organizations, which, as the report puts it, makes them “harder to detect, evaluate, and govern.”

    Its recommendation is one I’d frame and hang in the NOC: “Assume your inventory is incomplete. The long tail moves too fast for manual tracking.”

    Shadow AI Isn’t Going Away

    For years, the plan at most shops was to buy an approved AI platform, move everyone onto it, and watch shadow usage dry up. The Netskope AI Report: 2026, based on aggregate data from a subset of Netskope customers from June 2025 to July 2026, says that plan has stalled.

    According to Netskope, the shift from personal AI apps to organization-managed ones “stopped around March 2026, plateaued, and has trended back slightly in the opposite direction.” Its current breakdown of AI users:

    • 56% use only organization-managed AI apps.
    • 14% use both managed and personal apps.
    • 30% use only personal apps.

    Netskope’s read is that “shadow AI is here to stay.” It adds that discovery priorities have expanded to include AI agents, MCP servers and local AI infrastructure. It also reports that remote MCP users rose 250% and transactions 375% over a recent 10-week span, and that 41% of organizations are running Ollama to host models locally.

    That last pair is the part infrastructure folks should circle. A personal chatbot tab is a data-loss problem. A locally hosted model or an MCP connector tied into internal systems is an asset, running on your hardware or reaching into your data, that may never have gone through change management.

    The Humans Behind the Long Tail

    Telemetry tells you what’s running. Surveys tell you why. The PagerDuty Shadow AI Survey, published June 11 and conducted by Wakefield Research among 1,250 office professionals outside IT and technology roles at companies with at least $500 million in annual revenue across the US, UK, Australia and Japan, found:

    • 66% have used AI tools at work even though they believed doing so wasn’t permitted under company policy.
    • 88% have shared work-related information with public AI tools, including 34% who have input customer data.
    • 89% of those who’ve used AI for work first encountered the tool in their personal lives.
    • 72% believe they know AI better than their own tech teams.

    That last number stings, but it explains the pattern. People find a clever niche tool at home, decide it’s better than what IT approved, and bring it to work. No ticket gets filed, because in their minds there’s nothing to ask about.

    Why This Is the New Shadow IT, and Why It’s Worse

    This is my opinion, not a finding from any of these reports, but I’ve seen this movie. Shadow IT used to mean a rogue Dropbox account or a marketing team’s unapproved project tracker. You found it through expense reports, DNS logs or a firewall rule someone requested in a panic.

    The AI version is harder to see for three reasons the data supports:

    1. The tail is enormous and constantly churning. Some 519 GenAI tools in one vendor’s dataset, with most of the usage outside the big three. No approval board reviews that many products a year.
    2. A lot of the AI isn’t a separate app. Nightfall calls it “Invisible AI”: features inside tools you already approved, like Google Workspace Gemini, which it found in 73% of organizations. Your inventory says “Google Workspace,” not “a model reading our docs.”
    3. The tools connect to things. MCP servers and agents don’t just receive pasted text. They reach into CRMs, repos and databases. Nightfall’s line is that AI risk “isn’t about the tools — it’s about what they can reach.”

    What I’d Do Monday Morning

    • Start with egress, not policy. Proxy, DNS and CASB logs will show AI domains no matter what the acceptable-use policy says. Sort by volume and look at what’s below the big three.
    • Hunt for keys. Scan repos, CI variables and config stores for API keys to AI services you don’t have contracts with. That’s where the opening hypothetical turns real.
    • Inventory the embedded AI too. Review the AI features in your major SaaS platforms and decide which ones should be on.
    • Treat MCP servers and local model hosts like servers. They get owners, patching and network rules, just like anything else on the network.
    • Give people a fast yes. If 66% of people use tools they think are banned, a slow approval process just pushes the problem underground.

    You can’t secure what you haven’t found, and with AI tools the finding is the hard part. The good news is that the clues are already in logs we collect. Someone just has to go read them before the Friday ticket shows up.

  • Anthropic Akamai Cloud Deal: $11.6B of AI Infrastructure, and It’s CPUs, Not a CDN

    Anthropic Akamai Cloud Deal: $11.6B of AI Infrastructure, and It’s CPUs, Not a CDN

    Back when I was racking servers for a living, “the CDN” was the box you checked so the marketing site stayed up on launch day. Nobody asked what was inside it. That’s why the Anthropic Akamai cloud deal caught my eye this week: $11.6 billion of AI infrastructure committed to the company most of us still think of as the people who cache our JPEGs. The tempting headline is that AI labs just became the biggest CDN customers on Earth. The paperwork says something more specific, and frankly more interesting.

    What the Anthropic Akamai Cloud Deal Actually Covers

    Start with the primary source. In its September 24 press release, Akamai announced “$11.6 billion of contractual commitment over seven years” from Anthropic, to “support Anthropic’s accelerating CPU workload demands by leveraging Akamai Cloud’s distributed AI infrastructure and software.”

    Note what’s missing from that sentence: content delivery. Akamai’s Form 8-K filing goes further. It says two new project plans, signed September 18 under a master services agreement dated May 5, cover “dedicated cloud computing capacity and related managed support services.” Each plan runs for an initial seven years from its own service start date.

    The key numbers:

    • $11.6 billion committed over seven years. TechCrunch, citing Bloomberg, reports it’s the largest contract in Akamai’s history.
    • Up to $9 billion more if the relationship expands, for a potential total of about $20 billion.
    • About $5.5 billion in capital spending to build the capacity, plus roughly $1.7 billion added to 2026 capex to pre-buy components including memory.
    • A warrant for up to about 5% of Akamai’s stock, at $111.33 per common-share equivalent. About 2% is tied to the initial commitment, and roughly 1% more vests with each additional $3 billion Anthropic commits.

    So the $11.6 billion is real, but it’s a compute contract, not a CDN bill.

    The Fine Print Operators Will Recognize

    The filing has a line any of us who’ve signed a hosting contract will appreciate. Anthropic’s commitment is “subject to any termination described below and satisfaction of certain delivery and service availability requirements.” TechCrunch put it plainly: the commitment “isn’t ironclad.” Akamai has to deliver the capacity and keep it available, and either side can end the agreement under certain conditions. The 8-K also lets Anthropic terminate a project plan after a material outage, subject to conditions.

    That’s an SLA with real teeth. If you’ve ever explained to a CFO why a 99.9% target isn’t the same as 99.99%, you know the difference between “committed” and “paid” lives in clauses like that.

    Timing matters too. TechCrunch reports Akamai won’t see revenue from the deal this year. Executives told investors they expect $150 million to $300 million in 2027, starting in the second half, and an annual pace of about $1.7 billion by the end of 2028. That’s a long runway of building before billing.

    Why CPUs, and Why Akamai?

    Here’s the part I think gets lost. The AI infrastructure conversation is usually about GPUs, giant training clusters and power plants with a data center attached. This deal is about CPUs, the plain general-purpose chips. TechCrunch notes CPU demand has grown as AI agents take on more tasks like running code and browsing the web. It also notes Akamai didn’t say what Anthropic will use them for.

    That gap is where the CDN framing creeps in. In The Boston Globe’s take, IDC’s Ashish Nadkarni thinks Anthropic will use Akamai’s network to get closer to its customers: “It’s kind of similar to what you do with streaming music today, where the edge network essentially speeds up the delivery of the data.” The Globe also reports that Akamai expects the new equipment to draw at most an additional 110 megawatts, spread across dozens of data centers where it already leases space.

    That’s an analyst’s informed guess, not a disclosed use case. But it lines up with Akamai’s pitch about “a continuum of compute from core to edge” spread across thousands of points of presence. It also lines up with the hook I can’t shake: model traffic is starting to look like a content-delivery problem with an inference bill attached.

    So Is “Biggest CDN Customer on Earth” True?

    No, or at least nobody has shown it. None of the sources I read rank Anthropic, or AI labs generally, as the largest CDN customers anywhere. The deal isn’t described as CDN services at all. What the sources support is narrower:

    • It’s the largest contract in Akamai’s history, per Bloomberg as reported by TechCrunch, and the biggest single contract Akamai has ever won, per The Boston Globe.
    • It’s for dedicated cloud computing capacity serving CPU workloads, not content delivery.
    • How Anthropic will use it, edge inference or something else, hasn’t been disclosed.

    That’s why this post isn’t wearing the original headline.

    What This Means If You Run Infrastructure

    Picture a team that would have tried to rent a few hundred CPU nodes for an internal agent platform next spring. In that hypothetical, they might find the friendly regional cloud quoting longer lead times. They wouldn’t have done anything wrong. A buyer with $11.6 billion and a seven-year horizon would simply have gotten in line first. Akamai pre-buying memory with $1.7 billion of this year’s capex is a hint about where supply chains are tight.

    A few practical takeaways:

    1. Don’t assume AI capacity means GPUs. Agent workloads spend a lot of time on ordinary compute: calling tools, parsing pages, running code. Plan and budget for both.
    2. Read the availability clauses. If Anthropic negotiated an outage termination right, you can ask your providers what happens to your commitment when they miss theirs.
    3. Watch vendor concentration in both directions. Akamai now has one customer with a potential equity stake and a contract that’s big relative to the company. That’s a dependency worth understanding if you rely on Akamai.
    4. Treat edge inference as a design option, not a given. Running smaller models closer to users is plausible. Until someone discloses it, it’s still a hypothesis.

    The CDN didn’t become an AI company overnight. A company that spent decades putting servers close to users just sold a lot of that closeness, and a lot of new CPUs, to a customer that needs both. I’d call that the most sysadmin story of the AI boom so far: the winning move was boring infrastructure, in the right places, with an SLA.

  • Meta Muse AI Agent Home Address Privacy Leak: One ‘Allow Always’ Click, No Human Review

    Meta Muse AI Agent Home Address Privacy Leak: One ‘Allow Always’ Click, No Human Review

    Picture a new intern on day three. You hand them the shared inbox, a sticky note with your pickup details, and a breezy “just handle it.” If that intern then pasted the sticky note into every reply to every stranger who asked, nobody would call it a breach. They’d call it onboarding that went sideways, and somebody’s manager would get a very awkward meeting invite. That hypothetical intern is the best lens I’ve found for the Meta Muse AI agent home address privacy leak that went viral this week. The twist is that the intern here has no manager, and nobody is quite sure which ticket queue the fallout lands in.

    What Actually Happened in the Meta Muse AI Agent Home Address Privacy Leak

    Muse is Meta’s personal AI agent, launched earlier this month with a lot of emphasis on security. Over the weekend of September 26 and 27, a tech YouTuber posted that Muse, which he had authorized to run his Facebook Marketplace listing for a keyboard, had given his home address to a buyer, accepted a lowball offer, and let the buyer show up, all without telling him until late that night. The Verge’s write-up quotes the Muse-generated summary he shared, and it’s a remarkable piece of machine self-awareness: “You never explicitly instructed me to share the address with buyers — and I never asked you for consent to do so.”

    The key detail is where the address came from. Muse didn’t dig it up from somewhere. According to the user’s own follow-up, the agent had asked him for setup information, including pickup address, pickup windows and accepted payment types, plus a tone instruction to be “short, casual, and human.” He gave it all of that. Then Muse offered a choice between “Allow One Time” and “Allow Always.”

    He picked Allow Always, assuming the agent would still come back for approval before accepting an offer. As Business Insider reported, that setting let Muse send messages on his behalf using a template that contained the pickup address. “I didn’t think it would send it out to everyone that gave me an offer,” he wrote. “It’s worth checking.”

    Is It Doxxing, a Leak, or a Permissions Bug?

    This is where I’m going to be the annoying sysadmin in the room. “Doxxing” usually means deliberately digging up and publishing someone’s personal information to expose or harm them. That isn’t what the reporting describes. The address was information the user supplied for pickups, and it went to people responding to his Marketplace listing, not out to the public. Meta’s position, per a reply from David Singleton of Meta Superintelligence Labs quoted by Business Insider, is that there was “no breach of privacy controls.”

    That’s technically defensible and practically unsatisfying. The controls did what they were configured to do. The problem is that the person configuring them didn’t understand what “Always” covered, and the agent didn’t treat a home address as sensitive enough to ask about first. The Verge put it well: it’s an oversight if Muse didn’t register that a home address shouldn’t be handed out without express permission.

    A few more details round it out:

    • The price. The user had set a $700 minimum, and a $600 offer appeared to be accepted. Meta told him a separate display error had dropped the “7,” so the buyer saw “Sounds good, 00 it is!” Singleton said Meta fixed that issue.
    • The fix. The user said Meta’s Muse team reviewed the logs with him and plans to make the permission prompt clearer. PCMag’s updated story notes he also suggested a “Sent By Muse” badge on agent-written messages, though it’s unclear if Meta will do that.
    • The ending. He and the buyer patched things up, and the buyer later came back to buy something else. His advice: “double-check what permissions AI has in your life.”

    So I’d call it a privacy leak caused by a permission scope nobody fully read, which is the most normal IT failure there is. What’s new is who was holding the permission.

    Which Ticket Owns Real-World Harm?

    In a company, when an intern forwards the wrong thing, there’s a chain. The intern has a manager, the manager has a process, and the process usually has a form. With a personal agent, that chain collapses into one person and one button. The user is the requester, the approver, the change manager and the incident responder, and he found out about the incident from the agent’s apology.

    That’s the part operators should chew on, because the same shape shows up the minute we hand agents shared mailboxes, ticket queues or customer chat. A few observations from the reporting:

    • “Always” is a scope, not a vibe. The user read it as “always allowed to help.” The system treated it as “always allowed to send.” Those are very different change requests.
    • Data given for one purpose got used for another. The address was context for Muse. The template turned it into outbound content. No data-classification step sat in between.
    • Recipients couldn’t tell who was talking. The user said there was “really no way of knowing who sent what in the chat,” which is why he asked for a label.
    • The alert came after the impact. Muse told him it was “bad news” late in the evening, after the buyer had come and gone.

    What I’d Put in the Runbook

    1. Give agents a deny list, not just instructions. Home addresses, phone numbers and account details should need explicit, per-message approval, whatever the global setting says.
    2. Default to “Allow One Time” until you’ve watched the agent work. Treat the first week like a probation period, because it is one.
    3. Separate “draft” from “send.” An agent that drafts replies for review is a productivity tool. An agent that sends on its own is an employee, and it needs the oversight an employee would get.
    4. Label agent output. If people outside your organization can’t tell a bot from a human, you’ve created a social engineering problem for yourself.
    5. Know where the escalation goes. Before an agent touches anything customer-facing, decide who gets paged when it does something weird. If the answer is “whoever notices,” that’s not an answer.

    None of this makes Muse uniquely bad. It is, as The Verge notes, the latest of several security concerns raised about the product since launch, and that’s worth watching. But the core failure is one I’ve seen in plenty of service accounts: broad permissions granted in a hurry, sensitive data sitting in a template, and no human in the loop until the damage is done. The agent even offered to fix its own behavior afterward. I’ve had interns do less.

  • Google Cloud API Gateway MCP Tools for REST APIs: A Handy Preview That Turns Opted-In Endpoints Into Agent Doors

    Google Cloud API Gateway MCP Tools for REST APIs: A Handy Preview That Turns Opted-In Endpoints Into Agent Doors

    Picture this ticket. Somebody’s shiny new assistant “finds a way” to reset a user’s MFA, and if you traced it back, you might find it went through something like /internal/v2/admin/resetFactor, an endpoint documented in one wiki page from 2021 and a Slack thread nobody can find. In that scenario, the assistant wouldn’t have hacked anything. It would have read a description, decided the endpoint was a skill it had, and used it. That’s the scenario I kept imagining while reading about the new Google Cloud API Gateway MCP tools for REST APIs, which let a gateway you already run hand your API operations to AI agents as callable tools.

    The feature is useful, and the fine print matters more than the headline. Let’s go through what Google actually shipped and which doors it opens.

    What Google Actually Announced

    On September 24, Google’s developer blog published Turn your REST APIs into MCP tools with Google Cloud API Gateway. The short version: in Public Preview, API Gateway can now act as a remote Model Context Protocol (MCP) server. You annotate the OpenAPI spec you already deploy, redeploy, and the gateway starts serving MCP on a /mcp path. Agents built with frameworks like Google’s Agent Development Kit can then discover your operations and call them. You don’t have to build or host a separate MCP server.

    A few specifics worth pinning down:

    • It’s a preview, not GA. The MCP overview in Google’s documentation marks the feature as subject to the Pre-GA Offerings Terms, available “as is” with possibly limited support. The API Gateway release notes list it as a Public Preview feature dated September 11, about two weeks before the blog post.
    • It’s opt-in. Nothing happens to your APIs until you add the x-google-api-management MCP setting to your spec. Google didn’t flip a switch on anybody’s endpoints.
    • OpenAPI 3.x only. Specs still on OpenAPI 2.0 aren’t supported, so some shops will have a migration to do first.
    • Tools only. The gateway supports initialize, tools/list and tools/call. MCP resources and prompts aren’t supported, and neither are stdio transport or streaming and long-running calls. Google says resources, prompts, response streaming and Model Armor payload inspection are on the roadmap.
    • Some limits. Operations that return empty bodies, like HTTP 204, aren’t exposed. A gateway serves up to 1,000 tools. MCP and model routing can’t be enabled in the same API config.

    So the accurate version of the hot take is narrower: if you run Google Cloud API Gateway and choose to turn this on, your REST operations can become agent tools with a few lines of YAML. That’s still a big deal. It just isn’t every endpoint on the internet.

    How Google Cloud API Gateway MCP Tools for REST APIs Handle Auth

    This is the part I actually care about, and Google’s design is mostly sensible. According to the blog, the gateway transcodes each tool call into the matching REST request, and it’s “indistinguishable from a normal REST call.” That means the JWT or API key checks, quotas and logging you already configured still apply, and MCP and REST traffic share one policy path.

    The configuration guide lays out the authentication model by method:

    • initialize and the handshake acknowledgment are unauthenticated.
    • tools/call reuses whatever authentication the underlying operation requires. If the REST endpoint wants a JWT, so does the tool.
    • tools/list, the method that tells a client what tools exist, is unauthenticated by default. Google strongly recommends locking it down, and if you do, it has to be a JWT scheme. API keys aren’t supported for discovery in the preview.

    Google’s own blog says the quiet part plainly: an open tools/list “publishes your tool names and input schemas to anyone who asks.” Tool names and schemas are a map. Even if every call is authenticated, you’ve handed out a labeled floor plan of the building.

    The Global Toggle Is the Real Door

    Here’s where the “every endpoint” worry has some truth to it. You can enable MCP two ways. Per operation, you mark the specific paths you want exposed. Globally, with mcp: true, the docs say all eligible operations are exposed as tools. Eligible means GET, POST, PUT, PATCH and DELETE. By default the tool name is the operationId and the description comes from the operation’s description or summary.

    One gotcha deserves bold type. Configuring the mcp setting as an object, which is how you secure tools/list, also enables MCP globally. Google’s blog flags it right in a code comment. So the admin who does the responsible thing and requires a JWT for discovery has also, without trying, published every eligible operation in the spec. You then opt individual operations back out with x-google-mcp-tool: false. That’s default-open behavior hiding inside a security setting, and it’s exactly how the half-documented admin endpoint ends up in an agent’s toolbox.

    The docs’ own example is a DELETE operation described as “Permanently delete a shelf and every book on it.” Sure, it’s a bookstore demo. But swap “shelf” for “tenant” and you have the ticket I opened with.

    Descriptions Are Now Security Controls

    Google notes that a tool’s description “is the primary signal an LLM uses to decide when to call it.” That changes the job of your OpenAPI summaries. For years, “Deletes stuff (internal, do not use)” was a note for humans. An agent may read that as a capability with a mild warning. If an operation shouldn’t be called by an agent, don’t rely on the description to say so. Keep it out of the tool list.

    What I Would Do Before Turning It On

    1. Audit the spec first. Every operation in that OpenAPI file is a candidate tool. Find the leftovers: admin routes, debug endpoints, anything “temporary.”
    2. Prefer per-operation opt-in over global. If you must use the object form to secure discovery, add explicit opt-outs and review the resulting tools/list output.
    3. Require a JWT on tools/list in anything that isn’t a sandbox. Unauthenticated discovery is fine for a demo and a gift for recon.
    4. Separate agent credentials from human ones. Since the backend can’t tell MCP calls from REST calls, give agents their own identities so the logs can.
    5. Treat destructive methods as their own review. PUT, PATCH and DELETE exposed to an agent deserve the same scrutiny as handing a junior admin the keys.

    I like this feature. Rebuilding auth and quota logic in a separate MCP server was always going to produce worse security than the gateway you already hardened. But convenience moves the risk to your spec file. The gateway will faithfully publish what you describe, so make sure you’re describing only the doors you meant to install.