Steve Miller's Blog

A digital workshop for systems-minded tech commentary.

Blog

Systems-minded notes on technology, everyday life, and the occasional global outage — from Steve’s digital workshop.

  • Microsoft Says Attackers Abuse GSocket for Linux Backdoor Persistence That Survives Patching: Hunting for the Access That Outlives the Fix

    Microsoft Says Attackers Abuse GSocket for Linux Backdoor Persistence That Survives Patching: Hunting for the Access That Outlives the Fix

    Patching the hole an attacker came through is not the same as evicting the attacker. Microsoft’s latest threat research puts a sharp point on that: attackers are abusing GSocket, a legitimate open-source connectivity tool, to build Linux backdoor persistence that keeps working after the web application flaw behind the break-in is fixed. If you close an incident when the patch ships, the report suggests you may be closing it early.

    What GSocket Is, and What Microsoft Reported

    GSocket is the Global Socket Toolkit. Its project page describes it as a way for two users behind NAT or a firewall to establish a TCP connection with each other, using the Global Socket Relay Network, a free cloud relay, and a shared secret in place of IP addresses. Its gs-netcat tool can give a remote shell. It is released under the BSD-2-Clause license. The tool is not malware by nature, though Defender flags it as a “GsNetcat” hacktool. What matters here is how it is used.

    In a report published November 11, 2026, Microsoft Security Research authors Yash Gund and Sumith Maniath said Microsoft observed multiple incidents in which threat actors deployed GSocket on compromised Linux servers “to maintain access even after the vulnerabilities used for initial compromise were remediated.” The report does not name a threat actor. It says “multiple unrelated threat actors have adopted it,” and it names no vulnerability used for initial access beyond saying internet-facing web applications were exploited.

    Why a Patch Does Not Remove It

    Microsoft’s explanation is brief: “Because GSocket runs independently of the exploited application, patching the vulnerability does not remove the relay client or its persistence.” The web exploit gives command execution only while the hole stays open. The relay client does not need the hole.

    Microsoft describes three properties that make it attractive to attackers. Traffic passes through a shared public relay, so there is no attacker-owned IP address or domain to block. Both the server and the operator connect outbound, so inbound firewall rules never see the session. And the deployment script can rename the process, stage files in memory-backed directories, and install persistence that survives a reboot.

    What the Linux Backdoor Persistence Looks Like

    In the observed intrusions, attackers used curl to download and run a script, which profiled the host, picked a client for the architecture, and staged it in writable or memory-backed paths such as /dev/shm. Microsoft says attackers then disguised the process with kernel-thread-like names (one example is [kaluad]) and persisted through “cron, systemd, shell profiles, watchdogs, and redundant copies.” They also removed staging files. Microsoft’s examples include a defunct.service unit, a renamed payload at /usr/bin/defunct, a binary called systemd-remount-d, and a hidden /dev/shm/.gs-0 directory.

    Reconnection works through a secret rather than an address. The deployment script typically generates it and, per Microsoft, the operator can read it from the console, retrieve it from a temporary file, or have it delivered through Telegram, Discord, or a webhook. With the secret, the operator can reconnect later from any compatible client.

    In a few incidents, Microsoft says, attackers who had execution as the web-service account tried to escalate to root using CVE-2026-43284, one of the Linux kernel flaws known as Dirty Frag. Microsoft says “attempted,” and it does not say every attempt worked. Ubuntu’s advisory says the Dirty Frag flaws were publicly disclosed on May 7, 2026, covers two CVEs (CVE-2026-43284 and CVE-2026-43500), and says kernel updates are available and need a reboot. Check your own distribution for both.

    What Microsoft’s Telemetry Shows

    This is vendor telemetry, from Microsoft Defender for Endpoint, and it reflects Microsoft’s customers. From March through August 2026, Microsoft says devices with GSocket-related detections rose 356.2%, “although the increase was not consistent month to month.” Web applications and hosting made up 87.1% of attributable incidents. Within that group, PHP web applications and cPanel hosting were 63.9% and Apache 24.3%. Education and research was the largest sector at 30.7%, followed by technology and telecommunications at 22.9%. Microsoft cautions that exposure is the more likely factor than deliberate targeting, noting that academic institutions often run large, decentralized web estates maintained outside central IT.

    Microsoft also notes limits. Not every stage occurred in every intrusion, and persistence across reboot and successful relay sessions were not confirmed in every case. The published indicators are specific observations, “rather than a complete inventory.”

    A Hunting and Cleanup Plan for Patched Servers

    Microsoft’s report includes Defender alert names, Advanced Hunting queries, and an IOC list of domains, file paths, and SHA-256 hashes. Its mitigation guidance tells defenders to remediate the host as well as the vulnerability: investigate and remove the client, its persistence, and staged files, “then verify that outbound connections to the relay infrastructure have stopped.” Using that guidance, here is how I would run it.

    1. Audit persistence first. Review cron entries, systemd units with plausible service names, shell startup files, and watchdog scripts. Microsoft notes persistence is often layered, so killing the process alone does not remove access. Remove the persistence before the process, or it comes back.
    2. Hunt by behavior. Look for shells launched by PHP, Java, or Node web processes, execution from hidden or world-writable paths, and processes whose names imitate kernel threads but do not belong to the kernel. Run Microsoft’s published queries if you use Defender XDR.
    3. Constrain egress. Microsoft recommends deny-by-default outbound rules and monitoring connections to relay infrastructure. A web server should not need to reach arbitrary hosts.
    4. Harden temp paths. Mount temporary and memory-backed directories non-executable where feasible, as Microsoft advises.
    5. Patch the local privilege escalation path. Prioritize kernel updates on internet-facing servers, including the Dirty Frag fixes, and schedule the reboots.
    6. Treat credentials as exposed. This one is my judgment, not Microsoft’s: if an attacker held a shell on the host, rotate the secrets the web application can read, such as database passwords and API keys.

    When a Cleaned Server Is Still Suspect

    Microsoft’s guidance is to remove the tool and verify. If you cannot establish whether the attacker reached root, or what else was changed, my view is that rebuilding from a known-good image is the safer path. That is an opinion, and the cost is real downtime. The report gives you the checks for the specific tool, and it cannot tell you what else an attacker with a working shell did.

    The lesson is a change in what counts as done. A closed vulnerability is the start of the cleanup, so add a host check to the end of every web-compromise ticket.

  • CISA Says Cisco BroadWorks Vulnerability CVE-2026-20320 Is Being Exploited Months After Its Fix: Bringing Voice and Telephony Platforms Into Your Patch Program

    CISA Says Cisco BroadWorks Vulnerability CVE-2026-20320 Is Being Exploited Months After Its Fix: Bringing Voice and Telephony Platforms Into Your Patch Program

    The phone system is usually the last platform anyone patches. It belongs to a different team, runs on a vendor’s release train, and has a service window nobody wants to schedule. That habit now has a federal deadline attached. On November 10, CISA added the Cisco BroadWorks vulnerability CVE-2026-20320 to its Known Exploited Vulnerabilities (KEV) catalog, nearly three months after Cisco published its advisory and fixed releases. If your company, or your service provider, runs voice or unified communications on BroadWorks, here is what is confirmed, what is only reported, and what belongs in your patch program.

    What the Cisco BroadWorks Vulnerability Allows

    Cisco’s security advisory was first published on August 19, 2026, and rates the flaw High, with a CVSS base score of 7.5. It is an XML external entity (XXE) problem in the Open Client Interface (OCI) XML parser. Cisco says XML entries are improperly parsed “due to external entity resolution being allowed by default,” and an unauthenticated, remote attacker could send a crafted XML message to the Open Client Interface – Provisioning (OCI-P) service and read sensitive files from the filesystem with the privileges of the BroadWorks user.

    Four components are affected when they run a vulnerable release, regardless of device configuration: the Application Delivery Platform, Application Server, Profile Server, and Xtended Services Platform. For all four, Cisco lists releases earlier than RI.2026.07 as affected, with RI.2026.07 as the first fixed release. Cisco says there are no workarounds. Upgrading is the remedy.

    What CISA Added and What It Requires

    The KEV catalog entry lists the vendor and product as Cisco BroadWorks, with a date added of 2026-11-10 and a due date of 2026-11-24, which is next Tuesday. The entry marks known ransomware campaign use as “Unknown.” CISA’s alert says the additions are “based on evidence of active exploitation” and the entry’s required action cites Binding Operational Directive 26-04.

    That directive binds Federal Civilian Executive Branch agencies only. Everyone else is not on the clock, but CISA’s alert says it “encourages all organizations to adopt risk-based vulnerability management and prioritize remediation of KEV Catalog vulnerabilities.” For a service provider or a company that hosts voice for customers, the federal date is a useful forcing function even without a legal requirement.

    Who Says It Is Being Exploited

    Here the sources differ, and the difference matters. CISA’s listing rests on its evidence of active exploitation, and it has not published details about attacks. Cisco’s advisory went through two revisions on this point. Version 1.1 on November 12 was “updated to reflect known active exploitation.” Version 1.2 on November 13 was “corrected to state that this vulnerability was added to the CISA KEV,” adding that “Cisco does not have direct evidence of active exploitation.” So the advisory now says the flaw is on CISA’s list and does not confirm attacks itself.

    A threat-intelligence firm has gone further. This is vendor research, not a CISA or Cisco finding, and I have not verified it. In a post dated November 8, Malbear Labs, writing about a campaign it calls Operation Snowball, reports that an attacker used an AI-driven intrusion harness against a European telecom provider and exploited CVE-2026-20320 on a BroadWorks Xtended Services Platform, Release 23.0.1.1075, on port 2208. Malbear says it recovered the platform’s system identity private key and other credentials from that host. Malbear does not say CISA relied on its research, and neither CISA nor Cisco has linked the two.

    The practical reading: treat it as exploited, because CISA says so, and treat the details of who and how as unsettled.

    Eighty-Three Days

    From the advisory’s August 19 publication to the November 10 KEV listing is 83 days by my count. Any system still on a release earlier than RI.2026.07 has had a public advisory describing the bug for about that long. Why do telephony platforms fall behind? This is my opinion, not something the sources say. The platform often belongs to a separate voice team. The upgrade cadence is set by the vendor and often by a carrier customer. A telephony outage is visible immediately to every employee. Meanwhile the vulnerability scanner that drives patching for servers may never have seen the box.

    The flaw type adds to the risk. This bug reads files. It does not run code or crash the service, so nothing breaks, and nobody notices that configuration data or keys left the building.

    Bringing Voice Platforms Into the Patch Program

    These steps are my recommendations, based on the advisory’s facts and not on any guidance from Cisco or CISA.

    1. Inventory the four components. Find every Application Delivery Platform, Application Server, Profile Server, and Xtended Services Platform you run or depend on, including those a hosting partner operates for you. Record the release of each.
    2. Compare against RI.2026.07. Anything earlier is listed as affected. Since Cisco says there are no workarounds, the question is when you upgrade, and the answer should be before November 24 if you want to meet the federal date.
    3. Check who can reach OCI-P. The advisory says the exploit travels through the OCI-P service. Confirm that only provisioning systems that need it can connect, and review how that service is exposed to the internet.
    4. Assume file read means credential exposure. If a vulnerable system was reachable before you patched, consider which keys, certificates, and credentials live on that host and whether they should be rotated. Malbear’s report of a stolen system identity key shows what that can include, but that is a vendor claim about one campaign.
    5. Look for the quiet signs. Neither the advisory nor the KEV entry publishes indicators of compromise. Review logs for unexpected connections to the OCI-P service, and for new outbound connections from BroadWorks hosts, from August 19 onward.
    6. Put telephony on the same dashboard. Add voice platforms to the asset list, ownership list, and KEV watch that covers your servers and network gear. A separate team can still run the platform, but someone should answer for it when a KEV date arrives.

    What to Take From It

    A KEV listing for a fixed bug is a statement about the gap between availability of a patch and its installation. The gap is longest on platforms that sit outside the normal patch cycle, and a phone system fits that description. If you cannot say today which BroadWorks release you are on, find out before Tuesday.

  • Google Cloud Traced Its London Outage to a Data Center PLC Misconfiguration: The Building Controls Under Your Cloud Are Part of Your Uptime Too

    Google Cloud Traced Its London Outage to a Data Center PLC Misconfiguration: The Building Controls Under Your Cloud Are Part of Your Uptime Too

    We like to treat a cloud region outage as a software problem with a software fix: a bad deploy, a config push, a control plane that fell over. Google’s incident report for the October 20 europe-west2 (London) event is a data center PLC misconfiguration outage in the plainest sense, and it is a useful reminder that the layer under your availability zone is a building full of programmable logic controllers (PLCs) that nobody on your team will ever patch or see. Google’s report names a PLC misconfiguration as the root cause, and that changes what a resilience plan should cover.

    Anatomy of a Data Center PLC Misconfiguration Outage

    According to Google Cloud’s incident report for europe-west2, posted in final form on November 4, 2026, at 22:53 Pacific on October 19 (06:53 BST on October 20), the utility’s primary feed A to the data center hosting zone europe-west2-b failed because of an external underground cable fault. Google says the site was already in a “temporary single-feed (1N) bridged maintenance posture”: the feed B breaker had been opened for repairs after issues found in a quarterly generator load test on October 14.

    Then the controls took over. Google says the investigation “identified the root cause as a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders.” The undervoltage sensing inputs were wired on the line side of the open feed B breaker, and feed B was still energized upstream. The two master PLCs “sensed live voltage” and concluded a healthy utility source was available. They never sent the generators a start command.

    The UPS batteries carried the load “as intended” (Google puts their runtime at approximately five minutes) until they were exhausted, and the data halls lost power. Impact began at 23:00 Pacific and lasted 6 hours and 8 minutes, across 40 Google Cloud services that the report lists as affected in the region. Google says it found no impact outside europe-west2.

    What It Was Not

    This was not a cooling failure, and the report does not point to a cyberattack. The trigger was a cable fault outside the building. The PLC logic and sensor wiring, plus a maintenance posture with one feed open, turned a survivable utility loss into a zone outage. The report does not say how long the misconfiguration had existed, and I will not guess.

    Redundancy That Shared the Same Mistake

    One detail stands out to me as an operator: the facility had a dual-redundant PLC pair, and Google says the active and standby masters ran identical control logic. Two copies of the same wrong rule give you availability against a failed controller and no protection against a wrong rule.

    The second detail is the change. A site that has dropped from 2N to 1N for repairs is running a different system than the one that was commissioned, and the controls logic has to be correct for that posture too. NIST’s SP 800-82 Rev. 3 covers operational technology, a category it says includes building automation systems, and says configuration management helps “reduce risks from outages due to configuration issues.” That is the failure mode here.

    What Google Is Changing

    The remediation list is a good template for questions to put to any provider or colo vendor. According to Google’s report, the PLC was reconfigured and validated on October 20 and again on October 26. Google ran full generator load-transfer tests with utility cutover, and rewrote its emergency procedures to include manual generator restart and transfer within the UPS runtime, practiced twice per shift. It froze planned maintenance at sites with similar topologies and added elevated approvals for configuration changes outside standard designs. It also plans PLC code reviews at facility commissioning, with a second Google team validating within a quarter after a facility is declared ready (estimated completion December 18).

    Those fixes suggest good questions for your own vendors. When did you last test automatic generator start through the controls with the utility actually cut? How does your battery runtime compare with how long the manual procedure takes?

    Why a Single-Zone Fault Still Hurt Multi-Zone Customers

    Google’s geography and regions documentation says “Zones should be considered a single failure domain within a region” and tells you to deploy across multiple zones for fault tolerance. Yet Google says some customers in zones a and c, or on regional architectures, were affected anyway, and names two causes: demand surges on neighboring zones and control-plane dependencies across zonal boundaries.

    • Capacity: Google says some Compute Engine failover attempts “were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism.”
    • Placement drift: Apigee X is designed to survive a single-zone loss, but Google says a scheduling misconfiguration put two of three datastore replicas in the zone that lost power. The datastore lost quorum and the runtime returned errors for 7 hours and 19 minutes.
    • Slow failover: Regional AlloyDB primaries saw up to a 95-minute delay in starting automated failover because failover notifications were delayed.
    • Configuration that worked: Bigtable multi-cluster instances with HA routing failed over, while single-cluster instances in europe-west2-b had the expected zonal unavailability.

    A Resilience Checklist for the Layer You Cannot See

    These steps are my opinion, not Google’s guidance.

    1. Ask for the facility story. Ask your colo or cloud account team about generator auto-start testing, controls change review, and what changes during maintenance postures.
    2. Audit replica placement. Verify that quorum-based services are spread across three zones today, and alert on drift. The Apigee case is a placement bug, not a power bug.
    3. Run a zone-loss game day. Drain a zone in staging and watch how the survivors handle the surge. Price reserved capacity in the surviving zones if it matters.
    4. Remove single-zone and non-HA tiers from critical paths. Google’s report treats loss of zonal resources in a failed zone as working within design.
    5. Decide what needs a second region. The incident stayed inside europe-west2, so workloads already running in a second region would have been outside the blast radius, at real cost and operational effort. Be selective.
    6. Update your risk register. Uptime Institute’s 2026 annual outage analysis (industry survey data, not a Google measurement) found power was the cause of 45% of respondents’ most recent impactful data center outages in 2025, down from 54% in 2024. Among respondents with power-related IT outages in the past three years, the most common causes were UPS failure (39%), transfer switch failure (34%) and generator failure (28%); controls failure was cited by 8%. They are survey results, but they point at the switchover path.

    The Takeaway

    Google’s report is unusually detailed, a rare chance to study a facility-level failure from the customer’s side of the wall. A change on a building control system carries the same kind of risk as one on a production network, and testing has to match the posture the site actually runs in. Your uptime plan should name the layers you do not control and say how you will find out when they fail.

  • Exploited Outlook Zero-Day Leaks NTLM Hashes: Why November’s Patch Tuesday Is the Moment to Start Retiring NTLM

    Exploited Outlook Zero-Day Leaks NTLM Hashes: Why November’s Patch Tuesday Is the Moment to Start Retiring NTLM

    When a Patch Tuesday ships more than a thousand fixes, the one that matters first is the one attackers already use. In November 2026, that is an Outlook zero-day, and it is an NTLM hash theft bug: open the wrong file and Outlook can hand your Windows authentication hash to a server the attacker controls. Patch it this week. Then use the scare to fix the real problem, which is a three-decade-old authentication protocol most of us keep around only because nobody has inventoried what still depends on it.

    How the Outlook Zero-Day Enables NTLM Hash Theft

    The bug is CVE-2026-124194, Microsoft Outlook Spoofing Vulnerability, released November 10. Microsoft’s Security Update Guide marks it Exploited: Yes and Publicly Disclosed: No, with an exploitability assessment of Exploitation Detected. Microsoft rates it Moderate with a CVSS 3.1 base score of 6.5. Affected products are Microsoft 365 Apps for Enterprise, Office LTSC 2021, and Office LTSC 2024; the fixed builds listed are 16.0.14334.21048 for LTSC 2021 and 16.0.17932.21140 for LTSC 2024, with Microsoft 365 Apps covered through the normal Office security release channel.

    The mechanics are short. According to Microsoft’s FAQ, an attacker sends a specially crafted message file and convinces the user to open it. Outlook then connects to an attacker-controlled network location without showing the usual external-content warning, and Microsoft says the flaw “discloses a user’s NTLMv2 hash to the attacker who could use this to authenticate as the user.” The Preview Pane is not an attack vector, so somebody has to actually open the file. Microsoft has not published details about the attacks it detected, so I am not going to guess who is behind them or how many organizations were hit.

    CISA added the flaw to its Known Exploited Vulnerabilities catalog on November 10 and set a November 24 remediation due date for federal civilian agencies. If you are not a federal agency, treat that date as a sanity check, not a grace period.

    Prioritizing One Fix Inside a Thousand-Fix Release

    You may have seen this month described as a record. Microsoft’s own data does not support that. Counting CVEs in the November Security Update Guide data that were issued by Microsoft as the CNA and first published on November 10, I get 1,093. The same count for October 13 comes to 1,408. Both are enormous, but November is the smaller of the two, so it is not a record. Third-party tallies differ depending on whether they include Edge, Chromium, Linux, and other non-Microsoft CVEs that Microsoft republishes, so check what a headline number actually counts before you repeat it.

    The practical point survives the correction. Severity sorting will not surface this bug on its own; a 6.5 Moderate sits below a long list of Critical ratings. The flags that should move it to the top are the exploitation status and the KEV listing. A workable triage order for a release this size:

    1. Exploited first. Anything Microsoft marks Exploitation Detected or CISA lists in KEV goes to the front, regardless of score. This month, that is the Outlook fix.
    2. Confirm the Office channel. Office fixes ride a different update path than the Windows cumulative update. Verify the build number on actual machines, not just the deployment ring status.
    3. Then the internet-facing and identity tier. Domain controllers, Exchange, and anything reachable from outside come next.

    Why This Bug Class Keeps Coming Back

    The patch closes one path. It does not change the underlying condition, which is that a Windows client will still offer an NTLM response to a server it should never have been talking to. Microsoft’s own description of NTLM, in its January 2026 post on disabling NTLM by default, lists the problems plainly: no server authentication, vulnerability to replay, relay, and pass-the-hash attacks, and weak cryptography. The same post notes NTLM has been part of Windows authentication for more than three decades and is already deprecated.

    Imagine a finance user in a 40-person office who opens a message file that looks like a forwarded invoice. On an unpatched machine, the outbound connection happens, the hash leaves the building, and nobody sees a prompt. That is a hypothetical, but it is the shape of the risk: the user did one ordinary thing, and the protocol did the rest.

    That is why the network layer matters as much as the patch. Microsoft’s guidance on securing SMB traffic says to block TCP port 445 outbound to the internet at the corporate firewall, and notes it is unlikely you need outbound SMB to the internet unless a cloud service such as Azure Files requires it. The same guidance recommends outbound firewall rules for laptops on untrusted networks, because those devices are not behind your perimeter firewall. I would put SMB signing and Extended Protection for Authentication on the same list for any service that still accepts NTLM, since both exist to blunt relay.

    A Realistic NTLM Retirement Plan

    Microsoft’s roadmap has three phases: enhanced NTLM auditing now on Windows Server 2025 and Windows 11 version 24H2 and later; IAKerb and a local Key Distribution Center, both described as pre-release, to remove common reasons for NTLM fallback, slated for the second half of 2026; and network NTLM disabled by default in the next major Windows Server and client releases. Disabled by default is not removed. Microsoft says NTLM will remain in the OS and can be re-enabled by policy, and its timelines are explicitly subject to change.

    The steps Microsoft lists in that post are the right order of operations:

    • Deploy enhanced NTLM auditing to find where NTLM is still used.
    • Map dependencies across applications and services, and prioritize remediation, including pushing application vendors to update.
    • Migrate and validate that critical workloads succeed with Kerberos.
    • Test NTLM-off configurations in non-production environments before production.
    • Adopt the Kerberos improvements as they become available.

    Notice what is not on that list: flipping a block switch on Monday morning. Audit before block. Every blocking control will break something you did not know about, often a printer, a scanner that writes to a share, or an old line-of-business app that only knows NTLM. The audit phase is how you find those before your users do. None of this is risk-free, and a staged rollout with a written rollback is cheaper than an all-hands outage call.

    What I Would Do This Week

    Push the Office update and verify builds. Confirm outbound 445 is blocked at the edge and test it from a laptop on guest Wi-Fi. Turn on NTLM auditing where your OS versions support it and let it collect for a few weeks. Then start the dependency list. The Outlook zero-day will be patched in a week or two. The NTLM inventory is the work that makes the next one of these a smaller story.

  • Democrats’ AI Legislation Plans: Compliance Prep for the 120th Congress

    Democrats’ AI Legislation Plans: Compliance Prep for the 120th Congress

    The useful compliance response to AI legislation plans in Congress is not to guess which proposal will survive. It is to make your own systems explainable. What uses AI, whose data reaches it, what was tested, and who accepted the remaining risk? Those are operational questions worth answering before a lawyer has to reconstruct the deployment from invoices and Slack archaeology.

    In a November 6 report by Courtney Rozen, Reuters described Democrats’ post-election plans for AI and data-center legislation. Representative Lori Trahan’s deputy chief of staff, Francis Grubar, said she would pursue independent evaluation of leading AI models against safety standards and legislation clarifying responsibility for harmful AI behavior. Reuters also reported Senator Chuck Schumer’s priority of protecting consumers from higher energy costs tied to data centers. These are distinct policy directions, not a finished compliance checklist.

    The target is the 120th Congress, beginning in January 2027, not the current 119th Congress’s lame-duck period. The Twentieth Amendment sets the start of congressional terms at noon on January 3. Reported intentions do not establish final bill language, covered organizations, effective dates, or passage. I would not turn them into mandatory project requirements.

    AI Legislation in Congress Is Not a Compliance Specification

    Preparation has a firmer foundation than forecasting. NIST’s AI Risk Management Framework 1.0 is voluntary, not a new federal AI law. Its GOVERN 1.6 outcome addresses AI inventories; GOVERN 2.1 addresses documented responsibilities; MEASURE 2.1 addresses test sets, metrics, and evaluation tools. Those existing outcomes support useful records without pretending we know what Congress will enact.

    Keep separate tracks: legal obligations confirmed for your organization, and voluntary risk-management improvements. Counsel determines applicability; IT supplies evidence about actual deployments. A framework mapping can organize that evidence, but it is not proof of legal compliance or a promise that future legislation will recognize your template.

    Inventory Workflows, Not Just Vendor Logos

    I would make the business use case the organizing unit. A model name alone does not tell you whether it drafts internal meeting notes or influences a consequential customer decision. Following NIST’s inventory and context-setting approach, a practical entry should identify:

    • Purpose and ownership: the workflow, accountable business owner, technical operator, intended users, and prohibited uses.
    • Dependencies: the application, model and version where available, provider, integrations, and production status.
    • Data and authority: input categories, retrieval sources, output destinations, and whether the system can recommend or execute actions.
    • Evidence: links to evaluations, approvals, known limitations, incident records, and the next review trigger.

    Those fields are an implementation recommendation, not a proposed statutory form. Start with higher-risk workflows rather than spending months perfecting a catalog nobody updates. Include AI embedded in purchased applications; ask application owners and procurement what is enabled instead of assuming the model team sees everything.

    Preserve the Test, Not Just the Green Checkmark

    NIST’s Generative AI Profile, NIST AI 600-1, adds specific suggested actions. GV-1.6-003 discusses inventory details including underlying models, versions, access modes, and human oversight. MS-2.3-003 recommends sharing pre-deployment test results with people who have release approval authority. MS-2.5-003 recommends verifying sources and citations in generated outputs during testing and ongoing monitoring.

    For an internal evidence package, retain the evaluated configuration, test-set description, evaluation date, acceptance criteria, results, and unresolved failures. Record where test conditions differ from production. Separate what the provider evaluated from what your organization tested in its own workflow. A vendor benchmark and a deployment-specific evaluation answer different questions.

    Hypothetical example: a support team adds a drafting assistant that retrieves approved troubleshooting articles. Its pilot produces convincing answers, but one invents a refund policy. The useful record includes the failed case, the article sources available at the time, the revised restriction on refund answers, and the retest. A screenshot saying “pilot successful” preserves the celebration, not the evidence.

    Record Decisions, Not Rubber Stamps

    Turn sign-off into a decision record tied to the evaluated configuration. Identify who approved release, what use was authorized, what limitations remain, and who owns corrective work. Record the conditions that require another review: a different model, a new data source, expanded tool permissions, or a materially different use case.

    This translates NIST’s accountability outcomes into change management; it does not claim that Trahan’s reported plans require a particular approval chain. Scale the process to the risk. Requiring the same ceremony for a disposable internal draft and an automated decision affecting customers can bury important reviews under routine paperwork.

    Make the Evidence Useful Before Anyone Requests It

    Assign an owner and access controls to the records themselves. Prefer approved test data and appropriately redacted examples; keeping evidence should not create another uncontrolled collection of customer information. Define retention with legal, privacy, and security teams rather than deciding to save every prompt forever.

    Use the records to answer operational questions now: can we disable the integration, identify affected workflows after a provider incident, or reproduce the configuration that produced a disputed result? Documentation that cannot support those tasks is likely to disappoint during an audit too.

    As federal proposals develop, track actual text, scope, legislative status, and any enacted deadlines separately from political announcements. Independent testing, liability, and electricity costs may affect different organizations in different ways. The sensible investment now is a maintained inventory and a traceable decision history, not a declaration that your organization already complies with a law Congress has not passed.

  • Healthcare Cyberattack Planning: Patient Portal Downtime Lessons From University Hospitals

    Healthcare Cyberattack Planning: Patient Portal Downtime Lessons From University Hospitals

    Healthcare cyberattack planning has to cover patient portal downtime, not just the servers behind it. Cleveland’s University Hospitals offers a concrete example: keeping clinical sites open did not preserve every digital route patients used to reach them. When the portal disappears, a technical isolation decision becomes a scheduling, medication, communications and payment problem. Restoring a login page is only one part of restoring the service.

    What University Hospitals Actually Said

    In its cybersecurity incident notice, UH says it detected “unusual activity” on October 22, 2026, activated response protocols and disconnected systems from the internet “out of an abundance of caution.” That is the supported description. It does not establish ransomware, patient-data theft or a confirmed breach.

    The notice’s November 2 update said hospitals, emergency departments and physician offices remained open. It also described the operational split: most applications on UH-owned servers were functioning, but could not exchange information outside UH. An application can be running while the service its users need is unavailable. Infrastructure monitoring and patient experience are not the same dashboard.

    On November 7, UH announced MyChart was accessible again, according to WKYC’s report quoting UH’s restoration statement. October 22 to November 7 is 16 calendar days: more than two weeks, not three. Those dates do not establish an exact outage duration to the hour.

    The statement warned that some features might remain limited and new results or other information could be delayed while data was reconciled. WKYC explicitly noted that the announcement did not update electronic prescribing. Portal access was a recovery milestone, not confirmation that every connected workflow had recovered.

    Healthcare Cyberattack Planning Starts With Patient Portal Downtime

    UH’s downtime notice described working alternatives rather than simply asking patients to wait. Its phones remained operational, but very high call volumes produced long holds. Patients with urgent care, result or medication questions were directed to their doctor’s office, rather than a main number that could only transfer them. UH asked patients not to call merely to confirm existing appointments unless their care team had contacted them about a change.

    Video visits moved to phone, Zoom or in-person appointments with instructions from the care team. Outpatient pharmacies remained open. When electronic prescribing was unavailable, UH said providers could print, fax or call in prescriptions when needed. That describes UH’s fallback; another organization must have its clinical, pharmacy and compliance teams approve alternatives appropriate to its own requirements.

    UH also directed patients with Quest Diagnostics testing to Quest’s results service. That was not a universal replacement: UH Specialty Lab and inpatient results still required MyChart or the doctor. Payment at the time of service was unavailable at its cashless facilities, although the notice offered separate automated-phone and online payment routes. These were different dependencies, with different workarounds.

    A Working Telephone Is Not a Capacity Plan

    My practical starting point would be a workflow inventory owned jointly by IT and operations. For each patient-facing function, record what the patient needs, which outside connection it depends on, the approved alternative and who owns the resulting queue. “Call the office” needs staffing, routing and a way to track completion. Otherwise, the fallback is a hold queue with a reassuring name.

    • Appointments: distinguish existing bookings from changes, cancellations and new requests. Keep patient instructions consistent with what scheduling teams can actually deliver.
    • Medications and results: name the clinical escalation owner and the approved handoff. Track receipt and follow-up rather than treating a fax transmission as a completed patient task.
    • Payments: separate an unavailable collection channel from the underlying balance. Have finance explain permitted alternatives without implying every patient must use a portal.

    This is operator analysis drawn from UH’s documented workarounds, not a claim that UH used this exact inventory. I would exercise the alternatives under sustained demand, including shift changes and unavailable staff. A plan that works for a quiet afternoon may not remain workable after repeated days of diverted requests.

    Status Updates Are Part of the Recovery System

    The public messages also need reconciliation. On November 10, UH’s incident page still displayed its November 2 update and wording that MyChart was offline, despite the later restoration announcement. The dated announcement is evidence of restored access; the older page is not evidence of a fresh outage.

    There was still a narrower limitation afterward. In its November 8 update, cleveland.com reported a UH spokesperson’s statement that bill pay was the only remaining out-of-service portion of the portal. That does not establish that electronic prescribing, a separate workflow, had recovered. Neither an expected billing fix nor portal availability supplies a verified e-prescribing restoration date.

    I would give every status update an effective date, a named owner and a service-by-service description. When a newer notice supersedes an older one, link them and remove contradictory instructions. Patients should not have to debug the organization’s communications history to find a working route to care.

    Keep the Downtime Team Through Reconciliation

    HHS’s ASPR TRACIE Healthcare System Cybersecurity: Readiness & Response Considerations treats recovery as a continuation of downtime. It recommends planning the paper-document lifecycle, maintaining consistent communications and supporting the migration of manual documentation back into electronic systems. It also warns that partial functionality can hinder workflows or increase risk. These are planning considerations to adapt, not one configuration for every provider.

    My recovery checklist would therefore test complete workflows: request received, routed, acted on, documented and visible to the appropriate patient or team. Clinical leaders would decide how to reconcile medication requests and results safely; operations would retain staff for the backlog. The finish line is not merely an available portal. It is knowing what happened while the portal was unavailable, what remains unfinished and who is responsible for closing the loop.

  • RMM Tool Vulnerability: MSP Security Lessons From N-able’s N-central Shutdown

    RMM Tool Vulnerability: MSP Security Lessons From N-able’s N-central Shutdown

    An RMM tool vulnerability is an MSP security problem before it becomes a patching task. Remote monitoring and management concentrates the authority used to maintain customer machines. When that management plane becomes suspect, the same team must contain risk, preserve evidence, and keep customers working without its usual controls. N-able’s November 9, 2026 N-central shutdown makes that dependency concrete. The lesson is not that every management agent is malicious. It is that your recovery plan cannot depend entirely on the system you might need to disconnect.

    What N-able Actually Took Offline

    N-able’s status dashboard recorded the critical N-central incident starting November 9 at 10:15 AM EST. The company said it took its hosted N-central environments offline as a precaution while responding to publicly disclosed vulnerabilities. Its affected-services list covered hosted N-central in the Americas, APAC, and Europe. It also strongly advised on-premises customers to temporarily take their N-central servers offline until further guidance. That is not a shutdown of every N-able cloud product.

    The notice distinguished the unavailable hosted N-central classic environment from Ecoverse services available through N-able Login, including Asset View, Patching, Vulnerability Management, scripting, policies, and tagging. That distinction matters when building a fallback: unavailable management functions and available alternatives are not interchangeable, and neither should be guessed from a headline.

    In its November 9 update labeled 3 PM EST, N-able said it had no evidence of active exploitation or unauthorized data access. That was the company’s assessment at that point, not proof that every customer environment was clean. A precautionary outage is not, by itself, a disclosure of customer compromise. Nor does it establish when service would safely resume.

    That picture has since changed. N-able’s vulnerability remediation update says N-central 2026.4 HF2 addressed the internet-facing, unauthenticated vulnerabilities, after which it brought its hosted servers back online. It also describes a second publication on November 12 involving Legacy API abuse and Agent impersonation. On November 16, N-able released HF3, which fixes 17 more CVEs, hardens how Agents, Probes and appliances authenticate, and permanently removes the Legacy API. Remote control other than Take Control and the Intune integration remain disabled until a later release. N-able has also said it saw scanning activity tied to the public proof-of-concept code but no indications of compromise within its hosted servers. That statement does not cover customer environments, and The Hacker News has since reported active exploitation of the Legacy API flaws to deploy cryptocurrency miners on vulnerable servers.

    What the Able2Shell Research Establishes

    The name is real: the original Able2Shell write-up uses it. The researcher states that testing used N-central Appliance 2026.4.0.31, a pre-release build. The repository describes an anonymous XML external-entity and session-trust chain leading to code execution, plus stored cross-site-scripting chains that require a ProductAdmin to view attacker-controlled patch details before privileged execution occurs.

    Those are researcher-reported results with different prerequisites, not one universal attack path. The author’s expectation that older versions are affected is not a verified version-by-version assessment. I would not turn it into one. N-able’s precautionary action establishes operational urgency; it does not independently validate every repository claim or demonstrate that attackers used those chains against customers.

    RMM Tool Vulnerability Response Starts With MSP Security Boundaries

    The joint CISA Guide to Securing Remote Access Software, published June 6, 2023 with NSA, FBI, MS-ISAC, and Israel’s National Cyber Directorate, provides a useful baseline. It recommends reduced-privilege RMM configurations for common tasks such as read-only monitoring, multifactor authentication for accounts accessing customer environments, customer separation, and safeguards for mass scripting. These are agency recommendations to implement against your platform’s supported controls, not a promise that every RMM product exposes identical switches.

    My practical review would separate four kinds of authority:

    • Viewing: Who can see inventory and health information without changing a machine?
    • Changing: Who can launch a remote session, deploy software, or run a script across a customer estate?
    • Delegating: Who can create technicians, issue integration credentials, or enable vendor support access?
    • Crossing boundaries: Which accounts and integrations can reach more than one customer?

    Reduce unnecessary authority where the product supports it, and test changes against required maintenance functions. Do not blindly change an agent’s operating-system service identity and call it hardening. Constrain technician roles, automation scope, and network reach deliberately; document exceptions that genuinely require elevated execution.

    Observe the Work, Not Just the Approved Executable

    The CISA-led guide recommends establishing normal behavior, correlating suspicious activity, auditing remote-access tools, and keeping log servers and their alteration or deletion out of RMM tools’ reach. An approved management executable is not an explanation for everything it launches.

    I would correlate endpoint process activity with management audit records: which technician or integration requested the action, which customer and machines it targeted, and whether the work matched an approved change. Unexpected command interpreters, new administrator accounts, unusual remote sessions, or security-setting changes deserve investigation in context. These are behavioral review ideas, not published Able2Shell indicators of compromise.

    Keep a usable copy of the necessary telemetry outside the management plane’s administrative control. If your only record of a suspicious script lives inside the same console that issued it, the investigation has a rather circular dependency.

    Practice an Outage Without Adding Another Uncontrolled Door

    Consider a hypothetical MSP whose patch approvals, device inventory, and customer contact list are accessible only through its RMM console. Disconnecting that console may be the right containment decision, but the team must still identify critical endpoints and reach the people responsible for them. That is an availability failure even if no intrusion occurred.

    My fallback runbook would keep an access-controlled inventory and escalation contacts available independently, identify essential maintenance that can safely wait, and test an approved alternate administration route. That route needs its own authentication, logging, and customer authorization. Installing an ungoverned second remote-control agent everywhere is not a recovery strategy; it is an additional trust relationship.

    For restoration, follow the vendor’s current product-specific remediation instructions, verify the applicable build and configuration, and reconcile accounts, integrations, and queued automation before resuming broad execution. Where there is evidence of intrusion, involve incident responders rather than treating an upgrade as a clean bill of health. Restored console access proves availability, not the integrity of everything previously managed.

    The question I would put into the next exercise is simple: can we stop trusting the management plane without losing the ability to understand and operate the estate? That answer is more useful than another dashboard showing that the dashboard is online.

  • Non-Human Identity Security: Inventory Service Accounts, API Keys, and Bot Tokens

    Non-Human Identity Security: Inventory Service Accounts, API Keys, and Bot Tokens

    Non-human identity security starts with an awkward question: who owns the credentials that keep working after their creator leaves? Service accounts, deployment keys, and bot tokens do not appear on the employee offboarding checklist unless somebody puts them there. I would rather have an inventory that answers that question than a frightening industry ratio and a spreadsheet nobody trusts.

    A Big Ratio Is Not Your Cloud Census

    Entro Labs, the research team inside security vendor Entro Security, reported 144 non-human identities for every human identity in its H1 2025 report. This was vendor-sponsored research using telemetry from Entro’s enterprise customer base, not a representative survey of every cloud estate. Its methodology covers more than 27 million non-human identities across global customers, collected between January 1 and June 30, 2025, spanning cloud providers and SaaS platforms.

    The report does not disclose the number of customer organizations or a detailed counting rule for that ratio. It does not establish that non-human identities outnumber employees in most cloud estates. Human identities are not necessarily employee headcount, either. Treat the finding as a reason to inspect your environment, not a multiplier for estimating its size. More credentials do not automatically mean more risk; their permissions, exposure, and lifecycle matter.

    Make Non-Human Identity Security an Ownership Problem

    Keep the principal separate from its credentials. A service account is an identity; a key or token is a way of authenticating or exercising access. Rotating a credential need not retire the principal, and counting every replacement token as another permanent identity can make your dashboard nonsense.

    AWS’s IAM best practices call for reviewing and removing unused users, roles, permissions, policies, and credentials, applying least privilege, and using temporary credentials with workload roles. Those are AWS-specific controls, not a promise that every SaaS integration supports the same model. My practical inventory would connect these records:

    • Identity and owner: the principal, accountable team, application, environment, and business purpose.
    • Credential references: issuer, key identifier or approved secret-store reference, creation date, expiry where applicable, and revocation procedure. Do not copy secret values into the inventory.
    • Access and trust: allowed actions, target resources, and who or what can assume the identity or obtain a credential.
    • Dependencies and evidence: consuming jobs, last observed use, log coverage, replacement method, and a tested retirement plan.

    Reconcile cloud identity records with deployment configuration, secret-management metadata, and SaaS application registrations. An export from the cloud console is a starting point, not evidence that you found a bot token installed in a separate collaboration service.

    For AWS workloads, prefer IAM roles with temporary credentials where supported. Scope permissions to the required actions and resources, and constrain the trust relationship as carefully as the permissions policy. A narrowly scoped role still needs an appropriately restricted caller. Temporary credentials reduce the lifetime of a stolen credential; they do not make excessive permissions harmless.

    Cloudflare’s Missed Credentials Were Not Harmless

    There is a documented consequence to confusing apparently unused credentials with revoked access. In its disclosure of the November 2023 intrusion, Cloudflare said it had failed to rotate one service token and three service account credentials stolen during the earlier Okta compromise. They were mistakenly believed to be unused. The resulting intrusion reached its Atlassian environment.

    Cloudflare said no customer data or systems were impacted, and described access controls that limited movement. This was not proof that every bot token compromises an entire company. It was evidence that a credential left valid can remain useful to an attacker even after its owner thinks it has no job. An inventory needs an operational revocation state, not just a status someone typed.

    Rotate Without Turning Expiry Into an Outage

    Provider guidance is not interchangeable. Google Cloud’s service account key guidance recommends avoiding user-managed keys where possible, using usage metrics to identify unused keys, and routinely rotating the user-managed keys that remain. It also warns against simply setting key expiry on production workloads because expiry can cause outages; its recommendation there is lifecycle management through rotation. It makes exceptions for some hardware-protected keys.

    AWS’s guidance above takes a different formulation for necessary long-term access keys: update them when needed, using last-used information to make changes safely. Neither page supplies a universal rotation interval for all credentials. Choose a supported authentication mechanism and a lifecycle policy for that mechanism, rather than copying a calendar rule across unrelated services.

    For a planned replacement, my runbook would identify every consumer, deploy the replacement through the approved delivery path, verify successful authentication, and then invalidate the old credential. Where overlapping credentials are supported, keep that overlap controlled and brief. A suspected compromise needs incident handling and prompt containment, not an ordinary change-window delay. Confirm revocation using the issuer’s documented behavior; do not assume replacing a stored key instantly terminates every existing session.

    Test the Inventory Against a Real Dependency

    Imagine a hypothetical deployment bot whose maintainer has left. Ask the owning team to trace its credential reference to the issuer, explain its production permissions, and demonstrate replacement in a safe test environment. Verify the old credential is rejected and the intended job still runs. If ownership or dependencies cannot be established, that is remediation work, not permission to delete blindly.

    I would prioritize unknown owners and broad production access before chasing the smallest possible identity count. Track whether important identities have accountable owners, constrained access, usable logs, and tested revocation. A bigger inventory with those answers can be healthier than a smaller one full of mysteries. The target is controlled access, not winning a ratio contest.

  • Leaseweb’s Cloud Exit Survey: Test Your Cloud Provider Exit Plan Before You Need It

    Leaseweb’s Cloud Exit Survey: Test Your Cloud Provider Exit Plan Before You Need It

    On Nov. 4, 2026, Leaseweb published survey numbers that should make any ops lead uncomfortable: among the C-suite leaders it polled, nearly nine in ten said they could not move business-critical workloads off their largest cloud or infrastructure provider within three weeks. That is a different problem from getting through a bad day in one region. Outage response is about restoring service within agreed time and data-loss limits, sometimes by failing over inside the same provider and sometimes outside it. A cloud provider exit plan is about replacing your dependencies on that provider altogether. Mix up those two clocks and you end up running the wrong drill.

    What the Survey Actually Measured

    Start with the publisher and the sample, not the headline. In its Digital Independence Survey 2026 press release, Leaseweb, an IaaS and cloud provider with a commercial interest in alternatives to hyperscale lock-in, reported that 89% of respondents could not migrate business-critical workloads away from their largest cloud or infrastructure provider within three weeks. Only 11% said they could make that move in that window. More than half (55%) said it would take longer than three months or was not currently possible.

    The same release says 76% reported complete or high control over their ability to move data to another provider or environment. Leaseweb presents that next to the migration figure as a gap. I read it as two different tasks. Copying data out is one job. Moving a whole workload means rebuilding the compute, identity, networking, managed services, deployment pipelines, and monitoring around that data. A team can be confident about the first and still realistic about how long the second takes.

    Censuswide ran the fieldwork for Leaseweb among 800 C-suite leaders at organizations with 500 to 5,000 employees, 200 each in the UK, US, Germany, and the Netherlands, between Sept. 22 and 30, 2026. The release summarizes the findings but does not publish the questionnaire wording. Read “most companies” as most of this vendor-sponsored, self-reported sample of organizations in four countries, estimating how long an exit would take. It is not a census of every cloud customer.

    The October 2025 Outages Illustrate a Different Problem

    Provider exit is a planned migration measured in weeks or months. Incident recovery is measured against your recovery time objective (RTO), meaning how long until the service is usable again, and your recovery point objective (RPO), meaning how much data loss you can accept. Bringing DynamoDB clients back in us-east-1 tells you nothing about whether you could re-platform the estate onto another vendor by Friday.

    Amazon’s own summary of the Amazon DynamoDB service disruption in Northern Virginia (us-east-1) says impact began at 2:48 AM EDT on Oct. 20, 2025. AWS says the broader event ended at 5:20 PM EDT that day. In between there were separate periods of DynamoDB API errors, Network Load Balancer connection errors, and EC2 launch and connectivity problems. AWS attributes the DynamoDB portion to a latent defect in automated DNS management that caused endpoint resolution failures. That postmortem is about dependencies inside a single region: DNS, control planes, and shared building blocks. It is not a case study in leaving a provider.

    Customers also published their own accounts. Coinbase’s retrospective says it saw severe disruption from the us-east-1 outage on Oct. 20, 2025. Customer impact ranged from login failures to being unable to trade, with about 3 hours 17 minutes of degraded performance for users. Engineers were paged at 3:08 AM EDT, and Coinbase says all services were restored at 6:45 PM EDT after AWS fixed the systems it depends on. Coinbase also notes it could not reach AWS Support because the Support Console was offline. That is incident response and dependency triage during a provider outage, not a three-week migration to someone else.

    Microsoft’s Azure Networking team offers another recovery example from the edge side. Its December 2025 post on Azure Front Door lessons from the October outages covers customer-impacting incidents on Oct. 9 and Oct. 29, 2025, both caused by incompatible configuration metadata spreading through the global edge. The fixes Microsoft describes are configuration safety, tenant isolation, and faster automated recovery, including a proposed practical recovery target of about ten minutes for Front Door—not a guarantee or proof of achieved recovery times. That is RTO work, not advice to move off Azure this quarter.

    What a Cloud Provider Exit Plan Drill Should Actually Test

    The rest of this post is my own analysis as an operator. I’m not claiming Leaseweb’s 89% applies to your shop, and I’m not claiming lock-in caused any of those October outages.

    1. Write both clocks down. For each critical user journey, record the RTO and RPO next to an honest estimate of how long a provider exit would take. If the exit column is blank or says “call the vendor,” you have found a documentation gap.
    2. Inventory the sticky parts first. That means data gravity (object stores, databases, logs), identity (IAM roles, workload identity, SSO apps), DNS and TLS (registrars, certificate automation, traffic managers), and managed services with no clean equivalent elsewhere (queues, ML endpoints, proprietary APIs). Exit drills tend to fail on that glue, not on the VM list.
    3. Keep recovery tests and exit tests separate. Failing over to another region or another account with the same provider exercises RTO and RPO. It does not test an exit, because you are still on the same provider’s identity, control plane, and managed services. Imagine a tabletop where the team announces “we cut over to another provider by Monday” without export paths, IAM remapping, or named DNS cutover owners. That exercise is fiction, and the gaps it skips are your real findings.
    4. Run a bounded exit drill on one workload. In a hypothetical drill, you pick a non-critical service and time a restore onto a different provider or an independently operated on-premises environment. Moving to a different account or region does not count here. Write down what breaks: secrets, service endpoints, observability agents, payment webhooks. That stopwatch reading feeds your exit estimate. It does not prove you could move your most critical systems in three weeks.
    5. Budget for people and change windows. Leaseweb’s sample already says full workload migration is slow, even among leaders who feel confident about moving data. Your real constraint may be contract terms, data-residency rules, or the one engineer who understands the Terraform state.

    Use the survey as a reason to write the exit estimate down. Use the October postmortems as a reason to harden the recovery dependencies inside the provider you already pay. Keep those two drills on separate checklists, so the next outage doesn’t get answered with a three-month migration plan.

  • OpenSSH Vulnerability Patch for Musl Hosts: How Password-Disabled Accounts Can Be Exploited

    OpenSSH Vulnerability Patch for Musl Hosts: How Password-Disabled Accounts Can Be Exploited

    Portable OpenSSH 10.6p2 landed on Nov. 5, 2026 with a critical security workaround for a password authentication bypass on some Linux systems. The OpenSSH vulnerability patch is small. The operator work is larger: figuring out whether your fleet even runs the risky stack, and what “password-disabled” means in /etc/shadow versus turning password logins off in sshd. A reproducible bypass is not evidence of a verified exploitation wave; the primary sources cited here do not establish a within-hours attack timeline. What they do establish is a sharp, reproducible login path on musl hosts without PAM—and a same-day response plan you can actually run.

    How This OpenSSH Vulnerability Is Exploited—And What to Patch

    Start with the vendor note. The OpenSSH 10.6p2 release notes say the release contains a critical security workaround that affects Linux systems that use musl libc with PAM disabled. Damien Miller’s project text is careful: the team considers the root cause outside sshd, but the condition is exploitable through sshd and can be avoided there. The bug was reported by Özgün Kültekin of Wakefield. Fixed portable builds are labeled 10.6p2; Alpine published its advisory the same day.

    CVE-2026-123158, published Nov. 5, 2026, describes sshd in OpenSSH before 10.6p2 on Linux when musl libc is used without PAM: for some account configurations, an oversized password succeeds. Alpine Linux is named as an example platform. That is not a blanket claim that every OpenSSH daemon on the internet is open. Glibc hosts, OpenBSD, Dropbear, and sshd builds that actually use PAM for password checks sit outside this bypass path.

    The mechanism, as documented on the researcher’s *sshole site and restated in Alpine’s advisory, is an interaction—not classic key authentication “working as designed.” Accounts with * in the shadow password field are supposed to reject every password. Musl’s crypt() can return the string * as an error when a submitted password exceeds its length limit. In the non-PAM path, OpenSSH may borrow a real salt from another account that has a $5$ or $6$ hash so timing does not leak usernames. Hash the oversized password with that salt, get * back, compare it to the target account’s stored *, and the comparison can succeed. Researcher and Alpine guidance put the practical password length window at 257 to 1,024 bytes. Key-based logins are not the bug; confusing a crypt error token with a disabled-account marker is.

    Password-Disabled Is Not PasswordAuthentication No

    Three different account and config states keep getting mashed together in chat threads. Keep them separate on the ticket.

    • Password-disabled (* in shadow): No usable password hash. Common for key-only users and some service accounts. This is the account state the bypass targets when the rest of the stack lines up. Other login restrictions still apply: Alpine notes that the default PermitRootLogin prohibit-password protects root from this password path; an accessible account with passwordless doas can still provide a route to root.
    • PasswordAuthentication no: An sshd setting that refuses password logins entirely. Alpine and the researcher both list it as the immediate mitigation if you cannot patch yet. It is the opposite of “password auth still on,” which is a prerequisite for this path.
    • Locked with !: Alpine’s advisory says accounts whose password field is ! are not susceptible through this * comparison. Historically Alpine also learned that locking with ! can break key logins when PAM is off—which is why some images moved to * and walked into this interaction later. Do not “fix” a fleet by flipping every * to ! without understanding key login side effects.

    Alpine’s Nov. 5, 2026 security advisory is blunt: if you run sshd on Alpine, update openssh and restart sshd; upgrading the package alone does not replace the running daemon. You are in scope when you run the non-PAM openssh-server package (or openssh-server-pam with UsePAM no), PasswordAuthentication is still yes, and an account with a * password field exists alongside at least one account with a usable $5$/$6$ hash. Official cloud images with a default alpine user and passwordless doas raise the stakes after a console password appears elsewhere on the box. Alpine noted that no CVE identifier had been assigned at the time of writing; CVE-2026-123158 carries the same publication date. Prefer the fixed package matrix Alpine published for your release branch over guessing from portable tarball names alone.

    A Same-Day Response Plan for Your SSH Fleet

    The following is my analysis as an operator. Treat the timed drill as hypothetical unless your change board already authorized an emergency window.

    1. Inventory the stack, not just the banner. Ask which hosts run musl (Alpine, OpenWrt, Buildroot, Yocto appliances, some containers that expose sshd) and whether PAM is actually in the password path. A glibc Ubuntu bastion with UsePAM yes is a different work item than an Alpine edge node.
    2. Patch or mitigate in that order. Move affected builds to OpenSSH 10.6p2 or the distro’s fixed package, then restart sshd. If the package lag is real, set PasswordAuthentication no, validate with sshd -t, restart sshd, and verify with sshd -T. Check applicable Match blocks too, using connection-specific sshd -T -C parameters where needed. Keep an existing session and a recovery path available, then test a new authorized connection before closing the change. That mitigation is temporary discipline, not a substitute for the upstream workaround on hosts you intend to leave reachable.
    3. Hunt logs for impossible password successes. The researcher points operators at authentication logs where an account that should never accept a password shows a successful password login. Correlate with shadow: if the field is still *, treat that as a compromise indicator and rotate keys, tokens, and anything that account could reach. Do not invent a global breach rate from a lab reproduction.
    4. Hypothetical same-day drill: Within one change window, list every internet-reachable SSH listener you own, flag musl/non-PAM candidates, push the fixed package or the password-auth kill switch, restart sshd, and sample logs for password accepts on * accounts. That is a tabletop you can schedule—not evidence that honeypots were rooted hours after disclosure.

    What we know from primary notes: Nov. 5, 2026 brought OpenSSH 10.6p2, Alpine packages, CVE-2026-123158, and a clear musl-without-PAM exploit path against password-disabled (*) accounts while password authentication remains enabled. What we do not know from those same sources is a verified “within hours” mass exploitation wave. Patch the hosts that match the prerequisites. Keep the vocabulary honest so the next person on call does not “mitigate” the wrong setting.

Browse by topic