Incident analysis

The credentials MFA never sees

Two October breaches, ConnectWise Cloud Backup and PingCAP, ran on credentials that belong to software. Nobody signed in, so nobody was asked for a second factor.

An app or service account signs in with a secret or a certificate, not a person and a phone. Phishing-resistant MFA, the control most teams spent the last three years on, does not apply to it. When one of these credentials is stolen, the attacker inherits everything it was ever granted, at any hour, with no prompt anyone can refuse.

The half we do not do

TemprBac grants people time-bound membership in groups. It does not issue, rotate or vault credentials for apps and services. This post is about that other half, because October made it impossible to ignore, and because the people who administer those apps are where the two halves meet.

ConnectWise Cloud Backup: one vendor credential, many tenants

ConnectWise Cloud Backup, formerly SkyKick, backs up Microsoft 365. To do that it is installed in each customer's tenant as an application with, in Huntress's words, broad access to mailboxes, OneDrive, SharePoint, Teams, and directory data in every tenant it is installed in. Huntress reported on 16 October that a credential belonging to that application had been compromised.

The credential lived in ConnectWise's own tenant and signed in as the app. As Huntress put it, app-only sign-ins involve no user, no device, and no MFA prompt. On 15 October, working from a hosting address:

  • ConnectWise service principals assigned themselves directory roles, mostly Exchange Administrator, in roughly 50 tenants Huntress monitors.
  • In one tenant, a ConnectWise Teams app added itself to Global Administrator, then added new client secrets to two other applications, an MSP management tool and Huntress's own. The new secrets were set to expire in 2028.
  • Huntress confirmed unauthorized sign-ins in 162 tenants by the evening of 16 October, and later found almost 1,000 tenants with sign-in activity from the attacker's address, over 300 of them successful.

ConnectWise cancelled the credential and then shut off Cloud Backup's application access, which interrupted backups for its customers. It has said it cannot see into customer tenants to tell whether any one of them was accessed. How the credential was stolen has not been explained.

The detail worth sitting with is the 2028 expiry. The attack did not just use a standing credential. It made new ones, on other apps, designed to outlast the cleanup.

PingCAP: service account credentials, four weeks

PingCAP, which makes the TiDB database and runs TiDB Cloud, published its own report on 19 October. Between 9 September and 7 October, it said, an unauthorized party used stolen service account credentials to access certain PingCAP internal systems, copy data, and obtain source code.

The confirmed exports were TiDB Cloud account details (user names, email addresses, organization and project associations, SQL editor text and errors), billing records (contacts, contract terms, orders and pricing) and a table of encrypted API-key values. PingCAP found no evidence that data stored inside customers' TiDB databases was affected. Its remediation list is instructive: it removed excessive permissions between cloud projects, added storage access logging, and is adding secret scanning and alerts for large exports. PingCAP has not said how the credentials were obtained.

Four weeks is the number to remember. A service account does not go home at night, so nothing about its behavior looks odd at night.

Why MFA did nothing, and was never going to

A person's accountAn app or service account
Signs in with A password and a second factor A client secret, a certificate or a key
Stolen, the attacker still needs The phone, the passkey, or a person to approve a prompt Nothing
Usually holds What one person's job needs Whatever the integration needed on the day it was installed, often tenant-wide
Expires When the person leaves, if offboarding works When the secret's expiry date arrives, which may be years away
Looks unusual when It acts at 3 a.m. Rarely. It always acts at 3 a.m.

What to check in your own tenant this week

  1. List every app and service principal, with what it can do. Not what you meant to grant. In Entra ID that is each enterprise application's API permissions and any directory roles assigned to it. Sort by blast radius: anything with mail, files or directory write across the tenant goes to the top.
  2. Alert on new credentials. A client secret or certificate added to an app is rare and almost always worth a human look. Huntress's advice after ConnectWise was to review every secret and certificate added since the attack began, and the 2028 secrets are why.
  3. Alert on directory role assignments. A service principal giving itself Exchange Administrator or Global Administrator is the loudest thing that happened here. It should page somebody.
  4. Shorten what you can. Prefer workload identity federation or managed identities to long-lived secrets, and short-lived certificates to two-year ones. A credential that expires in an hour is a much smaller prize.
  5. Know how to switch a vendor off. Huntress's advice was to disable the ConnectWise apps rather than delete them, because deleting removes the consent you would need to restore them. Decide that for your critical vendors before the day you need it.

Where people come back in

Every one of those apps was consented to, configured and given its roles by a person with an administrator role. The roles that can add a secret to an app or assign a directory role are among the most powerful in a tenant, and in most organizations several people hold them permanently. That is the part time-bound access does address.

In Entra ID, directory roles can be assigned to role-assignable groups, so even Application Administrator or Global Administrator can be held through group membership. Make that membership something an administrator requests for an hour, with a reason and another person's approval, and two things follow. A stolen administrator laptop on an ordinary day cannot add a secret to anything. And when someone does add one, there is a grant on record that says who held the role at that moment and why.

It does not make the app's own credential any safer. That needs the five checks above. It makes the people who govern those credentials a smaller, better-recorded target.

Sources

Make the administrators a smaller target

Time-bound membership in the groups that carry your most powerful roles, with a record of who held them and why.

Join the waitlist