Security incidents

Take the authority away from whoever should not have it, keep the evidence that says what happened, put a known-good path back, and prove the old one is dead.

Treat uncertainty as an incident. A credential that might have been copied, logged, pasted into a ticket, or read by someone who should not have it is an incident until you can bound the exposure — and the containment steps below are cheap compared to being wrong.

The sequence, every time#

1. Contain. Take the authority away at the system that accepts it. Do this first; the cost of an unnecessary revocation is one rotation.
2. Preserve the evidence. Before any step that destroys state — rebuilding a host, purging a runner install — capture what that step would take with it.
3. Rotate or revoke the rest. Anything the exposed credential could reach, or that sat beside it, gets the same treatment.
4. Restore a known-good path. A new credential, a rebuilt host, a re-trusted version you can account for.
5. Verify both directions. The new path works, and the old one fails. One without the other is not containment.
6. Review the audit trail. Read the window around the exposure for what the credential actually did, not only for what it could have done.

Routine rotation follows the same shape at a calmer pace, with an overlap window where one exists — that is Rotate and revoke credentials. The difference here is that you cut first and accept the interruption.

Preserve this before you clean up#

• — The cloud audit trail for the window. It lives off the affected hosts, so it survives anything you do to them. Export the window and keep it with the incident — Audit & SIEM owns the export.
• — The local journal on each affected host. Copy /var/log/emisar/events.jsonl and its rotated siblings off the host, and run sudo emisar audit verify --all before you touch anything: it re-derives the hash chain and names the first break. The chain proves the file was not edited in place; it cannot prove a privileged attacker did not replace the whole file, which is exactly why the cloud record matters too.
• — The exact pack hashes and their trust evidence. Record what each affected host was advertising and what the account had trusted, hash for hash. "The redis pack" is not evidence; a content hash is.
• — The identifiers and the window. The runner identity, the run, operation, and request ids, and the UTC start and end of the exposure as best you know it. Everything else is searchable from those.

A leaked MCP API key#

The key calls MCP tools with its owner's role and runner scope. Whoever holds it can discover and dispatch exactly what that operator could — policy, approvals, and pack trust all still apply, which is the point, but everything inside those gates is reachable.

1. Revoke it — do not rotate it. Revocation is terminal and it reaches further than the one secret: every successor in that key's rotation chain goes with it, and any OAuth access and refresh token backed by it stops working. A rotation would leave the leaked predecessor usable until its successor's first call.
2. Read the audit trail for the window: what that key discovered, dispatched, and had approved. A key acting outside its owner's normal hours or runners is the signal.
3. Mint a replacement for the legitimate holder, and put it in the client config. A cloud client connected over OAuth reconnects instead — there is no secret to hand it.
4. Confirm the old key returns 401 on its next call, and narrow the replacement's runner scope if the incident showed it was wider than the work needed.

A leaked audit-export token#

An audit-export token reads your account's event history and does nothing else. It is a separate credential kind from an MCP key: presented to the MCP endpoint it is rejected as the wrong kind, so it can never list a runner or dispatch an action. The exposure is disclosure — who did what, when, on which hosts — not execution.

1. Revoke the token.
2. Read the export events for the window. A poll that returns events is itself recorded — a caught-up, empty one is not — so an unfamiliar pattern of pulls shows up in the same trail the token was reading.
3. Mint a replacement, give it to the collector, and let it complete one poll that persists its cursor before you consider the cutover done.
4. Confirm the old token returns 401.

A leaked reusable enrollment key#

An enrollment key registers runners. Whoever holds a usable one can enroll a host you do not own into your account, subject to your runner limit — and a malicious runner receives the actions dispatched to it and reports whatever state it likes. With a matching external identity it can also obtain a fresh token for an identity you do own and compete for its connection.

1. Revoke every affected key. If one reusable key was installed on several hosts, treat every host that used it as affected rather than waiting for per-host attribution.
2. Know what revocation does and does not reach. It blocks new registrations. It does not disconnect anything: each already-enrolled runner is connected on its own long-lived token, which the enrollment key does not control.
3. Review the registration events and the fleet inventory. Disable and delete any runner identity you did not expect — deletion preserves its history while making its tokens fail.
4. Re-enroll the hosts you keep, one fresh key per host rather than a shared reusable one. Setting the new key and restarting is enough: the runner notices the key changed, registers again under its existing identity, and replaces its cached token.
5. Finish when the inventory shows the new registrations and the old key fails at registration.

A runner token or host you no longer trust#

This is the case where the reversible lever is the wrong one. A per-runner token cannot be revoked on its own, and disabling an identity keeps its token: enable it again and the same credential is back in service. If the token may have been copied — or the host may have been compromised at all — the identity has to be replaced, not paused.

1. Disable the runner to stop dispatch now. That is containment, not the fix — it refuses the connection while keeping the identity retryable.
2. Preserve the evidence from that host while it is still intact: the journal, the verify output, and the hashes it was advertising.
3. Delete the runner identity. That is terminal: the token stops authenticating, the next connect gets 401, and the process exits rather than retrying. Run and audit history for that runner is preserved.
4. Rotate every credential the host held — pack and provider values in its environment file are the ones people forget, and they are rotated at the system that issued them, not here.
5. Rebuild the host rather than cleaning it, then enroll it as a new runner under a fresh key. Read the runs that host executed in the window before you close the incident.

A pack, a registry, or an unexplained hash#

Pack bytes are executable configuration, so this is the incident class with the widest blast radius. The good news is that the guard usually fires first: trust binds one exact content hash, so bytes that changed under a version you trusted stop being dispatchable until a human reviews them. An unexplained drift is a security event even when it turns out to be a mistake.

1. Record the exact hashes first — what each host advertises, and what the account trusted. These are your before-and-after, and they are immutable facts you can compare later.
2. Find every run that executed under the suspect hash. The audit trail carries the pack reference with the run, so the question "what did this actually do" has an answer.
3. Stop new dispatch of the suspect version through the trust and retirement controls that own it. Retirement and revocation stop new dispatch; they do not erase an immutable published artifact, and they do not rewrite the history of what already ran.
4. Reinstall the exact prior version and hash on the affected hosts, and trust that hash deliberately. The rollout and rollback mechanics are in Roll out and roll back packs.
5. If the registry itself is implicated, rotate its publishing credentials at the system that owns them and re-verify what it serves before you install from it again. Hosting a registry is covered by Host your own registry.

A leaked signing key#

A signing key and its certificate let the holder produce dispatches that a runner enforcing client signatures will accept. Every other gate still applies — the control plane still authenticates the caller, and policy and approvals still run — but the runner's own signature check will not stop them.

1. Decide on the exposure window. If the certificate expires in hours and the risk is tolerable, stop re-issuing to that operator and let it lapse — that is the designed path.
2. If it is not tolerable, run the CA rotation below. There is no smaller operation that takes a single certificate out of service.
3. Read the audit trail for signed dispatches in the window, and check the runs against what the legitimate operator says they ran.

A compromised signing CA#

The CA private key vouches for every certificate a runner will accept, so this is the same operation as above, done urgently and for the whole fleet. The key is meant to live offline; if it was on a machine that was compromised, assume it was taken.

1. Mint a new CA offline and add it to the trusted list alongside the old one, then reload. Both are trusted at this point — this step is what lets you move without dropping runs.
2. Reissue every operator certificate under the new CA and hand them out.
3. Remove the old CA and reload every runner. Until a host has reloaded without the old CA, that host still accepts certificates signed by it. A host you missed is a host that is still exposed.
4. Verify from both ends: a dispatch signed under the new CA is accepted, and one signed under the old CA is refused on every host you can test.

The CA model, the certificate scopes, and the reload mechanics are owned by Signed dispatch.

A leaked OIDC secret or SCIM bearer#

These sit on the same SSO connection and contain differently, so decide which one you are responding to before you touch anything.

• — OIDC client secret. Replace it at your identity provider and save the new value on the connection, then complete a real sign-in before removing the old one at the provider. Replacing the secret does not sign anybody out — if you need existing sessions gone, disabling or deleting the connection revokes the sessions authenticated through it, which is a much larger interruption and still leaves API keys and OAuth credentials untouched. Neither action suspends or deletes members.
• — SCIM bearer. Rotating replaces it immediately, with no overlap: the leaked value stops authenticating the moment you rotate, and provisioning fails until your connector holds the new one. That gap is the containment working. Then read the directory events for the window — role and runner-access changes pushed by a directory you did not recognize are the thing to look for.

The provider-side steps live in Single sign-on and Directory sync; offboarding a person, rather than a credential, is in Authentication.