Security incidents

Take authority away from anyone without permission. Keep the evidence, restore a known-good path, and make sure the old one is dead.

Treat uncertainty as an incident. Assume exposure when a credential was copied, logged, pasted into a ticket, or read without permission. Containment is cheaper than being wrong.

The sequence, every time#

1. Contain. Take the authority away at the system that accepts it — an unnecessary revocation only costs one rotation.
2. Preserve the evidence. Before any destructive step — rebuilding a host, purging a runner install — capture the evidence it would erase.
3. Rotate or revoke the rest. Rotate or revoke any credential that the exposed credential reached or sat beside.
4. Restore a known-good path. A new credential, a rebuilt host, a re-trusted version you can account for.
5. Verify both directions. The new path works, and the old one fails. One without the other is not containment.
6. Review the audit trail. Read the window around the exposure. Check what the credential did, not only what its access allowed.

Routine rotation follows the same shape at a calmer pace, with an overlap window where one exists — that is Rotate and revoke credentials. The difference here is that you cut first and accept the interruption.

Preserve this before you clean up#

• — The cloud audit trail for the window. It lives off the affected hosts, so it survives anything you do to them. Export the window from Audit export and keep it with the incident — Audit & SIEM owns the export.
• — The local journal on each affected host. Copy /var/log/emisar/events.jsonl and its rotated siblings off the host. Before any other change, run sudo emisar audit verify --all to re-derive the hash chain and print the first break. Verification covers only the retained journal or retained suffix. It cannot prove that a privileged host operator did not replace or truncate the entire local journal. That limit makes the cloud record important.
• — The exact pack hashes and their trust evidence. Record the hash each affected host advertised and the hash the account trusted. A pack name alone is not evidence. A content hash is.
• — The identifiers and the window. The runner identity, the run, operation, and request ids, and the UTC start and end of the exposure as best you know it. Everything else is searchable from those.

A leaked agent key#

The key calls MCP tools with its owner's role and runner scope, so its holder gains the same discovery and dispatch access as that operator. Policy, approvals, and pack trust still apply — everything within those gates remains reachable.

1. Revoke it — do not rotate it. Do that from AI agents. Revocation also removes every successor in the key rotation chain. Any OAuth access or refresh token backed by the key also stops working. Rotation leaves the leaked predecessor usable until its successor's first call. For a lost or stolen device, use Revoke all on the member's group instead — it revokes every key that member owns in one act.
2. Read Audit for the window. Audit shows dispatch gates, approval decisions, denials, and terminal outcomes. MCP discovery reads are not audited, so treat discovery during the window as unknown. A key acting outside its owner's normal hours or runners is the signal.
3. Mint a replacement from AI agents for the legitimate holder, and put it in the client config. A cloud client connected over OAuth reconnects instead — there is no secret to hand it.
4. Confirm the old key returns 401 on its next call. If its scope was too wide, narrow the replacement's runner or pack scope.

A leaked audit-export token#

An audit-export token starts emk-export-, reads your account event history, and does nothing else. It is a different credential kind from an agent key — the MCP endpoint rejects it as the wrong kind, so it cannot list a runner or dispatch an action. The exposure reveals who did what, when, and on which hosts; it does not grant execution.

1. Revoke the token from Audit export.
2. Read the export events for the window. A poll that returns events is recorded — a caught-up empty poll is not — so an unfamiliar pattern of pulls shows in the same trail.
3. Mint a replacement from Audit export and give it to the collector. Wait for one poll to save its cursor before completing the cutover.
4. Confirm the old token returns 401.

A leaked reusable enrollment key#

An enrollment key registers runners. Anyone with a usable key can enroll an untrusted host in your account, up to your runner limit. A malicious runner receives its dispatched actions and can report false state. With a matching external identity, it can get a fresh token for an existing runner and compete for the connection.

1. Revoke every affected key from Runners → Enrollment keys. If one reusable key was installed on several hosts, treat every host that used it as affected rather than waiting for per-host attribution.
2. Know what revocation does and does not reach. It blocks new registrations. It does not disconnect anything: each already-enrolled runner is connected on its own token, which the enrollment key does not control.
3. Review the registration events and the fleet inventory in Runners. Disable and delete any runner identity you did not expect — deletion preserves its history while making its tokens fail.
4. Re-enroll the hosts you keep, one fresh key per host rather than a shared reusable one. Setting the new key and restarting is enough: the runner notices the key changed, registers again under its existing identity, and replaces its cached token.
5. Finish when the inventory shows the new registrations and the old key fails at registration.

A runner token or host you no longer trust#

This is the case where the reversible lever is wrong: a runner token has no separate revocation, disabling an identity keeps its token, and enabling it restores the same credential. After a possible token or host compromise, replace the identity.

1. Disable the runner to stop dispatch now. That is containment, not the fix — it refuses the connection while keeping the identity retryable.
2. Preserve the evidence from that host while it is still intact: the journal, the verify output, and the hashes it was advertising.
3. Delete the runner identity. That cannot be undone: the token stops authenticating, the next connection is refused, and the process exits rather than retrying. Run and audit history for that runner is preserved.
4. Rotate every credential that the host held. Pack and provider values in its environment file are easy to miss. Rotate each value at its issuer.
5. Rebuild the host rather than cleaning it, then enroll it as a new runner under a fresh key. Read the runs that host executed from Runs before you close the incident.

A pack, a registry, or an unexplained hash#

Pack contents execute as real commands on your hosts, so this incident has the widest blast radius. Trust normally binds one exact content hash. A configured catalog can automatically authorize a matching installed pack, and its publisher chooses the risk, kind, and retirement floors the control plane reads — so a suspect catalog is itself a lever. Treat unexplained drift as a security event, even if the cause is a mistake.

1. If the configured catalog publisher is suspect, stop using or replace that catalog first. Do this before installing or trusting more packs.
2. Record the exact hashes first from Packs — what each host advertises, and what the account trusted. These are your before-and-after, and they are immutable facts you can compare later.
3. Find every run that executed under the suspect hash in Audit. The audit trail carries the pack reference with the run, so the question "what did this actually do" has an answer.
4. Stop new dispatch of the suspect version through the trust and retirement controls in Packs. Retirement and revocation stop new dispatch. They do not erase an immutable artifact or rewrite past run history.
5. Reinstall the exact prior version and hash on the affected hosts, and trust that hash deliberately. The rollout and rollback mechanics are in Roll out and roll back packs.
6. If the registry is implicated, rotate its publishing credentials at their issuer. Verify its contents before another install. See Host your own registry for the hosting steps.

A leaked signing key#

A signing key and certificate let their holder create dispatches accepted by a runner that enforces client signatures. The control plane still authenticates the caller. Policy and approvals still apply. The runner signature check cannot stop a holder with valid signing authority.

1. Decide on the exposure window. If the certificate expires in hours and the risk is tolerable, stop re-issuing to that operator and let it lapse — that is the designed path.
2. If it is not tolerable, run the CA rotation below. There is no smaller operation that takes a single certificate out of service.
3. Read Audit for signed dispatches in the window, and check the runs against what the legitimate operator says they ran.

A compromised signing CA#

The CA private key vouches for every certificate a runner accepts. Rotate the whole fleet urgently. If the key was on a compromised machine, assume theft.

1. Mint a new CA offline and add it to the trusted list alongside the old one, then reload. Both are trusted at this point — this step is what lets you move without dropping runs.
2. Reissue every operator certificate under the new CA and hand them out.
3. Remove the old CA and reload every runner. Until a host has reloaded without the old CA, that host still accepts certificates signed by it. A host you missed is a host that is still exposed.
4. Test every available host in both directions. Confirm that it accepts a dispatch signed by the new CA and rejects one signed by the old CA.

The CA model, the certificate scopes, and the reload mechanics are owned by Signed dispatch.

A leaked OIDC secret or SCIM bearer#

These sit on the same SSO connection but are contained in different ways, so decide which one you are responding to before you touch anything.

• — OIDC client secret. Replace it at your identity provider. Save the new value under Team → Single sign-on. Complete a real sign-in, then remove the old provider secret. Replacing the secret does not end existing sessions — to end them, disable or delete the connection, which revokes the sessions it authenticated while leaving API keys and OAuth credentials active. Neither action suspends or deletes members.
• — SCIM bearer. Rotate it from the connection under Team → Single sign-on. Rotation replaces the bearer immediately with no overlap, so the leaked value stops working at once. Provisioning fails until the connector has the new value — that gap is your containment working. Review directory events for the window, looking for role or runner-access changes you did not make.

The provider-side steps live in Single sign-on and Directory sync. Authentication covers offboarding a person instead of a credential.