Troubleshooting

Start from what you can actually see, run the smallest check that separates the causes, and land on the page that owns the fix — or on a report that already carries the evidence we need.

This page routes; it does not restate the reference. You are done when the owner page named beside your symptom explains what you are seeing and the check it prescribes passes. When that does not happen, each symptom says what "still broken" looks like — and the last two sections say exactly what to send.

The runner will not connect#

It never appears in the fleet#

The installer finished, but the runner is not in the Runners list at all. It never completed its first registration, so there is no identity to show.

• — First, read the host log. sudo journalctl -u emisar -n 200 names the failure. A 401 from registration means the enrollment key was spent, expired, out of uses, or revoked — mint a fresh one and reinstall with it. A 400 invalid_external_id means the host offered no usable identity: the runner registers under its configured runner.id, or its hostname when none is set, and that value must be nonblank and at most 255 characters.
• — Then check egress. A connect that hangs rather than failing is a blocked path, not a bad credential. The runner needs outbound TLS to your emisar origin and nothing inbound — the exact hosts and ports are in Network requirements.
• — Escalate when a freshly minted, unused key still returns 401, or registration returns a status the log cannot explain. Enrollment keys and their caps are owned by Runner fleet.

It is offline or keeps reconnecting#

The runner registered once and is now shown offline. A reconnect on its own is normal: the runner dials out again with bounded backoff and re-advertises its packs and actions, and an action already executing keeps running across that gap.

• — First, look at the host. systemctl status emisar says whether the process is even running, and sudo emisar doctor checks config, credential, packs, action binaries, the service, and reachability in one pass without opening a cloud session.
• — Then check the identity, not the host. A disabled runner is refused with 403 and keeps retrying with its existing token, so enabling it brings the host back with no shell access at all; a disabled account behaves the same way for every runner in it. A deleted runner is terminal: it gets 401, discards its cached token, and stops rather than retrying a revoked identity — that host has to enroll again.
• — Escalate when the service is up, the runner is enabled, doctor passes including reachability, and the fleet still shows it offline across several backoff cycles.

The service failed or is crash-looping#

systemd reports failed. After five starts in five minutes it stops retrying on purpose — a revoked credential should not hammer the control plane.

• — First, read the exit, then re-arm the unit. Fix what the log names, then sudo systemctl reset-failed emisar && sudo systemctl start emisar . Starting without resetting the failure counter does nothing.
• — Then check for a second process. A runner locks its data directory at boot, and only one process may own that dispatch and replay state at a time. A stray manual emisar connect beside the service is the usual cause.
• — Escalate when doctor reports every check green and the service still exits. Install layout, the service unit, and the paths it writes are owned by Install on a host.

A pack or an action is missing#

Four different states look the same from the catalog — the action you wanted is not offered. They separate on the host and on the Packs page.

• — The pack did not load. The runner reports it as degraded with the loader's reason rather than dropping it silently, so the console can say which pack failed on which host. Reproduce it locally with sudo emisar pack validate ./pack — duplicate action ids, a script escaping the pack root, and unexpected symlinks all fail closed and name the file and rule.
• — The pack is installed but pending. Installing bytes is not trusting them. Review the exact version and content hash on the Packs page and trust it deliberately; sudo emisar pack list shows what the host will advertise.
• — The hash drifted. Trust binds one exact content hash, so bytes that changed under the same version block dispatch until someone reviews them. That is the guard working — treat it as a change you did not expect until you can explain it.
• — One action is missing from an otherwise healthy pack. A runner advertises whether the program each action would start resolves on its PATH. When it does not, that action stays in the advertisement but drops out of the executable targets. Install the missing binary, or fix the service's PATH, and the runner refreshes on its own. Local admission rules can also hide actions by id or risk ceiling; a rule in the host's config always wins.
• — Escalate when the hash on the host matches the hash you trusted and dispatch still refuses. Pack rollout, trust, and rollback are owned by Roll out and roll back packs; the YAML contract is in Pack reference.

An action will not dispatch#

Dispatch from the console first, even if the agent is what failed. The run row tells you which gate stopped it, and that is faster than reading an error twice removed.

Over MCP the same conditions collapse deliberately: every stale or hidden target condition comes back as target_contract_changed with the exact call to refresh, so a stale caller cannot learn which hidden fact moved. Follow that refresh once, then retry once. signature_required names the runners that enforce client signing, and not_allowed means the current scope does not permit the request — it is not an invitation to probe.

Escalate when policy allows the action, its trusted hash is current, the runner is online and in scope, and dispatch still refuses with no matched rule on the run.

A run failed, timed out, or looks stuck#

Read the status first — the Runs page carries the full table, and three of them get confused with each other. Failed means the action ran and exited non-zero: that is a real result to read, not a gate. Error means it could not complete — the runner went offline, was disabled or removed, or never acknowledged the send. Timed out means the action ran past its declared limit and the runner stopped it.

• — A queued run is usually not stuck. It waits for its runner and is delivered as soon as that runner reconnects. Only when the host is still gone after a short grace window does the sweep end it as Error rather than leave it queued indefinitely.
• — A run that ends with execution_outcome_unknown may have done its work. That is what a runner reports when its process or host died after execution started and before a terminal result was durable. emisar never re-runs that dispatch automatically, and neither should you until you have checked the target system — the side effect is outside emisar's control.
• — Output stopped but the run is still running. Output is bounded per action and truncation is reported, so a quiet run is more often a quiet command than a lost stream. Check the host with sudo emisar events tail -f, which shows the local journal live.
• — Escalate when the same dispatch reports execution_outcome_unknown repeatedly, or a run stays Running with no progress while its runner is connected and idle.

The LLM client or the bridge#

• — Every call returns 401. The key was revoked, or a rotation successor retired it. Mint or rotate one on the Agents page and put the new secret in the client config — API keys owns scope and expiry, Rotate and revoke credentials owns the cutover.
• — The bridge is outdated or unsupported. The Agents page carries the version each key last reported; emisar-mcp --version confirms what is on disk. A running client keeps the binary it already loaded, so restart the client after installing — see Upgrade runners and MCP bridges.
• — A tool call comes back with an error code. invalid_args returns the exact paths to correct, so fix those rather than retrying the same call. The complete code list, with the one action each implies, is in MCP reference.
• — The response was lost after a mutation. Recover by operation id with get_operation, using the same credential lineage that made the call — never by repeating the mutation. A cancelled call, a closed client, or a dropped bridge does not roll back work the control plane already admitted: cancellation stops the bridge's wait, and closing the client's input lets requests already in flight finish.
• — Escalate when the same tool returns a transport error on every attempt, or an operation id you recorded cannot be found under the credential that created it.

Sign-in and directory sync#

Both surfaces keep their own symptom tables, because the fix is nearly always at the provider. Two cases are worth knowing before you open them.

• — SSO sign-in fails, or members loop back to the provider. A discovery failure means the issuer URL is wrong; a rejected callback means the redirect URI or client secret is; an accepted login that emisar still refuses means the ID token is missing the configured identifier or a verified email. If Require SSO is on and no one can complete a real sign-in, use an owner session you already have open to turn it off. The full table is in Single sign-on.
• — Provisioning or group push stopped working. A rotated SCIM bearer invalidates the old value immediately, with no overlap — the connector fails until you paste the new one in. A group mapping that does nothing is almost always keyed on a display name instead of the directory's externalId. The full table is in Directory sync.
• — Escalate when the provider reports success, the identifier claims match, and emisar still refuses the identity — or when Require SSO is on and no owner session remains.

The audit poller#

• — It stopped advancing. An empty page sets neither X-Next-Cursor nor Link, and the collector holds its last cursor: that is caught up, not stuck. Only a full page carries Link: rel="next", meaning more may be available right now.
• — Events repeat, or look missing. The export is a forward keyset read: persist the returned cursor and resume strictly after it. A collector that restarts from a timestamp instead of its saved cursor re-reads. Process or hand off the page idempotently, then persist its returned cursor; a crash between those steps may repeat the page, but it cannot skip unprocessed events. Expect to see an event more than once rather than assume exactly-once delivery.
• — It gets 429, 401, or 403. The endpoint allows 60 requests a minute per token, so poll on an interval rather than in a loop; past that it answers 429. A 401 means the bearer is unknown or revoked. A 403 means it authenticated but may not read here — an MCP key used where an audit-export token belongs, a plan without export, or a role without audit access. The two credential kinds never substitute for each other. Whichever it is, the poller must keep its cursor and retry; a failed request is not an empty page.
• — Escalate when a page returns events you can see in the console with a cursor that does not advance past them. The export contract is owned by Audit & SIEM.

Collect this before you report it#

A report with these attached is usually answered in one round trip; one without them costs a day of questions.

• — Versions of every component involved — the runner (emisar --version), the bridge (emisar-mcp --version), and the LLM client.
• — The UTC time window in which it happened, not "this morning".
• — Safe labels, not secrets — the runner name and group, and the account name. Never the enrollment key, the runner token, an API key, or a signing key.
• — The identifiers — the run id, the operation id, and the request id where you have them. They are how we find the same event you are looking at.
• — sudo emisar doctor output from the affected host, whole rather than the one line you think matters.
• — A bounded slice of the logs around the window — sudo journalctl -u emisar -n 200 and sudo emisar events tail --lines 100 . A whole journal is harder to read, not more complete.
• — The exact error text and whether it reproduces — every time, sometimes, or once.

Where to send it#

• — A product problem — something broken, unclear, or slower than it should be — goes to Support with the evidence above.
• — A suspected compromise or a vulnerability — a leaked credential, an action you cannot account for, an unexpected content hash — goes to security@emisar.dev, and the containment steps are on Security incidents. Contain first; the report can follow within the hour.