Runner fleet

The quickstart gets one runner online. This page is the rest of its life.

emisar console · Runners

Expand

Groups and labels#

Every runner declares a group (one string — cassandra-prod, web) and free-form labels (role=db, region=us-east-1) in /etc/emisar/config.yaml. Groups are load-bearing: runbook steps target them, per-member runner scopes follow them, and an LLM fan-out call can address every runner in one. Pick group names the way you'd name a fleet tier, not a host. The installer defaults the group to the hostname — fine for one box, worth changing the moment you have a second.

Enrollment keys#

• — Two key models. An emkey-enroll-… key is single-use by default — spent on its first registration (concurrent attempts with the same key can't both win), right for an autoscaler enrolling one host at a time. Mark it reusable and the same key keeps enrolling hosts until it expires or hits its max-uses cap — leave the cap blank for unlimited. Right for image bakes and stable fleets.
• — Shown once, traded for a token. The raw key appears once, at creation. On first connect the runner trades it for its own long-lived token (rnrtok-…, stored hashed cloud-side, mode-0600 on the host) — the key itself is never presented again by that host, whichever model minted it.
• — Mint at install time. Runners → Connect a runner generates the install one-liner with a fresh single-use key baked in. For fleet provisioning (cloud-init, Packer, Ansible), mint one reusable key under Runners → Enrollment keys and inject it as EMISAR_ENROLLMENT_KEY.
• — Revocation is cloud-side. Revoking a key blocks new registrations; hosts already enrolled keep running on their own tokens. Disable a runner to block it reversibly — the host service stays up and keeps retrying until you re-enable it. Delete a runner to revoke its token permanently: its next connect gets a 401 and the service exits rather than retrying forever.

Giving packs their credentials#

Packs that talk to a service — Nomad, Consul, Postgres — read credentials from the runner's environment, never from call arguments, so secrets never transit the cloud. Two steps on the host:

# 1. /etc/emisar/runner.env (mode 0600) — the values
NOMAD_ADDR=http://127.0.0.1:4646
NOMAD_TOKEN=<acl-token>

# 2. /etc/emisar/config.yaml — allowlist the names
execution:
  inherit_env:
    - NOMAD_ADDR
    - NOMAD_TOKEN

Only allowlisted names reach an action's process (on top of the PATH/locale baseline) — the runner's own environment and its token never leak through. Restart the service after editing; what each pack needs is in emisar pack info <id>.

Online, offline, and stuck runs#

The runner heartbeats every 30 seconds; the dashboard's online badge reflects it, and either side closes a connection that goes quiet. Disconnects are normal — the runner reconnects with exponential backoff and re-advertises its catalog, and in-flight actions keep executing through the gap with their results delivered on the new connection. If a runner stays offline past a two-minute grace, its in-flight runs are marked as errored with an explanation — nothing sits "running" forever.

Updating#

# runner binary — re-run the installer; configs and packs are preserved
$ curl -sSL https://emisar.dev/install.sh | sudo bash

# packs — update from the registry; the update reloads the runner (no restart, no dropped runs)
$ sudo emisar pack update linux-core

# provenance — built by our workflow, from our source
$ gh attestation verify emisar-<version>-linux-amd64.tar.gz --owner andrewdryga
# checksums — the bytes match what we published
$ sha256sum -c SHA256SUMS

Reload (SIGHUP) re-reads packs and re-advertises the catalog on the live connection. A changed pack hash lands as pending on the Packs page until someone trusts it.

The host-side toolbox#

$ journalctl -u emisar -f          # service logs
$ sudo emisar action list           # what this host advertises
$ sudo emisar pack list             # loaded packs + hashes
$ sudo emisar events tail -f        # the local JSONL journal, live
$ sudo emisar audit verify --all    # verify the journal's hash chain

(The CLI defaults to --config /etc/emisar/config.yaml, so these work as-is on an installed host.) The runner itself runs as the unprivileged emisar user, so a pack action that needs root — restarting a system service, editing system config — runs only if you grant that user the right: a scoped sudoers rule (e.g. emisar ALL=(root) NOPASSWD: /usr/bin/systemctl restart nginx, with the action calling sudo) or an equivalent polkit rule. For that plus the other host defense-in-depth controls — the local admission allowlist, systemd hardening drop-ins — see the security model.

Troubleshooting#

• — Never shows up: journalctl -u emisar -f. A 401 on a first connect means the enrollment key was spent, expired, out of uses, or revoked — mint a fresh one. A connect timeout means outbound 443 is blocked: the runner needs emisar.dev, the installer also needs github.com and api.github.com, and emisar pack install needs the registry — all outbound, no inbound rule.
• — Service is "failed": after five crashes in five minutes systemd stops retrying (deliberate — a revoked key shouldn't hammer the cloud). Fix the cause, then systemctl reset-failed emisar && systemctl start emisar.
• — Action won't dispatch: check the Packs page first — a pending (untrusted) or drifted pack hash blocks dispatch by design. Then check the member's or key's runner scope.
• — Pack refuses to load: the journal names the exact file and rule — duplicate action IDs, a script escaping the pack root, and symlinks (without allow_symlinks) all fail closed. emisar pack validate ./pack reproduces it locally.

Removing a runner#

# remove binary + service + cached token
$ sudo bash install.sh --uninstall
# …or purge everything, including the local journal
$ sudo bash install.sh --uninstall --purge

Then delete the runner in the dashboard. Its run history and audit events stay. --purge deletes the local journal, configuration, and runner.env secrets too, so export the journal first if you keep host-side forensics.

Changing the enrollment key#

Supply a different EMISAR_ENROLLMENT_KEY when reinstalling. The runner detects the changed key, re-registers its configured runner.id or current hostname, and replaces the cached token on its next connection.

A replacement host with a new hostname enrolls as a new runner automatically. No disk-backed identity file or identity-reset step is involved.