Manage the runner fleet

The quickstart gets one runner online. This page covers everything after: groups and labels, enrollment keys, pack credentials, connectivity, updates, and removal.

emisar console · Runners

Expand

Groups and labels#

Every runner declares one group string and free-form labels in /etc/emisar/config.yaml. Group examples include cassandra-prod and web. Label examples include role=db and region=us-east-1. Groups do real work: runbook steps target them, per-member runner scopes follow them, and an agent can address every runner in one group at once. Name a group after a fleet tier. The installer defaults the group to the hostname, which works for one host — change it when you add a second.

Install packs generously. Give each group every pack that matches what runs there — the postgres pack on the database tier, nginx on the web tier — and the observation packs everywhere. The more actions an agent can reach, the more it can do for you, and breadth is safe: policy decides what actually runs, so reads run at once while risky work waits for a person. sudo emisar pack suggest lists the published packs that match a host.

Enrollment keys#

• — Two key models. An emkey-enroll-… key is single-use by default: its first registration spends it, and concurrent attempts with the same key cannot both succeed — right for connecting one host. Mark it reusable to keep enrolling hosts until it expires or reaches its maximum use count (blank for unlimited) — right for launch templates and fleet provisioning.
• — Shown once, exchanged for a token. The raw key appears once, at creation. On first connect the runner exchanges it for its own rnrtok-… token, which handles every reconnect from then on. Runner credentials covers the token's lifetime, refresh, and recovery.
• — Mint at install time. Runners → Connect a runner generates the install one-liner with a fresh single-use key filled in. For fleet provisioning (cloud-init, Packer, Ansible), mint one reusable key under Runners → Enrollment keys and inject it as EMISAR_ENROLLMENT_KEY.
• — Revoke from the console. Revoking a key blocks new registrations; hosts already enrolled keep running on their own tokens. To take a runner itself offline — disable (reversible) or delete (terminal) — see Runner credentials.

To change the key a host carries, reinstall with the new EMISAR_ENROLLMENT_KEY — the runner re-registers under its runner.id or current hostname; Runner credentials has the step-by-step. A replacement host with a new hostname simply enrolls as a new runner.

Giving packs their credentials#

Packs that talk to a service — Nomad, Consul, Postgres — read credentials from the runner's environment, never from call arguments, and the control plane does not distribute or store them. An action can still print a value, and if redaction misses it, it can appear in run output. Two steps on the host:

# 1. /etc/emisar/runner.env (mode 0600) — the values
NOMAD_ADDR=http://127.0.0.1:4646
NOMAD_TOKEN=<acl-token>

# 2. /etc/emisar/config.yaml — allowlist the names
execution:
  inherit_env:
    - NOMAD_ADDR
    - NOMAD_TOKEN

Only allowlisted names reach an action's process, in addition to the PATH and locale baseline. The runner's own environment and token never reach it. Restart the service after editing. Run emisar pack info <id> to see what each pack needs, and sudo emisar pack verify to make sure the wiring works — it probes every installed pack's verify action and prints ok, failed, or skipped per pack.

Online, offline, and stuck runs#

The runner sends a heartbeat every 30 seconds, and the console's online badge reflects it. Either side closes a connection that goes quiet; the runner reconnects on its own with backoff and advertises its catalog again. In-flight actions keep running through the gap, and their results arrive on the new connection. If a runner stays offline past a two-minute grace period, its in-flight runs become errored with an explanation — nothing stays "running" forever.

Updating#

# runner binary — re-run the installer; configs and packs are preserved
$ curl -fsSL https://emisar.dev/install.sh | sudo bash

# packs — update from the registry; the update reloads the runner (no restart, no dropped runs)
$ sudo emisar pack update linux-core

Reload (SIGHUP) re-reads packs and re-advertises the catalog on the live connection. An unrecognized hash — your own pack, or an edited one — waits for trust on the Packs page.

To roll the binary across a fleet with canaries, batches, and rollback, see Upgrade runners. To do the same for packs, see Roll out and roll back packs. A binary upgrade never touches packs, and a pack update never touches the binary.

The host-side toolbox#

$ journalctl -u emisar -f          # service logs
$ sudo emisar action list           # what this host advertises
$ sudo emisar pack list             # loaded packs + hashes
$ sudo emisar events tail -f        # the local JSONL journal, live
$ sudo emisar audit verify --all    # verify the journal's hash chain

The CLI defaults to --config /etc/emisar/config.yaml, so these commands work on an installed host. The runner runs as its own unprivileged emisar user; granting an action root — a scoped polkit rule, or running the service as root — is covered on the Linux host page, and the security model covers the rest of the host protections.

Removing a runner#

# remove binary + service + cached token
$ curl -fsSL https://emisar.dev/install.sh | sudo bash -s -- --uninstall
# …or purge everything, including the local journal
$ curl -fsSL https://emisar.dev/install.sh | sudo bash -s -- --uninstall --purge

Then delete the runner from Runners. Its run history and audit events stay. --purge also deletes the local journal, configuration, and runner.env secrets. Export the journal first if you want to keep it.

That is the routine retirement. For a host you no longer trust, follow security incidents instead — evidence first, then containment.

Troubleshooting#

It never appears in the fleet#

The installer finished, but the runner is not in the Runners list at all. It never completed its first registration, so there is no identity to show. Read the host log first — the register line carries the status that stopped it:

sudo journalctl -u emisar -n 200

On the name conflict: a runner's name is unique in the account — by default it is the hostname of the host it runs on, and it cannot be renamed after the runner is created. Set runner.id in the config (or RUNNER_ID at install) to register under a declared name and identity instead of the hostname. A deleted runner cannot be restored — the host enrolls again as a new one, and Removing a runner covers the difference from disabling.

• — The log shows no failure — the connect hangs. A hang is a blocked path, not a bad credential. The runner needs outbound TLS to your emisar origin and nothing inbound — the exact hosts and ports are in Network requirements.
• — Contact support when a freshly minted, unused key still returns 401, or registration returns a status the log cannot explain. For enrollment keys and their caps, see Enrollment keys above.

It is offline or keeps reconnecting#

The runner registered once and is now offline. Reconnecting is normal: the runner retries on its own, advertises its packs and actions again when it connects, and an action already running continues through the gap.

• — First, look at the host. systemctl status emisar shows whether the process is running. sudo emisar doctor checks config, credentials, packs, action binaries, service state, and reachability, all without a session to the control plane.
• — Then check the identity, not the host. A disabled runner gets a 403 and retries with its existing token, so enabling it brings the host back without shell access — a disabled account behaves the same way for every runner in it. A deleted runner cannot be restored: it gets a 401, discards its cached token, stops retrying, and that host must enroll again.
• — Contact support when the service is up, the runner is enabled, doctor passes including reachability, and the fleet still shows it offline after several reconnect attempts.

The service failed or is crash-looping#

systemd reports failed. After five starts in five minutes the service stops retrying on purpose, so a revoked credential cannot hammer the control plane. Fix what the log shows, then re-arm the service — starting without resetting the failure counter does nothing:

sudo systemctl reset-failed emisar && sudo systemctl start emisar

• — Then check for a second process. A runner locks its data directory at boot — only one process can own the dispatch and replay state. A stray manual emisar connect beside the service is the usual cause.
• — Contact support when doctor reports every check green and the service still exits. For the install layout, the service unit, and the paths it writes, see Install on a host.