Autoscaling fleets

An autoscaling group — a GCP managed instance group, an AWS Auto Scaling group, an Azure Scale Set — creates and terminates hosts on its own. Give the launch template one reusable enrollment key, and each instance enrolls a runner during boot with no per-host key or approval.

For hosts that stay put, see Linux host.

How it works#

A host install uses a new single-use key for one machine; a fleet uses one reusable key, because every instance starts from the same template. On boot each instance runs the same installer, which downloads the runner, exchanges the key for the runner's own token, and connects. Holding a valid key is the whole enrollment check — no one approves each host.

Set EMISAR_GROUP in the boot script to set the instance's fleet tier. Runbooks and runner scopes then target the fleet instead of one host.

Create a reusable enrollment key#

In the console, open Runners → Enrollment keys → New key. Turn on Reusable so every instance can enroll with the same one. Two optional limits keep a shared key honest:

• — Max uses. Cap the total number of runners the key can ever enroll — a ceiling that survives a leak. Uses are never refunded when an instance terminates, so budget for every boot over the key's life, not the group's peak size.
• — Expiry. Set the date when the key stops enrolling runners, so a forgotten template cannot add runners forever.

The emkey-enroll-… secret appears once. Store it in the secret manager, and let the boot script read it at start. Never bake the key into an image.

The boot install#

The boot script runs the same installer as a normal host install, plus two flags: --yes and --packs. Without both, the installer stops to ask questions no one is there to answer.

Keep the enrollment key off the command line. Every user on the host can read any process's command line; only the process owner can read its environment. So the script exports the key first, then uses sudo -E to keep it in the environment through sudo:

#!/usr/bin/env bash
# the key reaches the environment, never argv
export EMISAR_ENROLLMENT_KEY="$(your-secret-tool read emisar/enrollment-key)"

curl -fsSL https://emisar.dev/install.sh \
  | sudo -E env EMISAR_GROUP=web \
      bash -s -- --yes --packs linux-core

The command line then does not carry the secret, and only root can read the key. The example reads it from your secret manager at boot because cloud user data is not a safe place for a secret.

The pieces of that command:

• — --yes skips the prompts.
• — --packs lists the packs to install, and is required with --yes. An empty value installs none; you can add packs later.
• — EMISAR_GROUP sets which fleet the runner joins.
• — EMISAR_RUNNER_LABEL_* variables add free-form labels — EMISAR_RUNNER_LABEL_ROLE=web.
• — EMISAR_RUNNER_ID sets the name and identity the runner registers under, when the image's own hostnames are generated or repeat.

The Linux host page covers every other installer flag and config setting.

GCP, AWS, and Azure#

The install line is identical on every cloud. All that changes is where the launch template runs a boot script — save the command above as emisar-boot.sh and wire it in:

• — GCP Managed Instance Group. Add it to the instance template as startup-script metadata — gcloud compute instance-templates create … --metadata-from-file=startup-script=emisar-boot.sh.
• — AWS Auto Scaling group. Put it in the launch template's user data (base64-encoded) — the UserData field of aws ec2 create-launch-template --launch-template-data.
• — Azure Virtual Machine Scale Set. Pass it as cloud-init custom data — az vmss create … --custom-data emisar-boot.sh.

Each cloud runs the script as root during first boot; the runner enrolls and appears in the fleet within seconds. For faster boots, bake the binary into a custom image with --no-start and let the boot script start and enroll it.

A new runner each boot#

A runner uses its hostname as its identity unless you set runner.id. A new scale-out instance receives a generated hostname and registers as a new runner. A reboot keeps the hostname and reconnects the same runner, so scaling out by ten creates ten runners — the expected shape of an ephemeral fleet. Two rules follow:

• — Never bake a credential into an image. At first connect, the runner stores its token in /var/lib/emisar. A snapshot taken after enrollment copies one runner token into every instance built from that image. Build the image with --no-start. Let each instance enroll during its own boot.
• — Use a stable runner ID when the hostname is unsuitable. If the group recycles hostnames, set runner.id from the instance metadata in the boot script. A cloud instance ID is a natural unique runner ID.

Reaping terminated instances#

A terminated instance does not remove its runner — the record just goes offline and stays in the fleet, where each dead host counts against the plan runner limit until removal.

Turn on retention to remove them. On Runners, an owner or admin sets Remove runners inactive for, and an hourly sweep removes runners past the window — never a connected or disabled one. It is off by default. Pick a window longer than an expected disconnect, a rolling deployment or a zone outage, but short enough to clear dead hosts promptly. A removed host that returns enrolls as a new runner.

Between sweeps, a high-churn group can still carry dead runners against its plan limit — trim them sooner with Clean up now on the same page. Removal, by sweep or by hand, keeps the runner's audit history.

After it is connected#

Manage the runner fleet covers work after the first connection — groups, labels, pack credentials, updates, reconnects, and removal. To take a fleet to production in phases, see the Go to production.