Autoscaling fleets

An autoscaling group boots and terminates hosts on its own — a GCP managed instance group, an AWS Auto Scaling group, an Azure Scale Set. To put a runner on each one, bake a single reusable enrollment key into the launch template and let every instance install and enroll itself on boot: no per-host key, no manual approval, no click.

How it works#

A host install pastes a fresh single-use key for one machine. A fleet can't do that — every instance boots from the same template — so it uses one reusable key instead. Each instance runs the same installer on boot: it downloads the runner, trades the key for the runner's own token, and connects. Enrollment is zero-touch — a valid key is the credential, so the runner is live the moment it registers; nobody approves each new host.

Each instance can name its own fleet tier with RUNNER_GROUP in the boot script (below), which is how a runbook or a runner scope later targets "the web fleet" instead of one host.

Create a reusable enrollment key#

In the console, open Runners → Enrollment keys → New key. Turn on Reusable so every instance can enroll with the same one. Two optional limits keep a shared key honest:

• — Max uses. Cap the total number of runners the key can ever enroll — a ceiling that survives a leak. Uses are never refunded when an instance terminates, so budget for every boot over the key's life, not the group's peak size.
• — Expiry. A date after which the key stops enrolling, so a template you forget about can't keep adding runners forever.

The emkey-enroll-… secret is shown once. Put it in the boot script — or, better, in the cloud's secret manager with the boot script reading it at start — never committed into an image.

The boot install#

The boot script is the ordinary host installer, run non-interactively. Environment values go before bash; the unattended flags go after bash -s --. The --yes --packs pair is the contract for a no-TTY install — without it the installer refuses to run rather than guess:

#!/usr/bin/env bash
curl -sSL https://emisar.dev/install.sh \
  | sudo EMISAR_URL=https://emisar.dev \
      EMISAR_ENROLLMENT_KEY=emkey-enroll-… \
      RUNNER_GROUP=web \
      bash -s -- --yes --packs linux-core

--yes skips the prompts a host install would show; --packs is the pack set to install and is required alongside it (an empty value installs none, and you add packs later). RUNNER_GROUP — plus any RUNNER_LABEL_* labels (e.g. RUNNER_LABEL_ROLE=web) — tags the runner for targeting. Everything else about the installer, its flags, and the config it writes is on the host install page.

GCP, AWS, and Azure#

The install line is identical on every cloud. All that changes is where the launch template runs a boot script — save the command above as emisar-boot.sh and wire it in:

• — GCP Managed Instance Group. Add it to the instance template as startup-script metadata — gcloud compute instance-templates create … --metadata-from-file=startup-script=emisar-boot.sh.
• — AWS Auto Scaling group. Put it in the launch template's user data (base64-encoded) — the UserData field of aws ec2 create-launch-template --launch-template-data.
• — Azure Virtual Machine Scale Set. Pass it as cloud-init custom data — az vmss create … --custom-data emisar-boot.sh.

Each runs the script as root on first boot, exactly as if you'd pasted the host-install command — the runner comes up, enrolls, and shows in your fleet within seconds. For faster boots, bake the binary into a custom image first (run the installer with --no-start during the image build, then let the boot script start and enroll) — just don't bake an enrollment key into it.

A new runner each boot#

A runner's identity is a UUID it writes under /var/lib/emisar the first time it starts. A disposable instance with an ephemeral disk has none, so it mints a fresh one and registers as a new runner every boot. That's the shape of an ephemeral fleet — scaling out by ten gives you ten new runners, not ten reconnects — and it's fine. Two things follow from it:

• — Never bake an identity into an image. If you install on a build host and snapshot its /var/lib/emisar, every instance launched from that image claims the same identity — and emisar rejects the second one to connect as a cloned data directory.
• — Want a stable id? On instances that persist — or where you want one runner to map to one machine across reboots — pin runner.id in config.yaml from the instance's own metadata in the boot script; the cloud's instance id makes a natural, unique runner id.

Reaping terminated instances#

A terminated instance doesn't remove its runner — the record goes offline and stays in your fleet. Two things then pile up: the Runners list fills with dead hosts, and each one counts against your plan's runner limit until it's gone.

To solve this turn on retention. On the Runners page, an owner or admin sets Remove runners inactive for to a window; a daily sweep then deletes any runner disconnected longer than that. Connected and disabled runners are never touched, and a host that comes back re-enrolls as a fresh runner. It's off by default — pick a window comfortably longer than your longest expected blip (a rolling deploy, a zone outage) so a brief disconnect isn't mistaken for a dead host, but short enough that dead hosts don't linger in the fleet.