Nomad system job

A Nomad system job places one runner on every client node. This example uses the release archive and keeps runner state on each client.

To read what a containerized runner can reach, see Install in a container.

Prepare the deployment#

Complete these steps before you submit the job:

1. Enable raw_exec on every client that should run a runner.
2. Create a reusable enrollment key.
3. Store the key as enrollment_key in the Nomad Variable nomad/jobs/emisar-runner.
4. Open the runner release manifest and copy the version and the linux-amd64 archive's sha256.

The system job#

The artifact stanza checks the archive digest before extraction. One template writes the runner config, and the other reads the enrollment key from Nomad Variables. raw_exec cannot mount volumes, so the runner keeps its token and durable dispatch state in /var/lib/emisar on the client itself.

variable "runner_version" {
  type = string
}

variable "archive_sha256" {
  type = string
}

job "emisar-runner" {
  type = "system"

  group "runner" {
    task "runner" {
      driver = "raw_exec"

      artifact {
        source = "https://emisar.dev/releases/runner/runner-v${var.runner_version}/emisar-${var.runner_version}-linux-amd64.tar.gz"
        options {
          checksum = "sha256:${var.archive_sha256}"
        }
      }

      template {
        destination = "local/config.yaml"
        perms       = "0600"
        data        = <<-EOH
schema_version: 1
runner:
  group: nomad
cloud:
  url: https://emisar.dev
  enrollment_key_env: EMISAR_ENROLLMENT_KEY
paths:
  data_dir: /var/lib/emisar
  packs:
    - emisar-${var.runner_version}-linux-amd64/packs
events:
  jsonl_path: /var/lib/emisar/events.jsonl
EOH
      }

      template {
        destination = "secrets/emisar.env"
        env         = true
        data        = <<-EOH
EMISAR_ENROLLMENT_KEY={{ with nomadVar "nomad/jobs/emisar-runner" }}{{ .enrollment_key | toJSON }}{{ end }}
EOH
      }

      config {
        command = "local/emisar-${var.runner_version}-linux-amd64/emisar"
        args    = ["connect", "--config", "local/config.yaml"]
      }

      resources {
        cpu    = 100
        memory = 128
      }
    }
  }
}

On first start the runner creates /var/lib/emisar on the client, owned by the Nomad client's user with mode 0750. Its token, dispatch state, and local journal stay there across allocation restarts and job updates. Stopping the job leaves the directory in place. To retire a node's runner, delete it from Runners, then remove /var/lib/emisar on that client.

Deploy and verify#

Set the release version and its digest, then validate, plan, and run:

export EMISAR_VERSION=<version>
export EMISAR_ARCHIVE_SHA256=<64-character-sha256>

nomad job validate \
  -var "runner_version=$EMISAR_VERSION" \
  -var "archive_sha256=$EMISAR_ARCHIVE_SHA256" \
  emisar-runner.nomad.hcl

nomad job plan \
  -var "runner_version=$EMISAR_VERSION" \
  -var "archive_sha256=$EMISAR_ARCHIVE_SHA256" \
  emisar-runner.nomad.hcl

nomad job run \
  -var "runner_version=$EMISAR_VERSION" \
  -var "archive_sha256=$EMISAR_ARCHIVE_SHA256" \
  emisar-runner.nomad.hcl

1. Run nomad job status emisar-runner. Confirm every client has one running allocation.
2. Open Runners. Confirm each client's runner is online.
3. Run linux.uptime on one client runner.
4. Open Audit. Confirm that the event records the runner, action, operator, and reason.

Driver and visibility#

The driver decides what the runner can reach. This is the same namespace question as any container deployment. raw_exec runs the binary straight on the client. Use the docker driver instead when the runner only needs mounted files and services.

Keys and pack credentials#

Use a reusable enrollment key. With a single-use key, the first node spends it and every other node fails to enroll.

Give packs their credentials through environment variables on the runner's inherit_env allowlist, never through action arguments. A Nomad template can read the values from Nomad Variables or Vault. The allowlist is covered in Giving packs their credentials.

After it is connected#

Manage the runner fleet covers work after the first connection: groups, labels, pack credentials, updates, reconnects, and removal. To roll the system job across datacenters in phases, see Go to production.