Nomad system job

A Nomad system job places one runner on every client node. This example uses the release archive and a persistent host volume.

To read what a containerized runner can reach, see Install in a container.

Prepare the deployment#

Complete these steps before you submit the job:

1. Enable raw_exec on every client that should run a runner.
2. Register a host volume named emisar-state at /var/lib/emisar on each client.
3. Create a reusable enrollment key.
4. Store the key as enrollment_key in the Nomad Variable nomad/jobs/emisar-runner.
5. Open the runner release manifest and copy the version and the linux-amd64 archive's sha256.

The system job#

The artifact stanza checks the archive digest before extraction. One template writes the runner config, the other reads the enrollment key from Nomad Variables, and the host volume keeps the runner token and durable dispatch state.

variable "runner_version" {
  type = string
}

variable "archive_sha256" {
  type = string
}

job "emisar-runner" {
  type = "system"

  group "runner" {
    volume "state" {
      type   = "host"
      source = "emisar-state"   # a host volume registered on each Nomad client
    }

    task "runner" {
      driver = "raw_exec"

      artifact {
        source = "https://emisar.dev/releases/runner/runner-v${var.runner_version}/emisar-${var.runner_version}-linux-amd64.tar.gz"
        options {
          checksum = "sha256:${var.archive_sha256}"
        }
      }

      template {
        destination = "local/config.yaml"
        perms        = "0600"
        data = <<-EOH
schema_version: 1
runner:
  group: nomad
cloud:
  url: https://emisar.dev
  enrollment_key_env: EMISAR_ENROLLMENT_KEY
paths:
  data_dir: /var/lib/emisar
  packs:
    - emisar-${var.runner_version}-linux-amd64/packs
events:
  jsonl_path: /var/lib/emisar/events.jsonl
EOH
      }

      template {
        destination = "secrets/emisar.env"
        env         = true
        data = <<-EOH
EMISAR_ENROLLMENT_KEY={{ with nomadVar "nomad/jobs/emisar-runner" }}{{ .enrollment_key | toJSON }}{{ end }}
EOH
      }

      config {
        command = "local/emisar-${var.runner_version}-linux-amd64/emisar"
        args    = ["connect", "--config", "local/config.yaml"]
      }

      volume_mount {
        volume      = "state"
        destination = "/var/lib/emisar"
      }

      resources {
        cpu    = 100
        memory = 128
      }
    }
  }
}

Deploy and verify#

Set the release version and its digest, then validate, plan, and run:

export EMISAR_VERSION=<version>
export EMISAR_ARCHIVE_SHA256=<64-character-sha256>

nomad job validate \
  -var "runner_version=$EMISAR_VERSION" \
  -var "archive_sha256=$EMISAR_ARCHIVE_SHA256" \
  emisar-runner.nomad.hcl

nomad job plan \
  -var "runner_version=$EMISAR_VERSION" \
  -var "archive_sha256=$EMISAR_ARCHIVE_SHA256" \
  emisar-runner.nomad.hcl

nomad job run \
  -var "runner_version=$EMISAR_VERSION" \
  -var "archive_sha256=$EMISAR_ARCHIVE_SHA256" \
  emisar-runner.nomad.hcl

1. Run nomad job status emisar-runner. Confirm every client has one running allocation.
2. Open Runners. Confirm each client's runner is online.
3. Run linux.uptime on one client runner.
4. Open Audit. Confirm that the event records the runner, action, operator, and reason.

Driver and visibility#

The driver decides what the runner can reach. This is the same namespace question as any container deployment. raw_exec runs the binary straight on the client. Use the docker driver instead when the runner only needs mounted files and services.

Keys and pack credentials#

Use a reusable enrollment key — with a single-use key the first node spends it, and every other node fails to enroll.

Give packs their credentials through environment variables on the runner's inherit_env allowlist, never through action arguments. A Nomad template can read the values from Nomad Variables or Vault. The allowlist is covered in Giving packs their credentials.

After it is connected#

Manage the runner fleet covers work after the first connection — groups, labels, pack credentials, updates, reconnects, and removal. To roll the system job across datacenters in phases, see the Go to production.