Skip to main content

Prometheus operations

v0.1.22

Server status, target health, alertmanager linkage, instant + range queries, rule listing, TSDB stats, plus admin-API actions for remediation: reload config, take snapshot, clean tombstones, delete series. Admin endpoints require --web.enable-admin-api + --web.enable-lifecycle.

11 allowed by default 3 need approval by default
Pack ID
prometheus
Vendor
emisar
OS
linux
Actions
14
Required binaries. Install these on the host before relying on the pack — an action that calls a missing one fails at run time.
curl

Install

emisar pack install fetches this pack, re-validates it, and verifies its content hash against the --hash below — the exact bytes this page was rendered against, so a tampered copy is rejected — before copying it into the runner's packs dir. The command reloads a running daemon itself; no manual restart.

on the runner host
sudo emisar pack install prometheus --hash sha256:27cad664b0c27471f4ddd38d7c4005f7785fee40eca9af0fdcf07fd527e11ca4

Setup

Every action calls the Prometheus HTTP API at $PROM_URL via curl on the runner host. Set PROM_URL to the server base URL; the actions send no auth, so a reachable endpoint is all that is needed.

Environment

Set these on the runner host, then add each name to execution.inherit_env so the value reaches the action.

  • PROM_URL default http://127.0.0.1:9090

    Prometheus base URL (scheme + host + port, no trailing path). Defaults to a local Prometheus.

Notes

  • PROM_URL only reaches an action when the runner allowlists it in execution.inherit_env — the action env is scrubbed to PATH/LANG/LC_ALL/TERM by default, so an env present on the host but not allowlisted is silently dropped and every action, read and admin alike, falls back to http://127.0.0.1:9090 .
  • No credentials are sent. If your Prometheus sits behind an auth proxy, expose it to the runner on a network path that does not require a header, or point PROM_URL at a trusted side door.
  • Admin actions (reload_config, snapshot, clean_tombstones, delete_series) only work if Prometheus was started with --web.enable-admin-api and --web.enable-lifecycle.
  • Range queries are limited to 7 days and 10,081 outer evaluation timestamps per returned series, with a 30-second provider-side timeout. This preserves a full week at one-minute resolution; query cardinality, range-vector lookbacks, and subqueries can still add cost.

Verify it works

Runs prom.build_info, a low-risk read that confirms the pack can reach its target. Run it on the host once the pack is installed; pack install runs it for you.

on the runner host
sudo emisar pack verify prometheus

Install and configure a pack walks through the whole sequence on a host.

Actions 14 total

View on GitHub
  • prom.alertmanagers exec low Low — read-only or trivially reversible

    GET /api/v1/alertmanagers

    List configured Alertmanager endpoints + their reachability.

    View source on GitHub
  • prom.alerts exec low Low — read-only or trivially reversible

    GET /api/v1/alerts

    List currently active alerts (pending or firing).

    View source on GitHub
  • prom.build_info exec low Low — read-only or trivially reversible

    GET /api/v1/status/buildinfo

    Show Prometheus version + build details.

    View source on GitHub
  • prom.clean_tombstones exec high High — service-affecting

    POST /api/v1/admin/tsdb/clean_tombstones

    Reclaim disk space after a delete_series call. Tombstones are removed and the data they marked is permanently deleted.

    View source on GitHub
  • prom.delete_series exec high High — service-affecting

    POST /api/v1/admin/tsdb/delete_series

    Mark series matching the matcher as deleted (tombstoned). Series no longer return from queries. Run clean_tombstones afterwards to actually reclaim disk. Use to remove cardinality-explosion mistakes or accidentally-recorded secrets. Match pattern is restricted to safe label-selector characters.

    View source on GitHub
  • prom.flags exec low Low — read-only or trivially reversible

    GET /api/v1/status/flags

    Show server command-line flags.

    View source on GitHub
  • prom.query_instant exec low Low — read-only or trivially reversible

    GET /api/v1/query

    Run an instant PromQL query. Use to answer 'what's X right now?'.

    View source on GitHub
  • prom.query_range script low Low — read-only or trivially reversible

    GET /api/v1/query_range

    Run a range PromQL query over a trailing window ending now. The server's retention is the only bound on the window; window / step may produce at most 10,081 evaluation timestamps per series, just under the 11,000 Prometheus itself refuses. Use for bounded trends and incident graphs.

    View source on GitHub
  • prom.reload_config exec high High — service-affecting

    POST /-/reload

    Trigger Prometheus to reload its config + rule files in place. Requires --web.enable-lifecycle. If the new config is invalid, reload fails and Prometheus keeps the old one.

    View source on GitHub
  • prom.rules exec low Low — read-only or trivially reversible

    GET /api/v1/rules

    List all loaded recording + alerting rules with last evaluation result.

    View source on GitHub
  • prom.runtime_info exec low Low — read-only or trivially reversible

    GET /api/v1/status/runtimeinfo

    Show uptime, last config reload, chunk count, WAL corrected/unprocessed.

    View source on GitHub
  • prom.snapshot exec medium Medium — changes state, easily reversible

    POST /api/v1/admin/tsdb/snapshot

    Trigger an on-disk snapshot of the TSDB into data/snapshots/. Requires --web.enable-admin-api. Used before risky upgrades or for offline diff. Uses hardlinks initially — no extra disk until blocks rotate.

    View source on GitHub
  • prom.targets exec low Low — read-only or trivially reversible

    GET /api/v1/targets

    List all scrape targets with state (up/down) and last error.

    View source on GitHub
  • prom.tsdb_stats exec low Low — read-only or trivially reversible

    GET /api/v1/status/tsdb

    Show the top-10 cardinal label/value pairs in the TSDB. Use to spot cardinality explosion.

    View source on GitHub