Prometheus operations
v0.1.22Server status, target health, alertmanager linkage, instant + range queries, rule listing, TSDB stats, plus admin-API actions for remediation: reload config, take snapshot, clean tombstones, delete series. Admin endpoints require --web.enable-admin-api + --web.enable-lifecycle.
curl
Install
emisar pack install
fetches this pack, re-validates it, and verifies its content hash against the
--hash
below — the exact bytes this page was rendered against, so a tampered copy is
rejected — before copying it into the runner's packs dir. The command reloads a
running daemon itself; no manual restart.
sudo emisar pack install prometheus --hash sha256:27cad664b0c27471f4ddd38d7c4005f7785fee40eca9af0fdcf07fd527e11ca4
Setup
Every action calls the Prometheus HTTP API at
$PROM_URL
via curl on the runner host. Set
PROM_URL
to the server base URL; the actions send no auth, so a reachable endpoint is all that is needed.
Environment
Set these on the runner host, then add each name to
execution.inherit_env
so the value reaches the action.
-
PROM_URLdefault http://127.0.0.1:9090Prometheus base URL (scheme + host + port, no trailing path). Defaults to a local Prometheus.
Notes
-
PROM_URLonly reaches an action when the runner allowlists it inexecution.inherit_env— the action env is scrubbed to PATH/LANG/LC_ALL/TERM by default, so an env present on the host but not allowlisted is silently dropped and every action, read and admin alike, falls back tohttp://127.0.0.1:9090. -
No credentials are sent. If your Prometheus sits behind an auth proxy, expose it to the runner on a network path that does not require a header, or point
PROM_URLat a trusted side door. - Admin actions (reload_config, snapshot, clean_tombstones, delete_series) only work if Prometheus was started with --web.enable-admin-api and --web.enable-lifecycle.
- Range queries are limited to 7 days and 10,081 outer evaluation timestamps per returned series, with a 30-second provider-side timeout. This preserves a full week at one-minute resolution; query cardinality, range-vector lookbacks, and subqueries can still add cost.
Verify it works
Runs prom.build_info, a low-risk
read that confirms the pack can reach its target. Run it on the host once the pack
is installed; pack install runs it for
you.
sudo emisar pack verify prometheus
Install and configure a pack walks through the whole sequence on a host.
Actions 14 total
View on GitHub-
prom.alertmanagers exec low Low — read-only or trivially reversible
GET /api/v1/alertmanagers
List configured Alertmanager endpoints + their reachability.
View source on GitHub -
prom.alerts exec low Low — read-only or trivially reversible
GET /api/v1/alerts
List currently active alerts (pending or firing).
View source on GitHub -
prom.build_info exec low Low — read-only or trivially reversible
GET /api/v1/status/buildinfo
Show Prometheus version + build details.
View source on GitHub -
prom.clean_tombstones exec high High — service-affecting
POST /api/v1/admin/tsdb/clean_tombstones
Reclaim disk space after a delete_series call. Tombstones are removed and the data they marked is permanently deleted.
View source on GitHub -
prom.delete_series exec high High — service-affecting
POST /api/v1/admin/tsdb/delete_series
Mark series matching the matcher as deleted (tombstoned). Series no longer return from queries. Run clean_tombstones afterwards to actually reclaim disk. Use to remove cardinality-explosion mistakes or accidentally-recorded secrets. Match pattern is restricted to safe label-selector characters.
View source on GitHub -
prom.flags exec low Low — read-only or trivially reversible
GET /api/v1/status/flags
Show server command-line flags.
View source on GitHub -
prom.query_instant exec low Low — read-only or trivially reversible
GET /api/v1/query
Run an instant PromQL query. Use to answer 'what's X right now?'.
View source on GitHub -
prom.query_range script low Low — read-only or trivially reversible
GET /api/v1/query_range
Run a range PromQL query over a trailing window ending now. The server's retention is the only bound on the window; window / step may produce at most 10,081 evaluation timestamps per series, just under the 11,000 Prometheus itself refuses. Use for bounded trends and incident graphs.
View source on GitHub -
prom.reload_config exec high High — service-affecting
POST /-/reload
Trigger Prometheus to reload its config + rule files in place. Requires --web.enable-lifecycle. If the new config is invalid, reload fails and Prometheus keeps the old one.
View source on GitHub -
prom.rules exec low Low — read-only or trivially reversible
GET /api/v1/rules
List all loaded recording + alerting rules with last evaluation result.
View source on GitHub -
prom.runtime_info exec low Low — read-only or trivially reversible
GET /api/v1/status/runtimeinfo
Show uptime, last config reload, chunk count, WAL corrected/unprocessed.
View source on GitHub -
prom.snapshot exec medium Medium — changes state, easily reversible
POST /api/v1/admin/tsdb/snapshot
Trigger an on-disk snapshot of the TSDB into data/snapshots/. Requires --web.enable-admin-api. Used before risky upgrades or for offline diff. Uses hardlinks initially — no extra disk until blocks rotate.
View source on GitHub -
prom.targets exec low Low — read-only or trivially reversible
GET /api/v1/targets
List all scrape targets with state (up/down) and last error.
View source on GitHub -
prom.tsdb_stats exec low Low — read-only or trivially reversible
GET /api/v1/status/tsdb
Show the top-10 cardinal label/value pairs in the TSDB. Use to spot cardinality explosion.
View source on GitHub