CockroachDB
v0.1.8Investigate a CockroachDB cluster over the `cockroach` CLI — node liveness / store capacity, under-replicated & unavailable ranges, running queries / sessions / transactions, contention & locks, jobs, statement statistics, table sizes, and cluster settings — plus a few gated operators (cancel a query / session / job, pause / resume a job, decommission / recommission a node). Reads are SQL against crdb_internal / SHOW statements; mutators use SQL or `cockroach node`.
cockroach
Install
emisar pack install
fetches this pack, re-validates it, and verifies its content hash against the
--hash
below — the exact bytes this page was rendered against, so a tampered copy is
rejected — before copying it into the runner's packs dir. The command reloads a
running daemon itself; no manual restart.
sudo emisar pack install cockroach --hash sha256:264dc0b2534d3d80674979b4b79213ce5e72b6d310bde13a382a95dba6562f23
Setup
Every action expands
COCKROACH_URL
on the runner host and passes it to
cockroach
as --url, so host, port, database, TLS mode and the client cert/key paths all live in that single postgres-style URL.
Environment
Set these on the runner host, then add each name to
execution.inherit_env
so the value reaches the action.
-
COCKROACH_URLrequiredPostgres-style CockroachDB connection URL. Use CLIENT-CERT auth (sslmode=verify-full + sslrootcert/sslcert/sslkey paths) — do NOT embed a password, since the URL is passed on the command line. The referenced key file is the secret; the URL itself carries only paths.
Notes
-
Create the SQL user, then mint its client cert with cockroach cert create-client <user> --certs-dir=<dir> --ca-key=<ca.key>; point sslcert and sslkey in
COCKROACH_URLat the pair that writes. -
COCKROACH_URLmust be allowlisted in the runner'sexecution.inherit_env— the action env is scrubbed to PATH/LANG/LC_ALL/TERM by default, so without it cockroach can't find the cluster. - Use client-cert auth in the URL (no embedded password): the URL appears in the action's command line / audit record, so a password there would leak. The client key file (referenced by path) stays on the host and is the actual credential.
- The SQL user needs VIEWACTIVITY + VIEWCLUSTERMETADATA for the reads, and CANCELQUERY / CANCELSESSION / CONTROLJOB (or admin) for the matching mutators. node decommission/recommission need admin.
- Reads use --format=tsv (parsed as text). There is no SQL surface for hot ranges, per-range QPS, replication lag, or clock offset — those live in the DB Console / Prometheus, so this pack deliberately omits them.
Verify it works
Runs cockroach.databases, a low-risk
read that confirms the pack can reach its target. Run it on the host once the pack
is installed; pack install runs it for
you.
sudo emisar pack verify cockroach
Install and configure a pack walks through the whole sequence on a host.
Actions 26 total
View on GitHub-
cockroach.cancel_job exec medium Medium — changes state, easily reversible
Cancel a job (CANCEL JOB)
Cancel one job by id (from jobs). The job stops and rolls back any partial work where applicable. Use to abort a wrong or stuck schema change / backup / import.
View source on GitHub -
cockroach.cancel_query exec medium Medium — changes state, easily reversible
Cancel a running query (CANCEL QUERY)
Stop one in-flight query by its query_id (from cluster_queries). The query ends with an error; the session/connection stays open. Use to kill a runaway query without dropping the client.
View source on GitHub -
cockroach.cancel_session exec medium Medium — changes state, easily reversible
Cancel a session (CANCEL SESSION)
End one client session by its session_id (from cluster_sessions) — its current query is cancelled and the connection closed; an open transaction is rolled back. Use to evict a stuck or abusive connection.
View source on GitHub -
cockroach.cluster_queries exec low Low — read-only or trivially reversible
Show running queries cluster-wide
Show in-flight queries across all nodes (the SHOW CLUSTER QUERIES surface) — query id, node, user, start time, client address, application, and the query text (truncated). Use to find a long-running or runaway query.
View source on GitHub -
cockroach.cluster_sessions exec low Low — read-only or trivially reversible
Show open sessions cluster-wide
Show open SQL sessions across all nodes (the SHOW CLUSTER SESSIONS surface) — session id, node, user, client address, application, active queries, and session start. Use to see who is connected and from where.
View source on GitHub -
cockroach.cluster_settings exec low Low — read-only or trivially reversible
Show cluster settings
Show all cluster settings with their current value, type, and description (SHOW CLUSTER SETTINGS). Use to confirm a setting's value during an investigation. Read-only — this pack does not change settings.
View source on GitHub -
cockroach.cluster_transactions exec low Low — read-only or trivially reversible
Show open transactions cluster-wide
Show open transactions across all nodes — id, node, session, start time, application, statement count, and retry count, from crdb_internal.cluster_transactions. Use to spot long-open or heavily-retried transactions.
View source on GitHub -
cockroach.contention_events exec low Low — read-only or trivially reversible
Show who is blocking whom (contention events)
Show recent transaction-contention events — blocking vs waiting txn, how long the wait was, and the object (database / table / index) and key contended, from crdb_internal.transaction_contention_events. The core "what's blocking my writes" view.
View source on GitHub -
cockroach.contention_leaderboard exec low Low — read-only or trivially reversible
Show most-contended objects
Show the most-contended tables/indexes by cumulative contention time, from crdb_internal.cluster_contention_events. Use to find the hotspot behind widespread contention.
View source on GitHub -
cockroach.databases exec low Low — read-only or trivially reversible
List databases (SHOW DATABASES)
List the cluster's databases. Cheap connectivity + auth check, and the pack's verify action.
View source on GitHub -
cockroach.jobs exec low Low — read-only or trivially reversible
Show recent / running jobs
Show cluster jobs (schema changes, backups, restores, imports, row-TTL) from crdb_internal.jobs — id, type, status, running status, timing, fraction complete, description, and any error. Optionally filter by status. Use to find a stuck or failed job.
View source on GitHub -
cockroach.locks exec low Low — read-only or trivially reversible
Show contended locks (lock table)
Show contended entries in the lock table — range, object, the pretty key, holding txn, lock strength, and how long it's been held, from crdb_internal.cluster_locks. Use to find the lock behind a stall.
View source on GitHub -
cockroach.node_decommission exec high High — service-affecting
Decommission a node (cockroach node decommission)
Mark one node decommissioning and start moving its replicas off. Effectively irreversible once it completes — a fully decommissioned node cannot rejoin under the same id. Uses --wait=none, so the call returns after initiating; track progress with node_status.
View source on GitHub -
cockroach.node_liveness exec low Low — read-only or trivially reversible
Show node liveness / draining / decommissioning
Show per-node liveness from crdb_internal.gossip_liveness — epoch, draining, decommissioning, membership, and last update. Use to see which nodes are live and which are leaving the cluster.
View source on GitHub -
cockroach.node_recommission exec high High — service-affecting
Recommission a node (cockroach node recommission)
Reverse an in-progress decommission so the node resumes accepting replicas. Only valid before the decommission completes. Use to abort a decommission started by mistake.
View source on GitHub -
cockroach.node_status exec low Low — read-only or trivially reversible
Show node status (liveness, ranges, decommission, disk)
Show every node's status — address, build, liveness, replica/leaseholder counts, range counts (incl. unavailable / under-replicated), liveness bytes, and decommission / draining / membership state. The first stop for cluster health.
View source on GitHub -
cockroach.node_status_one exec low Low — read-only or trivially reversible
Show status for one node
Show full status for a single node by id — same columns as node_status, scoped to one node. Use to inspect a node flagged elsewhere.
View source on GitHub -
cockroach.pause_job exec medium Medium — changes state, easily reversible
Pause a job (PAUSE JOB)
Pause one running job by id (from jobs). It can be resumed later with resume_job. Use to relieve load from a heavy backup / schema change without losing its progress.
View source on GitHub -
cockroach.replication_constraints exec low Low — read-only or trivially reversible
Show replication constraint violations
Show zones whose ranges violate their replication constraints (e.g. a region/locality requirement that can't be met), from system.replication_constraint_stats. Empty means all constraints satisfied.
View source on GitHub -
cockroach.replication_stats exec low Low — read-only or trivially reversible
Show under-replicated / unavailable range counts
Show cluster-wide range health — total, under-replicated, unavailable, and over-replicated range counts, summed from system.replication_stats. Any non-zero unavailable count means data is offline; under-replicated means recovery is in progress.
View source on GitHub -
cockroach.resume_job exec medium Medium — changes state, easily reversible
Resume a paused job (RESUME JOB)
Resume one paused job by id (from jobs). Use to continue a job paused with pause_job.
View source on GitHub -
cockroach.set_cluster_setting exec critical Critical — data loss or irreversible
Set a cluster setting (SET CLUSTER SETTING)
Change one CockroachDB cluster setting cluster-wide — SET CLUSTER SETTING <name> = <value>. An unrestricted config write (rate limits, GC TTLs, feature flags) that can materially change cluster behavior, so risk:critical → default-denied. An operator allows it by explicit policy, and WHICH settings/values are permitted is a portal policy decision, not a hardcoded list. Read cockroach.cluster_settings first to see the current value.
View source on GitHub -
cockroach.statement_stats exec low Low — read-only or trivially reversible
Show top statement fingerprints by exec count
Show the busiest statement fingerprints — application, execution count, mean service latency, and the query text — from crdb_internal.statement_statistics (the data behind the DB Console's Statements page). Use to find the highest-volume or slowest statements.
View source on GitHub -
cockroach.store_status exec low Low — read-only or trivially reversible
Show per-store disk capacity / range counts
Show each store's capacity, available and used bytes, range and lease counts, and writes/sec from crdb_internal.kv_store_status. Use to spot a node running low on disk or carrying too many ranges.
View source on GitHub -
cockroach.table_ranges exec low Low — read-only or trivially reversible
Show range count + size for one table
Show the ranges backing one table — range id, leaseholder, size in MB, and replica nodes — via SHOW RANGES ... WITH DETAILS. Use to find a table's data size and how its ranges are spread across nodes.
View source on GitHub -
cockroach.table_row_counts exec low Low — read-only or trivially reversible
Show estimated row counts per table
Show estimated row counts for every table from crdb_internal.table_row_statistics, largest first. Cheap (uses table statistics, not a COUNT). Use to find the biggest tables.
View source on GitHub