CockroachDB
v0.2.1Investigate a CockroachDB cluster over the `cockroach` CLI — node liveness / store capacity, under-replicated & unavailable ranges, running queries / sessions / transactions, contention & locks, jobs, statement statistics, table sizes, and cluster settings — plus a few gated operators (cancel a query / session / job, pause / resume a job, decommission / recommission a node). Reads are SQL against crdb_internal / SHOW statements; mutators use SQL or `cockroach node`.
cockroach
Install
emisar pack install
fetches this pack, re-validates it, and verifies its content hash against the
--hash
below before copying it into the runner's packs dir. That hash covers the exact bytes
this page was built from, so a tampered copy is rejected. The command reloads a
running daemon itself, so you don't need to restart it manually.
sudo emisar pack install cockroach --hash sha256:ea50355405a346f062493744797527899ae4d924a34af8ef4d805d51f4bcff32
Setup
Every action expands
COCKROACH_URL
on the runner host and passes it to
cockroach
as --url, so host, port, database, TLS mode and the client cert/key paths all live in that single postgres-style URL.
Environment
Set these on the runner host, then add each name to
execution.inherit_env
so the value reaches the action.
-
COCKROACH_URLrequiredPostgres-style CockroachDB connection URL. Use CLIENT-CERT auth (sslmode=verify-full + sslrootcert/sslcert/sslkey paths) — do NOT embed a password, since the URL is passed on the command line. The referenced key file is the secret; the URL itself carries only paths.
Notes
-
Create the SQL user, then mint its client cert with cockroach cert create-client <user> --certs-dir=<dir> --ca-key=<ca.key>; point sslcert and sslkey in
COCKROACH_URLat the pair that writes. -
COCKROACH_URLmust be allowlisted in the runner'sexecution.inherit_env— the action env is scrubbed to PATH/LANG/LC_ALL/TERM by default, so without it cockroach can't find the cluster. - Use client-cert auth in the URL (no embedded password): the URL appears in the action's command line / audit record, so a password there would leak. The client key file (referenced by path) stays on the host and is the actual credential.
- The SQL user needs VIEWACTIVITY + VIEWCLUSTERMETADATA for the general reads, VIEWJOB for cockroach.jobs, and CANCELQUERY / CANCELSESSION / CONTROLJOB (or admin) for the matching mutators. node decommission/recommission need admin.
- Reads use --format=tsv (parsed as text). There is no SQL surface for hot ranges, per-range QPS, replication lag, or clock offset — those live in the DB Console / Prometheus, so this pack deliberately omits them.
Verify it works
Runs cockroach.databases, a low-risk
read that confirms the pack can reach its target. Run it on the host once the pack
is installed; pack install runs it for
you.
sudo emisar pack verify cockroach
Install and configure a pack walks through the whole sequence on a host.
Actions 26 total
View on GitHub-
cockroach.cancel_job exec high High: can disrupt service
Cancel a job (CANCEL JOB)
Cancel one job by id (from jobs). The job stops and rolls back any partial work where applicable. Use to abort a wrong or stuck schema change / backup / import.
View source on GitHub -
cockroach.cancel_query exec high High: can disrupt service
Cancel a running query (CANCEL QUERY)
Stop one in-flight query by its query_id (from cluster_queries). The query ends with an error; the session/connection stays open. Use to kill a runaway query without dropping the client.
View source on GitHub -
cockroach.cancel_session exec high High: can disrupt service
Cancel a session (CANCEL SESSION)
End one client session by its session_id (from cluster_sessions) — its current query is cancelled and the connection closed; an open transaction is rolled back. Use to evict a stuck or abusive connection.
View source on GitHub -
cockroach.cluster_queries exec medium Medium: changes state, easily reversible
Show running queries cluster-wide
Show in-flight queries across all nodes (the SHOW CLUSTER QUERIES surface) — query id, node, user, start time, client address, application, and the query text (truncated). Use to find a long-running or runaway query. Rated medium because live query text can contain literal request values no redaction list can enumerate.
View source on GitHub -
cockroach.cluster_sessions exec medium Medium: changes state, easily reversible
Show open sessions cluster-wide
Show open SQL sessions across all nodes (the SHOW CLUSTER SESSIONS surface) — session id, node, user, client address, application, active queries, and session start. Use to see who is connected and from where. Rated medium because active queries can contain literal request values no redaction list can enumerate.
View source on GitHub -
cockroach.cluster_settings exec medium Medium: changes state, easily reversible
Show cluster settings
Show all cluster settings with their current value, type, and description (SHOW CLUSTER SETTINGS). Use to confirm a setting's value during an investigation. Read-only — this pack does not change settings. Credential values are masked in the output; the setting name stays visible so you can see it is set.
View source on GitHub -
cockroach.cluster_transactions exec low Low: read-only or trivially reversible
Show open transactions cluster-wide
Show open transactions across all nodes — id, node, session, start time, application, statement count, and retry count, from crdb_internal.cluster_transactions. Use to spot long-open or heavily-retried transactions.
View source on GitHub -
cockroach.contention_events exec medium Medium: changes state, easily reversible
Show who is blocking whom (contention events)
Show recent transaction-contention events — blocking vs waiting txn, how long the wait was, and the object (database / table / index) and key contended, from crdb_internal.transaction_contention_events. The core "what's blocking my writes" view. Rated medium because the pretty key can contain real row-key values no redaction list can enumerate.
View source on GitHub -
cockroach.contention_leaderboard exec low Low: read-only or trivially reversible
Show most-contended objects
Show the most-contended tables/indexes by cumulative contention time, from crdb_internal.cluster_contention_events. Use to find the hotspot behind widespread contention.
View source on GitHub -
cockroach.databases exec low Low: read-only or trivially reversible
List databases (SHOW DATABASES)
List the cluster's databases. Cheap connectivity + auth check, and the pack's verify action.
View source on GitHub -
cockroach.jobs exec medium Medium: changes state, easily reversible
Show recent / running jobs
Show running cluster jobs and jobs completed in the past 12 hours (schema changes, backups, restores, imports, row-TTL) from SHOW JOBS — id, type, status, running status, timing, fraction complete, description, and any error. Optionally filter by status. Use to find a stuck or failed job. Rated medium because job descriptions and error text can contain operator-authored values no redaction list can enumerate.
View source on GitHub -
cockroach.locks exec medium Medium: changes state, easily reversible
Show contended locks (lock table)
Show contended entries in the lock table — range, object, the pretty key, holding txn, lock strength, and how long it's been held, from crdb_internal.cluster_locks. Use to find the lock behind a stall. Rated medium because the pretty key can contain real row-key values no redaction list can enumerate.
View source on GitHub -
cockroach.node_decommission exec high High: can disrupt service
Decommission a node (cockroach node decommission)
Mark one node decommissioning and start moving its replicas off. Effectively irreversible once it completes — a fully decommissioned node cannot rejoin under the same id. Uses --wait=none, so the call returns after initiating; track progress with node_status.
View source on GitHub -
cockroach.node_liveness exec low Low: read-only or trivially reversible
Show node liveness / draining / decommissioning
Show per-node liveness from crdb_internal.gossip_liveness — epoch, draining, decommissioning, membership, and last update. Use to see which nodes are live and which are leaving the cluster.
View source on GitHub -
cockroach.node_recommission exec high High: can disrupt service
Recommission a node (cockroach node recommission)
Reverse an in-progress decommission so the node resumes accepting replicas. Only valid before the decommission completes. Use to abort a decommission started by mistake.
View source on GitHub -
cockroach.node_status exec low Low: read-only or trivially reversible
Show node status (liveness, ranges, decommission, disk)
Show every node's status — address, build, liveness, replica/leaseholder counts, range counts (incl. unavailable / under-replicated), liveness bytes, and decommission / draining / membership state. The first stop for cluster health.
View source on GitHub -
cockroach.node_status_one exec low Low: read-only or trivially reversible
Show status for one node
Show full status for a single node by id — same columns as node_status, scoped to one node. Use to inspect a node flagged elsewhere.
View source on GitHub -
cockroach.pause_job exec medium Medium: changes state, easily reversible
Pause a job (PAUSE JOB)
Pause one running job by id (from jobs). It can be resumed later with resume_job. Use to relieve load from a heavy backup / schema change without losing its progress.
View source on GitHub -
cockroach.replication_constraints exec low Low: read-only or trivially reversible
Show replication constraint violations
Show zones whose ranges violate their replication constraints (e.g. a region/locality requirement that can't be met), from system.replication_constraint_stats. Empty means all constraints satisfied.
View source on GitHub -
cockroach.replication_stats exec low Low: read-only or trivially reversible
Show under-replicated / unavailable range counts
Show cluster-wide range health — total, under-replicated, unavailable, and over-replicated range counts, summed from system.replication_stats. Any non-zero unavailable count means data is offline; under-replicated means recovery is in progress.
View source on GitHub -
cockroach.resume_job exec medium Medium: changes state, easily reversible
Resume a paused job (RESUME JOB)
Resume one paused job by id (from jobs). Use to continue a job paused with pause_job.
View source on GitHub -
cockroach.set_cluster_setting exec critical Critical: data loss or irreversible
Set a cluster setting (SET CLUSTER SETTING)
Change one CockroachDB cluster setting cluster-wide — SET CLUSTER SETTING <name> = <value>. An unrestricted config write (rate limits, GC TTLs, feature flags) that can materially change cluster behavior, so risk:critical → default-denied. An operator allows it by explicit policy, and WHICH settings/values are permitted is a portal policy decision, not a hardcoded list. Read cockroach.cluster_settings first to see the current value.
View source on GitHub -
cockroach.statement_stats exec low Low: read-only or trivially reversible
Show top statement fingerprints by exec count
Show the busiest statement fingerprints — application, execution count, mean service latency, and normalized query shape — from crdb_internal.statement_statistics (the data behind the DB Console's Statements page). Stays low because CockroachDB fingerprints replace literal values with placeholders, so this returns query shape rather than request data. Use to find the highest-volume or slowest statements.
View source on GitHub -
cockroach.store_status exec low Low: read-only or trivially reversible
Show per-store disk capacity / range counts
Show each store's capacity, available and used bytes, range and lease counts, and writes/sec from crdb_internal.kv_store_status. Use to spot a node running low on disk or carrying too many ranges.
View source on GitHub -
cockroach.table_ranges exec low Low: read-only or trivially reversible
Show range count + size for one table
Show the ranges backing one table — range id, leaseholder, size in MB, and replica nodes — via SHOW RANGES ... WITH DETAILS. Use to find a table's data size and how its ranges are spread across nodes.
View source on GitHub -
cockroach.table_row_counts exec low Low: read-only or trivially reversible
Show estimated row counts per table
Show estimated row counts for every table from crdb_internal.table_row_statistics, largest first. Cheap (uses table statistics, not a COUNT). Use to find the biggest tables.
View source on GitHub