Linux debugging toolkit

Install

emisar pack install fetches this pack, re-validates it, and verifies its content hash against the --hash below — the exact bytes this page was rendered against, so a tampered copy is rejected — before copying it into the runner's packs dir. The command reloads a running daemon itself; no manual restart.

sudo emisar pack install debugging --hash sha256:ae3b9a7c8f11eb1de5099024adee2e5d0231bdcc6ebe9276d5d93073cb965dda

• debugging.disk_free exec low

  df + mounts

  Returns `df -hT` output. Filesystem type, size, used, avail, and mountpoint for every mounted filesystem. Use as the first check when a write fails with ENOSPC or when /var/log has gone dark.

  View source on GitHub
• debugging.dmesg_oom exec low

  OOM-kill events from dmesg

  Filters dmesg for OOM-killer events. Returns the kernel log lines showing process id, name, RSS, and OOM score for every killed process. The "why did mysqld vanish?" answer. Falls back to `journalctl -k` when dmesg is not permitted (needs CAP_SYSLOG / root, or journal read access via systemd-journal / adm). Read-only.

  View source on GitHub
• debugging.dmesg_tail exec low

  Recent kernel messages

  Returns the last N kernel log lines. Surfaces OOM kills, hardware errors, network link flaps, and dropped packets. Reads the kernel ring buffer via dmesg; when that is not permitted (modern kernels gate it behind CAP_SYSLOG) it falls back to `journalctl -k`, which works when the runner can read the journal (root, or a member of systemd-journal / adm). Read-only.

  View source on GitHub
• debugging.drop_caches exec high

  echo <n> > /proc/sys/vm/drop_caches

  Force the kernel to drop pagecache / dentries / inodes. Use only when a benchmark or memory-fragmentation test requires a cold cache — never on prod for "I want more free RAM". Production RSS appears to drop briefly, then the cache repopulates and the next workload is slower until it warms back up.

  View source on GitHub
• debugging.iostat exec low

  iostat per-device sample

  Runs `iostat -xz 1 N` to capture N one-second samples of extended per-device statistics. Surfaces await, %util, queue depth. Use to prove or disprove "the disk is slow" before chasing application code.

  View source on GitHub
• debugging.kernel_taint exec low

  Kernel taint state

  Reads /proc/sys/kernel/tainted. A non-zero value means a binary module, a proprietary driver, or a kernel crash has compromised the integrity of the running kernel. The number is a bitmask; this action returns both the raw value and the decoded flags.

  View source on GitHub
• debugging.kill_pid exec high

  kill -<signal> <pid>

  Send a signal to one process by PID. SIGTERM is the polite default — the process gets a chance to flush state. SIGKILL is unrecoverable — use only when SIGTERM is ignored. Watch for PID reuse: confirm the target via `pid_status` first.

  View source on GitHub
• debugging.loadavg exec low

  Load + memory + uptime snapshot

  Reads /proc/loadavg, /proc/meminfo, and /proc/uptime to produce a one-shot system snapshot. Cheap. Use as the very first check when triaging a host alert.

  View source on GitHub
• debugging.lsof_port exec low

  Who owns a TCP port?

  Returns the PID/process that has a given TCP port open (listening or connected), via `ss -tnp` (not lsof, despite the name). Use to answer "EADDRINUSE: who's on 8080?" or to confirm a stuck connection to an upstream. Read-only.

  View source on GitHub
• debugging.mem_top exec low

  Top processes by RSS

  Returns the top N processes sorted by resident-set size (RSS). Use when /proc/meminfo or `free` shows pressure and you need the offender. RSS does not double-count shared pages, so a "leak" candidate showing high RSS is worth investigating.

  View source on GitHub
• debugging.netstat_connections exec low

  Established connection summary

  Returns counts of TCP connections grouped by remote peer + state. Useful for spotting connection storms (single host) or TIME_WAIT pressure. Read-only.

  View source on GitHub
• debugging.netstat_listen exec low

  Listening sockets

  Returns TCP and UDP listening sockets with the owning process (`ss -tulnp`). Use to confirm whether an expected daemon is actually bound and on which interfaces. Read-only.

  View source on GitHub
• debugging.pid_cwd exec low

  Process cwd + exe

  Returns the working directory and executable path of a PID. Use before drawing conclusions from a process name — `nginx` could be any of several binaries depending on PATH order.

  View source on GitHub
• debugging.pid_environ exec medium

  Process environment

  Show the full set of environment variables a PID was started with — reads /proc/<pid>/environ and turns NULs into newlines. This deliberately surfaces the process's entire environment, which commonly carries injected secrets (DB URLs, API keys, cloud credentials); scope it by policy and prefer pid_status / pid_limits when you don't need the values. The runner's redaction is a fail-closed backstop, not a guarantee — it is pattern-bound and can miss a bespoke secret whose name and value match no rule.

  View source on GitHub
• debugging.pid_fds exec low

  Process open file descriptors

  Lists what each FD in /proc/<pid>/fd points at. Sockets show up as `socket:[N]`, files as their path. Use to spot FD-leak candidates or confirm a daemon has the log file you expect.

  View source on GitHub
• debugging.pid_io exec low

  Process I/O accounting

  Reads /proc/<pid>/io — bytes read/written (logical + physical), syscall counts, cancelled writes. Use to find which process is driving disk I/O. Pair with `debugging.iostat` for the disk side.

  View source on GitHub
• debugging.pid_limits exec low

  Resource limits applied to a PID

  Reads /proc/<pid>/limits — every ulimit applied (open files, stack size, NPROC, memlock, msgqueue, niceness). Use to confirm whether a process is actually getting the higher limits its service file requested.

  View source on GitHub
• debugging.pid_stack exec low

  Process kernel stack

  Reads /proc/<pid>/stack — the current kernel-side stack trace. Use to find what syscall a stuck process is hung in (futex, read, write, lock_kernel). Needs CAP_SYS_ADMIN or root to read.

  View source on GitHub
• debugging.pid_status exec low

  Process /proc status block

  Dumps /proc/<pid>/status — capabilities, uid/gid, signal masks, RSS, peak RSS, voluntary/involuntary context switches, OOM score. More detail than `ps` for one PID. Read-only.

  View source on GitHub
• debugging.pid_threads exec low

  Per-thread stats for one PID

  Lists every thread of one PID with CPU%, policy, priority, comm name. Use when one PID's CPU is high but it's unclear which thread inside it is hot. Read-only.

  View source on GitHub
• debugging.ping_host exec low

  Ping a host

  Sends N ICMP echo requests to a target. Use to confirm L3 reachability when a higher-layer probe (TCP/HTTP) fails. Target is pattern-restricted to safe hostname/IPv4 characters to prevent argument injection.

  View source on GitHub
• debugging.processes_top exec low

  Top processes by CPU

  Returns the top N processes sorted by CPU%. Standard `ps` output — pid, user, %cpu, %mem, rss, command. Use as a first-touch check before going deeper with per-PID inspection.

  View source on GitHub
• debugging.sar_recent exec low

  sar 3-sample CPU snapshot

  `sar -u 5 3` — three 5-second CPU utilization samples. Surfaces user/system/iowait/steal/idle split with statistical smoothing. Use when `vmstat`'s noise hides the signal. Requires the sysstat package.

  View source on GitHub
• debugging.slabtop exec low

  Kernel slab cache top consumers

  `slabtop -o -s c | head -40` — top 40 kernel slab caches by cache size. Use when /proc/meminfo shows high `Slab` but no userland process accounts for the memory. dentry / inode pressure is the usual answer.

  View source on GitHub
• debugging.socket_summary exec low

  Socket counts by family + state

  `ss -s` — aggregate counts: TCP/UDP/raw/frag, timewait, by state. Cheaper than the per-connection enumeration. Use as a one-shot "are we close to a port-tuple exhaustion?" check.

  View source on GitHub
• debugging.swap_status exec low

  Swap usage summary

  `swapon --show` plus the per-process swap usage (top 20). Read-only. Use to spot a host actively swapping — swap-in I/O is the slowest userland-visible memory tier.

  View source on GitHub
• debugging.sysctl_set exec high

  sysctl -w <key>=<value>

  Change a runtime kernel parameter. Change is not persistent — reverts at next boot unless mirrored in /etc/sysctl.d/. Wrong values can crash the kernel (net.* tunables especially). Read the current value first.

  View source on GitHub
• debugging.tcp_retrans_top exec low

  TCP connections with retransmits

  `ss -i state established` filtered to flows showing retrans counters. Use when network latency is high — surfaces which peers are seeing TCP loss without a tcpdump. Read-only.

  View source on GitHub
• debugging.tcp_summary exec low

  TCP state counts

  Returns the count of TCP sockets in each state (ESTAB, TIME-WAIT, CLOSE-WAIT, FIN-WAIT-*, SYN-*). High CLOSE-WAIT usually means the application isn't close()ing; high TIME-WAIT means short-lived client connections. Read-only.

  View source on GitHub
• debugging.top_open_files exec low

  Top processes by open-file count

  Aggregates `lsof` by PID and returns the top 20 PIDs holding the most file descriptors. Use to find FD-leak candidates before EMFILE breaks something. Read-only.

  View source on GitHub
• debugging.vmstat exec low

  vmstat sample

  Runs `vmstat 1 N` to capture N one-second samples. Shows run/block queues, free memory, swap pressure, context switches, and per-CPU user/system/iowait. Use to spot a CPU-bound vs IO-bound vs context-switch-storm problem.

  View source on GitHub