Apache Spark

Install

emisar pack install fetches this pack, re-validates it, and verifies its content hash against the --hash below — the exact bytes this page was rendered against, so a tampered copy is rejected — before copying it into the runner's packs dir. The command reloads a running daemon itself; no manual restart.

sudo emisar pack install spark --hash sha256:6b24467c62bd6eff06d69bd43e470ad27722a3ffd78367ea33ebaf54b0f4334b

• spark.application script low

  GET /api/v1/applications/{app_id}

  Show one Spark application — its name, user, and every attempt with start time, end time, duration, and whether it completed. Read it to confirm an application id and see whether the run you care about finished or is still going.

  View source on GitHub
• spark.applications script low

  List Spark applications (GET /api/v1/applications)

  List Spark applications with their id, name, user, start and end times, and attempts. Against the history server this is the record of what ran; against a live driver it returns the one application that driver is running. Every other action in this pack takes the app_id this returns.

  View source on GitHub
• spark.environment script high

  Dump an application's Spark configuration (GET .../environment)

  Dump the Spark configuration an application is running with, plus its runtime versions and resource profiles. Answers "what settings did this job actually get" when a tuning change did not take effect. High risk because it is a configuration dump: Spark's own spark.redaction.regex masks the property values it recognizes, and the JVM system properties and classpath are dropped on the runner host, but an object-store key or keystore password stored under a property name that pattern does not match is returned as configured.

  View source on GitHub
• spark.executor_threads script medium

  Dump an executor's threads (GET .../executors/{executor_id}/threads)

  Dump the stack traces of every thread in one live executor. This is the read for a task that is running but making no progress — the stack says whether it is blocked on a lock, waiting on a shuffle fetch, or spinning in user code. Only a live driver serves it; the history server does not. Heavier than the other reads: collecting the dump briefly pauses the executor's threads.

  View source on GitHub
• spark.executors script low

  List an application's executors (GET .../executors)

  List an application's executors with cores, active and completed tasks, failed tasks, GC time, storage memory used against total, disk used, and their host and log URLs. Read it when an application is slow or losing work: a single executor with most of the failed tasks names the bad host.

  View source on GitHub
• spark.job script low

  GET /api/v1/applications/{app_id}/jobs/{job_id}

  Show one Spark job — status, timing, the stage ids it owns, and its task counts including failed and killed. Read it after spark.jobs narrows to the job you care about, then follow its stage ids into spark.stage.

  View source on GitHub
• spark.job_kill script high

  Kill a running Spark job (POST /jobs/job/kill on the driver UI)

  Kill one running job in the application on the live driver UI. Every task in the job is cancelled and the job is marked failed; whatever the job had already written stays written, so a partially completed output is the normal outcome. The action reads the job back afterwards and returns its state, so a request against an already-finished job is visible rather than assumed. Needs spark.ui.killEnabled, which is on by default.

  View source on GitHub
• spark.jobs script low

  List an application's jobs (GET /api/v1/applications/{app_id}/jobs)

  List a Spark application's jobs with their status, submission time, duration, and task counts — completed, active, skipped, and failed. Filter by `status: failed` to find the job that broke, or `status: running` to see what a stuck application is still working on.

  View source on GitHub
• spark.master_app_kill script high

  Kill an application from the standalone master (POST /app/kill)

  Kill one whole application on a Spark standalone master. Its driver and every executor are terminated and its cores and memory return to the cluster — which is the point when one application is starving everything else — but the application's work stops wherever it was, so partial output stays partial. The action reads the master's state back and reports whether the application is still active.

  View source on GitHub
• spark.master_driver_kill script high

  Kill a cluster-mode driver from the standalone master (POST /driver/kill)

  Kill one cluster-mode driver on a Spark standalone master. The driver process and the application it runs are terminated. Use it for a submitted job whose client is long gone — a driver stuck in a retry loop, or one submitted by mistake. The action reads the master's state back and reports whether the driver is still active.

  View source on GitHub
• spark.master_state script low

  Show standalone cluster state (GET /json/ on the master)

  Show a Spark standalone master's cluster state — every worker with its state, cores and memory used against total, the running and completed applications with the resources each holds, and any submitted drivers. This is the capacity read: an application stuck in WAITING with no free cores means the cluster is full, not that the application is broken.

  View source on GitHub
• spark.master_worker_decommission script high

  Decommission a standalone worker host (POST /workers/kill on the master)

  Decommission every standalone worker on one host. Spark stops scheduling new tasks there and migrates shuffle and cached blocks off before the executors go away, which is the graceful way to take a node out for maintenance or a scale-down. Still a capacity change: the cluster loses that host's cores, and running tasks on it are rescheduled elsewhere. Needs spark.decommission.enabled and an allowed value of spark.master.ui.decommission.allow.mode; both are off or restrictive by default.

  View source on GitHub
• spark.sql_execution script low

  GET .../sql/{execution_id}

  Show one Spark SQL execution with its node-level metrics — rows produced, bytes read, shuffle sizes, and time per operator — and optionally the physical plan. This is where a query's cost is attributed: the node with the row explosion or the full scan is named here.

  View source on GitHub
• spark.sql_executions script low

  List SQL executions (GET .../sql)

  List an application's Spark SQL executions with their description, submission time, duration, and the jobs each one spawned. For a SQL or DataFrame workload this is the layer that maps a slow query to the jobs and stages underneath it.

  View source on GitHub
• spark.stage script low

  GET /api/v1/applications/{app_id}/stages/{stage_id}

  Show every attempt of one stage with its status, task counts, timing, shuffle and spill metrics, and — with with_summaries — the task metric quantiles. A stage with more than one attempt has been retried, which usually means lost executors or a fetch failure.

  View source on GitHub
• spark.stage_kill script high

  Kill a running Spark stage (POST /stages/stage/kill on the driver UI)

  Kill one running stage in the application on the live driver UI. Narrower than killing the job: the stage's tasks are cancelled, and whether the job survives depends on whether Spark can retry the stage. Use it to shed one runaway stage — a skewed shuffle, a stage stuck on a dead host — without losing the whole job. The action reads the stage back and returns its attempts. Needs spark.ui.killEnabled, which is on by default.

  View source on GitHub
• spark.stage_task_summary script low

  Get task metric quantiles for a stage attempt (GET .../taskSummary)

  Get the task metric quantiles for one stage attempt — duration, GC time, shuffle read and write, input bytes, and memory and disk spill at each quantile. This is the skew read: when the maximum task duration dwarfs the median, the stage is waiting on a few partitions, not on the cluster.

  View source on GitHub
• spark.stage_tasks script low

  List tasks in a stage attempt (GET .../taskList)

  List individual tasks in one stage attempt with their executor, host, duration, status, and — for a failed task — the error message. Sort by `-runtime` to put the slowest first, or filter `status: failed` to read why the stage broke.

  View source on GitHub
• spark.stages script low

  List an application's stages (GET /api/v1/applications/{app_id}/stages)

  List a Spark application's stages with status, task counts, duration, shuffle read and write bytes, spill, and input and output sizes. This is where a slow job is diagnosed: the stage with the outsized shuffle read or spill is usually the one to look at.

  View source on GitHub
• spark.storage_rdds script low

  List cached RDDs (GET .../storage/rdd)

  List the RDDs and DataFrames an application has cached, with their storage level, partition count, and how many bytes sit in memory versus spilled to disk. Read it when executors are short on memory: a large cache that is mostly on disk is paying the cost of caching without the benefit.

  View source on GitHub
• spark.submission_kill script high

  Kill a REST submission (POST /v1/submissions/kill/{id})

  Kill one cluster-mode submission through the standalone REST submission server. The driver and its application are terminated, and the server answers with whether it found and killed the submission. This is the path for a job submitted with `spark-submit --deploy-mode cluster` when you have the submission id but not the master UI. Needs spark.master.rest.enabled, which is off by default.

  View source on GitHub
• spark.submission_status script low

  Get a REST submission's status (GET /v1/submissions/status/{id})

  Get the state of one cluster-mode submission from the standalone REST submission server — whether the driver is queued, running, finished, or failed, and the worker it landed on. Use it to follow a `spark-submit --deploy-mode cluster` job whose client already exited. Needs spark.master.rest.enabled, which is off by default.

  View source on GitHub
• spark.version script low

  GET /api/v1/version

  Show the Spark version a UI is running. Use it to confirm a driver or history server is reachable and which release it is on before reading a version-specific field.

  View source on GitHub