Google Cloud Dataproc operations
v0.1.3Inspect classic Dataproc clusters, daemon metrics, jobs, failures, retained driver logs, operations and regional autoscaling policies, then govern narrow start, stop, resize, cancellation and deletion controls. Selected projections exclude arbitrary configuration and submitted job payloads. Serverless batches, cluster creation, job submission and policy writes are outside this pack.
bash
gcloud
curl
jq
python3
date
sha256sum
Install
New to emisar? Follow the quickstart to connect your host and AI agent, then install this pack below. Agents can use its actions through the CLI or MCP.
emisar pack install
downloads and checks the pack, then compares its content hash with the
--hash
below before copying it into the runner's packs directory. The hash identifies
the exact pack shown on this page. If the download was changed, the command rejects
it. It also reloads a running runner, so you do not need to restart it yourself.
sudo emisar pack install gcp-dataproc --hash sha256:9b434d14815386ae0c3611779b83a70991e9ccd3e6d904137f8ee54abf8437ae
Setup
Uses the runner host's gcloud identity. On Google Compute Engine, attach a dedicated service account with the cloud-platform OAuth scope and the permissions below; gcloud obtains its short-lived credential without a key file. The actions send it through curl stdin to the Dataproc and Cloud Storage HTTPS APIs, never argv or output.
Environment
Set these on the runner host, then add each name to
execution.inherit_env
so the value reaches the action.
-
CLOUDSDK_CONFIGOptional dedicated gcloud configuration directory.
-
CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDEOptional workload-identity or service-account credential file for hosts without an attached identity.
Notes
- Enable the Dataproc API in the target project. Ordinary metadata and daemon metric reads need dataproc.clusters.get/list, dataproc.jobs.get/list, dataproc.operations.get/list and dataproc.autoscalingPolicies.get/list. A custom role can grant exactly the actions the runner needs.
- Driver log tails additionally require storage.objects.list and storage.objects.get on the provider-selected staging bucket. Python 3 runs the isolated bounded reader; caller arguments never select a bucket or object. Logs may contain sensitive application text and follow the account policy for medium-risk reads.
- The controls additionally need dataproc.clusters.start/stop/update/delete and dataproc.jobs.cancel/delete. No cluster creation, job submission, service-account impersonation or iam.serviceAccounts.actAs grant is required by this action surface.
-
For a host without an attached identity, create a dedicated gcloud configuration with
gcloud auth login --cred-file=...using a workload-identity configuration or a service-account credential from Google Cloud IAM → Service Accounts. Allowlist configured credential/config variables inexecution.inherit_env; ordinary gcloud authentication does not use GOOGLE_APPLICATION_CREDENTIALS. - Targets are explicit project and region arguments. Remote authentication is not host authority; no protected-file, sudo or root grant is needed. This remote pack has no host detection signal.
- Requires curl 7.76 or newer. Read pages contain at most six rows to stay within the 8 KiB structured-result budget; follow next_page_cursor with unchanged selectors. Long or unsupported provider cursors are omitted with cursor_omitted=true.
- Driver log reads accept an explicit byte offset from the current listed tail to inspect earlier windows without fetching skipped objects. At most 4 KiB is fetched and 1 KiB is emitted before redaction; the offset is not a gap-free cursor and fetched bytes are not a continuation step.
- Lifecycle operations are asynchronous. A successful request is acceptance, not completion; inspect the returned operation or job until it reaches a terminal state. Start, stop and delete requests require the UUID from cluster_details to protect against a reused name. Worker resize uses the cluster name and a single worker-count update mask; it has no UUID argument.
Verify it works
Runs gcp-dataproc.clusters_list, a low-risk
read that confirms the pack can reach its target. Run it on the host once the pack
is installed; pack install runs it for
you.
sudo emisar pack verify gcp-dataproc
Install and configure a pack walks through the whole sequence on a host.
Actions 18 total
View on GitHub-
gcp-dataproc.autoscaling_policies_list script low Low: read-only or trivially reversible
List Dataproc autoscaling policies
List one page of regional autoscaling policies with exact IDs, primary and secondary bounds, weights and cooldown. Return no labels or arbitrary policy fields.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes page_size integer No page_cursor string No See the action source for defaults, validation, and limits.
-
gcp-dataproc.autoscaling_policy_details script low Low: read-only or trivially reversible
Show a Dataproc autoscaling policy
Show one regional autoscaling policy's worker limits, weights, cooldown and YARN scale factors and graceful decommission timeout. Identify Spark Standalone policies but leave their YARN-only factors null. No policy is changed.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes policy string Yes See the action source for defaults, validation, and limits.
-
gcp-dataproc.cluster_delete script high High: can disrupt service
Delete a Dataproc cluster
Delete one classic cluster guarded by its exact UUID. Destroy cluster VMs and cluster-local data and interrupt jobs. External Cloud Storage data is not deleted by this action. Return the operation without waiting; inspect operation_details for completion.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes cluster string Yes cluster_uuid string Yes See the action source for defaults, validation, and limits.
-
gcp-dataproc.cluster_details script low Low: read-only or trivially reversible
Show a Dataproc cluster
Show one classic cluster's state, UUID, capacity, machine types, attached service-account email, image, autoscaling policy ID and deletion schedule. Return selected metadata only; arbitrary properties, labels, startup configuration and credentials are excluded. Free-text fields are clipped to their schema bounds.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes cluster string Yes See the action source for defaults, validation, and limits.
-
gcp-dataproc.cluster_metrics script low Low: read-only or trivially reversible
Show Dataproc HDFS and YARN metrics
Show the cluster daemon HDFS and YARN metric snapshot from clusters.get. Preserve int64 values as decimal strings; return up to eight metrics per family with omission counts. Missing metrics report available=false, never zero usage. Metric names are clipped at 48 characters with an explicit flag. This vendor snapshot is a beta field and is not a historical Cloud Monitoring query.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes cluster string Yes See the action source for defaults, validation, and limits.
-
gcp-dataproc.cluster_resize_secondary_workers script high High: can disrupt service
Resize Dataproc secondary workers
Resize only the secondary worker count on one RUNNING classic cluster, including scaling to zero. Patch config.secondary_worker_config.num_instances and no other field. Shrinking can interrupt work after the graceful deadline; growing incurs VM charges. An attached autoscaling policy can change capacity again. Return the operation without waiting.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes cluster string Yes secondary_workers integer Yes graceful_seconds integer No See the action source for defaults, validation, and limits.
-
gcp-dataproc.cluster_resize_workers script high High: can disrupt service
Resize Dataproc primary workers
Resize only the primary worker count on one RUNNING classic cluster. Patch config.worker_config.num_instances and no other field. Shrinking can interrupt work after the graceful deadline; growing incurs VM charges. An attached autoscaling policy can change capacity again. Return the operation without waiting; inspect operation_details for completion.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes cluster string Yes workers integer Yes graceful_seconds integer No See the action source for defaults, validation, and limits.
-
gcp-dataproc.cluster_start script high High: can disrupt service
Start a Dataproc cluster
Start one stopped classic cluster guarded by its exact UUID. Return the long-running operation without waiting; inspect operation_details for completion. Starting resumes VM billing and scheduled work can resume.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes cluster string Yes cluster_uuid string Yes See the action source for defaults, validation, and limits.
-
gcp-dataproc.cluster_stop script high High: can disrupt service
Stop a Dataproc cluster
Stop one running classic cluster guarded by its exact UUID. Return the long-running operation without waiting; inspect operation_details for completion. Stopping interrupts cluster workloads; it is not job cancellation or deletion.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes cluster string Yes cluster_uuid string Yes See the action source for defaults, validation, and limits.
-
gcp-dataproc.clusters_list script low Low: read-only or trivially reversible
List Dataproc clusters
List classic Dataproc clusters in one project and region with state, worker counts and image version. Return one bounded page and an explicit continuation; no software properties, labels or metadata are returned.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes page_size integer No page_cursor string No See the action source for defaults, validation, and limits.
-
gcp-dataproc.job_cancel script high High: can disrupt service
Cancel a Dataproc job
Request cancellation of one Dataproc job. The returned state can still be CANCEL_PENDING or CANCEL_STARTED; inspect job_details until cancellation finishes. Cancellation interrupts the workload and does not roll back its external writes.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes job string Yes See the action source for defaults, validation, and limits.
-
gcp-dataproc.job_delete script high High: can disrupt service
Delete a Dataproc job record
Delete one terminal Dataproc job record. Active jobs are rejected by Dataproc and must be cancelled separately. Remove the job resource used for inspection; this action does not delete output data or cluster VMs.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes job string Yes See the action source for defaults, validation, and limits.
-
gcp-dataproc.job_details script low Low: read-only or trivially reversible
Show a Dataproc job
Show one job's lifecycle, placement and up to four YARN application states and progress values. Return no job arguments, properties, tracking URLs or driver-output locations. Free-text fields are clipped and omitted applications are counted. An unavailable placement UUID is null, including while placement is pending.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes job string Yes See the action source for defaults, validation, and limits.
-
gcp-dataproc.job_driver_log script medium Medium: changes state, easily reversible
Tail a retained Dataproc job driver log
Tail raw application driver output, which can contain personal data, requests or credentials; the runner applies its normal log redaction and account approval policy. Resolve the current job's output-only driver URI internally, list at most five pages of 1,000 chunk metadata rows, and read at most four nonempty, generation-pinned chunks ending at the selected tail offset from Cloud Storage. Skip later chunks using metadata only and use explicit byte ranges for windows ending before an object EOF. Return at most the requested lines and bytes, additionally capped at 1,024 UTF-8 bytes before redaction to leave room for expanding secret masks. Return explicit truncation and partial-first-line flags; JSON escaping can further reduce the tail, and counts precede runner redaction. A skipped tail is truncated but does not itself make the first line partial; that flag describes a possibly incomplete prefix. Offsets select windows in the current listed stream and are not gap-free continuation cursors; fetched bytes can exceed emitted bytes. Window boundaries can split lines or UTF-8 characters; invalid UTF-8 is replaced. Support classic ASCII gs:// driver prefixes and numeric chunk suffixes, including logs retained after cluster deletion. Missing job metadata, unsupported URIs, incomplete listings, chunk gaps, encoded objects and changed snapshots fail closed. No bucket, object or URL is caller-selected.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes job string Yes lines integer No bytes integer No skip_tail_bytes integer No See the action source for defaults, validation, and limits.
-
gcp-dataproc.job_errors script medium Medium: changes state, easily reversible
Read Dataproc job failure details
Read bounded raw job failure text from status.details and the last four lifecycle states. Error text may contain application data or credentials and receives the runner's redaction; it is not a driver log fetch. Return at most 512 characters of details and count omitted history entries.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes job string Yes See the action source for defaults, validation, and limits.
-
gcp-dataproc.jobs_list script low Low: read-only or trivially reversible
List Dataproc jobs
List retained Dataproc jobs in one region, including jobs from deleted clusters, with optional exact cluster, state and insertion-time selectors. Each call reads one provider page of up to 100 jobs when selecting a cluster and returns up to six matching summaries plus a continuation, scanned count and unreachable count. An empty page with more_available is partial coverage. Job arguments, properties, driver output and submitted payloads are excluded.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes page_size integer No page_cursor string No cluster_filter string No state string No start_time string No end_time string No See the action source for defaults, validation, and limits.
-
gcp-dataproc.operation_details script medium Medium: changes state, easily reversible
Read a Dataproc operation result
Read one operation's completion, cluster, type and bounded provider error message. The message may include submitted application text and receives runner redaction. A pending operation is not completion proof; a done operation with an error code is a failed operation. Return no response payload or error details envelope.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes operation string Yes See the action source for defaults, validation, and limits.
-
gcp-dataproc.operations_list script low Low: read-only or trivially reversible
List Dataproc operations
List one page of regional Dataproc operations with IDs, lifecycle state, type, completion flag and error code. Return only selected operation metadata, never response payloads or raw error messages.
Link to this action View source on GitHubArguments Name Type Required project string Yes region string Yes page_size integer No page_cursor string No See the action source for defaults, validation, and limits.