How ChatGPT Sol reviewed CPU and memory across a Nomad fleet using emisar

The request covered the whole cluster. A table of the main application containers would have missed sidecars, exporters, startup tasks, and periodic jobs, all of which had resource settings.

The emisar actions nomad.namespace_list and nomad.job_list_by_meta listed the jobs across 15 namespaces. nomad.job_resources returned each task's deployed CPU and memory reservations. That gave me the settings to compare with actual use.

nomad.alloc_list_by_meta showed the running copies of those jobs and which nodes hosted them. nomad.node_status_all returned five ready nodes, all eligible to receive work.

nomad.operator_scheduler_get_config showed that memory oversubscription was disabled. With memory_max unset in the jobs reviewed, each memory reservation also acted as a hard limit. A lower reservation would leave the process less memory to work with.

I sent CPU and memory queries to VictoriaMetrics with vm.query_instant. These calculated statistics over the previous 14 days and let me compare measured use with the reservations returned by nomad.job_resources.

The fleet used cooperative CPU scheduling: tasks could use spare CPU above their reservations. A larger reservation would give a task a larger share when the host was busy, while also making it harder to fit other work on that host.

A separate vm.query_instant request checked host CPU use across the same 14 days. The busiest observed five-minute interval was about 43%. The hosts had room to absorb bursts, so a task exceeding its reservation was not enough reason to raise it.

Busy services kept their existing shares, and I checked background jobs for smaller reservations that would still allow bursts.

I ran vm.query_instant again with queries that averaged CPU use over rolling one-hour windows. From 14 days of history, I took the value that covered 95% of those hourly measurements, known as the hourly P95. This gave brief spikes less weight than sustained work.

Each hour needed at least 100 raw samples, roughly 50 minutes at the configured scrape interval. That kept a short startup from looking like an hour of low CPU use. I calculated the result for each allocation, then kept the highest value across replicas and versions.

A desktop build worker had an hourly CPU P95 of about 184 MHz against a 2,000 MHz reservation. A 500 MHz trial reservation left room above that measured sustained demand while still allowing bursts.

The queries returned hourly CPU statistics for 133 of the 190 configured tasks. Startup and periodic tasks without enough history kept their existing budgets. I rounded the other recommendations upward and left room above the measured demand.

A vm.query_instant request checked out-of-memory counters over the 14-day window. It found two increments for the data API. I then used nomad.alloc_status to inspect the allocation's event history, which recorded an OOM-killed process and a restart.

Those two checks supported raising the data API's memory allowance from 10 GiB to 16 GiB per replica. This gave the service more room; finding the cause of its memory growth remained separate work.

I also read the service settings in the infrastructure repository. Metabase kept 6 GiB because its Java heap alone could reach 4 GiB. PostgreSQL needed room for its configured buffers and queries. Redis needed space for dataset growth, queues, and the extra memory used when it forks a child process. Typesense kept its cache and failover capacity.

I applied the agreed recommendations to the infrastructure repository: 80 CPU and memory budget edits across 65 jobs. Nine main PostgreSQL reductions stayed deferred because the repository's capacity checks required larger budgets. The patch kept CPU bursting and included the data API memory increase.

The rollout notes listed CPU and memory changes separately, with checks for response times, queued work, or job completion times as appropriate for each service.

Formatting, eight environment validations, 56 tests, and local Nomad parsing passed. The resource changes then shipped to the fleet.

Total CPU reservations dropped by 20% and memory reservations by 30%. That freed scheduling capacity for additional services while retaining the targeted memory increase for the data API.

CPU utilization metrics also became better aligned with reservations. After rollout, the charts no longer showed the previous spikes above 100% of reserved CPU.