How ChatGPT Sol, Astra and Terra used emisar to migrate from GKE to bare metal

I read the application code and deployment configuration, then used the existing Nomad environments in the same infrastructure repository as a reference. I adapted that setup for the applications moving from Google Cloud and prepared their development and production deployments.

We deployed these application copies on Nomad while they still connected to Cloud SQL. Public traffic continued to go to Google Cloud. This let us check the new application deployments before moving their databases and traffic.

The emisar action gcp.sql_instances returned four running PostgreSQL instances across development and production. These were the source databases the applications still used.

nomad.job_list_by_meta listed the staged application jobs, and nomad.job_status_one returned their allocation counts and deployment results. These checks confirmed the destination applications were running before we began the database move.

I wrote Terraform changes to create two PostgreSQL targets for development and two for production. These included persistent storage, SQL health checks, backups, and private migration connections to Cloud SQL. I tested the configuration and published separate development and production pull requests in the infrastructure repository.

I then used tfc.plan_summary to inspect the plans for those changes. Each development and production plan added 26 resources, with no updates, deletions, or replacements. The new targets could be built alongside the existing databases before copying data or switching applications.

We brought up the two development targets first. nomad.node_status_all returned five ready nodes. nomad.csi_volume_status showed that a new PostgreSQL volume could be scheduled and that the storage plugin was healthy on all five nodes. consul.service_health returned passing TCP and SQL readiness checks.

Once the development targets passed their checks, we proceeded with the separate production plan. After approval, tfc.apply_run queued it. I repeated tfc.run_details until it returned applied. Then tfc.create_plan_only_run started a follow-up plan, and tfc.run_details confirmed that it finished with no changes.

A vm.query_instant request returned successful full-backup timestamps for all four development and production PostgreSQL targets. The database infrastructure was ready for the initial data copy.

Before starting the copy, I used vm.query_instant to check the replication watcher's source-query results and last-success timestamps.

We copied development data first, then production data. Six application databases lived across the four PostgreSQL targets. The initial copy covered 281 tables; checks found matching row counts, zero replication lag, and no replication errors. Google Cloud remained the writer, and replication kept the new copies current.

While replication continued, grafana.alerting_state returned alerts for a failing watcher query and a missing or inactive replication slot.

nomad.job_allocations found the watcher's current allocation. nomad.alloc_status showed it running with zero restarts and a healthy deployment. But nomad.alloc_logs returned repeated records with query_success: 0 and metrics_pushed: 1. The process was publishing metrics even though its source queries were failing.

In one database, a separate check found the existing 46 replicated tables caught up, but four new source tables were missing from the replica. We added their schema, extended the existing publication and subscription, and updated the watcher. Follow-up checks found all 50 tables in that database ready, zero lag, and no active replication alerts. The applications still used Cloud SQL while these copies stayed in sync.

On the operator's cutover command, we stopped writers in GKE and the staged Nomad applications, paused scheduled jobs, and synchronized Redis and MQTT broker state.

I used the emisar action nomad.job_allocations to locate the six replication watchers and nomad.alloc_logs to check their progress. All six showed zero lag. We then disabled the PostgreSQL subscriptions and aligned the target sequences so new rows would receive the correct IDs.

I switched the application configuration to VA1 PostgreSQL and started the required applications, workers, and schedules. The net.http_probe checks against VA1 passed before we moved the nine public DNS records. The applications now wrote to the VA1 databases, and public traffic reached Nomad. The old GKE writers remained stopped.

After the traffic switch, I updated the database service-discovery templates to supply the current PostgreSQL addresses. Final net.http_probe checks returned HTTP 200 from development and production web, admin, health, and library endpoints. MQTT checks and scheduled jobs also passed.

After cutover was complete, we scaled six application and authentication services to two replicas each, placed on separate nodes. One service moved its sessions into Redis so requests could reach either replica. Two telemetry collectors ran on separate hosts and sent application metrics to VictoriaMetrics.

Repeated net.http_probe checks recorded a 39-second API 502 window during the first scaling rollout. Both API replicas restarted together after Consul membership changed. I added a two-minute restart stagger, and the corrective rollout stayed healthy throughout the checks.

I added a production dashboard with 31 panels. I used vm.query_instant to check the live service and exporter metrics. The results showed all six services with two running replicas each, none under-replicated, and two healthy metrics exporters. Repeated net.http_probe checks covered development and production web, admin, and API endpoints.

With VA1 serving production, I removed the obsolete GKE delivery path from five application repositories. The old writers and scheduled jobs had remained stopped since cutover.

We removed the six disabled replication subscriptions. I used postgres.replication_slots to check that each publisher's matching slot was gone. The retired slots had been retaining roughly 122 GB of production write-ahead logs. The remaining old application workloads were then scaled down through GitOps.

The resource handoff transferred image repositories, media storage and CDNs, CI identities, and release credentials into the Terraform configuration that would remain in use. I compared the resource addresses returned by tf.state_list for the old and maintained workspaces to check the transfer. The resources themselves stayed in place.

We removed migration proxies, stopped the old Cloud SQL instances, and scaled down legacy compute. Both GKE clusters were then deleted. The stopped databases, backups, and rollback disks remained for the final cleanup.

The final cleanup deleted the stopped Cloud SQL instances, old virtual machines, disks, and load balancers. We also removed obsolete DNS and configuration, preserved the active API uptime check, and archived the old infrastructure and GitOps repositories.

Final net.http_probe checks returned HTTP 200 from development and production web, admin, API health, and media endpoints. The application services were running on VA1, and the old GKE clusters and migration proxies were gone.

Google Cloud retained image delivery, media, monitoring, 76 database backups, and 40 snapshots. The active services had owners in the Terraform configuration that remained in use.

After Google's required delay, we finished the private-network cleanup through GitOps, removing the old connections, reserved ranges, and production network.