The GCP to AWS migration: firewall, health checks, and SSL idle drops

We migrated our main PostgreSQL database from GCP Cloud SQL to AWS RDS in June. The migration itself took an afternoon. Recovering from it took two weeks.

Three problems stacked:

1. The AWS security group only allowlisted a single old home IP. When GitHub Actions runners tried to connect, they were silently black-holed — not connection refused, just timeout. The migration checklist had "update firewall rules" as a line item, but nobody had enumerated every caller IP beforehand.

2. The R price-fetch job opened a database connection at startup, spent 2-3 minutes fetching ~1,000 coin prices, and then tried to save — only to find the connection had been dropped by the new AWS idle timeout. This had never happened on GCP Cloud SQL because it had a different connection-handling policy. The fix: reconnect before write and set sslmode=require.

3. The health check was reporting green throughout. It was checking "did the step script run?" not "is the database reachable and returning fresh data?" A health check that doesn't fail on the actual failure mode is worse than no health check — it creates false confidence that delays investigation.

We fixed all three — opened the firewall, added reconnect-before-write logic, and replaced the fake health check with a real pipeline that verifies connectivity, auth, and data freshness. But the migration bill was 11 days of missing data and a scramble to backfill.

The playbook now has a "pre-cutover" step: enumerate every consumer's IP or network range and open the firewall before the switch. Every migration gets a two-gate deletion rule: a backup must exist and a parity check must pass before the old resource is retired.