The SSH key that nobody rotated

Our ML trading bot deploys via SSH to a VM. The deploy workflow had been green for months — every commit showed a passing check. Except it wasn't deploying.

The workflow had two jobs: a Cloud Run dashboard deployment (triggered on path changes) and an SSH deploy (triggered on a different path filter). The dashboard job was succeeding constantly because we pushed frequent frontend changes. The SSH job had been failing for three weeks — ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey] — and nobody noticed, because the workflow's overall status was green.

Path-filtered multi-job workflows hide a broken job behind a healthy one. The root cause was trivial: someone had reprovisioned the VM without copying the deploy public key. The VM's authorized_keys no longer matched the GitHub secret.

It took three weeks to discover because monitoring only checks the workflow-level status, not per-job. The fix was straightforward — regenerate keypair, update secret — but the pattern is dangerous: any CI pipeline with conditional job execution can mask failures for the less-frequently-triggered path.

We now tag each deploy job with an independent status badge and alert on per-job failure, not workflow-level green.