Timeline
| Time (UTC) | Event |
|---|---|
| 14:04 | Routine schema migration deployed to add an index to the sessions table. |
| 14:06 | Migration acquires a metadata lock. Login p99 latency crosses 8 s. |
| 14:08 | Automated alert fires. On-call engineer acknowledges within 90 seconds. |
| 14:14 | Cause identified as the running migration. Decision made to let it finish rather than kill mid-DDL. |
| 14:29 | Migration still running. Reversed the decision and killed it. |
| 14:31 | Lock released. Login latency begins recovering. |
| 14:38 | p99 back under 400 ms. Error rate at baseline. |
| 14:47 | Incident closed after nine minutes of clean metrics. |
What actually went wrong
The migration was tested against a staging replica with a representative row count but not representative write throughput. On staging it completed in 900 ms. In production, the sessions table takes roughly 4,000 writes per second, and the online DDL path could not acquire its final metadata lock while those writes were in flight. It queued, and every new login queued behind it.
The 8% figure is because sessions are sharded across twelve database clusters and the migration ran on them sequentially. It had completed on one shard and was stuck on the second when we killed it.
The decision we got wrong
At 14:14 we chose to let the migration finish, reasoning that killing a DDL mid-flight carries its own risk. That cost fifteen minutes. In hindsight the migration had no bounded completion estimate, and 'wait for something with no ETA while customers are impacted' is not a defensible call. We should have killed it at 14:15.
What changed
- All migrations now run with an explicit lock_wait_timeout of 3 seconds and abort rather than queue.
- Migrations are validated against a shadow replica that receives mirrored production write traffic, not just production row counts.
- The runbook now says: if a migration is implicated in a customer-facing incident and has no completion estimate, kill it. No deliberation.
- Session shard migrations run one shard per deploy window, so a bad one can affect at most 1/12th of accounts.
All four changes shipped within nine days. The lock_wait_timeout guard has since aborted two migrations harmlessly, both of which would have caused the same incident.
Affected accounts received an SLA credit automatically on their next invoice. Nobody had to ask for it, and nobody had to have noticed.