On 11 September 2024, a scheduled night maintenance left 230 of more than 1,300 managed databases on a Crossplane-based platform in an invalid state. The reconcile ran from a stale state; the resources were marked for deletion, and the garbage collector removed them with no safeguard and no way to stop it once it started. The on-call team watched the databases disappear one by one.

This talk walks through that night and the 26-page post-mortem that followed, together with two smaller incidents on the same platform: deletion protection silently broken by an upstream label change, and connection secrets lost during an upgrade. The common thread is that each layer of abstraction adds its own state lag, and declarative does not mean predictable. It covers how the data that mattered was recovered, what changed in architecture, testing and maintenance procedure, and how the customer relationship survived.