Why out-of-band changes keep happening despite the tooling
During an incident, the fastest fix is often a direct manual change in the console — and it should be, in a genuine emergency. The failure isn't making that change; it's not reconciling the declared state afterward once the incident is resolved, leaving the drift to be discovered weeks later by an unrelated deployment.
This is a process gap specifically at the incident-closure step, not a gap in the IaC tool's detection capability.
The process fix that actually closes it
Making "reconcile the declared state" an explicit, checked step of incident closure — not an optional follow-up — closes the gap at the point it's actually created, and is far more reliable than hoping drift detection catches it before the next deployment surfaces it as a surprise.
Drift detection finds the symptom. The incident-closure checklist is what actually prevents the cause.MNB DevOps Practice
Key actions
- Make state reconciliation an explicit, checked step of incident closure
- Allow emergency manual changes but require follow-up reconciliation
- Run drift detection regularly, but treat it as a backstop, not the fix
- Audit incident closures for unreconciled manual changes