S3 Bucket Deletion - Postmortem and Recovery

Hello Teams,
On Tuesday there are accidental deletion of S3 bucket across many non-production accounts in our organization. We’re sorry for this incident, and we’re offering to help recovery of deleted Terraform state, and we’re filling the postmortem document and keep you up to date.

Recovery

Any teams that their Terraform state has been deleted during the incident, please join into #terraform-state-recovery channel if you haven’t. Any teams that’re interested on learning or helping, we’re welcoming you to the dicussion.

Postmortem

Document
We apologise the postmortem is currently in progress, as we split our focus on helping the recovery.
Teams that impacted, we’re kindly asking for your participation to complete the postmortem, put supporting information, and provide the business impact of this incident.

About Region Locking Progress and Plan

This incident was caused by cleanup script when we’re doing region locking across accounts, see this announcement. We’re planning to postpone this works. Focusing on recovery and then postmortem are our current priority, and then after that we will review the process again before making further changes.

---

We want to reiterate the apology again, and we’re offering our help to restore Terraform state. Please contact @rafikurnia @bernard @darwinwirawan or join into #terraform-state-recovery for further help.