Recently we got a critical incident where one of our postgres in tvlk-ast-prod got accidentally deleted due to lack of communication. Post mortem: https://docs.google.com/document/d/1pixnQNAfcYMulk2JbbEDlSYH01UMRHpSHZdt3HPBqhU

We should have clear communication between all of our team members in regard of any infra changes or terraform execution in production environment. This should minimize any human errors which may cause unwanted incident.

Using GitHub Infra Issue

We would like to enforced infra issue creation before executing any infra changes in production, including AWS maintenance tasks such as restarting RDS instance, etc. Though service deployment to production is not mandatory to have infra-issue for now, but this may changed in the future.

Please create the infra issue from our GitHub repository (example)

https://github.com/traveloka/tvlk-ast-prod-terraform-aws/issues

https://github.com/traveloka/tvlk-ast-prod-terraform-aws/issues/352

Assign to one Supply PDA and related Supply Lead to review the infra issue

Auto assignment to Supply PDA is ongoing to be implemented

Please collaborate, double check, and cross check about the risk and carefully review the Terraform Plan
<optional> PDA and Supply Lead might want to ask for @ihabibi approval

Let us improve our process together, and let me know your ideas and inputs. Thanks team.

GitHub Infra Issue As Tracking

Background

Using GitHub Infra Issue