In order to keep our shared tvlk-midas account clean, operational overhead low and organizational costs down, Fintech DevOps team is announcing roll-out of compulsory Fixed Term Log Retention Policy.
What does it mean?
Fintech DevOp team will periodically conduct an automatic Log Group sweep, during which:
All CloudWatch Log Groups older than 6 months, with 0 stored bytes and no writes within 30 days will be deleted.
All CloudWatch Log Groups found without a retention policy, will be given a compulsory 30-day retention policy (meaning all stored log events older than a month will be deleted).
What should you do?
If you have product or regulatory requirements to retain your logs for periods exceeding 30 days, make sure to either:
Explicitly request custom log retention policy for the Log Groups that need it.
Note: you will need to properly justify custom retention periods - both the reason and the duration in this sheet. Request you all please update retention period if required more than 30 days. By default it will be considered 30 days.
Note: DevOps team have been exported all log groups with /tvlk/
prefix to cloudwatch-export-RANDOM
bucket already.
Why is this happening?
No resource is infinite or 'unlimited'. Having large number of unused resources wastes company's money, people's time and shared computing resources.
Currently 11TB cloud watch log groups are present in Midas account without retention period. This data is keep accumulating from beginning since the account was created.
Number of CloudWatch Log Groups per region is limited, maximum throughput of API calls used to describe each log group is limited. Separating useful signal from noise when trying to collect information about a particular Log Group consumes limited human time. Each log event stored by AWS past its "useful-by" date wastes money company could have instead spent on its people.
It would be great help if we can provided information by 3rd June 2020. 4th June this sheet information will be implemented in Midas infrastructure.
Please reach out to Fintech DevOps Team for any query or clarification, we are here to help you. Thanks all