[psa] Change VPC FlowLogs Logs Delivery To S3 To Reduce VendedLogs Cost

Overview

Currently all of our VPC is sending VPC FlowLogs to CloudWatch Logs. AWS already support delivering FlowLogs to S3 [1].

FlowLogs is one of the Vended Logs which have tiered pricing. At most this can reduce cost up to 90% of the cost.

In our org we can save around 50% of VendedLogs Cost for FlowLogs

Get Current Vended Logs Usage

Open Cost Explorer
On Usage Type, choose APS1-VendedLog-Bytes
Get the bytes / GB value of this usage type

Estimating Potential Cost Saving

Estimate Cost Vended Logs Delivered to CloudWatch

Use https://calculator.aws
Choose Amazon CloudWatch
Put the value of APS1-VendedLog-Bytes on the Vended Logs: Data Ingested

Estimate Cost Vended Logs Delivered to S3

Use https://calculator.aws
Choose Amazon CloudWatch
Put the value of APS1-VendedLog-Bytes on the Logs Delivered to S3: Data Ingested

The potential cost saving is = VendedLogs to CWL - VendedLogs to S3

NOTE : There is additional cost for the S3 bucket, The data on the S3 bucket is compressed so the cost is small compared to the VendedLog cost.

What Do You Need To Do

Update VPC terraform module to version v0.5.0 (This will create new FlowLogs to S3)

Ensure your VPC already use terraform-aws-vpc-module version v0.4.0.
Update Your VPC to use terraform-aws-vpc-module version v0.5.0.

Monitor The New FlowLogs

Ensure the status of the FlowLogs is Active and no error shown.
You might want to run both legacy FlowLogs and new FlowLogs for 1-2 hours to have overlap data

Update VPC Terraform module to version v0.6.1 (This will remove old FlowLogs) - DO NOT USE v0.6.0

Ensure your VPC already use terraform-aws-vpc-module version v0.5.0.
Update Your VPC to use terraform-aws-vpc-module version v0.6.1.
Run terraform plan. It will show list of resources that will be destroyed.
To keep the FlowLogs Log Group do the following steps

a. Check for destroyed resource module.<module_name>.aws_cloudwatch_log_group.flow_logs. Replace <module_name> with your VPC module name.
b. Get resource detail. Run terraform state show how module.<module_name>.aws_cloudwatch_log_group.flow_logs . Take notes the value of name, retention_in_days and tags.
c. Create Terraform config outside module similar to below. Put the value from step b on the respective parameter.

resource "aws_cloudwatch_log_group" "flow_logs" {
  name              = ""
  retention_in_days = ""
  tags = {
    "Description"  = ""
    "Environment"   = ""
    "ManagedBy"     = ""
    "Name"          = ""
    "ProductDomain" = ""
  }
}

d. Move the state using command terraform state mv module.<module_name>.aws_cloudwatch_log_group.flow_logs aws_cloudwatch_log_group.flow_logs. There should be no changes in aws_cloudwatch_log_group.flow_logs new resource.

Re-run terraform plan to check that the changes to be applied is already as you would expect. then apply the changes.

NOTE: Depend on your FlowLogs Log Group retention policy, you can delete the log group after the Log Group is empty.

How to Monitor

From AWS Console, Choose VPC
Choose The VPC that you want to monitor
Choose FlowLogs
Ensure the status is Active

References

[1] https://aws.amazon.com/about-aws/whats-new/2018/08/amazon-vpc-flow-logs-can-now-be-delivered-to-s3/
[2] https://aws.amazon.com/cloudwatch/pricing/
[3] https://aws.amazon.com/about-aws/whats-new/2018/01/cloudwatch-introduces-tiered-pricing-with-up-to-90-percent-discount-for-vpc-flow-logs-and-other-vended-logs/