Kafka Decommission (1 March 2021)
Background
We have been double writing tracking events to both Kafka and Google Pub/Sub all this time. It is costing us a lot of money to have two infrastructures, not mentioning maintenance cost of the self-managed Kafka. Data team folks have migrated data ETLs out from Kafka downstream. So, we are ready to decommission Kafka to save cost.
We are going to turn off the Kafka servers on 1 March 2021 12PM (GMT+7)
What do I need to do?
Monorepo
We have turned off Kafka publishing from backend-monorepo (see this PR) by changing the config files. We need your help to release your services to include the commit c7b970f before 1 March 2021.
Multirepo
We need your help to change your tracking config files with simple change like this and release your services with the change before 1 March 2021.
Do I need to change/release my services?
Yes, if these two conditions are met:
- your services still run on AWS tvlk-dev/tvlk-prod (services on AWS multi account setup will not be affected)
- your code is using
org.traveloka.tracking.TrackingComponent
(from snapshot JAR if you have migrated to multirepo setup)
What will happen if I don’t change/release my services before 1 March?
- For Avro events: your services might fail to encode tracking events and the tracking events are not sent, even to Google Pub/Sub.
- For JSON events: your services might produce a lot of errors due to inaccessible Kafka servers.
Decommission plan
- 1 March 2021, 12 PM: Detach security groups from schema registry and Kafka load balancers
- staging security groups have been detached
- production security groups have been detached but re-attached back because it brought some services down, postponed to 3 March
- 8 March 2021, 10 AM: Detach security groups from schema registry and Kafka for production
- 9 March 2021, 10 AM: Stop schema registry, Kafka, and Zookeeper instances
- 16 March 2021, 10 AM: Terminate schema registry, Kafka, and Zookeeper instances