Fprowan-memcached scaling: from 4 to 6 nodes
Background
Based on CPE Lebaran 2018 Planning fprowan status is moderately at risk because there is 40 eviction per minute (total for all node). Another case that we have:
- Flexible fare now enable for GA and plan to enable on QG before push freeze. It requires more cache to be stored, because flexible fare will add more cache key (double search when user tick high flexibility option: regular and flexible search). By our estimation, this will increase another ~1GB.
- Airline will add new routes due to Peak Season (Lebaran)
- To accommodate new airlines & routes in the future
From the value of eviction we need to upgrade memcached as minimize as we can since it will impact more call triggers to frs if the search result isn't found in fprowaan.
Changes
Current number of cache nodes is 4 and we plan to add 2 more nodes. This will gain us additional ~30 GB.
Risks
- It’s expected that some search result cache (around 15-20%) will need to be re-searched in our machines, but it shouldn’t siginificantly impact our search performance and the performance should be back to normal in a short period of time.
Steps
- Create jira to site-infra requesting additional nodes thanks to @adelia.herlisa - DONE
- Adding nodes by site infra, scheduled 2018-05-15 00:00 UTC+07 (flight team: monitor) - DONE
- Restarting frs-app after adding nodes is done 2018-05-15 if needed
- Restarting fprint-app after adding nodes is done 2018-05-15 if needed
Reference