On-call Report 16-22 May 2022
Action Taken
- cut-off v3.55.x
- large event monitoring
- HeyFest 2022:
- Scale-out instances
- Join war-room to monitor issues
- NowPlaying Fest 2022: new batch available
- Issues:
- there's a lot of incoming requests
- there's rate limit at xpebook getBookingPage and createBooking
- Action:
- manual scale out instances
- increase getbookingpage rate limit
- Automation checks
- xpebook > rpc_server > getexperiencerefundtoproviderdata exception
- xpebook > rpc_server > getbookingpage server exception
- Search code occurence of "Provider id is not supported with spec"
- Found active axes product ids at demand
- Deactivate products
- xpebook > rpc_client > ipiprf getRefundStatusByBookingIdAndItineraryId server exception, raised by Nurlina
- Merchandising load test support
Inquiry
- XS-399 Products not showing on storefront
- The availability flag from POT is not working expectedly: https://29022131.atlassian.net/browse/POT-2988
- Unless POT fix the issue, there might be issue where a product is not searchable at autocomplete
- Merchandising: autocomplete is not working expectedly -- some products don't appear
- Globaltix: booking failed at demand
- Merchandising load test support
- They'll continue the load test next week
- Findings from May 19th 22:00-00:30 load test
- xpedisc cannot handle request at 6 instance, scale out to 12
- xpedisc to xpedata got exceptions, potentially caused by xpedata cannot receive more request. scale out 3 to 6
- https://tvlk.slack.com/archives/GR4UD90RL/p1652880052367069
- Cinema
- Health: Missing issuance on teleconsul from 6-11 May 2022
PagerDuty
Pending Task
- Louis Tomlinson Live in Jakarta Tickets reschedule is not processed accordingly
- XPE operationalPeriod issue >> handle short period (< 7 days)
- [BE] assessment why does mongoDB.upsert() gives error about duplicate
- [BE][XPE][TEX] Investigate searchExperienceDetail error from xpesrch -> xpedata