usriac-mongod maintenance #MSG

We are going to do maintenance on usriac-mongod (My Inbox feature) to resolve erratic cpu usage which has been confirmed to be resolved by restarting the mongo process.

Jira : https://29022131.atlassian.net/browse/USR-747

When : 7 Aug 2018, 20:00 UTC +7

Steps done will be :

restart secondary mongo usriac-mongod-02 – DONE (26 Jul, 16:24)
restart secondary mongo usriac-mongod-01 – DONE (7 Aug, 20:07)
step down current primary and elect usriac-mongod-02 as the new primary – DONE (7 Aug, 20:19)
restart usriac-mongod-03 – DONE 20:24
adjust datadog cpu monitor threshold to 50% – DONE (7 Aug, 20:27)

Risks :
During maintenance there might be some failures for My Inbox read operations from secondary db (affecting non critical flows).

References :

https://app.datadoghq.com/dash/403511/usr-usriac-mongod---mongod?live=true&page=0&is_auto=false&from_ts=1531052862899&to_ts=1533644862899&tile_size=m
https://docs.mongodb.com/v3.0/tutorial/perform-maintence-on-replica-set-members/
https://docs.mongodb.com/v3.0/tutorial/force-member-to-be-primary/

issues or questions can be raised on this post, or at #user-eng-channel, mention @messaging-eng