dwhrt-mongod Maintenance

We had several incident where mongod process on dwhrt-mongod secondary died because of it failed to do logrotation due to incorrect mongod configuration.

Site infra team will deploy new configuration on dwhrt-mongod cluster.

Plan

We will do maintenance on secondary nodes first before doing maintenance on primary nodes.

For each secondary nodes:

Stop mongod services
Rerun configure playbook
Check logrotation on the instance
Start mongod service

On primary:

Step down primary node
After the nodes become secondary do the maintenance process for secondary node above.

Impact:

One of secondary nodes will be unavailable during maintenance time.

Duration:

40 minutes. Each secondary nodes will be unavailable around 5-10 minutes during maintenance

Time:

Monday, 2017-05-15T14:40:00 (UTC+7)

Notes:

Please contact Site-Infra on #site-infra-channel if after this maintenance you have a problems accessing dwhrt-mongod cluster.

Related TOSD ticket:

https://29022131.atlassian.net/servicedesk/customer/portal/11/TOSD-1288