No scrapping via AWS or Office IPs

As mentioned in #technology earlier, AirAsia recently filed an abuse report to Amazon and blacklisted all AWS IPs (not just ours) due to our scrapping load. This also put us in violation of AWS Acceptable Use Policy, which can have a very serious consequences. We’re taking a few immediate actions to mitigate this.

1. Route-off AirAsia on all staging machines [pic: @chandrawibowo] (done)

2. For “shared” staging machines (i.e.: staging06, staging19, testing-app-01):

3. For the other staging machines (e.g.: staging01, etc):

4. For the development machines:

Aside from the immediate actions above, we’re also pursuing the following high-priority tasks:

Note:
If your team has any kind of mission-critical scraping needs that are currently running from our office or AWS IP, and you are not sure what to do, please add your use case(s) in this spreadsheet for us to take a look.