Tech:Incidents/2015-05-ddos

Note: this is a draft, and the issues are as of 24 May more or less still ongoing.

Orain suffered 2 days of downtime/consistent issues for non-IPv6 (users which are not visiting Orain with IPv6 enabled), probably due to DDoS attacks.

(Times are UTC+0)

Timeline
Note: small(er) issues have been reported before.
 * 20 May
 * 06:15 Nagios: nagios sends last alerts out
 * 06:56 Southparkfan: discovers Orain is down and sends mail to staff@undefinedorain.org notifying of downtime
 * 07:06 Southparkfan: realizes that prod6 seems down, sends mail to Dusti's and addshore's personal email addresses
 * 10:15 Addshore: mails back and says Orain is up. It is unknown whether addshore accessed Orain with or without IPv6.
 * 10:25 Addshore: mails a picture of prod10 graphs, which includes a graph of the inbound and outbound private and public traffic. There is a public inbound traffic spike, with one hitting 800mb/s inbound traffic
 * 19:57 Southparkfan: FastLizard4 tells me Orain is accessible when using IPv6, but not when using IPv4.
 * 19:58 Southparkfan: tries to SSH into prod10 by using either prod10.orain.org or its public IP, but none of them work. SSH'ing into prod10 by using prod8 as a proxy works though.
 * 22:33 Southparkfan: proposes to FastLizard4 (since he was able to do things, and Southparkfan wasn't) to redirect *.orain.org to prod13-temp.orain.org
 * 22:40 Southparkfan: above setup will break IPv6 support, Southparkfan proposes to revert the whole DNS repo to b83d1ca08fe6bc728427d061b040d0245078e031
 * 22:45 FastLizard4: tries to push commits to the DNS repo, but gets stuck with permission errors.
 * 21 May
 * 06:46 Southparkfan: revert /config to b83d1ca08fe6bc728427d061b040d0245078e031
 * 06:46 Southparkfan: grant operations full access to DNS repo (it already should have, but just add the group in 'Colloborators' too
 * 07:01 Southparkfan: All The Tropes and TestWiki are both confirmed back online, Meta is still down
 * 10:44 FastLizard4: confirms Orain is up
 * 14:36 Southparkfan: confirms Orain is up
 * 22 May
 * TO-DO
 * 23 May
 * TO-DO