Tech:Incidents/2014-07-07

On Monday July 7th, Orain experienced frequent database issues for 4 hours.

Timeline

 * 14:09 JohnLewis: issues seem to be resolved
 * 14:08 JohnLewis: reboot prod3
 * 14:00 JohnLewis: prod3 seems a bit slow and unresponsive
 * 13:50 JohnLewis: I become a aware of the issue and start investigating
 * 11:24 Addshore: Investigates the issue briefly
 * 11:15 JohnLewis: Southparkfan emails sysadmin@undefinedorain.org reporting issues
 * 10:25 JohnLewis: Nagios - MySQL is returned as 'critical'
 * 10:02 JohnLewis: Pingdom reports Orain is down