Unplanned outage July 1, 2009, Unplanned outage Options

[Imhotep]

Greetings all,

Just wanted to pop in and apologise for the outage this morning. I've been looking through logs, stats, server hardware, but cannot seem to find a root cause for the outage. I found the system locked hard enough to have to hit the reset button to get it to reboot. So, I'm going to keep digging, but didn't want anyone to think that the issue went unnoticed or that it wasn't being thoroughly researched to figure out why so it doesn't happen again.

Apologies for any inconvenience this may have caused,

I

[Imhotep]

Heyo. Yea, definitely some sort of bug in the operating system (the server is running Linux) - looks like they introduced it in the latest set of bug fixes and security patches. So far we've been able to isolate it down to the kernel (for those who are interested in systems architecture and don't already know Unix systems, the kernel is basically the clearing house everything else goes through - video talks to it, disks talk to it, memory talks to it, etc etc etc), looking somewhat towards the video subroutines, something in the graphics portion of the driver seems to cause a memory leak and 100% CPU. Not sure so far what specifically it is, the system just sits at the login screen so it's not an application driven crash. So for more troubleshooting I've disabled everything on the console except text mode and sent off logs and a core dump. We'll see if that gives any additional clues.

Majorgeeks is a good site. I've been working with some people I met from fedoraforum and the kernel mailing list. It's a software driven issue, so it's unlikely to cost anything to fix, but I'd rather see the Raid applied sooner rather than later, if you know what I mean