Second Life power outage Tuesday night

At around 11pm the colocation facility in which SL is housed lost power for almost an hour. We've just now brought everything back online. I hope to have an explaination from them tomorow, which I will post - we're all curious as to how a building with 21 megawatts of generator capacity can go completely dark.

High-tech flywheel-based power protection doesn't help when it's intentionally turned off. We have an explaination, but it seem our confidentiality agreement prevents me from posting it verbatim. So here's a breakdown in my own words, assuming I'm reading their somewhat obfuscated document correctly.

22:54PM – A faulty fire-suppresion valve in the mechanical penthouse incorrectly pressurizes the dry-pipe sprinkler system used outside the compuer rooms. The fire system sets off the alarm and summons SFFD.
22:55PM – To avoid possible electrocution, the fire system shuts down all power in the building. Everybody evacuates. Note that the sprinklers have not actually engaged.
23:05PM – SFFD shows up and lets everyone back in.
23:18PM – Colo people drain the sprinkler system.
23:32PM – Colo people kill the alarm.
23:38PM – Colo people start turning things back on.
23:46PM – SL gets power again. 98% of our servers come back online, 1% fail to start, and 1% are now completely dead.
00:01AM - Linden Lab staff realize we've had a power outage instead of just a network outage. Off to the colo to deal with the few older, mostly "bronze age" servers which haven't come back up.
00:45AM - We get SL running again.
01:05AM - We open SL to login and start cleanup on residual problems.

Diary of a power outage.