SL Power Outage
|
|
Jack Digeridoo
machinimaniac
Join date: 29 Jul 2003
Posts: 1,170
|
04-06-2005 07:25
From: someone At around 11pm the colocation facility in which SL is housed lost power for almost an hour. We've just now brought everything back online. I hope to have an explaination from them tomorow, which I will post - we're all curious as to how a building with 21 megawatts of generator capacity can go completely dark.
Perhaps weakened electronics from the large kp spike the previous night?  From: someone
This plot contains the estimated planetary K-index. This index is derived, at the U.S. Air Force Space Forecast Center, using data from ground-based magnetometers at Meanook, Canada; Sitka, Alaska; Glenlea, Canada; Saint Johns, Canada; Ottawa, Canada; Newport, Washington; Fredericksburg, Virginia; Boulder, Colorado; and Fresno, California. These data are made available through the cooperation of the Geological Survey of Canada (GSC) and the US Geological Survey. K-indices of 5 or greater indicate storm-level geomagnetic activity. Geomagnetic storms have been associated with satellite surface charging and increased atmospheric drag.
If that wasn't it, I blame HAARP.
_____________________
If you'll excuse me, it's, it's time to make the world safe for democracy.
|
|
Kris Ritter
paradoxical embolism
Join date: 31 Oct 2003
Posts: 6,627
|
04-06-2005 07:28
I blame Jack. He's just trying to deflect suspicion.
|
|
Jack Digeridoo
machinimaniac
Join date: 29 Jul 2003
Posts: 1,170
|
04-06-2005 09:39
The storm might have also set off a volcano, And caused an earthquake, Beware the end is nigh!!
_____________________
If you'll excuse me, it's, it's time to make the world safe for democracy.
|
|
Jack Digeridoo
machinimaniac
Join date: 29 Jul 2003
Posts: 1,170
|
04-06-2005 09:45
Also, I was listening to Coast To Coast AM, and last night at exactly the same time SL lost power, this call was received: From: someone
George: Let's move now to our wildcard line, your on coast to coast, it's your turn, hi there.
Caller: Hi this is Debra in Hollywood
George: Hollywood, hey did you feel any rumble by the way?
Caller: OH YEA, just felt a real good one
George: Yep, that's what I'm hearing
And at the end of that very same call, From: someone
George: Ok, thanks for your call, appreciate it. We must have gotten some kind of rumble all my computers are out right now too.
Guest: Wow
George: Strraaaange times.
Something IS amiss.
_____________________
If you'll excuse me, it's, it's time to make the world safe for democracy.
|
|
Jack Digeridoo
machinimaniac
Join date: 29 Jul 2003
Posts: 1,170
|
04-07-2005 06:53
The "official" response from Ian Linden. From: someone
High-tech flywheel-based power protection doesn't help when it's intentionally turned off. We have an explaination, but it seem our confidentiality agreement prevents me from posting it verbatim. So here's a breakdown in my own words, assuming I'm reading their somewhat obfuscated document correctly.
22:54PM – A faulty fire-suppresion valve in the mechanical penthouse incorrectly pressurizes the dry-pipe sprinkler system used outside the compuer rooms. The fire system sets off the alarm and summons SFFD.
22:55PM – To avoid possible electrocution, the fire system shuts down all power in the building. Everybody evacuates. Note that the sprinklers have not actually engaged.
23:05PM – SFFD shows up and lets everyone back in.
23:18PM – Colo people drain the sprinkler system.
23:32PM – Colo people kill the alarm.
23:38PM – Colo people start turning things back on.
23:46PM – SL gets power again. 98% of our servers come back online, 1% fail to start, and 1% are now completely dead.
00:01AM - Linden Lab staff realize we've had a power outage instead of just a network outage. Off to the colo to deal with the few older, mostly "bronze age" servers which haven't come back up.
00:45AM - We get SL running again.
01:05AM - We open SL to login and start cleanup on residual problems.
So computers all over California go down after a large "rumble" and the SFFD manages to make it to LL colo in 10 minutes. Ok.. And "colo" people drain the system and "colo" people kill the alarm. Last I checked "colo" people didn't "drain pipes" or "kill alarms". The sprinklers didn't activate so why did they need to drain them? Doesn't the colo use halon in the server rooms not water sprinklers. That's not something you want web monkeys playing with. I don't buy it - what was the rumble, why did the outage stem from Hollywood to SF? Is there something wrong with the water table?
_____________________
If you'll excuse me, it's, it's time to make the world safe for democracy.
|
|
Rickard Roentgen
Renaissance Punk
Join date: 4 Apr 2004
Posts: 1,869
|
04-08-2005 13:47
colo people is ambiguous and could refer to maintenance staff at the facility. it's a dry pipe system meaning there's normally no water in the pipes because it's around electronics. the system incorrectly pressurized the pipes though the sprinkers themselves didn't go off. an old sprinkler head might go off if bumped or simply because there is pressure so the pipes have to be drained returning it to dry pipe status.
Halon systems were probably not originally installed and it's a pain and spendy to convert after the fact. Not to mention halon can be a bit hazardous to your health.
|
|
Pete Fats
Geek
Join date: 18 Apr 2003
Posts: 648
|
04-08-2005 15:39
From: Rickard Roentgen Halon systems were probably not originally installed and it's a pain and spendy to convert after the fact. Not to mention halon can be a bit hazardous to your health. Most people use FM-200 these days. Halon is a death trap. |
|
Teeny Leviathan
Never started World War 3
Join date: 20 May 2003
Posts: 2,716
|
04-08-2005 22:45
From: Pete Fats Most people use FM-200 these days. Halon is a death trap. Actually FM-200 isn't much safer. If its designed to suppress combustion, it ain't necessarily safe to breathe.
_____________________
The Default Avatars were created by Linden Lab
They evolved.
They rebelled.
There are many copies.
And they have a plan.
|
