Outage on Tuesday Night/Wed Morning

Due to a power loss suffered by our communications carriers, SL was down for approximately 2 hours between Tuesday 11:30pm to Wednesday 1:30am . We apologise for the inconvenience, and are investigating the incident.

We are restoring service as quickly as possible.

Regards,
Mick Linden

That's all very well, Mick, and 2 hours might not seem a lot, but the psychological damage is going to take WEEKS or MONTHS to repair!

*looks around at all the quivering SL residents as they rock backwards and forwards humming to themselves quietly*

SL is down so I took a stroll outside and it was actually pretty nice.

Good graphics, nice physics... but darnit the asset server LAGS. I rezzed a cube and i'm still waiting.

...

-sits and stare-

-hums-

I used to work in the NOC at a major national ISP and one day we very nearly lost power to the data center, even though we had enough generator power to cover it. A truck had slammed into a power pole on our street, and knocked out the entire facility. Our backup generator started immediately, but the automatic transfer switch (ATS) failed, so the data center was left on battery power. In the hour that followed, we exhausted a full bank of batteries and were at the last dregs of the second and last bank. Sysadmins were shutting down services, it was that bad. Finally one of our electrical engineers managed to get the ATS to switch over to generator power and we were back in business.

Would be interesting to see their postmortem if they make one available to LL.

Begins to recover from her severe withdrawals syymptoms as the sl log in screen logs fully in.

Well, that certainly explains why my late night skydiving session went kerflooie. Thats okay, I needed to go to bed anyhow.

I haven't connected yet. I hope all will be well 2000 meters above the ground when I wake up

I've posted the postmortem here:

/3/4d/41673/1.html

The system used by our facility doesn't rely on electrical switching to go to backup power, which is nice, but doesn't help in this case. This is the second time we've been taken down by an EPO, but last time it took over 8 hours to get power back.

Ian, I'm happy to hear it was not a network issue. Also sympathize with ya on the power issue, but it sounds like the Colo did the right thing there.

Flywheel based power backups, huh. I've heard of them being used for power conditioning on large ocean vessels, but not as a UPS. Thanks for the postmortem.

I'd love to know if that faulty valve was made in America. *snickers*

I'm surprised the server room isn't isolated from the fire system entirely...

glad it was not peak hours and was resolved quickly

The flywheel is only good for a short run (less than a minute) but it's enough to get the generator up to speed and avoids all the nasty problems of switching and DC/AC conversion you get with batteries. In theory. The setup is this, not too unlike a ship:

Diesel Motor (idle) -> Clutch -> Electric Motor (utility power) -> Flywheel -> Dynamo

Turning everything off because of electrocution makes complete sense, btw.

I once sent in a team to a colo and blue sparks were flying everywhere. I had a hard time explaining to the CEO why the restart took 4 more hours than it should have because I wouldn't let them back in until an engineer had declared it safe..

There is a lot of juice running around in those places. You can easily tell by the huge air conditioners they use to absorb all that excess heat.