Hardware Failure DownTime

Hardware Failure


Cory Linden:

Hardware Failure and Downtime Thursday AM


We just blew a disk on one of database machines. No data has been loss, but performance will be degraded until we can switch over to the backup machine tomorrow AM. As a result, login may be slow or blocked during heavy load tonight (which we're mostly through).

We are going to down the grid tomorrow morning around 6am PST and will be back up before noon PST.

We apologize for the login problems and are working to get everything fixed ASAP. Thank you for your patience.

/me is sleeping waaay in tomorrow

Again? This is becoming a habit.

I guess I'll take this time as an extra opportunity to do some more housework... the beasts (read: RP friends) are coming in on 31st, so it'll be much needed. Before and after

Same here

Stuff like this happens. Life happens. Second Life Happens.

In any case, good to know than remain in the dark -- Cory, all the best to you and the Linden family in getting the computers back up!

*hugz*

I'm trying to connect to the IRC, I've got the channel up with a list of people but no one's talking, and it's rather dead. I think there's a problem with the connection somewhere.

Downtime - ah hah, an opportunity to eat, rather than starve in front of the keyboard

Then it's either open the copy of Myst IV I got for Xmas, or wash the car.

Ahhhh, it's raining, second option dealt with.

What is this 'First Life' thing anyway ?

Ohhh no!! Do I have to go to sleep then?

Can't get in!

I am simply blown away after reading this thread in it's entirety.

The problem is that it keeps happening over and over and over.

Downtimes are becoming an all too frequent occurrence.

"We just blew a disk on one of database machines. "

I wish these people in San Fransisco would just stick to humans for crying out loud!!

Yea out again =(
that s all i see =(
AGAIN !!!!!!!!!!!!!! =(

I m building a City actually, do you think i have time to spare on the forums ?????

Hmmm... I've been thinking about this a bit lately myself.

Now I know that hardware dies, ok fine. Shizzle happens.

But what I'd like to know is:
Has all of the 8 M that LL received been earmarked in such a way that it can't be spent on hardware upgrades? I mean, 2 M could do wonders for the hardware infrastructure, I'm sure.

Because if that's the case, we deserve to know. I am starting to suspect it is. I am starting to suspect that the money has been earmarked for weird spinoff projects like the teen grid, or marketing projects or something.

But if not, please tell us and we can all breathe easier.

Shizzle does happen, but it's important for end users to know that even when shizzle happens, it doesn't mean everything should grind to a halt. I'm not an expert, but from my medium-level knowledge of servers and redundancy I belive it should be possible for a single drive going out to cause little or no effect on the running of a database server. Drives can be redundant, and mission critical servers can be redundant meaning that if a drive or a whole server fails, another should be able to either automatically switch in, or be manually switched in in a very short time.

Why isn't this being done at SL?

I like your attitude, Torley

hmmm... one disk on one database server barfs and 6 hours to replace?

suggests using a raid array.. replacing a hot spare would go unnoticed and increase uptime(ie customer satisfaction)just my 2 cents..

How much longer do we have to wait there is only so much RL a girl can take

Note from last nite says about noon game time...

Ever since I first saw that message, I've been wondering the same thing as Buddha. Replacing a single drive on a single server shouldn't take 6 hours in the first place, and it shouldn't need to take down the whole grid with it for that entire time. Heck, the grid was working fine last night after the drive crashed after you were finally able to log in!

OMFG thats 3 hours away!!! *cries*

But isn't this a wonderful opportunity to catch up here in the forum??<eg>

<just had to throw that in>

> Has all of the 8 M that LL received been earmarked in such a way that it can't be spent on hardware upgrades? I mean, 2 M could do wonders for the hardware infrastructure, I'm sure.
>
> Because if that's the case, we deserve to know. I am starting to suspect it is. I am starting to suspect that the money has been earmarked for weird spinoff projects like the teen grid, or marketing projects or something.

It's surprising how quickly money like this disappears. Bear in mind that a lot of people have probably been working on the promise of money arriving in the future, so they would have taken a wodge of this.

As an aside, note that whilst drives should be RAIDed, I've seen a ton of RAID controllers die, well ok three, but still they're surprisingly unreliable, and dont entirely guarantee that your server will never die. Of course, Compaq support is generally pretty decent, so the server shouldnt be dead for very long, maybe a couple of hours.

Azelda

I figured last night something was wrong with a server.

I live out in Gray sim, within about 80m of the old Gray airfield, and yesterday when I logged in in the evening, I noticed that all the fences around the airfield were gone. Baleeted. Vamoose. Well, the signs that hung on the airfield were still there but the fences itself were gone. Seemed kind of strange that a build that has been there for ages seemed to be starting to fall apart.... made me think of Gov. Linden's mansion.

Figured "Hmm, maybe there's a database server problem...."... and "Hmm.. good thing this isn't Jurassic Park."

They've probably got a RAID array, and one of the disks in the array has failed. Rather than keep the machine in production they're taking it out of the pool before another drive fails leaving the array broken and possibly unrecoverable. In the mean time they're probably prepping one of the "backup" servers to take it's place so they can rebuild the array offline.

/guesswork, but sound administration. Inconvenience rather than disaster.