Central Hardware Failing?

aren't borkends what they use to keep their "Asset Servers for Dummies" books from falling off the shelf?

my guess: they blew up the networking between the sims and the storage servers

and right now they are negotiating an army of equipment testers with the vendors






They run like 10.000 server, if you got to test every network card and cable

This "talking with their hardware vendor" reminds me of something...

In a online game I played we had a clan. And we had our website and forums hosted by some company in Switzerland.
On a certain moment they replaced some hardware and from that moment we had all kind of problems. They also "talked with thier hardware vendor" because the new hardware caused problems that were related to the new hardware and should not occur at all...

I think something like that is going on here as well.... but the experts may correct me .

Morwen.

I don't find it absurd to point out that even a small company with a few hundred customers manages to provide a failsafe service, and that I'd expect at least the same security measures from a company who claims to have 2 million customers, many of them business customers.

It's absurd that LL has to start negotiations with a hardware vendor when something goes awry, instead of just switching over to a backup solution.

[7:29am PST] Unfortunately it looks like the SL Grid will be down for as many as 5 more hours while we work with the manufacturer on the hardware failures from earlier today. As we get more information we’ll continue to post here.

Broccoli

[7:29am PST] Unfortunately it looks like the SL Grid will be down for as many as 5 more hours while we work with the manufacturer on the hardware failures from earlier today. As we get more information we’ll continue to post here.

This just popped up, so those of us in Euro zones may be out till the middle of the night at least!

BTW Ishtara, having seen you around the forums I was in your store for the first time when the grid blew!

(SNAP Broccoli)

This does not seem like a small issue to me. LL is clearly unable to handle the rapid growth we are seeing over the past few months. "Negotiating with their hardware vendor" at this time does not make sense! Hopefully they don't come out and say that we lost everything we had in-world!

OHHH this is looking worse by the minit i do hope i do not lose my land and items as i love my land it took me ages to find that plot i really wanted and now to think that it could be gone omg i hope not well if it is and my items and money have allso gone im gonna leave sl i hate the idea of starting all over after iv just build my bussnies up.

LL is a smaller company than the one you worked for. With best practices like having a couple of disk "laying around" for backup, I don't think your company is one LL should strive to emulate.

Relax people, the grid is actually up and fine (I'm in it right now.) They're not allowing new logins but as far as I can see everything is intact.

-peekay

Well at least we have a target now; I will be back at 11pm (my time) and see if they meet it. I guess if the Lindens have been up all night, 2am wont kill me, but I would hate to sit up all night looking at an offline status

I'm so glad i'm not the only one that gets hit with Murphy's law when it comes to storage cabinets. Anyone know what kind they are using?

I shall be in WoW all evening, unless I hear SL is back up, I'm not staying up till half way through the night "just in case"... I'm keen, but not that keen.

Broccoli

I'll continue to boot up SL occasionally to see if the probs are sorted but if this was any other equipment failure in RL then after a certain number of hours you usually can claim compensation for inconvenience caused / outage time etc.....wonder if there's any chance of a few extra L$'s on everyones balance when it eventually is all go ( said very tongue in cheek so please don't flame me )

Fluff

The ToS says after 24 consecutive hours of no service, we may get something (i think it's like 1/30th of our monthly fees per day).

I look forward to my $1.34 of compensation. If, of course, we get round to 23.00 PST/SLT today, with no open logins.

Broccoli

But, thats another 5 hours! What am I going to do?

OMG!!!

I know, I will sleep now and then I can be online all night

Peekay I am not sure which game you are playing but nothing shows for me and I have been offline now since 1.30 this morning in my real time. It is a serious issue to have a world-wide site down for this amount of time with the amount of investment that players have put in. Certainly there should be back-ups and on-going maintenance. I am sure people will leave SL if only because of the inconvenience. I'll stay but many will only bother to play for free.

Ya Peekay its down for me allso and has been all day im in the uk so whats goin on is it up for enyone els on here??

I've certainly not been able to get in at all, although it seems like some that were logged in at the time of shutdown have remained in.


Broccoli

Peekay must be in the test grid.

Broccoli is correct. From what I've heard from a few friends, people like Peekay who were logged in at the time of the shutdown were allowed to remain on the grid.

I've also heard from others currently on that their inventories and L$ balances are unaffected. So we can all breathe a little easier.

Also, in regards to the "compensation from LL" thread above.....it would be nice if LL would pro-rate our monthly payments, for those of us who are Premium members.

What's even better is if those of us who are in-world landlords, ad vendors, sign renters, etc., etc. give our tenants some compensation for the downtime. Currently, I only have one tenant and three vacant properties. But I will be giving my one tenant a few free days on his lease for the inconvenience.

Regardless if LL gives us anything for the downtime, those of us who have the power to give our tenants some compensation in-world should do so, and I think most of us will.

@Tamii: I'm in the main grid. Very few of us are on right now.

-peekay

Re- backups.

I'll bet like any large corporation that relies on network technologies they have contingency plans for MOST potential failures. However, no company has a total 100% unbreakable backup plan and obviously LL has fallen foul of Murphy's Law. As as lready been mentioned, they're likely on a Skeleton staff this week too.

If they're discussing with Hardware Vendors I can only assume that either:
1. They've recently installed something new that has broken .. big time so they're working with the vendor wo work out what happened and whether to just replace or g owith something else.
2. They don't have anyone on staff at LL who can fix this so need to go to their suppliers who should be able to sort it one way or another.

It'll be back. I managed to get on a few hours earlier, got lucky and logged during the brief period it was allowed. Had a crash a few hours ago and haven't been able to log back in but while I was there everything was present, had some issues rezzing items from my inventory and teleporting but otherwise it was OK.

It's just a shame for me personally that as I'm in the UK it's been down most of my day and I had a lot planned. I guess it'll have to wait until tomorrow, not the end of the world!

That's a really nice thing to do

ok central hardware failure example. They fried every network cable in the building, thier T1 (or what ever thier net gate is) caught on fire (yes this happened and the 24 hour less tech support took 3 days to get to us), thier NAS raid storage sytem (harddrives with no PC) may have gone offline cause the net controller card failed (gods I think they would need a monkey to fix that...or a large shipment!), or my personal favorite:

"WHO LET THAT KENDER IN HERE!"


PS: Kenders are annoying child like beings that fully grown act like kids and just start randomly pushing buttons, stealing things, etc. Imagine letting your 8 year old in Linden Labs data server center and saying "go nuts". There you go.

I am sure LL is pulling out their hair and are doing all they can to get this resovlved fast and get us back online. This is one of those "unforseen" issues that happen with all electronics.