What is going on ?

I cant remember the grid being down this long ( since last night) . Does anyone know what is causing such a big problem?

Elinah

Try reading the blog.

Just did ... I guess I wonder if this happened suddenly or got worse suddenly ?Or has it been getting worse slowly? I do remember a few months ago not being able to log on, but it just seems like this time there are many different problems all at once.

Elinah

They (LL) have a rule never to update something on fridays..... have seen that somewhere in the Blog a week or two ago sorry could not resist

/sarcasm on
The statement now talk about Routing and ISP issues. Anyway don't they have it every month..? I am sure the SLA of this ISP is as fair as the TOS LL provide to every user and nobody loose Money due to this bad error. ))
/sarcasm off

have fun in RL

OH I get it this is all due to the new update oh okay that explains it.

Elinah

This has nothing to do with any update. It is a problem with one of LL's ISP.

If the problem is with an ISP for LL then this could take a while. I wouldn't be surprised if SL is unavailable for at least another 12 hours or so or even until tomorrow.

The latest word from the blog indicates that they don't know: "We are still trying to put our fingers down on the cause for the problem." SL has had significant problems each day for over a week, and the past two days have been the worst.

Actually the blog states that they know what the problem is in a general sense. It is a problem with LL's ISP notwithstanding that many others on the forums are spreading misinformation tying it to the new viewer or havok updates.

Well this is getting too technical for me but is it fair to say that if there has been a problem with the isp lately there also would have to be the same problems in the past? And you have to admit a new update coming at this time wont help . You know how much fun we all have dealing with new problems every time there is a new update lol

Elinah

I wouldn't be surprised if this was a "Planned outage" to shut everyone out of the grid *during peak time* to see if they can pinpoint the thing that's been causing all the "transaction failures" in a "Controlled" environment.

But of course, PR wouldn't *LET* them say that.. so they made up a story about "their ISP's toplevel routing issues"....

I've got no proof that anything like that DID happen... but, I wouldn't be surprised.

Interesting point. That they want us out to pinpoint the transactions issue but then why dont they just say so ? We have had transactions problems in the past havent we? I think it may be due to lag. When there is lag I cant buy anything.

Elinah

Cuz at this point its gotta be embarrassing for them.

I heard a rumor that it was rats. Rats chewing on cables.

Public relations maybe? if they can blame it on some nebulous technical issue that isn't their fault, half the residents will rally to their side. If they came out and admitted that all the "fixes" of the last few weeks have been "waiting til the problem stops itself" then there would be bad press. I dunno, I'm not saying it's a well thought out corporate decision.. if it was, history shows it wouldn't have come from Linden Lab™.

But then again, I'm not saying that this *IS* what's happening.

I'm just saying I wouldn't be surprised to hear that it was.

Well at least beta still works. I can get some things accomplished that way . Even more so maybe because I dont get impatient and upload a texture to the main grid and use it before its really done, you know thinking its good enough for now until I put the finishing touches on it lol .

Elinah

PS Bree lol are rats better than termites?

The fact that the Beta works is actually the most confusing thing about this. If it were an ISP level issue, wouldn't the Beta Grid be affected as well?



At a phone center where I used to work, a squirrel got into the power box or transformer or something, and blew out power to the whole block.

I think they had a massive pie fight at Linden Lab and got pie stuff all over the servers and were too embarrassed and made up this whole thing while they clean up all the pie bits from the machinery.

I have no proof of this but it wouldn't surprise me if it happened.

/me puts on the old sandwich board "The end of the grid is nigh!"

hmm, I heard they forgot to pay the ISP bill.....
/me runs away

back to listing items on ebay!

Maybe LL should upgrade from Dial-up .......

I dont think thats a ISP Problem.
Before they resticted us to Login we was connected ... and we "only" couldnt make transactions, redresss, teleport and so on.

That looks really not like a ISP Prob that looks like an "Upadate" Prob.

I would be ashamed too and try to blame other if I would post

"The new viewer - a well-oiled machine?"

and one day after i have a over 12 hours downtime

Thats why I dont believe all what I get to read in the Blog



Greetings
Brad

Hmm. I briefly logged in about 5 hours ago, and no one was around. Not able to log in now though.

Seems that something very bad is happening. Hours and hours of unplanned downtime says there is a really big problem with the new viewer, that I was very angry about having to download yesterday. When Windlight was bieng tested, I couldn't even get it to work on my computer. I was worried it would fail this time too, but luckily it did actually start up, but looks awful.

I can see there are alreayd lots of threads about it, but I wanted to say that the new viewer makes all my avatar attachments look truly terrible. Everything looks shiny, and the more you turn up the quality of the grahpics, things get unbearably brighter, and brighter, and brighter...

Black attachments that used to seamlessly blend with black clothing now stand out as a completely different colour. My skin looks orange or pink around the edges and a lot of aesthetic effects that used to work in the old viewer now look wrong. Face lamps no longer rescue people from having prominent lines and "monkey shadows" around their lips and mouths.

After this big downtime is dealt with, I think there needs to be a lot of thought put into either having an on/off switch for the new viewer conditions, or completely new avatars.

But oh well, cos it doesn't look like the Lidens have even found out what the big problem is at the moment, let alone planned or executed a fix for it... O.o

What kind of cut-rate ISP do they use? Ive never heard of an ISP with so many problems. They are constantly having problems with this ISP, and whatever bozo outfit they use for their SF to Dallas VPN. Surly, if this was a major carrier, this massive outage would be affecting hundreds of thousands of web sites and been headline news on tech sites. But it's not. Nothing on Google news, nothing on CNet, nothing on zdnet, informationweek, or networkworld. Why?
The blog has just been updated with more garbage: "Many high-profile services and corporations are feeling the pain this morning, and full scale efforts are underway to isolate and fix the root cause." Who? Again, why hasn't this massive outage been reported on? Google wasn't affected, at least not in a visible way to their users. Neither is yahoo, AOL, amazon, ebay, or any other high profile web business.

This lack of any information is insane. This has been going on for nearly 18 hours. A competent ISP would have this nailed down and solved by now. LL needs to find a new carrier, or move their entire operation to a carrier neutral facility, like Equinix, where they can use multiple tier-1 carriers so they are fully redundant.

But we have no ETA. We have no cause. They have no clue. Not a good situation. I strongly suspect that LL will get one HELL of a service credit from this, and they, in turn, need to credit users / landholders. This is very damaging to the SL economy and SL businesses.

Maybe LL use use this massive outage to do something useful, like optimize the asset servers. Surely, their INTERNAL network is working... Right???? Then again, probably not. It never is.

LL: Act like a real business. Get some redundancy. Get a real carrier. Get things stable.

Hi Brad, welcome to the forums! I'm halfway between your opinion and others on this thread. Well, it's all speculation but this is a speculation thread:

No routing issue should keep a datacenter customer offline this long. Datacenters keep their network architecture pretty uniform so that they can train mulitple people to maintain it, which enables them to provide 24x7 coverage. So, I bet this investigation has gone beyond the ISP/datacenter, though I'm sure LL's "top-level routing issue" was the immediate cause of the outage.

Lately there've been brief time dilation and packet loss spikes on sims throughout the grid. That points to some weirdo traffic congestion issues that are not being managed well.

So, Nika's roleplay on how a problem like this might evolve:
/begin fantasy

LL to datacenter (DC): So FIX your damn network!

DC two hours later: Fixed. But it will happen again, because your traffic is impossible to manage. You are exceeding your allocation in this weird pattern: <graphs>

LL: WE HAVE AN SLA (service level agreement).

DC: YOUR TRAFFIC IS IMPOSSIBLE. You are not abiding by the terms of our contract. kthxbai.

LL to their own staff: What is up with our traffic? <hands engineering the graphs>

Now, once a problem goes to engineering/development, you need specific people, because unlike a datacenter, there is no way everyone in development can know about the entire product. This is inevitable in any engineering organization. So, the ideal guy will be Sam. Sam is a divorced guy who sees his kids once a month, and this is his weekend. But he thinks the problem might be in such-and-so subsystem. OK, Joe has also looked at that subsystem. Joe lives in some country far away and is asleep. OK, let's try Ted. Ted agrees to look at it but warns that it will take a couple of hours to come up to speed on the code, cause he's been doing other things for the last year.

In parallel, LL decides to look at another possible cause of the problem, something with the database. The team looking at that mentions that actually the longer this goes on the worse it will get, the volume of trashed transactions is getting so huge that the coherency of the asset databases is now at risk. ARGH! Shut the grid down, and run database verification/reconciliation.

By the way, Ted called his friend Stan in QA who has been whining about corrupt packets for the last two months. Stan has a set of diagnostics that he's been just itching to run on the maingrid. Fine, let Stan run them while the grid's down, maybe they'll show something. The diags take 1.5 hours to run.

And on and on, as more people get involved. Hopefully somewhere there's a problem coordinator keeping track of all the separate efforts at diagnosis/repair.

/end fantasy

Condolences to all at LL as well as anyone whose livelihood depends on the grid.

havok running havok with a "few unforseen issue`s"