Massive Packetloss!!!

Has anyone from LL even so much as acknowledged the issue with packet loss? If so, are they doing anything about it?

Normally, I'd let you get away with the answer that you drop ICMP traffic, giving priority to your UDP traffic, but when I see the same PL in game and on a trace, I'd have to discard that as an excuse.

And please don't insult me with claims of it being "my computer, my internet connection, my eye color, etc"

http://img124.imageshack.us/img124/16/packetlossvu1.jpg

Let alone packet loss, Teleports dont work 99 out of 100 attempts. Sims are crashing like OFTEN. C'mon guys we want to play. I didnt play sl for like 5 days now. I just log in and try to teleport, if it doesnt work i log out. You'll loose some residents that way...

It's not just you, it is grid wide

And today is the worst I have ever seen! Sheesh, how many times can I possibly make that statement and still have it be true?

Today, SL is absolutely complete unusable. I have been unable to do literally anything for hours. Cannot upload, cannot even see my own avatar at all (not even attachments), cannot see anything in my inventory (it never updates from the server), simply... cannot... do... anything!

So, yeah, I am just venting and letting you know that it is not on your end at all. SL is simply borked at the very deepest levels.

And the problems will only get worse. Nothing is being done to stop the flow of growing populations which is causing everything to screech to a halt. I also am concerned about how Lindens shut of the blog comment ability regarding todays issues-our opinions and concerns they just don't want to hear..........

Well, let's see if I get this post deleted and/or banned from the forums for this...

The bottom line is that their network connectivity to the outside world is jammed, and when that happens, routers start dropping packets, and things just run slow as crap. Now, what causes this, you ask? It can be a few things:

1) A multi-peered connection has lost one of its peers, therefore knocking the availiable bandwidth down, sometimes quite substantially. In my line of work (webhosting, network engineering, streaming media services, online game servers, etc), I've seen a 40 gigabit circuit drop to less than 5gbps due to this. It's not a pretty picture, and everything just goes to hell in a big hurry.

2) The person and/or company just isn't paying for enough bandwidth for their needs. This is the old "watermelon through a garden hose" analogy, where you have 10gbps of sustained outgoing traffic, and you try to jam it through 8gbps of pipe. Usually what's needed here is for the purchasing party to just bite the bullet and upgrade to what they need. Unfortunately due to ignorance or lack of research into bandwidth pricing, they end up being severely overcharged because a lot of people still have the mentality that "bandwidth is expensive!" from about 1998, when as of late, quite the opposite is true.

From the things I've read on here, and from general consensus opinions of how LL runs things, I'm inclined to think #2 is the most likely option. What they need to do is get with a company like Equinix that peers directly with the majority of the cable and dsl providers in the country for VERY good rates (think on the order of cents per mbit as opposed to dollars) and just buy enough capacity to sustain it and have some room to grow. Picking up 3 or 4 10gbps connections, then buying bandwidth in bulk from major providers gets you good rates when 3/4 of your traffic is being sent directly to the cable and dsl providers without even touching the "real" internet.

In a nutshell, LL just needs to get their crap together and learn the "in-and-out" of how to properly build a high bandwidth service instead of buying from a Tier 2 provider and getting bent over on price and quality.

Gypsy, I turned comments off for the blog because it's just an announcement. We can talk about the problems here if you like!

What's happening is that our bandwidth became saturated today. This plus there's a crash bug that we are still trying to squash. Our bandwidth supplier is going to bring some back-ups to primary level tomorrow which should help the problem initially, and then we'll be adding another fiber line shortly (I don't have a date, but hopefully in the next week or so).

I quoted Robin's post in order to be able to read it, because for some reason many of Robin's posts are in 3pt type, at least in Firefox.

In the meantime, does that mean that SL is shut down until then? At least, it seems that way at the moment.

[Edit] I finally saw the blog update, and it seems as if you are trying to get things back online. Thanks!!!

Robin,

The packet loss isn't just a "today" problem. It's been about a month now. Maybe TODAY you saturated your pipe and thats compounding the problem, but packet loss has been a daily problem.

Might I suggest actually analyzing some of your infrastructure with RMON or SNMP.

That's rather disingenuous. It's quite clear that either a) you are having serious internal network issues causing packet loss, or b) you've been overcommitted on your bandwidth needs and saturating your pipes for quite some time. As stated above, packet loss isn't a "today" phenomenon; it's been going on for at least a month, if not three.

"*A* crash bug"? We've been fighting with crash bugs with our sims going on 7 months now. We've reported a problem that causes crashes with sims that is easily reproducible. Response to such? "We're working on it / looking into it" and then silence. For months.

Right now I am logging crashes on our three sims, seeing 50+ crashes on some days. Is this the level of service we should get used to? If it isn't, please let us know, because a lot of us have had no choice but to get used to it.

Quite correct Jokey, this problem did not just start "today".

If you check this tread-

/142/a1/145456/1.html

you can see that on 10-25-06 many of us addressed this issue a month ago, yet no response was given by LL as to what the problem was, or why it was happening. I am glad to see this brought up again.
We ALL are experiencing this, it is making scripting, building, TP'ing, walking, and even CHAT impossible. I know personally I have had no desire to even attempt to create new content, for I am left completely frustrated.
Today has been a nightmare, the 30 mins I did decide to log on - I logged on to find i had $0L - couldnt buy items - couldnt rez items - rezzed in as nothing but the prims I had attached to myself. So another question, aside from this crazy packet loss problem, what is now going wrong with the asset server? Last night we were hit with blue messages saying "we know there is a problem and we are working on it", and then.....exactly not another word said. Today log on, same exact issue...what exactly is being "worked on"? I dont know about anyone else but, I see nothing as "the same, better, or improved".
*sigh*

I think I found the problem!

http://lindenlab.com/employment/sys_eng

Maybe nobody is driving the bus?!?!?

LOL makes you wonder. I to have had this packet loss for some time. It was better after one update and everyone was complaining about it and LL issued a patch to that update the next day or so. But then the next update (server this time) it started again and has only got worse. Sorry I don't have dates, but a month does sound about right.

-DB

I'm fairly certain that the packet loss became noticable for me directly after the very first update they did after I started out in SL. I remember it pretty well as things ran rather smoothly and then suddenly it was all borked.

I joined SL a month and a half ago, the update was a couple of weeks at the most after this I think. When i started, things loaded properly, the lag was bearable even in crowded places and packet loss was something that was yet to happen. There was a large update and .. whoa what the hell happened? Textures load after a week, walking into so many walls my face is now flat and that red bar in the top right corner has now burned a permanent copy of itself onto my screen.

Burn the update, whatever it was.


Heh, and as for the jobs available, it does look like there's too many positions free in rather important roles.

Elaine Dickinson: "There's no reason to become alarmed, and we hope you'll enjoy the rest of your flight. By the way, is there anyone on board who knows how to fly a plane?"
Airplane (1980)

The update where the packetloss started for me was the mid-October one I believe. Since then, I average spikes from 4%-76% (yesterday was my record). I had never even experienced packetloss before that doomed October update. To imply that this is new is well...not accurate

If you look at the forums, no less than 2x a day someone posts a question about why this is happening. People have been screeching about LL paying attention when this very suddenly started happening about a month ago immediately post update. It was always, "check your hardware; update drivers; or it's your connection." It isn't...

Take care,
Gillian