What is up with all the LAG???????

Has anyone noticed how BAD the lag has been lately? I have been wondering if it was just me.

Stage 4 has been horrible since the move to it's currrent location. At alot of my events, I have notecards pre done to spead up the response time, from questions to answer. The cutting and pasteing from chat to notecard is a challange. We want alot of ppl to come to the events. The lag is making it hard for ppl to stay at the event. The Sims crash if there are alot of avs, with sounds. If anyone has an object to share at a show and tell, they can sometimes bring down the house so to speak.

The class area in Perry is worse. I get the horrible noise every now and then, near the benches where we are supposed to be able to sit. Instructors have been bummped out of the game while teaching. Not cool! The one thought is will they all still be there by the time I can get back?

One main thing ppl, ask when I am out and about boasting about how cool SL is. "Is there much Lag" I have always been able to say no, that would at least get them interested in looking into SL. Most gamers that are willing to check out SL have palyed on several other sites that are, either cheap or free. The maintenance of the game is the answer I give for the cost saying that it is one reason it is as "problem free as it is" IMHO ( compare to others). I have gotten 3 new friends to actually try out the game in the last week, of the hundreds it seems I tell. Figures the lag would be at its ALL time worst the week they come IW.

I would think the location would have been part of it. The areas I have mentioned are in more populated locations. But I live in Brown. Not many ppl up there. The lag can be as bad there also.

Whats going on???

Every time I have lag trouble, I run some tests. On a few occasions it's been my ISP's problem, but 90% of the time, it's a problem at Level3 (LL's ISP.)

I think Level3's network gets overloaded during peak times, and they're probably just not interested in fixing a problem that only occurs about 5% of the time. (During non-business hours, no less, so the only one of their customers it affects is the one that's running an online "game" instead of a traditional business.)

Of course, the power disaster a couple of weeks ago already convinced me that Level3 doesn't exactly have all their ducks in a row, anyway.

I've always got pretty bad lag in most areas. I've never had a moment in SL where the red lag-meter isn't showing 4-8% packet loss. If I fly to far with out stopping, I zoom off into nowhere land and have to wait a couple minutes to come back, so I have to make sure I stop and rest every half-sim or so.

And now I even get pretty bad lag and slowdowns in my own office building ever since the aliens made a giant build in Rose. Not saying its them but its a big coincidence

Ya Grim, I've been meaning to ask since I don't live in that area. Who is Level3? Seems pretty rinkydink to me. Why isn't Linden Labs right on the backbone like Sprint or AT&T or some such? Level3 probably is just a provider for a larger ISP so LL should cut Level3 out and use the same one they do.

Probably a simple matter of cost. In order to play with the big dogs, you pretty much have to commit to extremely high-volume usage. In most cases, the backbone providers don't want to deal directly with "small-time" projects.

Their probably also the cheapest co-lo facility in the SF area that (theoretically) has enough bandwidth to handle SL.

AT&T used to run a co-lo facility that SL would smoke on. Unfortunately for LL, it was located in West Virginia (I think) and I'm not even sure if they still run it as a co-lo.

I had a business card not long ago from a company that specialized in installing fiber in corporate high-rises, so companies could get T3 speeds in their office for about the price of a normal T1. Unfortunately, I'm not a network administrator any more, and I don't think I have the card. Can't remember the name of the company, either. :-/

Edit: BTW, I don't live in the area, either. I live in Georgia.

Most sims that aren't linden owned are under 100fps. That is WRONG. Something is VERY wrong with the sims, the sim FPS has degraded verrrrrrry low. Lindens, whats going on!?

Ok yeah lag is begining to get really bad lately and someone mentioned to me that its teh over usage of bandwidth on the "prims" that Welsh/Taber are so vehemitly aruging about with taxation. I mean not just Welsh and Tabers prims but the entire game as a whole.

To give my 2 cents worth, I admit I've had these problems myself. And it doesn't always show itself as packetloss or poor bandwidth. Worse during an event, but I usually expect that. However I do hear this complaint alot by others...I donno, I thought maybe it was an ISP problem in their area or something.

Not to mention the fact that in the last few days the mblaster worm has been wreaking havoc on connection speeds across the Internet as a whole.

I've noticed after the last patch im running around with 12-25 bandwidth showing on the meter. Before the last patch that may have happened once every few days.

I don't think it's so much the number of prims/textures/sounds in a certain area as it is the number of avs that the data is being sent to:

Say, in a particular location, for a single sim, with the default view setting of 512, there are 8000 prims, 6000 textures, and 100 sounds that need to be sent to a single av. Let's say that's, oh, 3MB of data.

If you have one av in the area, the sim tells the streaming server, once, to send all this data to the one av.

If you have ten avs, the sim tells the streaming server, 10 times, to send all this data to each av.

And on top of that, instead of sending out 3MB of data, the streaming server now has to send 30MB of data, because it's going to 10 different places. The more people in the sim, the worse the problem is. Once you reach a "saturation point" on the network, sim, and streamer loads, it gets exponentially worse each time a new person has to get it all.

Not to mention that the sim actually has to determine who needs to get what.

And, I think the standard/max cache sizes may need to be increased. I've been in areas where I can look in one direction for awhile, turn around for 30 seconds, then look back, and have to re-download some textures.

And what's going to happen once the world grows, and the population grows. Sure, the sims can handle it as long as the population is dispersed. But one day a Linden is going to announce the SL Lottery is taking place, 100+ avs are going to run to Mocha, and it's going to crash spectacularly, because it's too many people for a single sim to handle at once. (Especially when they all need to be streamed the exact same prims, textures, and sounds.)

I think, in the long run, although it's going to make several people unhappy, that we're going to have to have a "fat" client, and offload more processing to the client instead of everything happening on the sim.

When I got really poor performance someone told be to check my trash. Sure enough it had a TON of stuff in it. As soon as I emptied it, I noticed improvements. Other things that have helped were cleaning up my temp folders and cache.. This of course is beyond the usual suggestions of turning down draw distances and making sure things like lights & shadows are off.

I agree the sims have been increasingly more laggish.
Stage 4 has been doing some wierd stuff and sims like DaBoom and Taber are really bad.

The chat in the events is coming up differently for different people, so if there is Trivia, there are all sorts of problems to see who really was first. I am also getting that situation where you get somewhere before your attachments and then you get stuck waiting for them before you can go any farther.

Could it be that the world is filling up again? Maybe it's time for some new land.

I keep my trash pretty empty and my settings are always at the low end, draw distance 192, blah blah. I had to double check at one point, it was acting like I had them all the way up.

fen-

I've only been in-world a litle over a month, and I've also notice increasing lag.

I presumed it was the tectonic-plate movement as the UK is slowly getting further away from the US (er, okay....i'll just get my coat!)

Actually, I believe we are getting over an inch closer every year! In the year 3003, we can create a bridge!

Why does that math sound wrong James? 1,000 inches 1 inch a year for a 1000 years) is only 83 feet. Why don't I think that being 83 feet closer to the UK will make a bridge possible? hm.

The sims are running a bit slow right now due to some debugging code we put in to help find a frequent crash. We should be taking it back out in the next release so things will pick up again after that.

As for network lag, the current explosion of internet worms may have something to do with it, but if you are seeing long ping times to our servers (as reported by an external application, not SL), please send a traceroute, if possible, to [email]support@lindenlab.com[/email] and mention this post. We have an SLA that gaurantees maximum latencies to other backbones in the continental US and Europe. If Level(3) is in fact slowing down, I'd love to know about it.

Also, to clarify, they supply our bandwidth, but not our colo space - we moved out of the Level(3) colo after they unplugged one of our cross connects for no apparent reason.

Geez, Ian. Sounds like you guys just have no luck with CoLos.

Is there a cutoff point that justifies sending in the traceroute? I mean, I think a jump from 50 to 150 is a symptom of a problem, but I don't know if you guys and Level3 feel the same way about it.

And you don't want to get flooded with a whole lot of "almost a problem" e-mails, do you?

I would say that if you're seeing a >70ms latency within the L3 net, in the US, then send mail. >100ms for Europe. If you don't know how to determine that, don't worry about it - local ISPs will do all kinds of fun stuff to your latency and I can't fix that. I'm betting that most slowdowns are not in fact L3.

Oh sure, so what about Canada eh?

edit: 3 of the last Level3 hops, for me, are 82, 124 and 94 ms

Uh.... Canada 70ms, Mexico 100ms; outside of North America and Europe, I've got no gaurantees.

Have a look at this one. This is similar to what I get at home, but in the evenings, the jump from hop #10 to #11 is larger.

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                              ********** -    0 |  113 |  113 |    0 |   11 |   31 |    0 |
|                              ********** -    0 |  113 |  113 |    0 |    1 |   31 |    0 |
|                         *************** -    0 |  113 |  113 |    0 |    1 |   16 |   16 |
|      bar6-serial4-0-0-18.Atlanta.cw.net -    0 |  113 |  113 |    0 |   15 |   38 |   31 |
|            agr2-loopback.Atlanta.cw.net -    0 |  113 |  113 |    0 |    7 |   47 |   47 |
|            dcr2-so-7-1-0.Atlanta.cw.net -    0 |  113 |  113 |    0 |    6 |   47 |   46 |
|         dcr1-loopback.Washington.cw.net -    0 |  112 |  112 |   15 |   21 |   62 |   62 |
|    bpr1-so-3-0-0.VirginiaEquinix.cw.net -    0 |  112 |  112 |   15 |   25 |   47 |   47 |
|                           208.173.52.74 -    0 |  112 |  112 |   15 |   22 |   62 |   47 |
|                      unknown.Level3.net -    0 |  112 |  112 |   15 |   23 |   47 |   47 |
|   so-3-0-0.mp2.SanFrancisco1.Level3.net -    0 |  112 |  112 |   78 |   87 |  110 |  110 |
|   pos9-0.core2.SanFrancisco1.Level3.net -    0 |  112 |  112 |   78 |   84 |   94 |   94 |
|gige7-2.ipcolo1.SanFrancisco1.Level3.net -    0 |  112 |  112 |   78 |   84 |  110 |  109 |
|                 data.agni.lindenlab.com -    0 |  112 |  112 |   78 |   88 |  235 |  109 |

Grim that pretty much exactly what I got. Same jump - same place.


Tracing route to secondlife.com [63.211.151.3]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms gatekeeper [10.1.1.10]
2 1 ms 1 ms 1 ms 149-99-33-25.dsl.tor1.sprint-canada.net [149.99.33.25]
3 4 ms * 14 ms agg1.tor1.sprint-canada.net [206.186.255.240]
4 7 ms 5 ms 5 ms gigabitethernet0-0.bb2.tor1.sprint-canada.net [204.50.251.2]
5 12 ms 8 ms 10 ms gigabitethernet11-1.bb1.tor1.sprint-canada.net [204.50.128.57]
6 18 ms 30 ms 24 ms sl-gw9-nyc-7-1.sprintlink.net [144.232.169.73]
7 18 ms 20 ms 17 ms sl-bb20-nyc-11-1.sprintlink.net [144.232.7.93]
8 17 ms 17 ms 17 ms pos6-2.core1.NewYork1.Level3.net [209.244.160.153]
9 24 ms 29 ms 33 ms ae0-54.bbr2.NewYork1.level3.net [64.159.17.98]
10 82 ms 86 ms 79 ms so-0-0-0.mp1.SanFrancisco1.level3.net [209.247.9.117]
11 124 ms 100 ms 115 ms pos8-0.core2.SanFrancisco1.Level3.net [209.247.10.230]
12 79 ms 80 ms 94 ms gige7-2.ipcolo1.SanFrancisco1.Level3.net [209.244.14.226]
13 94 ms 78 ms 82 ms arachnid.lindenlab.com [63.211.151.3]

Trace complete.

The big jumps are the trans-continental links - the light delay from SF to NY is about 35ms across the surface of the earth. These do seem a little high, but not high enough to noticeably affect SL. I'll keep an eye on these routes but send mail if you see spikes around primetime (early evening).