The Lag Monster Myth-- Part 2 ;) -- Particles

Lag is undeniably one of the major problems on SL. Everyone knows it. LL admits it. But what do we do about it? In the interest of assisting LL in discovering the source of this problem and fixing it to everyone's advantage... the following.

One recommendation is to focus on the source of the lag rather than patching symptoms. Until the source of lag is located, lag will continue. One fear often voiced is that LL knows where the source of the lag is... but is at a loss what to do about it. That is perhaps where client expertise might come in. There are many people out here who are experts in their fields. They are a tremendous potential asset to LL and SL. But they can only help of LL is open about what's going on internally and allows them to consider the issues. Who knows what solution might be found. It is well known that two heads are better than one.

PARTICLES!!!
Just this week I noticed something about particle behavior. As one of those odd coincidences-- within just 30 minutes one of Elf Clan's best scripters-- a specialist in particle manipulation-- IMed me with exactly the same observation: particles are not consistent in behavior. Since particles are supposedly generated client-side, this left us with a major question: why do particles "lag"?

The day before I had built a set of 6 torches. The flames were bright, consistent and lit up the area. The very next day, the same exact torches, in the same exact sim, in the same exact location, with only 2 avatars on the sim, were flickering, dying out, LAGGING.

Apparently our scripter was noticing the same effect with her particles. We had both witnessed such many times before... but this was so noticeable it suddenly hit us: what would possibly cause particles to lag?

I'm running a dual-core computer with 1 gig RAM and a high-level Nvidia graphics card. My internet speed is blazing fast-- faster than it has any right to be. My system drivers (including graphics) are all up to date. Other users were experiencing the same problems I was experiencing. So the problem isn't client equipment.

That brings it down to two conceivable areas:
1) Either SL client-side software is buggy, or...
2) There's a lot more to server-side rendering of particles that most people claim.

As far as I know, these are the only two logical explanations (if there are others, I welcome feedback).

I have long suspected that particles and other graphics issues are heavily server-side influenced. Sure, the actual rendering of the particle may be client side. But at what coordinates is that particle rendered? If an avatar has a glowing sword and moves, who tells the client where that glowing sword moved to? The server, of course. So while actual rendering might be client side, it stands to reason the server has to provide the client with constant information as to where and how to render that particle.

Further evidence: A particle alters, shifting in color, size, intensity and wind direction. Surely that all is not computed on the client side. If it were... each client would be seeing different particle effects. (I dunno, perhaps they do). But it stands to reason that the server is constantly updating the clients with rendering information-- including particles.

If there are severe server lag issues, that is going to lag everything... including script function, walking, flying and yes, rendering particles.

Which brings us back to the initial supposition made so many months ago: primary lag is not client side nor client content: it's server side and programming issues.

Last time we had this conversation, there were quite a few people who denied there could be server issues. They stated that our sim was maxed on prims and scripting (even though we controlled such scripting on that sim and at times... the sim ran just fine and lag free). The point was made over and over that if client content were the problem, the sim would always lag. Yet quite often, it ran just fine.

Now, I am hosting a sim that uses only about 8000 prims (out of the possible 15,000). We are heavily script-controlled. Users are regularly asked to remove items that can cause undue lag. Bottom line: our client content is monitored. Quite often the sim runs just fine. But quite often it lags as well, significantly, to the point that even particles render poorly.

Denials to server-side lag issues were further impacted when it was learned that SL stacks 4 sims to a box, using one single hard drive, bus system and network access card. It became obvious that there are times those four sims might butt head to head for asset use.

Another piece of evidence that happened 2 weeks ago established this even further. I logged on to SL... and something seemed to glitch. I couldn't identify it... but you know how when you do something regularly... even the slightest alteration is apparent. Something in the logon was... odd. This became amazingly apparent when I tried to move: I was moving at super-speed. Not only was there no lag... there was anti-lag. A walk was a run. Flying zoomed me across the sim in no time. Graphics rendering (except for textures) was instantaneous. That made me wonder... what was it that happened that day that removed every trace of lag? One possible answer: system governors, intended to slow down user activity. Such governors are essential to operating any 3-D system. But could it be that sometimes those governors are over-active, slowing users down the way histamines slow down people during allergy times? I can only speculate.

All that I know was that on that one day, there was ZERO lag. In fact, I was moving too fast for comfort; it was difficult to control. So obviously some kind of speed governing is necessary... but not to the point that it hampers activity.

So I find myself, after almost 2 years on SL, still holding to the same observations: while client content CAN cause significant lag... the primary and major source of lag still appears to be server-side and/or SL software oriented. And until the source of that lag is found and corrected-- no client efforts at controlling lag are going to solve the issue. We're almost wasting our time by even trying.

So what I'm throwing out for discussion is the concept of particles lagging. All things being identical, what could possibly make particles work just fine one minute... then fizzle the next?

(please note: the claim of "changes in client content" has of course already been considered, negated, discarded as not the case here. So we're looking for other possible explanations)

My position is it's server/software issues and that this is where LL needs to focus their attention. Any other ideas?

I always get into it with people about their particles (which are being used more intelligently these days it seems).

I believe the key issue is texture. If the particles are textures, and the textures are larger than 512x512, the person should be pimpslapped.

Area of effect has a lot to do with it as well, I believe. If you're flinging particles round and round half way across the sim... tada. If you want it to rain, that's cool. But it can't rain all the time (name the lead actor in that movie).

last, but by no means least: laggy particle scripts aren't normally just particle scripts. If they are listening scripts, spank! If they do other things that create lag: spank. But we see particles, so we blame particles.

Client lag can be minimized with a beefy machine. Server lag can be minimized by better texture/script combos. And true lag - lag of a connection - is a combination all as the two try to continuously synch.

When you're losing packets... you're out of synch. That's true lag. Splitting it into two separate entities is good for explaining to the non-technically minded, but the real issue is synchronization.

You make good and valid points. To clarify this issue:

The particular particle system mentioned above was an open-script, standard "glowies" particle. No textures being used to speak of-- just standard particle generation. To my knowledge, it is one of the most non-lagging particle generators on the planet (I've used it for ages on various things). Still, it lagged, whereas before it had not.

Yes, you're right that some particle scripts and generators are very badly behaved. That would constitute a problem with client content. But in this case, that wasn't the case. These scripts were very well behaved and generated standard particles rather than textures.

OK. What you're saying makes sense - but it's more than likely a bandwidth governor. I don't have any insight into LL, but a bandwidth governor fits the data you've given... I wonder, on your 'antilag' day - how many people were on the sim and in SL in general?

The 4 sim per server is ok, I guess (it seems a bit klugey). Sure, they can pump more hardware at the problem, but... unless it can be determined that this is a loading issue (as ini server load, number of connections, transfer, etc...) throwing more hardware at the problem just hides it.

Since lag is also largely a synchronization issue, I wonder about how internet connectivity throughout a sim varies the lag. For example, if someone from Brazil has their AV standing next to their newfound friend's AV who's body is in Iceland, would that affect lag? I think it would; synchronization between the two on the sim would also be dependent on independent internet connections. Get 40 people from around the world on a sim...

Consider: http://www.internettrafficreport.com/

With the new free accounts (I used to be one) with people from all over (I've met people from Brazil, Iceland, etc), this is bound to be an issue. And then the smoking gun leads us to none other than everyone's favorite people: Internet Service Providers.

I'm guessing - I may be wrong - that a lot of oldtimers here are in the U.S. where the backbone is *right there*. Then I can break into WSIS, but that might be a bridge too far for some.

Yeah, I've thought of bandwidth (are they providing enough bandwidth per user to handle something as complex as SL? As far as I can determine, it's quite often less than 56k). And certainly, there are times when lag is a bandwidth or packet loss issue.

However, unless the issue is on SL's end... that doesn't explain why several people can be standing around and all lag at the same time and in the same manner. That pretty much eliminates client-side issues (unless of course, the client software has a severe bug or two that affects all clients equally). It also eliminates pretty much packet loss or local internet issues... because everyone is experiencing the same thing.

This is compounded when people grid wide IM me with the statement, "Wow, lag is really bad today"... verifying my own experience for that particular day. When that happens, my consultant radar comes on and says, "Ok, the only thing that can cause that is either a massive internet issue (not likely) or serious deep-core central server issues (very likely). That's pretty much what it seems to narrow down to-- core issues that need to be hunted down and fixed, just as they hunted down and fixed the ghosting issue.

The downside is that it took them over a year to find the ghosting issue-- and even then it happened only when users started throwing a fit. I think this lag thing is pretty much the same thing. So instead of coming out with new toys, someone needs to go down into the basement and start checking for leaky pipes and blown fuses (metaphorically speaking). My money is on deep-core issues. Or as you stated-- simply not providing users with enough bandwidth to do the job. For the money we're paying... that should never be the case.

Other particles. Your client has to divide the available particles among EVERY particle emitter in draw distance. Whether you can see it or not. Because until the particles are generated there's no easy or straightforward way for the client to know if the particles are going to be visible to you eventually or not. If your draw distance is 128, and you're near the edge of a sim, the particles that are competing with yours could be on another sim. If you have a good graphics card, your draw distance is probably 256 or more, and you could be rendering particles in three or four sims!

3) there's more to client-side rendering of particles than you're aware of. In fact there's a lot more to client-side rendering period than you're aware of.
The client calculates almost all of it. The server has no idea where the sword is. The animation that places the sword here or there happens *entirely* on the client. The only the the server is doing is sending position and velocity updates... and not many of them. If the client misses some, it keeps the avatar moving in the same direction until you get an update. That's where rubber-banding comes in.
Maybe reasonable, but wrong. No matter what the server does, even if the server stops sending updates at all, particles are still rendered correctly. You can start an animation going with particles showering off your sword top, DROP YOUR INTERNET CONNECTION, and the particle system will keep running.
It won't lag particles, it won't lag animations, it won't lag rotating non-physical objects. All these things happen without any involvement from the server except to set them in motion.
Other particles. Even particles you're not aware of, that you can't see, that you will NEVER see, are still rendered if the emitter is in your draw distance.

I've been thinking more on the lag/bandwidth issue... and a lot of things regarding that make a lot of sense. I may be totally off base here; there is absolutely no way for me to know. Mere speculation based on visible evidence.

I first noticed the current lag issue near the end of April 2005. We were setting up a new sim at the time, and that sim was majorly prim and script controlled. It was running very well. Then one day, bam, lag hit hard (I think it was around April 27)... and SL has never been the same since. It had all the indications of someone at LL flipping a switch. That switch could very possibly have been a bandwidth-per-user reduction.

When I first joined SL there were less than 25,000 members. Lag was not a major issue (teleport freezeups and ghosting was, but lag was secondary to those). There were periods of lag, for sure. Always have been. But not like todays lag-you-to-the-floor and force you to log off type of lag.

Now SL has what, more than 250,000 users? I have not seen the demographics on how many unique users are on SL at one time (I'd sure like to see such. When are the peak periods on SL anyway? I'd love to see an hour-by-hour 7 day demographic). But if SL has reached a point where the number of users is saturating their available bandwidth... that could explain a lot of things.

That would explain why on some days the entire grid seems to lag (lots of people logged on and having to share bandwidth).

It would explain why an increased number of avs on a sim cause extreme lag-- more bandwidth required of that server (the graphics rendering really should have very little to do with it. I've played PvP games with up to 200 participants with no discernable lag).

It would explain why people experience lag grid wide, no matter where they are.

So the bandwidth concept does explain some things. However, it doesn't explain everything. For example:

It doesn't explain why everything will be going fine, then suddenly, bam! Everyone on a sim lags, Time Dilation drops down to 5, Sim FPS drops to 0, and no one can move... and then 30 seconds later everything is fine again.

It doesn't explain why that happens 24/7/365. (Some folks claim maybe something "changed" on the sim. Sorry folks, client content doesn't significantly change enough to lag an entire sim that many times a day, every day of the week. This points to server/asset/software issues).

It doesn't explain why there will be lag during what should be slack periods, with a minimum number of players on the grid.

So more than likely, severe lag is caused by more than one issue. But I do believe there is a good chance that current SL bandwidth is not sufficient for the current number of users. If that's not the case, LL is certainly welcome to provide technical data to indicate that bandwidth is sufficient to meet user needs in three areas: core server bandwidth, sim-server bandwidth and net-access bandwidth.

(In other words, the amount of server access bandwidth in shared data servers, the bandwidth allowed each sim, and the bandwidth allowed for all of SL to access the internet at any one time).

It's a thought to ponder, for sure. Again, as much as we're paying to use SL, at the least Premium users should be provided all the bandwidth needed for seamless interaction. But again, I don't think limited bandwidth is the sole culprit. There very well could be two or three lag monsters that need slain. But in earnest, I doubt there's more than that. I fully suspect just 2 or three core issues that are responsible for the vast majority of the lag. Once those are found and addressed, lag will go the way of the Ghosting monster.

You're saying that the server has little to do with this. Yet there seem to be loopholes in the logic.

Just because a client might not receive information from a server (ie, if there's a packet loss or some other such)... it doesn't mean the server isn't sending that info. What the client receives or doesn't receive has nothing to do with what the server is doing in the background.

The client continuing to render particles even if it's offline is true... but that's not the whole of it. That is nothing more than the client side software continuing to render particles based on the last set of information it received from the server. IF that client were still online, that server would still be sending it particle update information... especially on particles that are constantly changing, both in form, color and position. I seriously doubt that there are "not many updates" the server sends a client. From what I understand... the standard server-client update is scheduled at 20 per second.


If ALL particle rendering is done client side, with very little update from the server, then the faster my computer, the more particles I should be able to render... and if I have a really fast computer I should have no difficulty rendering particles at all. Since I do have a really fast computer... then why are particles still lagging?

I understand what you're saying about "all particles on a sim". I was already aware of that info. However, 2 things on that: if a sim is an island with no surrounding sims, then only the particles on that sim would be involved. In addition, clients have the ability both to reduce their draw distance and to specify how many particles they render at a time. On my sim... excessive use of particles is prohibited. So if I have my draw distance set on 64 or 128 and my particle count set correctly (I believe the standard is 4000... but I have at times set it to 2000 or 8000 to see if there is a difference)... the particles should render like gangbusters. Yet they still lag at times. Then at other times they work just fine. Since in both instances there were only a couple of avs on the sim and the same particle generators were involved... the logical conclusion is that it's not other particles causing the problem. The particles worked fine. Then they lagged. Then they worked fine. It's something other than client-side rendering of particles.



What happens if those items STOP rotating? What happens if they shift position? What happens if they're attached to an avatar or change color or size? Argent, the client cannot know such things unless the server tells it. So I think a little more is involved here than just setting items in motion. As a Linden once said, even if a script isn't active, every cycle the server has to check and see if it has become active. So every script, active or not, takes time. The same holds true for every single prim, texture, particle generator, etc. While the actual particles may indeed be totally generated on the client... the server has to tell the client how to do that... just like with everything else on the sim.

Further evidence: if it was the client that did everything... then the client should run totally without lag, shouldn't it? If it's not the server or SL client software, then realistically speaking, the client that just a few seconds before was running SL with no lag... should continue to run SL with no lag. And there should never, never be an instance in which SL lags for everyone. The fact that lag does exist, and that it exists for everyone regardless of their computer power or internet speed... I would have to believe indicates more than client-side rendering issues.

That's the thought presented here: things aren't stacking up to observable evidence. There appears to be more to server-side issues than commonly believed.

to me the problem is into the avatars, they really need to be severely capped in script time and prim numbers

I have to agree with you there. I know the notion is unpopular, but avatars walking around with 200 prim wings and hoochie hair have long been known to be super-major lag factors. Evidence indicates that one heavy-prim av can cause more lag than thousands of static prims on a sim. It's pretty obvious that the avs on SL are beyond the ability of their current technology to properly render. You get 20 avs on a sim, you notice it.

But the reality is, LL is unlikely to do anything about that. People love complex avs.

But here's the thing: those avatars would be a lot less laggy if LL would find out what's causing the CORE lag.. the primary, deep lag issue that is causing the major lag problems to start with. Because even with heavy-prim avs, I can walk around a sim if everything is running smoothly. But if deep-core lag hits, you can have one av on a sim wearing no prims and still lag to a standstill.

But you're right. I've long felt that avatars should be limited to 200 prims. Or maybe 100 prims for non-paying members and 200 for Premium members. There should be bonuses for being Premium.