The Lag Monster Myths

If you all will excuse me, I'm gonna bow out of this thread. All the evidence that could be presented has been presented. If some individuals refuse to acknowledge or accept that evidence... well, not the first time I've seen such. I find myself debating with folks who have never owned a sim and who haven't had the experience nor done the research to support their claims. We own three sims. We did the research. We did the statistical analysis. Few others have. As through all of history, when people refuse to accept the facts, they wind up paying the consequenses. You can proclaim how wonderful SL is till the cows come home. That won't bring back the thousands of users who've abandonded the system, nor will it retain clients who now are becoming increasingly more frustrated with a system that is apparently being managed for the good of the company rather than the good of the client. When a company forgets who pays the bills-- it's due to eventually be reminded.

Have fun guys. I have more important things to attend to at this hour than to re-hash old arguments.

actually i own two, and have prolly done more real actual sim analysis than almost anyone in SL including LL themselves...

LL is responsable for providing, for your $195, one sim, of the type that was most current when you bought it, and for mantaining the system thereof.... where they lapse in that responsability i have NEVER seen them be anything but quick and apologetic

what *you* are demanding however is that they run *WHATEVER YOU WANT* at x arbitrary speed, and thats bull. You are getting what you paid for, you just aren't getting what you 'want' and have some problems confusing the two

Take heart. After 1.7, Linden Lab will no longer be able to use "too many scripts" as an excuse for poor sim performance. 1.7 has changed the script scheduling routines. Scripts will get a set amount of cpu time and will no longer be able to bring a sim to its knees.

I wonder what they'll point to, instead.

And I disagree that we're getting what we paid for. It's more accurate to say we're gettin what we accept. We should be asking for more. For example, we have no SLA (service level agreement) with LL on a private estate. That's ridiculous and runs counter to industry standard. No one has made enough noise about it, so LL gets away with it.

You can argue that an SLA is an enterprise-level aggreement. It is. But, how can we *build* an enterprise with no reasonable expectations of service? Unfortunately, the TOS seems to fly in the face of any expectations of service.

Here's a thought. Do you really think that Wells Fargo is running their little island without ANY form of SLA between them and Linden Lab? If you were Wells, would you throw money down a hole like? I'll bet ya that the TOS has been waived for our banking friends and they've GOT a service agreement. LL, care to comment?

Do you think Wells has shell access on their server? Do you think they have access to performance logs and customer logs? Do you think they've got a DEDICATED support team from Linden Lab? You bet yer ass they do. And they should. It should be EXPECTED with a private estate. ALL private estate owners should get that level of service.

We just havent demanded it. Jessica Mullligan (a famous game-industry analyst) writes a great blog piece on the poor level of service in MMOS here:
http://www.skotos.net/articles/bth.html

She's right. We aren't gettin what we pay for. We're gettin what we put up with. And the MASS MARKET (which LL desperately wants and needs) won't put up with it.

I'ma go lie down, now.

Just wondering, do you have proof of any of this? People that own large amounts of land do get special treatment. Island owners and people that own more then half a sim have the concierge service. I have not used this service in quite a while, so I am unsure how they are now. But island owners do have the service that most people don't have access to.

not even a chance. They will get exactly the same amazing level of dedication and support, that everyone else who owns at least half a sim get, the concierge. And because they are a large successful business in their own right, they'll recognize good service and dedication when they see it, instead of digging around for fantastical conspiracy theories and other silly notions.

Paulie, since Eltee is being so blatantly insulting, and I cautioned him/her/whatever in a private message that I'd deal with the same respect we're dealt with, here's my uptake on this... from one of us paranoid conspiracy theorists.

My land group spent weeks and then MONTHS analyzing sim speed, raw data, statistics, etc. We posted it here and sent copies to Linden Lab. We copied down sim data, stats, called Lindens to our sims. NOT ONE TIME did we hear back from Linden Lab on ANY of our findings (except of course, from Lee Linden when we finally decided to take our findings public). From a professional standpoint, I can only assume one of two reasons for this:

1) It is LL standard policy to not acknowledge customer research for whatever legal reasons (which constitutes really, really bad customer support policy) or
2) Because LL did not wish to acknowledge the validity of such findings.

When people such as Eltee claim to have so much knowledge that no one else comes close to their level of expertise, and further, when he along with Dnate continue to ignore empirical data... I quickly realize we aren't dealing with the level of professionalism that I insist on from my business associates. In our field, we use the term "tech hacks". Lots of basic knowledge. Incredibly self-assured to the point of ignoring that others do have similar--or even superior-skills. And they're usually the first to accuse others of being totally clueless as to how things work.

That's all I've seen from these two. I've seen no presentation of data. I've seen no facts. Eltee's so-called figures are constantly presented while totally ignoring the facts that

a) Eltee's claims are based on one single rushed sim measurement which took all of some 15 minutes and
b) It totally disagrees with all other data readings taken from that point on
c) He continues to totally ignore revisions we made to our sim which proved his intial assumptions incorrect.

Now in my book, 3 strikes makes a "tech" I wouldn't hire on a bet.

When a so-called expert starts behaving in such a manner, I come to realize that it's a waste of time to further discuss the issue with them. All I've seen from these two are personal opinions and conjectures and an insulting attitude toward those who don't agree with them.

I'll respect people's right to their opinions to my dying breath. I don't respect the unprofessional way in which these opinions have been presented, the lack of proper research and data which accompanies their claims, nor their totally ignoring valid data gathered by others for months on end. Such a "hack" way of approaching system analysis is one of the main reasons why so many companies have serious computer problems that never seem to disappear: their techs can't see beyond their manuals to the real issues underlying system failure.

you don't have any valid data, we keep trying to tell you what you need to get, but you are so wrapped up in your self-importance you really don't care. Whatever

i've done absolutely nothing but try and help you and your area, i believe strongly in theme builds and wish for as many as possible to succeed in SL.

in retun i've gotten absolutely nothing but scorn an contempt... in one breath you say i talk nothing but tech gibberish you don't have the time to bother to try and understand, and in another breath you dismiss everything im saying as silly heresey.

Thats called hypocracy

you go to a random sim, read off some random values and post them as if they have any meaning at all when you don't even begin to look at the actual content. Then in the same breath you talk as if months of long collected data in SL that have *literally* been detailed enough to suprise the lindens and get them to look back and often revise their own understandings of the way certain sims are performing, and say its a hack 15 minute job.

here is just a fraction of the work i've done with this indeed helping LL sometimes see problems they were not aware of in their sims..

i'd ask you to kindly read through them before you go and try and blow off the help other people are honestly trying to give you... but then they are so full of meaningless tech gibberish im sure they have nothing at all worthy of your high and mighty concern

/111/28/49510/1.html

/111/0e/49045/1.html

/111/c3/41406/1.html

/54/c8/25865/1.html

/130/ca/29461/1.html

/111/a3/25483/1.html

/111/fb/11584/1.html


and thats only a cursoury search back in time... I would ask, respectfully, you actually note the kinds of arguments and evidence presented by myself, and many others, and also note the responses from LL as more accurate, and specific evidence and probable explanations for sim performance are given.

i've tried several times to kindly inform you of how to spot some of the actual extant simulator problems, but you've blown me off both times...

if you want to pull the cotten outta yer ears and actually discuss the known existing simulator back-end problems and methods of recognizing them, i would have heen happy to, and tried to, several times today, but at this point you got a long way to go before i bother to try and help the likes of you again wayfinder.

the other VERY telling thing wayfinder, is you absolutely, steadfastly and unilaterally refuse to accept that the content you choose has ANYTHING to do with simulator performance, which is not only childishly arrogant, its pretty much immediately going to turn off any chance anyone is going to really try and help you, because you absolutely refuse to help yourself

Eltee, anyone who has read this thread knows your statement here is totally bogus. You'll excuse me if I don't waste my time looking up your threads. Based on what I've read here, I think I see how your "research" works and I have no need to waste my time on any more of it. You'll also excuse me if I just let you go ahead and throw your tantrums. I'm not impressed.

please, point me to a single instance when you admitted your content was causing your simulators to lag. Not the ones where lee said it, where i said it, and where other people have said it, i'm looking for the post where you said at least 'yes i am putting a heavy load on the sim BUT...'

seriously, just be willing to at least question your own involvement... one of the reasons LL has been so willing to help listen to us is we *ALWAYS* question our own content first, we go through it with a fine toothed comb, we remove all the active scripts we have in the sim we're testing, we test it with bare avatars (ruth from the library) with complex avatars, with 2 avatars with 20 avatars, with varying types of scripts running, with no scripts running etc...

but believe me we are willing to question our own content and that makes all the difference when you are trying to get a point across... unwillingness to accept your part in blame for a problem you are raising, is the surest way to get your point ignored

Taking this right to the core:

Eltee claims sim performance (or lack thereof) especially in ElvenGlen is related to content. Despite Eltee's somewhat exciteable claims, I've never denied that content has some effect on sim performance. That's a given. What I've stated is that they are not the primary cause of no-discernable-reason lag.

Now, thinking logically (and I know this is going to be tough ) if content is to blame, then the more content we add to ElvenGlen... the slower it runs, right? I mean, all the time. Can't possibly run properly at all, because well, the content is always there, isn't it?

No college degree needed to figure that out. No rocket science mentality. Simple common sense that anyone can understand. If content is to blame, then as long as that content remains the sim will continue to lag.

Readings on ElvenGlen as of 9:40pm tonight SL time:

SIM FPS: 495
Run Tasks: 1.4ms
Physics FPS: 45.1
Physics Step: .7ms
Prims: 13,426
Active scripts: 525

Hmm... well durn. That just blows Eltee's claims all to bits.

Of course, I have no doubt that now we'll hear all kinds of ranting reasons why our sim is running at a decently fast speed. I've seen Eltees ranting continue in the face of prior such presented data.. data that was presented over and over and over.

But the fact is this: if CONTENT is the primary cause of sim lag as Eltee claims, and it is NOT server-side oriented, then there is no reason for a sim to run at this speed and then tomorrow drop down to 25 fps and stay there for days... then suddenly jump back up to fast speeds again, is there?

And all the tantrums you want to throw Eltee are not going to change these simple facts. So tantrum away. I think it might even be funny to see what claims you pull out of a hat next to enable you to ignore factual data.

actually if you woulda listened to me, three times now i attempted to explain to you the two extant back-end sim problems that can cause exactly what you are saying, all three times you spat 'you are saying techno-babble' back at me like it was some kind of rebuttal.

reads those links i posted, especially the most recent three... those problems are THERE, identified, and solved in 1.7, which is what we have been *TRYING* to tell you for the last day already

How were they identified and solved? through persistance, a willingness to work with LL not shout at them, and some good old fashioned techno babble.

_edit_
ooh and prolly 800 hours of LL's time, trying to make SL better, thanks guys, your work is appreciated, by mosta us ^.^

Why do you keep omitting time dilation?


... I was just in your sim. I was getting 26-28 client FPS and no discernable 'lag', time dilation of 0.95-1.00.

My avatar has 450 prims in attachments.

I was able to fly, move, without any stutter or lag.

Granted there weren't a lot of agents in the sim (4), but you said it was unusable with two?

Though your stomach is made for eating burgers... if you eat more than a certain amount, you will explode. It is totally up to you to keep track of that amount. Nobody is going to say "you can eat 15.2 before you explode."

In other words Eltee.. to put it plainly and simply.. the problem was SERVER SIDE and not content.

Durn, you're not the most obstinate person I've ever met, but I think you score top 10.

This really, upon inspection, doesn't have anything to do with the issue at hand.

To use your illustration however: there IS a dna-based coding in your stomach which tells you when you are eating too much, and which insistently prevents you from eating so much that you explode. It's hard coded.

Now, if someone is blatantly stupid enough to ignore what the stomach says, yeah, they're gonna suffer. But the coding is still there.

However, to get back to reality rather than ethereal fantasy, Second Life is not a stomach. It is a computer program. And computer programs CAN be coded to prevent security risks. These are called GOVERNORS and they exist in computer systems all throughout the world.

So sure, you give someone the ability to put scripts on a sim. But you govern that so that they can't put 5000 scripts on a sim. And you govern the LANGUAGE of those scripts so they cannot point a gun at someone on a no-weapons sim and blow them totally offline. You put in script testers so that if a script requires excessive sim resources, it is halted. Common computer administrative functions.

Simple concept. What's the problem?

we've been saying all day that most of the problems that have been out there are fixed in 1.7... your sim however does have a heavy content burden, like i said there isn't any one huge object causing problems just you are pushing the thing hard so yes the scheduler in 1.6 really doesn't like that very much and there is a specific problem with the physics task skipping passes it should not, that is addressed in 1.7 where it runs synchronously with the main sim engine.

but then all of that is techno babble, not english so let me spell it out

big

scripts

cause

lag

the

new

server

'stuff'

is

going

to

deal

with

it

better.




and on the horizon is a technology that could potentially run much *MUCH* faster... thats not going to make it in 1.7 but it is in development, and if you'd stop yellin at LL and listen to them, it may, one day be possible to have 10x the active, sim processor using content in a sim that you do now, and have far *LESS* lag inspite of it

what you basically missed in all this is that i am 'on your side' i took the time though, to talk to lee, and when he said 'i need better data' i asked him specifically what that meant

what that does mean, so you will know if you see problems in the future, is screen captures of the sim stats, lots and lots and lots of screen captures of the sim stats...

everything else is just 'numbers' and anyone can say any numbers...

live SL captures of sim stats tho (as you could see in some of the other threads i link to) *ARE* actual hard-data though, and if enough of them can be had to illustrate a point... say significant time dilation with over 1000 sim fps... well then LL is going to actually listen and look.

i know yer a 'big important guy' but you are exceptionally arrogant when it comes to how you react, Lee literally took the time to ask you for more data, and you shouted at him and said forget it you werent bothering..


now is that any way to get yourself heard?

mebbe if its yer employee and they are obligated to put up with yer BS, but believe me its definately not the way to approach someone in the real world with a problem.

Wayfinder, I have found SL performance to be a religious issue which holds up in conversation and forums less well than oh... religion discussions.

You can bring all the limited tools at your disposal to bear on the problem, but someone can just turn around and say "nopey". And you there you are.

We live in a universe of finite resources, and to take one simple example, Linden Lab has finite network capacity (as does every network). While their current pipes may not fill at 100 players, or 1000, they must at 10,000 or 100,000, or 1,000,000 players. This seems to be self-evidently true and seems to be in principle measurable. But we don't have the measurements so we're left to speculating as to how many avatars can dance on the head of a pin.

After having Linden staff deny that there was any such thing as "asset loss" or "world slowly rezzing" phenomena, I've simply given up. Rationality doesn't apply here. Do yourself a favor and vote with your feet or your wallet because argumentation no matter how valid will only get you "does not" responses from various segments of the SL community. Please understand, I think you are likely correct in your analyses, but that and 4 bucks will get you a cup o' joe.

*sigh*

Paulie, you're insane. There is no "Wells Fargo". Those people have only really seen the game once or twice. I developed the whole thing and I am the sole provider of tech support for it.
I do not have shell access or any form of privilege over other SL users.
As for "Liaison" type of support, they do have a dedicated support team... that they hired themselves. Anshe does the same thing with Dreamland. You can too.

I'll bite. The conversation being thrown around in this thread just plain hurts.


Lag. Fun topic, sans the personal attacks (ahem). Let me share my experiences tackling the beast.

All debug notes refer to the Ctrl-Shift-1 menu. These values do appear in other portions of the debug browser, but those are out of the scope of this post.


My machine:

For all intents and purposes, I connect to Second Life using:

- 1.7 ghz P4 processor
- 512 MB RAM (with a 1.2 gig page file)
- Radeon 9600 Pro AGP 128mb
- High-Speed Cable Connection with 2.7 mb/s downstream



Client-Side Lag:

Here's the stuff that hammers my client hard. Sims with lots of this stuff are generally taboo to me:

- Particles
- Arbitrary Geometry (more faces)
- Physical Interpolation (movement)
- Animated Textures
- Frequent (Mundane) Texture Changes
- Excessive Use of Textures (Texture Buffer Lag)
- Very Large Textures (Texture Buffer Lag too)
- Client Settings that aren't Scalable (eg. local lighting; high av detail)
- Any combination of the above

For client lag, the best tool is the bar graph that breaks down memory usage. I believe it's Ctrl-Shift-2.





Server-Side Lag:

By comparison, here's the stuff that I have confirmed as hurting a sim. I'm going to break this into two sections, one for stuff we have control over, and one for the stuff LL alone has control over:



Stuff We Control:


- Heavy Script Activity

Appears in Debug as: Run Tasks (%), Active Objects, Active Scripts

Commonly caused by calculation-intensive scripts and those that frequently process client requests. The common offenders are:

* Frequent listener calls in a short period of time
* Large sensor areas called often
* Multiple scripts churning out intensive calculations (eg. if "Deep Thought" were written in LSL)
* Short timer events
* Multiple Rez Object/Die calls
* llRemoteLoadScriptPin/llGiveInventory multithreads ad nauseum (affects global asset server, not the sim per se)
* DoS-like communications/teleportation/ban scripts
* Physics-based scripts (see below)



- Physics

Appears in Debug as: Physics (%), Physics FPS, Active Objects

Physics is by far the most lag-intensive feature of Second Life. Used properly, it won't leave a dent on a sim. Abusing this feature can damage a sim to the point of requiring a rollback of the data. Common offenders:

* Interpenetration of objects (commonly through movement scripts)
* Collisions involving difficult geometry
* Several collidable objects
* Frequent physics engine calls in a short time frame



- Avatars

Appears in Debug as: Run Agts (%), Main Agents, Child Agents, Agent Updates/Sec

The second most laggy feature of Second Life is other people. This stems from two main sources:

* Worn scripts adhering to poor scripting standards. This actually appears in Run Agents, NOT Run Tasks.
* Avatar activity



- Textures

Appears in Debug as: Send Imgs (%)

Textures cause a sim to lag because they are streamed to every agent viewing them, including those in an adjacent sim. The common offenders, similar to the client lag section:

* Excessively large textures
* Poor texture-vs-object ratio
* Broken textures (very rare, if not fixed)





Stuff Out of Resident Control:

This is the stuff LL is currently responsible for.



- Asset Server Lag

Appears in Debug as: Receive Msg, Retransmit, Pending Downloads, Pending Uploads

The asset server is a fickle creature, but the signs of global lag are fairly obvious. This is usually the cause of sluggish rez times, long delays before saving scripts/notecards, poor login performance, failed and slow uploads, lost inventory, and just about anything else that globally breaks in Second Life.

The (known) causes for this are:

* Inventory Assault (high changing of inventory stats/person)
* Poor scalability (slightly better now)
* Too many asset requests in the backlogs
* Global Denial-of-Service attacks (happened once about half a year ago)
* Global Bugs
* A Linden tripping over the power cord



- Sim (Server) Performance Levels

Appears in Debug as: Time Dilation, Sim FPS, Sim CPU

The fact we don't have direct (shell) control over this factor is erksome, but regardless, a fact. This is the kind of thing they call the grid monkeys to fix. Common causes:

* Stuff we majorly screw up (see above)
* Server-level bugs (eg. servers not restarting properly)
* Broken CPU alottment (can be EXTREMELY bad)
* Corrupted/Faulty data
* Unhandled bugs/exceptions (eg. the old chat buffer overflow)

Note: The typical sim-to-server ratio is 4:1 and 2:1. Some water sims are roughly 8:1.





How to Deal with Arbitrary Lag:

Armed with the above, the best thing to do is watch the debug menu very carefully, and see if you can tie an event in the sim to what's going on in the debugger.

By far the most common, arbitrary lag, is physics. If you see a spike in physics CPU, figure out what's causing it. Period.

The second most common is, again, agents with a lot of running scripts. The easiest fix is to flag your land "No Script" and see what happens. (this only works up to about 15 meters above ground level)

And, should you require Linden assistance, be prepared to give proof you're having a problem. Giving them something to go by is infinitely more helpful than "help, my sim isn't working..."






Mythbuster-busters:

And now, the rant. I did not like much of what I've read in this thread. At all.


Active Scripts/Objects is typically a good indicator of lag.

Rationale: While not perfect, a high active script count means the probability of lag-inducing scripts is also high. QED.

Note: Running Scripts in child prims (of linked sets, on avatars, etc) are not added to the Active Script counter, further skewing this data in favor of lag. The best indicator is Run Tasks.


Know thine code.

If you have no idea what a script or object does, it would behoove you to find out. 90% of all lag problems in Second Life are due to ignorance.


Develop good testing habits.

Watch the debug tool often. Take and set down objects if you're unsure of what they do to those counts.


Data transmission is a major lag source.

Which is why textures and agent actions can lag at the sim level.


LL cannot fix everything for us.

They're a small house and happen to be human beings. A wise resident is one that learns to rely on themselves. A wiser resident is one that brings to Second Life what they are well prepared to lose (and hopefully restore) if the situation presents itself.


And above all else:

Respect the opinions of others!




That's about it.

Jeffrey: Thanks for the excellent post. It's probably the best what-influences-what breakdown in the entire thread.

Wayfinder: I'm not sure who else we can forward you to at Linden to explain things. Hopefully the 1.7 script-handling features do make a big difference. As Jeffrey's list points out many examples, if you have scripts, it is not accurate or fair to label your content "static".

Malachi: I tend to agree. Bringing everything you can to a discussion, only to be told "nope", is frustrating. ;^)

That said, I'll keep watching this thread, but it seems providing a Linden voice would be largely antithetical to the current discussion.

Once thing that hasn't been mentioned before that can create problems:

IIRC, TWO sims are hosted per Opteron server - one processor each. However, if you're sharing the server with another simulator with horrible content and a ton of avatars, which is causing the machine to do "deep thinks", this can also affect the power you can harvest from the server for your simulator. Please correct me if I am wrong about this.

Wayfinder, just a thought, but how about setting your entire sim to No Script and seeing how that affects performance?

Regards,

-Flip

Great post, Jeffrey!

Based on your experience, what *is* an excessively large texture?

People talk about large textures a lot and how it causes both client and server side lag, but I never see anyone quantify what a large texture is.

Woud be helpful to me to put some real numbers to it. What x,y dimensions are excessive?

Thanks!