Lag Studies

So, because lag nazis are out in full force these days, I decided to run a few experiments so that we actually had some concrete numbers to go on.

I'm worried that lag nazis will use this post to flame other people, but hopefully knowledge is always ends up being a good thing. But I digress.

So, I went to Moopf's island (Numbukulla) to try and injure his sim with my scripts. (Thanks Moopf! )

So, the lag that I'm trying to measure is server lag. I'm using Server FPS as my metric.
Some important details:

Numbukulla
Oct 27, 2004, SL v. 1.5.5
Sim CPU: 0.5
Agents: 1 (ie. me)
Child Agents: 0
Unloaded Sim FPS: ~6000, fluctuates up and down by 500 for no apparent reason. (random cruft in the sandbox)

I am assuming that the compiler does not perform any optimization in any way.

One thing I should suggest before we look directly at any numbers is that if your sim FPS is > 100, everything is well. The server is busy toiling, but it's doing fine. There's no need to bring out your pichforks just yet. (eg. if your server FPS never drops below 200, there's no need to have to mix words with anyone

In all these experiments, it's 1 script, in 1 default prim. The script only contains a state_entry() function, no listen/timer/sensor/touch handlers of any sort.

I would guess that the error on these measurements is maybe +- 30%. Experiments were perfomed in the order listed:

Beginning Experiments:
Server FPS = ~6000fps.

--- Spinlocks ---
256 spinlock scripts: ~300 fps
Deleted: ~6000 FPS
32 spinlock scripts: ~4000 fps
16 spinlock scripts: ~5000 fps
256 spinlock scripts: ~300 fps
320 spinlock scripts: ~70fps
256 spinlock scripts: ~300 fps
192 spinlock scripts: ~900 fps
128 spinlock scripts: ~2000 fps
64 spinlock scripts: ~3500 fps

Deleted: ~6000 FPS

--- Listens ---

Listen on a high channel (no listen handler)
16 listeners: ~6000 fps
32 listeners: ~5400 fps
64 listeners: ~4700 fps
128 listeners: ~ 3500 fps
256 listeners: ~2300 fps
Deleted: ~6000 FPS

--- Empty scripts ---

Empty script:
256 empty scripts, ~6500 fps (Now you know how inacurate these measurements can be)
Deleted: ~6000 fps

--- 1 second timers (no timer event) ---
32 1 second timers: ~5400 fps
64 1 second timers: ~4400 fps
128 1 second timers: ~3300 fps
256 1 second timers: ~2300 fps
Deleted: ~7000 fps

--- 1 hour timers (no timer event) ---
1 hour timers (no timer event)
32 1 hour timers: ~5700 fps
64 1 hour timers: ~4600 fps
128 1 hour timers: ~3500 fps
256 1 hour timers: ~2300 fps
Deleted: ~6500 fps

--- 1 hour llSleep(); ---
32 1 hour llSleeps: ~5700 fps
64 1 hour llSleeps: ~4500 fps
128 1 hour llSleeps: ~3400 fps
256 1 hour llSleeps: ~2400 fps
Deleted: ~6000 fps

--- llSensorRepeat() 96 m sensor, all avs (ie. me), 1 second ---
16 sensors: ~5300 fps
32 sensors: ~3500 fps
128 sensors: ~2400 fps
256 sensors: ~1400 fps
Deleted: ~6500 fps

... At this point, someone reminded there was a lunar eclipse outside. So I abandoned my experiment and went outside Attached is a picture I took using my craptacular Canon S100 (I was cool in 2000, honest) with excessive light polution. It's the top right corner of the moon - we had a full one earlier today before the shadow of the earth ate it.

Overheard: Asset & other centralized servers are crumbling at this very moment. I didn't personally experience anything unusual. So, grain of salt and all.

Cheers,

wow, so there's not really a difference between a llSleep and a llSetTimerEvent. well llSleep automatically cleans itself up i guess but still, same-ish lag.

thanks for the info Francis. any chance of doing a listen test on chanel 0 to settle the argument of whether non zero channels are less laggy than listening to channel 0?

and what's a spinlock script?

it would be intresting to see how a for loop compares to sleep and timer.

You mean an "infinite loop", where it loops a certain number of times, does something, then repeats the loop?

That would be far more inefficient.

you couldn't guarantee how long that would take could you? it would vary depending on lag i would think.

My real-world tests with my server-based, inworld rental system (100 rental cubes and 1 server prim) have also proven that there is no sim performance impact between using an infinite loop in conjuction with an llSleep and using an equivalent timer.

Oh my god, my poor sim!!!!

Sim FPS wanders around a lot I've noticed that. You can stay in one place, not move, no one else be in the sim and it'll waver +/- as much as 1000. I know the bats I put out on the roost took a good 3000 off the level yesterday and the eye colour changing on the batfish that somebody came and built for Halloween also knocked a noticeable chunk off.

It's good to get some figures on this though Francis, thanks for posting.

I think His Grace came out with some similar statistics a while back, but I was too busy to comment at the time. The thing I find most peculiar about these results is not that llSleep and the different lengths of timer end up having the same effect (after all, they are doing the same thing) but that they have any effect at all on performance.

The classic solution to the timer problem is to have a time-based or delay-based queue, so that at each clock tick you only need to look at whether the task that is closest to being woken up should actually be woken up. This means that you spend a constant (and tiny) amount of time every clock tick irrespective of the number of waiting tasks, at the cost of having to scan the queue when something goes into a delay. You can do clever things with bins to make the latter happen less often, but the basic scheme is a net win even if you don't because as a general rule tasks wake up very much less frequently than the clock ticks.

The net effect is that, in operating system land, any number of tasks waiting to be woken up an hour from now have no effect on performance because all that happens every clock tick is a comparison or a decrement.

These results show that this isn't how LSL organises timers, which either means the Lindens don't know about time-ordered queues (pretty unlikely, I'd think) or that making this be as efficient as it could be just never got to the top of the priority list. Bummer.

Some stuff about interpretation, which someone did mention in His Grace's thread. The sim FPS isn't the most enlightening statistic to show for this kind of test. It is much more useful to show FPS and 1.0 divided by FPS, on the theory that costs are related to the basic sim frame cycle.

For example, if 128 timers gives us 3500fps then that's 286 micro-seconds per frame (286us). Going up to 256 timers gives us 2300fps, which is 435us. So, we've added 128 timers and (435-286=149)us; adding each timer cost a little over 1us in each frame (quite a long time by modern standards, probably hundreds of instructions worth depending on the machine). If you run the numbers, this is pretty constant (within Francis' +/-30%) across the range. That's consistent with the original theory that this is a cost associated with the frames we see in sim FPS rather than some slower and more regular clock, although FPS wobbles around enough that it is hard to be sure. Doing the same test set but averaging FPS over a minute or so for each test would reduce the wobble enough to draw some really nice graphs, I'd guess.

This behaviour seems to be true for listens, as well, but interestingly enough not for the spinlock scripts (definition please? I know what I think a spinlock is, but I can't think what the term would mean in LSL). My guess is that the spinlock scripts are CPU bound rather than waiting for events, and the LSL runtime is having to do a lot more work after some threshold: if you plot the spinlock results either as FPS or us/frame small numbers of them seem to be OK and then things don't so much fall as plummet. I'd guess this is LSL running out of time slice and having to context switch between the tasks, or something of that kind. If we can see the script in question, this will probably become clearer.

All the above is way more analytical than this probably deserves. The basic conclusion (sleeps, timers and listens have a non-zero cost, but not a very big one so lag nazis please don't get uptight about every object with a timer in it) was pretty obvious without it I spent a decade or so doing something pretty related to this, though, so I thought I might as well throw in my 2c.

Francis, I am very disappointed that you only measured sim FPS and not Run Tasks, Time Dilation and all the other stats :|
But thanks for the study, interesting stuff!

Hi guys,

A Spinlock: while ( TRUE );

I agree mostly with Hanks and Alexander's suggestions, and they're supported by the numbers I've seen.

My point is just to run some experiments in a way that we can reason about. Experiments should be re-run periodically - new software, etc

Eggy,

Physics, time dilation pretty much stood at normal levels- I don't think it dropped must past 44 fps and 0.99 respectively. I don't know what "Run Tasks" means exactly, nor most of the things in the alt-2 menu. Explanations and sources are most welcome

You're kidding me?
Run * are the times the server spends processing those things.
Run Tasks is for scripts, Run Agts is for Agents, etc.
IIRC Sim FPS is 1 over the sum of all those times.

Ah, I dig now. # of ms processing time per sim FPS cycle Makes sense.

I wonder how general sim processing and physics calculations are interleaved, since they seem to operate using different notions of "FPS".

also listens become more and more laggy with more agents.. many types of sensors as well.

A good guideline is honestly.. just don't loop things more often than you need to. an empty timer causes little lag... a timer with a single thing, a say, or a sensor, or a color change, etc... causes dramatically more.

If something only needs to show when yer online, and when yer not.. run it once every 30 seconds.. not 10 times a second... be realistic folks.. also looping color changers too... texture offsetter with a color grid is much much less sim *AND* client stressful than a full objet update (which changing the actual color property via script does)

If possible, have your script go to a non active state when it doesn't *need* to be.. or at least a less active state. Listens, timers, etc don't carry over between states, so if somethin only needs to flash when you are around, have a touch (which is inactive) trigger it to move to a state that does things, and then have it eventually time back out to a state that doesn't.

It doesn't look like any 'one' script can kill a sim.. no... but in a given sim there may very well be 500 scripts, each doing things continually that really *really* don't need to happen when their respective owners aren't around, and theres no one there to use them.

Sure one or two doesn't make a difference... but i will say that at the moment, *empty*, lusk gets 400 sim fps, with 2ms run tasks... and perry gets 6000 sim fps with 0.1ms run tasks... and most of that is a single group's build... 400 may sound 'okay' but with 5 people its already sub 100, and jus gets worse.

not that any one script is doing it all... again...

but just a *few* simple changes, by a single group, would probably double, or triple, the available resources for *everyone* especially visitors, to use

I might be talking out of my butt here, but it seems to me from the numbers in the original post, and what eltee said about lusk, that there is another factor besides the things you think would cause lag.

Is it possible that the mass of the object that a particular script is acting on might have something to do with lag? Or some other property of the object that is not directly related to the efficiency of the code?

It may be the case that even if you had all the scripts in lusk optimized that the sim fps wouldn't change appreciably.

nah the lindens came out an confirmed what we had been saying about sims all along, bit before the holidays. The factor was the oldest sims simply can't handle running scripts/people nearly as well as the new ones, and they are (purportedly) trying to remove them from the rotation

Agatha,

There are a lot of things that don't cause server side lag, but will bog down your client very significantly. If you do a forum search for "hoochie hair", you will find that there is compelling evidence that attachments made of dozens or hundreds of small cut torii will pretty much ruin your day. I know that when I go to places that use the torus a lot, I can barely move.

Alan this isn't a thread about client lag so much as about sim based lag.

LL is also trying to pool the sims so that a sim won't be put onto a machine that is of lesser quality.

strife, I know. Agatha mentioned properties of the object a script was in and, since she is new, I didn't think it would hurt to mention it.