"dormant" scripts aren't as dormant as you think

I've done extensive testing, and I've reached this conclusion: dormant, completely inactive scripts still take up significant sim processing power.

The evidence is pretty simple. Find a relatively unloaded sim, one with a "Total Frame Time" less than 22.2 seconds under View->Statistics. Watch the script time for awhile so you know the approximate baseline value.

Create an object, and put a script in it that contains only a blank state_entry() event (this is the minimum requirement to compile a script). Duplicate the object out to about 128 copies. By the time you're done, you should see that the script time has increased by about two milliseconds per frame.

What does this mean? First of all, it means that scripts like particle systems, texture animations, sit targets, and the like, that operate on object attributes and run a function only once still have a noticeable impact on a sim's performance. Even after they've done their one-shot action, they still burden the sim. Each script's contribution is slight, but given enough, they will affect script performance across the sim. One way to solve this is to delete such scripts after they've had their effect. I prefer to uncheck the running checkbox, so that I can still get at the script. I've tested, and scripts with the running checkbox unchecked don't affect the sim.

One note: llTargetOmega() on nonphysical objects is a client-side-only effect, and in the past, I believe it would continue to have its effect even after the script was removed, but that's not the case anymore. So llTargetOmega() scripts need to stay running.

Another thing this discovery brings to light is attachments. From watching the "Active Scripts" value in my sim, I've seen that, on average, avatars can contain more than 100 scripts apiece. I've seen some that contained over _1000_ scripts in their attachments. This has a very noticeable effect on the sim. My av has a fair number of scripts, although I've tried to pare them down to the bare essentials.

One tough example is transparency alternater scripts. For example, some dragon avs have wings that have two states, spread and folded. Scripters often will put one script into each prim so that the transparency toggling happens all at the same time. Sometimes color change scripts are done this way too. The thing is, those scripts, even though they're almost entirely dormant, have a significant impact on the sim.

I've known this for awhile, but I finally decided to write something about it because of something that happened recently in my sim. I keep track of the script time rather religiously, because I intend to script a lot of ambient effects, and I need to make sure the sim doesn't start crawling. I saw that our script time was way higher than it normally was, and my friends and I tracked it down to a combat system that one was wearing.

When worn, the thing took up over 10ms of script time per frame. That's over half of a sim's script processing power. The object contained over 400 scripts designed to act together to deliver a push simultaneously on a target. The system is advertised as low-lag, but it's not. It's a decent assumption that inactive scripts won't hurt a sim, but it turns out to be wrong. It surprised me as well.

My understanding is that once a sim's Total Frame Time gets to 22.2ms, scripts start to get crowded out and compete for processing time. More scripts can fit into the sim, but each script starts to have less processing time than it wants. Eventually, the sim becomes unable to run all of the scripts, causing time dilation.

One attachment with 400 dormant scripts takes up half of a sim's processing power. That's pretty unfair given that, in general, these kinds of scripts are only going to be actually doing something for a fraction of the time they're worn. Worse yet, there's no indication to nontechnical customers that what they're wearing is contributing significantly to sim lag -- especially since not many scripters know about this either.

It's unfair to land and sim owners that someone can simply walk into their sim and grind it to a halt this way. This is probably why sims can't handle lots of avs. I urge you to check through all of the scripts contained in your attachments, and make sure they're all necessary. Now that you know what kind of effect these things can have, try to limit your number of scripts. This really concerns me, because, no matter how efficiently I script my props in my sim, given enough avatars, my scripts won't run properly, and my effects won't work. In my observation of the sim statistics panel, most avatar take up a significant amount of sim resources.

For another datapoint, did you try putting 128 copies of an empty script into is a single prims?

Useful, and interesting, results there Lex, but it seems to me that the fundamental problem is that there is (a) no defence against (or in fact any easy way to detect) 'resource hog' scripts, whether contained in attachments worn by your visitors, or in your purchased items, and (b) no matter how 'efficiently' SL handles dormant scripts, there is no limit to how many of them people can associate with an object.

It seems to me that the simplest solution to (b) would be just to put a limit on the number of scripts per object. In most cases, objects only appear to use absurd numbers of scripts as a way of defeating quite the quite sensible performance throttles imposed on particular LSL function calls, so I don't think this would cause too much of an outcry.

The problem of dealing with (a) is more difficult - sometimes the issue is more to do with naive coding, than malicious intent. Maybe the best way to deal with it is to do what you have done, and try to raise awareness of efficiency issues within the scripting community.

Perhaps LL can expand LSL to make a variant of SetScriptState that can target scripts outside the root prim.

This way a color changer (for example with 100 scripts total) gets reduced to 1 active script when color change is not in use.

A problem with your procedure.

Your control state is the 'blank sim', with just you in it. As far as I'm aware, you're putting 100+ blank scripts out, but ALSO 100+ objects. Have you ever considered the objects themselves to be contributing to lag?

A way to properly check:
1. Define a control state - 100 blank prims, default cubes or something, so we don't concern ourselves with textures, etc.
2. Note the sim dilation.
3. Place 'blank' scripts in all the prims.
4. Note sim dilation.
5. Stop all the blank scripts from running in the prims.
6. Note time dilation.

Ofcourse scripts in prims are going to cause an extra server load, because you've made more data, with parameters, in the sim. Think about having one file on the computer and making 100+ shortcuts to it, or different copies entirely, its still taking up more memory. You need to test the effect they have when they're actually running, and not from adding more prims.

I believe Lex is correct. I figured this out some time ago when I was watching the script build up in Svarga. There is no such thing as an "inactive" script. If a script is in the queue it's being polled and this seems to take a disproportionately large amount of processing time.

I realised I had 100s of scripts in prims that do nothing other than start a texture animation or particles. Stopping and/or deleting these scripts does not stop the effect but removes the scripts from the queue and makes a noticeable difference to sim perfomance.

Recently I've started using llRemoveInventory( llGetScriptName() ) a lot more than I used to!

Waterfall builders take note !

The objects wouldn't be showing under "Script Time" in the stats screen.

I would agree with this assessment of a control if Lex had quoted TD as her parameter. I'd love to see the numbers for 100 blank prims but I'm willing to bet hard currency that blank prims don't affect the script time parameter one iota which is what was used to indicate that scripts that contain no active code still chew script time in the sim.

As an aside, you can find out how much you personally contribute, if it's a reasonable amount quite simply. Go find an otherwise empty void. Check it's really empty. Look at the number of active scripts, and the script time. That's you that is. Want to really check? Take all those attachments off and see... I've not done it recently, but last time I did scripts and script time fell to 0 as expected. There have been upgrades since then though, so it could have changed.

I just put up the way it would be handled in a scientifically controlled environment. Without the objects there, whilest they don't directly impact script performance, due contribute to the processing done by the server, and as far as I'm aware, its the same servers that handle scripts + objects. Its just there to rule out the possibility anyways.

Lol, spot the UK cultural reference. Thanks for making me choke on my coffee.

I don't believe it's possible to put this many scripts into one prim. I've seen the same excess script time effect from an object that contained 20 child prims with 20 scripts each, so it seems that the number of scripts per prim makes no difference (as one might expect). I just used the prims as a convenient way to duplicate the scripts.

I've done this, just for the sake of argument, except I checked Script Time as I did in my original experiment, because it's the only sensible metric to use for this. Unsurprisingly, 128 empty boxes have precisely no effect on the script time, and 128 boxes with one empty script apiece do have a noticeable effect on the Script Time.



I want you to know that I did this in the most scientific way possible. That's what it took to convince me that my statements above are actually true, because, as a programmer, it seems pretty surprising to me that dormant scripts in an event-based language should have such a constant effect on the sim. I agree with Eloise, though. It doesn't make any sense at all to rez empty boxes and look at their effect on the "Script Time" stat, because those boxes don't have scripts in them at all. I'd expect empty boxes to affect "Sim Time (Physics)" and maybe "Sim Time (Other)", but I haven't tested this directly.

I'm always happy to debate scientific method - that's one of the things the B.Sc and D.Phil are good for! My problem with your proposed control isn't that you're proposing a control - it's that your suggested control isn't controlling for the metric used - script time, it's controlling for at least one other (image time, that contributes to total frame time) - but not the directly quoted measure.

That said, the history of science if filled with such debates and debating over the suitablity of controls is probably the most common of them.

Thinking about Lex's experiment has made me consider two or three other things.

1) Is there a linear relationship between the number of scripts and the script time, even if they're all apparently dormant to expectation? If 100 empty scripts causes 1ms of script time, does 200 cause 2ms, 300, 3ms etc. or is there a more complex relationship?

2) It's obvious that not all (functional) scripts will be equal, but I'm wondering if writing "modules" has a higher cost than writing a bigger script. Each little script presumably has a time cost since even the otherwise useless one Lex is talking about does - whereas the single bigger script, even if busy all the time, is only running one event at a time, so less total load?

3) I really need to write myself a decent finder HUD! The one I was using did other things too, but at 350 scripts (yes 350 in one object!) it was worth about 1.2ms on its own on script time!

Apropos of this discussion, there is a recent post in Babbage's Blog about the problems of dealing with the unusually large number of independent scripts typical of the SL environment in the port to Mono. What isn't clear, and I guess is the original issue raised by Lex, is where the overhead arises for scripts which register no event handlers, and thus never need to have their execution context saved or restored. I can imagine, though, that optimising for these was not probably not uppermost when the engine was originally written.

It indeed drops to zero. I've been working to improve the scripts of a popular sailing boat lately and I always hide myself away in a completely empty void and make sure that I've got no scripted attachments or HUDs while I'm working there.

What is annoying though, is that the IPS stat appears to be buggy, as well as the number of running scripts. If I work in a void for any length of time (1hr +) sometimes the IPS seems to double. If I simply sail out of the sim, leaving it completely empty for a minute, then sail back, then the IPS drops back down to expected levels.

I haven't cross checked against script time to see if the same bug occurs there, nor have I found the specific set of steps needed to reproduce this bug

I think what Lex is documenting (and well, I might add!) is the "script overhead" that is imposed by the mere existence of a script, let alone if it does anything or not. This overhead is to be expected for all running scripts because, at a minimum, they are generally in some kind of "running" queue, and that queue must be searched whenever a process needs to be located. The longer the queue, the greater the overhead. I would hope that they use some form of O(log n)-type searching algorithm, but it may not be computationally efficient for an optimum number of scripts active in any particular sim. Also likely, each script has its own event queue (I think the Wiki documents this), and that means that each local event queue has to be checked for events to trigger event actions to process. It is just the nature of what the script system is doing that will always entail some amount of baseline overhead, regardless of what each individual script itself is doing (and if you think script programming in a game is a complex subject, try process scheduling in a modern OS sometime! ).

I have been concerned about this issue myself, wondering what kind of overhead even empty or idle scripts have on the performance. I spend a lot of time wearing big quadraped Dragon avatars and other full-prim avatars, which have not only high prim counts, but high script counts as well (like on the order of 500-700+). We know that these AVs cause a lot of lag, even when we are alone in a sim. I am also working on projects (like a HUD) which has a number of scripts in it as well. I would greatly love to keep my resource usage as small as possible, and design/code to that end whenever and wherever I can. However, for some things, it just is not possible to have a feature without going crazy in the script department, due to the way LL designed the game and the script system itself.

I not only am concerned about the time resource issues, but the space and loading issues as well. One of the Ice Dragon avs eats up around 10MB of memory in sandbox allocations alone (assuming they don't dynamically allocate the 16k sandboxes in 4 4k pages as needed). The time to load all those prims from the asset server, allocate all those sandboxes, and kick off all the 600+ scripts is quite hideous, and can take upwards of a minute to fully load all of the attachments (assuming the asset cluster is not heavily loaded). Get 20-30+ people wearing them in the same sim, and you're now talking major sim lag for everyone.

Anyway, the upshot of all of this is that, until they allow people to make complex items/avatars with meshes, or even give us the ability to change all prim params from another prim, to get the ability to change textures (for example) on all parts of a massive-primcount av, we will have to go the script-in-each-prim route. Everywhere I can, I will avoid going crazy like that, but it is unavoidable in some cases.

It also might be nice if we could do some kind of client-side scripting. HUDs don't really need any server-side script resources for presentation and basic operation. Just when they need to trigger something in the world, would we need the server-side doing anything.

But I can think of ways that an event-based language could be done such that there's not an O(n) check for stuff to do every frame, I think. How about a simulator-wide "event run queue", which would just be a long list of entries including a script name and what event needs to be run? Every time the sim figures out that a script will need an event called on it (ie, when someone touches it), it puts an entry for the script into the sim-wide event queue. Next time scripts are being processed, it just needs to look through this one mega-queue and start events going for each event it finds.

Say the script already had an event running when someone touched it. Instead of adding the touch event to the sim-wide megaqueue, it adds it to the script's personal queue. When the event that's already running finishes, the sim can check the script's personal queue, and if there's something in it, it can take the first event and add it to the sim-wide queue. I think this results in no O(n) scan for events each frame, while still getting everything done. Thoughts?

Well, the problem with that is it is harder to locate and limit events in a combined queue. If there is indeed a unified event queue (and no individual script queues), then there would have to be some other mechanism to limit the number of events per script (counter, maybe?), since we know there is a per-script limit of 64 (or thereabouts). Also, plucking events for a specific script from a unified queue would be slow, requiring a sequential search.

If you are already adding events to a dedicated script queue, it doesn't make much sense to put it back into a unified queue, just to take it back out later for the script. I also have to assume that script event execution is time-sliced to prevent starvation to otehr scripts, so it would seem more logical to have script-dedicated queues, as pulling the events out of a unified queue could allow some scripts to get preferential execution if they are fairly event-heavy.

Of course, this is all speculation. No matter what, it doesn't change the fact that using less scripts, no matter if they are active or no, is a good thing.

I find this subject fascinating, because I to care about sim performance. Thanks for sharing this vital information. It's kind of like how people hang them self with prims. Sim really run better with less prims. Running the max prims just bogs down the server. The hole thing make me wish we could have scripts generate and or replicate other scripts. (dance machines come to mind)

OT:
About scientific testing, if a theory is sound, it will hold up to any number of counter theories. 100 script in 1 prim, 100 prims with one script each, or 100 scripts in on prim with 99 empty prims. If his theory is sound, then all three conditions should yield the same result.

In quake 3 arena, I actually found a major bug that no one else knew about. I only found it by experimenting with every thing. That and of corse trying many different combinations of settings. This is what I discovered, by turning down certain settings in sound, I could increase the 3d rendering frame rate.
Specifically it is how quake 3 areana mixes left and right audio togther.
There was allot of debate over this issue; because many people felt that sound had nothing to do with 3d rendering. Just it turns out sound rendering is part of the quake 3 arena's engine; and thus it did have an effect on other parts of the engine. By sound taking less time, it meant more time was free to render visuals.
Just like prims, physics, and scripting are all part of the greater sim engine.

id buy that, since noone has had true direct access to hardware since the dos days

It would be nice to hear from a linden on this, but am i right in thinking LSL is getting a overhaul to use mono?

It would be a good idea making people aware of llSetScriptState and llGetScriptState, it wasnt until i read this post and actively looked that i found the functions. I assume that setting the script to a non running state removes its overhead? It would be good to see it extended to allow access to scripts in other prims. (i am assuming that from the description it only works on the local inventor, In fact this might already be possible using llRemoteLoadScriptPin, but i dont know whether its just me but http://secondlife.com/badgeo/wakka.php?wakka=llRemoteLoadScriptPin really doesnt seem to make a lot of sense. Does anyone have a decent example of this?)

Its a pretty good assumption that there are a fair few people who dont know or dont consider this kind of thing simply because its never been presented to them. I've pretty much taught myself from the wiki, reading through other scripts and from help in the sandbox, and noone has ever mentioned anything like this.

This is precisely what Lex was saying and what I confirmed in my earlier post.
All these pedantics about scientific accuracy are so geeky it's laughable. The fact remains a script is never 100% "inactive", though I have found stopping the script seems to be enough to remove it from the queue, you might not need to delete it.

seconded !

But... maybe my meaning didn't sink in well enough. I was suggesting that the system could have one unified queue that has only the first event for each script, and also a per-script queue for all of the rest. It's not useless to have both, because it prevents an O(n) scan every frame for events that need to run. The only purpose of the sim-wide queue is to build up the results of that O(n) scan across each script's queue for events without doing the scan. If our assumptions about scanning across all event queues for new events is correct, I'm prettty sure that my suggestion can only help.

That shouldn't be the case. A script that has no active event handlers shouldn't be in a run queue. If they're sitting there running through all scripts and checking each one whether they have work to do on a given frame, that's not good. Because well-written scripts should be idle (that is, not in any active queues) most of the time.

Ideally, the main sim loop should be something like:

calculate timeout based on length of run queue;
while(there's a script on the run queue)
{
  take script from run queue;
  run script until waiting or timed out;
  if(script is not waiting) put script on run queue;
}
for(each event) // timer, sensor, ...
{
  for(each script on event queue)
  {
    if(script is ready to run)
    {
      if(event is not periodic)
        take script off event queue;
      put script on run queue;
    }
  }
}

A script shouldn't use any CPU time unless it's waiting on an event that's actually occurred this frame.