Babbage Linden asks: what can LL change in LSL to reduce lag?

I may have to revise my statement concerning dormant vs idle scripts in light of llTargetOmega (which I'm being told continues running even if the script is set to not running, but dies when the script is removed? that's.... kind of freakish)

in which case there is either still some script operation under not running (and hence cpu usage) or some type of temporary prim property storage with a callback to the originating script (or the space is created by the script, which suggest higher bandwidth requirements even if the script is dormant, which seems not good)

I was unaware of formal definitions for these terms. I was referring to scripts with no events pending, nothing to do. I suppose by "dormant", you mean ones with no *possible* events? Evidently, you mean "not running". I call those "not running", since it's unambiguous.

I disagree. A script with no events pending (with possible exception for listens) should not be "visited" at all, even if the check is very efficient.

Is 2 microseconds "efficient"?

Note that it's probably not the time to run the instructions that takes the time, but the time to pull the relevant information from memory, especially if it's in disk cache.

Well, that's a special case, but definitely one to note. This is a corner case, really: should this be script time or physics time? In a way, it's both.

If it's more than a few dozen nanoseconds to do a table lookup on an event, then they need to fix it. The time is going to be dominated by memory access, since the data is probably not in L2 cache (5 cycles, or about 2-3 ns away). For list following and table lookup, the speed of the memory (throughput) isn't the critical factor, it's the cost of a cache miss (pipeline stall) you really need to worry about, because main memory is too far away for prefetch to do any good on this kind of tight loop. Say they have about two cache misses to wait through (follow the link to the next script, indirect fetch through the table). That means about 30 cycles stall if they're using AMD or Core Duo CPUs, 80 cycles if they're using Pentium 4. 2-3 GHz CPU (probably 3 GHz if P4, 2 GHz if AMD or Core Duo), so that's 15-27ns per table lookup per script.

If the relevant information is not in memory, then the problem isn't script time, it's script memory, and the sim will be lagging its guts out from swapping if that happens more than very occasionally.

Today's processors are considerably faster than memory access. Even if data is in memory, it takes a long time to fetch it. Even if it's in L2 cache, it takes a long time, relative to instructions.

In any case, it takes 2 microseconds, which IMHO is too long and should be fixed, as I said in my first post. This wastes far more CPU time than anything else in LSL.

/me finds .

Vote!

/me would have to see this to believe it. I always thought llTargetOmega continued after the script was gone.

As for running vs. not, I suspect that's confusion about how llTargetOmega behaves when it's set on a physical object. When it's on a physical object, it does collisions and stuff which, I guess, could cause a bit of extra processing time to show up on a script that has collision events even if it's set to not-running. Could be wrong, though. On a non-physical object, it should be 100% client side after it starts spinning.

The llTargetOmega weirdness is correct for nonphysical objects. Remove the script and the movement stops. Set the script to not running and it keeps spinning, _but_ it won't resume if you take and re-rez.

For physical objects, the movement _does_ stop once the script is set to not running.

Yes, I know that. But not THAT slow. Main memory is not 4000 cycles away.

L1 cache in a Pentium 4 is 2 cycles away. L2 cache is 5 cycles away.

WHAT takes 2 microseconds? What exactly are you measuring?

Using "Top Scripts", we see that scripts that have no events scheduled, or which can't possibly even have events to handle (e.g., state with only an entry handler) almost always read at 0.002 milliseconds. Sometimes they read at 0.001.

My undrestanding is that this is how much CPU time they took on the last pass of the scheduler.

I can't see any way that event handling could have that effect. Sounds more to me like they're measuring the time it takes to instrument the script. Like they're making a system call for each script every frame, to see how much time it's taking, and that system call is taking 1-2 ms.

Eg:

Get clock time.
See if script needs to run, if so:
* Run it.
Get clock time.

Instead of:

See if script needs to run, if so:
* Get clock time.
* Run script.
* Get clock time.

it gets weirder, apparently it it not a matter of client side not getting updated, as it was tested with a second client that tp'd in AFTER the script was stopped, both see it stopped upon script removal though... but in yet another twist, when the second viewer was a linux client that happened to be on a different virtual desktop (so effectively minimized) while the script was being removed, the view didn't update, and the object continued spinning (separate issue, but still funny)

I take no credit for the observations, I still can't get in, someone else told me about it and I suggested things to test.

I Know some of that idle time we see is for the script time counter, and if it operates off of some type of cycle increment (which would make sense for dilation effects) it''d explain why it needs the constant poll, even when idle... I'm not sure how necessary the script time counter is (it may be what's used to time built in function delays), but if if it's not a necessity it'd be nice if it only ran when we set it up.

Hrm..

/me wonders if it's because TargetOmega isn't a prim property like color/size and, when you mod the object by changing its inventory, an object update happens and wipes out the old TargetOmega setting. That'd sorta explain the remove vs stop behavior. Sorta. I guess. Maybe.


/me tries to stear the thread back on track! Is it time to sum this up and send it to Babbage?

I knew there was something else that should've been tested... full object updates (should be workable with a second script and a llSetTextCall if anyone cares to test)


kick it, strip it, and ship it

(although the more that I think about it, that script time counter needs a look... it's gotta be responsible for at least some of that idle overhead, and I bet I'm on track with it counting by cycles and the possible ties to script delays)

What do you mean "counting by cycles"?

er frames, I meant frames...

if the if the script time counter (which reports seconds) is only updated every X frames (when it's visited, I don't think a frame would be a whole second) it'd be an excellent mechanism for updating it that would account for both dilation effects on it, and part of the idle overhead. at least it seems sensible to my sleep deprived mind.

If what Lear's reporting is actually real, then I think just fixing the logic as I suggested (so that it only actually performs script time updates and related calculations on scripts that actually have pending events) would be more than enough improvement.

They could reduce the default plywood texture resolution I suppose.

Perhaps make standard floor/wall prim sets that are stored on everyones hard drives. Not to say you couldn't still use your own but the standard ones would rez instantly in comparison for common sizes like 1x1x.3, 2x2x.3,4x4x.3, 5x5x.3, 6x6x.3, 10x10x.3, 12x12x.3 or 20x20x.3. Custom prims would still be used for extras or odd pieces etc, but you could easily halve the download traffic on an average building. So all the server would have to say is "there is a part#523 at location (x,y,z) with it's orientation perhaps or just seperate part#'s for walls.

Perhaps delete & relink all the near identical duplicated "red brick wall" textures and such from the texture database.

Perhaps a simple one line command that tells a script to sleep unless an avatar is in the sim or within 50 metre's.

A simple way of making all the unselected faces of a prim/object to a default lowres texture at 1x1 repeat mapping.

Maybe a way to report all the objects on a parcel and their locations, to help cleanup un found junk in the sky/ground.

I'm not against upcapping the flight ceiling, but I doubt it will eliminate flight assists. I use the /flight fast, and /flight god modes in my multi-gadget all the time, and I wouldn't stop using it if the flight ceiling was removed. It would eliminate lesser flight assists that probably most people use, but saving 1 script per avie... I doubt it. While most avatars probably use some lame freebie flight assist that just breaks the ceiling, multi-gadget, along with mystic tool and a few others combined make up alot of multiple function devices that people use. Again, I'm for the ceiling cap being removed, but I doubt it would save all that much (although it would save some).


I think another issue here that hasn't really been touched on is resource management. ALOT of people in SL are total rank amateurs and write sloppy cpu intensive scripts without any thought of how efficient the script is. While I'm not off in a corner counting cpu ticks or studying the innards of the severs like Void and a few others do, I do try to write tight, efficient code with the least amount of scripts as possible. Take my clocks for example. I'm up to the 3rd generation. My originals had open listens on channel 0 all the time! Now, they have no listens at all unless a menu is present. Not only that, I've got a few ideas that will improve the efficiency still more, and the originals were written 3 years ago when I was a total scripter noob. So, novice scripters write sloppy code, and on top of that, there's script kiddies out there who cobble cow dung together by throwing dinky scripts together (i.e. a rotation script and a particle script and a running water sound script all tossed into a pool).

On that issue, I think what's definitely need is a script cost number similar to ARC. I mentioned this to Babbage 3 times at the last meeting I attended, and he either didn't catch the text (3 times???), or didn't like the idea, because he completely ignored my suggestion.

I think a script cost number would do leaps and bounds to get scripters to improve their efficiency. Down the road, when script limits are imposed, instead of a set number of scripts that can be ran by a specific avatar, a max script cost could be set. This would cause scripters to write tight code (and even advertise the script cost), that non-scripters buying objects could use to compare against similarly functioned products. The biggest thing that could blow script cost would be number of scripts used, followed by the amount of cpu load the active script produces. Hair resizers would disappear overnight because no one in their right mind would buy hair that costs THEM such a huge amount of their script allotment.

I'm thinking it's the root source (or one of them) of what he's seeing.. and if so it may not be correctable without more than a bit a bit of reworking, but it does fall within the same issue.

random thought:
physics can contribute greatly to general sim lag (not necessarily script lag), so should probably be looked at as part of the package of overall per user/sim limits.... currently I think they're only talking about script limits, but if all a parcels scripts are physics stuff, that may drag down a sim even inside the limits they are considering.

Flight Feather doesn't have any such modes... deliberately so. It would have been easy for me to add aggressive boost, but I chose to make it quiet and transparent to use and *non-laggy*.

If your flight script is listening on channel 0 for commands like "/flight" then it's far laggier than it should be. No matter how tight you think your code is, that open listen eats far more CPU than you'd waste by "sloppy" code, and you'd be better using a "lame freebie flight assist".

I respectfully disagree. They SHOULD accout for the time it takes to figure out whether a script should be run, otherwise we wouldn't know the cost of these scripts. It just shouldn't take ANY time for scripts without listens.

Again, I disagree. My guess is that testing to see if a script needs to run is what's taking time, not measuring the time. If I'm right, it's a very good thing that it's measured -- we need to see that having scripts with nothing to do costs CPU time!

What they need to fix is the fact that scripts with nothing to do take most of the CPU time allocated for scripts!

How many times do I need to repeat that simple suggestion? I don't seem to be making my point because everyone seems to be missing it.

Here's the beef:

In a typical sim, most of the CPU time for scripts is used for scripts with nothing to do!

Get it? We're wasting CPU time doing nothing. So, FIX IT if at all possible.

Now, I have a strong suspicion that for listens, it's just not feasible. I remember someone saying that if they're not using a hash-based system for listens, they're morons, but I have never quite figured out a way to employ hashes that doesn't cause more problems than it solves. And I'm sure I'm missing some of the facts that would complicate it even more.

But most events should either be obviously pending or not, and the fact that ANY event is pending should be a single flag, or possibly queuing the script in a run queue. Yes, this adds script time, but only for those with events. Our problem is for those *without events*.

A carousel should be used for timers, if it's not already.

Simon's comments in http://jira.secondlife.com/browse/SVC-3819 explain the times that are supposed to be included in top scripts vs. the total script time display. Scanning overhead isn't intended to be included in top scripts (it _is_ supposed to be in total script time), if additional time is being displayed it may be residue from this bug (which should be partially/mostly fixed now, if not yet perfect).

I can't see that taking two milliseconds. But more to the point, it is impossible to calculate the time taken by an operation (whether that is "testing to see if its to run" or "running the code" without making a system call.

And making a system call is at far more expensive than accessing main memory. You switch execution context, switch memory context, switch privilege context, validate the call and arguments (to avoid modern analogs of the old DEC-10 password bug), marshal the response, then go back through the context switch again. System call overhead is why real-time microkernels like QNX kick butt, and why people keep going back to microkernel designs (real ones, not jokes like Mach and Windows NT) despite their complexity.

According to benchmarks, the "read" system call (which basically does a memory to memory copy if the data is already in disk cache) overhead in Linux is about as expensive as copying an extra 100-150 bytes from memory to memory, for each call.

Accessing main memory on a GHz processor doesn't take milliseconds. If the memory is really slow, say "5-5-5-5", it'll take worst case 20 memory bus cycles, which adds up to maybe 1/20th of a µs if you have a slow memory bus. If you're following a linked list of prims, you're doing a link fetch and a table lookup that aren't cached, though with prefetch the table is probably going to be pulled in on the same lookup as the next prim link.



I'm not disagreeing with that, IF that's actually a real problem. I don't trust Top Scripts to tell me that. Have you rezzed 1000 objects on a quiet sim and seen long term script time jump by a couple of milliseconds?

What I'm saying that the problem, if it is a real problem, is not likely to be caused by checking if the script is ready to handle events, it's caused by the instrumentation itself. Just making ONE system call is EXPENSIVE. Putting a system call in a critical path (and that's what they're doing if the numbers returned by Top Scripts are real) is just plain bad design.

I agree with this. I'm saying that you have no basis to assume that event handling is what needs to be fixed.