i love island sims (or more research into lagging)

Some profiling tools that would dump the total cpu time for each script in a sim would be nice. If I had a script sitting at 10minutes of cpu time after 1 hour of resetting the script, I would know it was in need of optimizing.

And if not every script then please just this:

llGetCpuTime(); PLEASE!!!

SimFPS is not a valid statistic for measuing lag. SimFPS is a composite inverse number basically 1/n where n is the amount of total server time it takes to render a frame.

'1000 simfps reduction' can be a timeslice of 0.1ms, 0.005ms or 0.0001ms depending on starting number and what else is going on.

What you need to use to measure objects or LSL fuctions' lag profile is a snapshot of that objects' or scripts impact upon the overall simstats timeslices.

A proper measuremenet would look like the following (roughly)

1000 listens:

1.9ms runtasks


1000 object updates (color change)
8.1ms runtasks
2.5ms runAgents (one agent)

etc etc.

Obviously the granularity of a single objects impact in regards to ms *SHOULD* be very low (tho this is not always the case, *pokes* a 'hacked' 200 leaf fran tree) so it is most useful to run 10, 100, or even 1000 of the obejcts and then divide it ou of the resultant saying for example a listen takes 0.0019ms runtasks (based on the made up statistic above)

Certain slices of the simStats profile are more lag inducing than others, for example physics is particularly damaging because that is the practical cause of time dilation, over-extention of the physics slice, or in the case of an overall heavy burdened server the amount of time between physics engine calls)

as to impact on simfps thats barely meaningful essentially being only as mentioned previously a rough inverse composite value, and not in any way a precise measurement

Hank, you're just pontificating in abstract. In reality, timers DO cause "Avatar Experience" lag.
The other day I rezzed 4 cubes with a one-second-timer script (empty event handler) at the Galleria City.
Sim FPS dropped from 500ish to 100ish and the avatar experience afterwards would best be described as Lag-a-Licious.
I then changed it to 5 seconds and I noticed that the events were probably firing off more or less in synchrony since I would get periodic bursts of lag.
I have no idea why an empty event handler would cause such a performance hit and I dont think this is normal.
Someone get LL to chime in on this one, and report the results back to this thread

alpha is irrelevant to sims, its just geometry on a sim... its the client that has to actually render it and take the huge performance hit

(its a common misconception that rendering something 100% alpha or using an alpha texture makes it 'invisible' to the client and thus not rendered, in reality its the oposite, making something transparent means its actually rendered to the client twice each frame (alpha requires basically a second rendering pass after the first one is over, hence some of the weird transparency bugs you see in games including SL)

Please drop a copy of those 4 cubes with empty timer scripts that caused an 80% reduction in sim performance on my profile in game Eggy when you get a chance.

Aww, come on Kelly. You can do better than that. Let us know what we can do to analyze lag. What we can do to improve scripts. What can be done to measure the effect of scripts on sims. We don't have all the tools you guys have, and we are dangling here.

sim fps is a simple measurement.

your argument runs the way almost all arguments against any particular performance test goes. "that's metric isn't valid, vigorous, and/or meaningful."

which, to be honest, is a valid criticism.

note the goal currently isn't necessarily just to find things that cause lag, but to determine how to measure it and predict it.

does a low simfps predict lag. yes.

thus understanding how simfps is driven gives information about lag.

research into lag (aka performance) isn't a magic bullet type problem. there isn't one thing you can do to determine lag.

lag is a systemic problem ranging from "simple" usage to script event handling to script function calls to beating on the physics engine to the number of scripts running at one time to infinity and beyond.

as a generally rule, putting these kind of system introspection tools into a non-debug client and/or server with free user access to it is questionable. it just contributes to code bloat with very little utility to most of the user base.

well no the basic idea is that simfps is the simple sum total of all of the timeslices the server has, divided into 1 second.

Rather than fuss with that 1/n metric just use the real statistic when it comes to things like listens and color changes and timers, which is RunTasks. Run tasks is the amount of time it takes the current simulator to process all of the scripts within it. So if you are attempting to meaure the latency of a script command.. *IT* is what tells you how much that given script is slowing the server down.

This is not a percentage measurement, its not this is now 5% slower than it was

its a very concrete measurement. A script action that adds 1ms run tasks will *ALWAYS* add 1ms run tasks, whether the current sim runs at 10000 simfps or 100. Unlike simfps where at 10000 (aka a total server time to process of 0.1ms, a simple 1ms additional script object can bring that down to 900 (aka a total server time to process of 1.1ms), the same script tho may only bring a sim at 100 simfps (a total server processing time of 10ms) to 90 (a total server processing time of 11ms)

so as you can see depending on what else is running.. the *SAME* object can cause a greater than 10x slowdown in an empty sim, from the high end to mid range, or a barely 10% slowdown, in a rather busy sim towards the low end

i do know that.

but i'm not interested in single object affects on the sim per se.

i'm interested in the end user experience of the sim. and sim fps is a good predictor of lag in an of itself.

of course i know how to increase the time dilation on a sim with even a high sim fps.

sims have static loads. the lower the sim fps, the higher the likelihood that any one event or group of events will cause lag in a dynamic fashion.

runtasks is a dynamic load measure, which is important, but i'm not there yet. and again i might not be there anytime soon. i'm being methodical and plodding. and i don't spend all my time on this.

* * *

again, if anyone has some simple scripts they would like me to run in multiple objects, send me an im.

please keep in mind, this server (me) is very loaded (i have many projects) and this lag research is only getting a few timeslices here and there.

anyone who want to contribute to the community's knowledge with hard numbers is welcome to do so. don't wait for me to give you permission; you don't need it.

it's not like i'm managing a team project here or that i'll mind if you run your own tests. you won't be stepping on my toes, if you get some measurements against runtasks. any contribution you can make toward getting hard numbers is welcome not just by me, but i'm sure other people would like the information also.

Thank you for the work you did to get us this info His Grace.

Can you tell me if listeners that do "show/hide" lag a sim if they are not in range of people speaking? Say like at 700m?


Thank you in advance

Briana Dawson

This thread is a real eye-opener, and I thank His Grace (and the other gurus) for the information given here.

It appears that I'm about to be gifted with an old PowerPC-based Macintosh by my employer. My intention is to install Linux on it (which I'm told performs remarkably well on PowerPCs), and then to turn it into a dedicated Ruby script server.

I can hack my way around in Ruby, but I've never done much with XML-RPC, and I wonder if anyone here has any figures (or just a feel) for performance of, say a dozen objects (rezzed for event only, not perennial performing their own motion and texturing calculations, versus a dozen XML request-response loops hitting the sim at say, one-second intervals during the demo.

I hope my question made sense. A semi-related question on what I've read:

1. I want my house to do Cool Stuff in my absence, but not if nobody's around. Now, from what I understand, LSL scripts cannot initiate XML-RPC exchanges. That means to me that a device on my property should quietly email my external server, and wake up some linked objects to start listening for XML R/R loops to get the show going. Then, after the tourists leave: shutdown.

That's nothing more than a rough sketch in my head, and again (as if it weren't apparent) I stress my inexperience with this concept. If anyone wants to tune (or shred) this idea, I'm still a couple of weeks away from even attempting something. I'm not trolling for system design hints per se; my question is: does that sound like a decent stab at Good Citizenship, or am I going to get ridden out of Europa on a rail?

Ultimately, what I'm fishing for is an idea as to whether I can leverage a sim's available bandwidth by moving certain calculations Out-World and then transmitting them back in, or if that's going to be more net load on the sim than just letting the objects do their thing right in-world.

There; I FINALLY got to some efficient wording for my line of inquiry, and if you've stayed with me this far, I appreciate your patience

-yoshi

Edit: Changed "per sey" to "per se" so that I don't look like a complete goob.

There is some discussion on the Wiki about performance of XML-RPC; one conclusion seemed to be that it is pretty hard to get lots of XML-RPC into a sim in a short period of time. This is probably some kind of deliberate throttling rather than related to lag, but it probably means that you should regard XML-RPC as potentially quite high latency. If you're talking about a dozen transactions a second, it sounds like you'd hit this effect for sure.

My guess would be that unless you plan to do substantial real calculation there is not much point trying to take it off-world. For simple object control, I don't think it makes sense.
Yes to the first part . As to the rest, remember that XML-RPC receipt is done on a channel the recipient creates, and that you need to notify the off-world script of the channel's key. So, you need to llEmail the off-world script anyway, and you therefore probably don't want to bother closing the channel just because it isn't in use for a little while.
Speaking as the guy in the plot next door, its going to depend on exactly what cool stuff your house continually does in the corner of my eye while I'm sitting scripting
Unless you tell us what the computation is that you might shift, and how often the resulting communication would happen, we can't guess any more than you can. My no-data guess, though, is that unless you get real sophisticated, using XML-RPC is probably not going to improve either the performance of the sim or the behaviour of the objects.

BTW a timer set at 1000.0 seconds will only read every 1000 seconds and will only have a "load every 1000 seconds when it reads the selected code. A timer at 1 second will be 1000 times heavier on CPU cycles then the 1000! Think of a listen as a 0.01 timer!

This actually may not be the case. A lot of the linden functions do polling (more than just listen and sensor apparently.), someone needs to compile a big list of whether the individual event is polled (listen, sensor, collision, etc) or pushed to the object (link_message,stop_move,etc.), although I suspect it may only be possible for LL to compile that list.

-Adam

well you can add touch to the list of pushed events, not pollers. Basically you can do rather authorative testing with a sensor set to 'active' listing all nearby objects that are taking cpu time (polling, actually calling dynamic physics, applying object property changes, etc)

I made a public domain one i've been playing with for a few weeks now that works reasonably well (and hilights the sensed object with a laser)

it wouldn't be that hard to just create a script for each type of event in seperate objects, and see which ones the simulator sees as active, and which it sees as passive

for example, listen, timer, sensorRepeat will come in active, no matter what their specific properties are called as. Touch, linkmessage etc will always come in passive.

It seems to me that if you enable an event but don't fire it, and you observe a sim FPS drop relative to the same script without the event enabling code, then you've found an event that takes sim CPU when it isn't being fired... one that is implemented by polling in other words. Of course, you need an empty sim and a lot of identical objects. Sounds like exactly the testing His Grace has been doing?

i have a very simple script that can turn any object into an 'active script' scanner alexander. You don't need more'n one prim to use it an the sim can be as crowded or empty as you please

this's my active script scanner... its designed to be placed in an attached object, so you can fly around and spot potentially lagy transient objects like crashed vehicles and such

Due to the fact it uses a repeat sensor itself its *not* designed to be a permanent in-world placed script, as it will indeed burden the cpu on its own if left out continually. Aka consider it a 'tool' for a proverbial avatar attached toolbelt

(and do forgive its rather large size, most of that is to setup the particle system call.... that could be reduced down to a few lines with a dedicated particle call instead of my basic generic particle script but i've been too lazy to really bother with it)

//eltee Statosky's active scripts scanner
//PUBLIC DOMAIN

integer range=16;
integer particles=1;
integer activeTarget=0;

///////////////////////////////////////////////////////
// Effect Flag Collection variable
///////////////////////////////////////////////////////
integer effectFlags;
integer running=TRUE;

///////////////////////////////////////////////////////
// Color Secelection Variables
///////////////////////////////////////////////////////
// Interpolate between startColor and endColor
integer colorInterpolation;
// Starting color for each particle 
vector  startColor;
// Ending color for each particle
vector  endColor;
// Starting Transparency for each particle (1.0 is solid)
float   startAlpha;
// Ending Transparency for each particle (0.0 is invisible)
float   endAlpha;
// Enables Absolute color (true) ambient lighting (false)
integer glowEffect;

///////////////////////////////////////////////////////
// Size & Shape Selection Variables
///////////////////////////////////////////////////////
// Interpolate between startSize and endSize
integer sizeInterpolation;
// Starting size of each particle
vector  startSize;
// Ending size of each particle
vector  endSize;
// Turns particles to face their movement direction
integer followVelocity;
// Texture the particles will use ("" for default)
string  texture;

///////////////////////////////////////////////////////
// Timing & Creation Variables Variables
///////////////////////////////////////////////////////
// Lifetime of one particle (seconds)
float   particleLife;
// Lifetime of the system 0.0 for no time out (seconds)
float   SystemLife;
// Number of seconds between particle emissions
float   emissionRate;
// Number of particles to releast on each emission
integer partPerEmission;

///////////////////////////////////////////////////////
// Angular Variables
///////////////////////////////////////////////////////
// The radius used to spawn angular particle patterns
float   radius;
// Inside angle for angular particle patterns
float   innerAngle;
// Outside angle for angular particle patterns
float   outerAngle;
// Rotational potential of the inner/outer angle
vector  omega;

///////////////////////////////////////////////////////
// Movement & Speed Variables
///////////////////////////////////////////////////////
// The minimum speed a particle will be moving on creation
float   minSpeed;
// The maximum speed a particle will be moving on creation
float   maxSpeed;
// Global acceleration applied to all particles
vector  acceleration;
// If true, particles will be blown by the current wind
integer windEffect;
// if true, particles 'bounce' off of the object's Z height
integer bounceEffect;
// If true, particles spawn at the container object center
integer followSource;
// If true, particles will move to expire at the target
integer followTarget        = TRUE;
// Desired target for the particles (any valid object/av key)
// target Needs to be set at runtime
key     target;

///////////////////////////////////////////////////////
//As yet unimplemented particle system flags
///////////////////////////////////////////////////////
integer randomAcceleration  = FALSE;
integer randomVelocity      = FALSE;
integer particleTrails      = FALSE;

///////////////////////////////////////////////////////
// Pattern Selection
///////////////////////////////////////////////////////
integer pattern;



///////////////////////////////////////////////////////
// Particle System Call Function
///////////////////////////////////////////////////////
setParticles()
{
// Here is where to set the current target

// Feel free to insert any other valid key
    //target="";
// The following block of if statements is used to construct the mask 
    if (colorInterpolation) effectFlags = effectFlags|PSYS_PART_INTERP_COLOR_MASK;
    if (sizeInterpolation)  effectFlags = effectFlags|PSYS_PART_INTERP_SCALE_MASK;
    if (windEffect)         effectFlags = effectFlags|PSYS_PART_WIND_MASK;
    if (bounceEffect)       effectFlags = effectFlags|PSYS_PART_BOUNCE_MASK;
    if (followSource)       effectFlags = effectFlags|PSYS_PART_FOLLOW_SRC_MASK;
    if (followVelocity)     effectFlags = effectFlags|PSYS_PART_FOLLOW_VELOCITY_MASK;
    if (target!="")       effectFlags = effectFlags|PSYS_PART_TARGET_POS_MASK;
    if (glowEffect)         effectFlags = effectFlags|PSYS_PART_EMISSIVE_MASK;
    llParticleSystem([
        PSYS_PART_FLAGS,            effectFlags,
        PSYS_SRC_PATTERN,           pattern,
        PSYS_PART_START_COLOR,      startColor,
        PSYS_PART_END_COLOR,        endColor,
        PSYS_PART_START_ALPHA,      startAlpha,
        PSYS_PART_END_ALPHA,        endAlpha,
        PSYS_PART_START_SCALE,      startSize,
        PSYS_PART_END_SCALE,        endSize,    
        PSYS_PART_MAX_AGE,          particleLife,
        PSYS_SRC_ACCEL,             acceleration,
        PSYS_SRC_TEXTURE,           texture,
        PSYS_SRC_BURST_RATE,        emissionRate,
        PSYS_SRC_INNERANGLE,        innerAngle,
        PSYS_SRC_OUTERANGLE,        outerAngle,
        PSYS_SRC_BURST_PART_COUNT,  partPerEmission,      
        PSYS_SRC_BURST_RADIUS,      radius,
        PSYS_SRC_BURST_SPEED_MIN,   minSpeed,
        PSYS_SRC_BURST_SPEED_MAX,   maxSpeed, 
        PSYS_SRC_MAX_AGE,           SystemLife,
        PSYS_SRC_TARGET_KEY,        target,
        PSYS_SRC_OMEGA,             omega   ]);
}


///////////////////////////////////////////////////////
// Particle Effect Function
// - Edit the values here to change the effect
///////////////////////////////////////////////////////
ParticlePointer(key id)
{
//Color
    colorInterpolation  = TRUE;
    startColor          = <0.5, 1, 0.5>;
    endColor            = <0.5, 1, 0.5>;
    startAlpha          = 1.0;
    endAlpha            = 1.0;
    glowEffect          = TRUE;
//Size & Shape
    sizeInterpolation   = TRUE;
    startSize           = <0.04, 5, 0.0>;
    endSize             = <0.04, 5, 0.0>;
    followVelocity      = TRUE;
    texture             = "";
//Timing
    particleLife        = 0.25;
    SystemLife          = 0.0;
    emissionRate        = 0.001;
    partPerEmission     = 1;
//Emission Pattern
    radius              = 3.0;
    innerAngle          = 0.0;
    outerAngle          = 0.0;
    omega               = <0.0, 0.0, 0.0>;
    pattern             = PSYS_SRC_PATTERN_DROP;
//Movement
    minSpeed            = 3.0;
    maxSpeed            = 3.0;
    acceleration        = <0.0, 0.0, 0>;
    windEffect          = FALSE;
    bounceEffect        = FALSE;
    followSource        = TRUE;
    target              = id;
        // llGetKey() targets this script's container object
        // llGetOwner() targets the owner of this script
//Particle Call
    setParticles();
}


default
{
    state_entry()
    {
        llSensorRepeat("", NULL_KEY, ACTIVE, range, 2*PI, 1);
        llListen(1, "", "", "");
    }
    
    listen(integer chan, string name, key id, string msg)
    {
        if (llGetSubString(msg, 0, 4)=="range" && id==llGetOwner())
        {
            range=(integer)llGetSubString(msg, 6, 100);
            llSensorRepeat("", NULL_KEY, ACTIVE, range, 2*PI, 1);
        }
        
        if (llGetSubString(msg, 0, 5)=="target" && id==llGetOwner())
        {
            activeTarget=(integer)llGetSubString(msg, 7, 100);
        }

    }
    
    sensor(integer num)
    {
        string NameList;
        string header;
        integer counter;
        for (counter=0; counter<num; counter++)
        {
            NameList=NameList+"\n"+(string)counter+" "+llDetectedName(counter);
            //llParticleSystem([]);
        }
        header=(string)num+" Actives {"+ (string)range+"m}"+ "{t"+(string)activeTarget+"}";
        if (num<8)
        {
            llSetText(header+NameList, <0.5,1,0.5>, 1.0);
            ParticlePointer(llDetectedKey(activeTarget));

        }
        else if (num<16)
        {
            llSetText(header+NameList, <1,1,0.5>, 1.0);
            ParticlePointer(llDetectedKey(activeTarget));
        }
        else
        {
            llSetText(header+NameList, <1,0.5,0.5>, 1.0);
            ParticlePointer(llDetectedKey(activeTarget));
        }
    }
    
    no_sensor()
    {
        llSetText("No Actives {" +(string)range+"m}", <1,1,1>, 1.0);
        llParticleSystem([]);
    }
}

Sorry, I should have replied to your post at the same time. Are you just using llSensor to look for things marked ACTIVE, or something more complicated? The documentation I've seen for ACTIVE is vague enough I would like to confirm the results... and doing the larger scale testing might do that.

Edit: posts crossed I have something similar derived from the thing finder Christopher Omega posted a while back, does 16 at a time.

its using a repeat sensor as you can see

basically an active is anything at all that hits the sim's cpu. You'll notice while some script calls (such as a listen) will firmly plant an object in my scanner's list. Other things like editing an object's size/shape cause intermittent hits as well, hitting the cpu as the changes are made but once the object is done being processed, returned to its idle state.

the same thing applies to physics... an object actively being moved by the physics engine is considered active.. but once it comes to a rest it is 'idle' untill something else makes it move

another good tip-off can be object beacons.

All scripted objects get beacons. Red becons are for 'standard' scripts (aka scripts in objects that are not physical)... you'll see many MANY red becons as you look around, and you'll notice mebbe 1/2 of those objects will be targeted by my scanner... in the most common case these are obejcts with listens set... you'll also see many objects wich have red becons and yet don't get targeted by my scanner... these are 'good' scripts and usually involve touch events rather than listens.

green becons go to unscripted physical objects

orange beacons go to scripted physical objects (most commonly vehicles or bullets)

and yellow becons go where the physics engine is detecting collisions

Awesome work, His Grace! Thankyou for sharing your results!

[/i]

OK, what does this mean? The number that is really useful is what percentage is this taking? The answer is:

% = (100 * (b - a )) / (n * b * a)
Where:
b is the before FPS
a is the after FPS
n = the number if items (prims in this case)

Substituting for this example:

(100 * (11000 - 6000 )) / (128 * 11000 * 6000) = 0.000059% of 1 second per prim.

This is better expressed in time:
time = (10^9 * (b - a )) / (n * b * a) ns
time = 591 ns
This seems reasonable, given a processor running at 3 GHz that would be about 1800 cpu cycles.

So, how many prims would it take to take the FPS to 100? 0.01s / 591ns = 16,920 prims (in this example).

So even though the FPS fell substantially the actual effect of the 128 prims is _very_ small. This is because the change was made on a system that had a very light load.


Next we need to look at just how to interpret FPS vs Lag--it might be a straight forward as one might think.

A basic size piece of land of size 512sqm is 1/128th of a sim and it supports about 160 prims. That means, in total a sim allows somewhere around 20,000 prims. Given not every prim has a script attached to it, prims alone shouldn't tax a server.

As mentioned before, the biggest tax on a server is avatars. They are built out of numerous prims and often contain numerous other ones all animating about.

Has LL ever published any data for the number of avatars supported per sim?

One thing that's interesting, though, is they've never said each sim is a single machine. It's a node on a grid. Popular sims, in theory, could be upgraded to multiple machines (assuming their system supports this level of division).

The other cause of lag that people haven't talked about much here is simply bandwidth. Bring a handful of people into a texture heavy area on a sim all at once and you have a heavy peak bandwidth usage. All of the animation data has to be sent down to the clients, so a bunch of people in a club with animated casino objects and everyone dancing is going to lag as much by bandwidth as by processor load.

Not to be forgotten is that each sim can, in theory, be in a different location. Standard net weather issues could cause some servers to behave worse than others.

For those that are in the same location, an event in one sim can cause bandwidth lag in another sim.