Scripts and sim performance

I apologize if this is the wrong forum but it seemed the best choice.

I had a quick question on scripts and sim performace. I tried looking up some stuff but came up blank, probably because I do not know the correct terminology to plug in. Bear with me.



I have a class 4 sim that crashes randomly, lags like crazy, and is basically just going downhill fast. When I went into the "show top scripts" menu, it told me there are 780 scripts running at 26.4 ms.

Back on another sim, which is also class 4, it tells me there 645 scripts running, however the ms is only 2.4. What a big difference! This sim is practically lag free.

It seems to me this "ms" count is playing a part between a laggy sim and a non laggy sim.

My question is, what is the reccomended "ms" number? is 26.4ms way too high? Is this why the sim crashes , lags, and is just ultimately a bad place to try to live/conduct business on?

Would upgrading to class 5 server help or is the key here to cut down on the scripts that are attributing to the sim running at 26.4 ms ?

The top scripts of the sim that has 26.4ms seem to be pose balls, beds, etc. Things you would expect to see out at a shop or residence. Would seem slightly abnormal to have the renters pick up everything......

Anyone care to shed some light on this?
Thanks.

It's hard to peg a good target for script time because sims can run (almost) lag-free despite having very high script times as long as the scripts aren't doing things to slow down other sim operations. Indeed, the laggiest sims will have a very low script time (but also 0 "spare time" because the sim is too busy with other things to give scripts much time to run. But in this case, yeah, if the sim is using 26.4ms for scripts, it can't be good, given that the whole frame should only be 22.2ms.

IIRC, at the moment Top Scripts doesn't report agent-attached scripts (that was to be in 1.26, right?). So if the problem sim has anybody in it when you're measuring, scripts in their attachments could be the source of the high script numbers.

You might also consider filing a support ticket and get LL to help track down the problem.

It certainly seems worthwhile to find and fix the problem rather than upgrade the sim, especially if this is a grandfathered Class 4. Upgrading would be no guarantee of improvement.

thankyou for the reply. It is grandfathered class 4. Would upgrading cause me to have a higher tier fee?

Yes, I did file a ticket with LL and was instructed to upgrade to class 5. Was then told by another SL resident that doing so would not guarantee improvement, as you said

I will put in another ticket but they have never been helpful to me. Figured I would get more help here on the forums .

Basically "ms" is milliseconds. Every script takes a certain amount of processing time to execute. Some scripts will take a lot of time, some less. The less processor time your sim uses to execute scripts, the more time it has to send you "frames" of movement and process all the stuff you and the other avatars are seeing visually.

26 milliseconds is a HUGE number. I've seen heavy heavy scripted sims with tons of vendors and holodeck rezzers and teleport systems and sci-fi whistles and bells and all kinds of stuff going on and it averages about half that.

Ways that you can get up that high though is if you have a lot of animated prims that are always moving, lots of roaming pets (we had a dog from a popular pet manufacturer that added about 1.2ms - so 10 dogs roaming and there you go).

Chances are too you could be suffering somewhat from what we're calling "Ghost Scripts" - it's a bug in the current SL where sometimes (but not always) you will delete an object, but the simulator keeps the script running and executing even though the object and the stuff the script is doing doesn't even exist anymore.
http://jira.secondlife.com/browse/SVC-3867

Take a look at that link and it will tell you a few things about how you can see if this is affecting you (it's a real pain in the butt to track them down, but... it can be done). And of course, you might also want to vote for the issue - but it does affect a lot of people - and most probably don't even know about it.

Good luck on getting your script lag down. Hopefully there are a few things on your sim that you can identify and remove (or find alternate versions that aren't so laggy). Moving to a class 5 would help the speed - but 26ms is a big number regardless - and you'd be seeing lag even on a class 5.

Hope something in here is useful to you!

Thank you very informative!

One of the main problems for me is I dont own any object on the sim but a few. It is all renters. So this is going to be kind of tricky. This [bug] may be my issue, I will have to check it out again when I get in world. Something is just very weird about this sim.

Thanks again.

Is it your land, and are they vendors? Vendors are awful for causing lag, for two reasons. I have yet to see a vendor take less then 0.1 ms, even when idle, and second, all the texture loading. Yes scripts cause lag, but I think a bigger problem is all the textures, those seem to cause the most lag out of everything.

I'll wade in again. First, about the grandfathering: you know, I'm not sure if they actually "un-grandfather" sims at upgrade still -- I'm sure some RA people (Ciaran, for sure, and Elanthius) or LL themselves would know the scoop. The KnowledgeBase () says they do:


TBH, I'm surprised that scripts can even *get* 26ms in a simulation frame. It is definitely sim-side lag at issue here, not texture lag. (Which is not to say that vendors can't cause sim-side lag--and even script lag specifically--and also, indeed, texture download lag is a huge problem these days. It's just not *this* problem).

To be a little less lazy about it, I checked the 1.26 Server release notes (https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/1.26) and indeed that release should:
which may be a help in tracking down the problem *if* the offending scripts are in an avatar attachment. As best I can tell from http://status.secondlifegrid.net/2009/04/09/post582/, 1.26 is slated to roll out on Tuesday and Wednesday (April 14 & 15).

Of course, if there aren't agents in the sim while the lag is experienced, that's not going to help. But then, Top Scripts should give some numbers that would suggest which objects are causing the problem. (One can have a runaway script in the most innocently named object, so it's the numbers that matter here.)

Also, this isn't one of the old OpenSpace sims, by any chance, is it? I recall that there were some Class 4s running old OpenSpaces that were converted to Class 5s during the price hike. The sim performance statistics on OpenSpaces are notoriously unreliable.

Thanks alot Qie. Nope the sim is called "Existence" and its a full 15,000 primmed class 4 region.

I`m not on SL right now but I will look more into this in regards to your post.

I was also wondering if a faulty sim could perhaps cause scripts to have a higher run time. I noticed there was a pool rezzed, when I sat in it the run time was 4. something on it, it was the top running script, which got the number to above 26ms. I was going to try to get the owner of the pool, who happens to be a friend, to rez it on my other sim (the un-laggy one) and see if it had the same high run time.

Existence crashed again yesterday, I believe it to be crashing atleast once a day.

There is some good advice and tips here. I haven't seen a statement that's not true and possible. And, in fact, the real truth is going to be a combination of all of them, most likely. Lots of vendors, some laggy scripts/items, maybe a ghost script or two, and so on. The big key here is that even if you were to apply for an upgrade (and end up paying more) 26ms is going to bog that out too. So you will have to try to come up with some way to do it.

Another big thing with vendors is yep - even idle you'll looking at .1ms each (Hippos are that for sure - JEVN's are a little less, but still can do it) - so in a "mall" type sim, it's simply going to be an issue. If you have mall people with vendors that auto-cycle (i.e. they scroll through the prodccts when no one is around - that will add to it also).

Your friend's pond could be a contributer, but unless it's chock full of majorly laggy scripts, it probably isn't the culprit on its own. It still might be a good idea to check with your friend and pull it for a day and see if that does make a noticeable change to the bottom line. As I said - some thing can really make a big impact. Even then, that would not likely bring it lower than 24ms with just the one move.

Good Luck! And great thread - this will probably prove useful to a lot of people who might one day find this stuff and read it.

Youre right, ive come to find out that I need to get that number shaved down whether on class 4 or class 5. And theres no way im going to upgrade if it means an increase in tier price. I already think the tier prices were ridiculous as they were @ $195/month.


I dont see any vendors on the sim, only the Hippo rental boxes which I did notice one renter had a ton and they were all .1+ ms. I dont know how fair it is for me to ask her to remove them. I guess I just need to think up a plan and perhaps revise the rental rules.

There is one renter in particular who is using the most script time out of everyone.


Btw, the pool is not at 4.0 ms when idle, it was only when I sat in it (It had pose balls)

I guess ultimately this is a matter of getting the script run time cut down drastically. but I still wonder if all of these items were placed on another sim..would it be *this* bad. I need to do some experiments.

You might try sending Babbage Linden a message about this Jessaitui. As noted it should be impossible to exceed the 22.2 ms limit, but it is possible that this could be a very good example of what they are trying to fix in MONO now with scripts being able to "grab" more then they are allocated.

That's actually the pre-1.26 trick for measuring script lag from attachments: sit on a poseball and read the script time for the poseball. That 4.0ms was mostly your attachments. (Well... it's theoretically possible for scripts in the poseball to wait until somebody sits on them to trigger some wild processing, so a super simple poseball is what to use for a "pure" measurement of attachment scripts.) Anyway, come 1.26, this trick won't be necessary anymore.
Wow. I'd be really interested in results from others in a position to measure script times from those boxes. I know they're "networked" so they have a little something to do even when there's nothing happening on the sim (unlike standard stand-alone rental boxes), but I'd have expected that to be way down in the noise. Maybe it's a configuration thing, or something the Hippo people would want to investigate; they're reputed to be responsive to customer concerns, and it wouldn't be in their interest for their products to be banned to reduce lag.

The frequent reboots definitely raise the question of cause and effect: is it a sick sim that's causing the extreme time dilation, or is it the extreme time dilation that's somehow making the sim sick? Informative, either way--and possibly of interest to LL Development.

Is the OP seeing time dilation? That wasn't mentioned above.

If so, a possible culprit is script memory usage, causing memory cache thrashing.

What kinds of lag are you seeing? Just slow response for scripts, or slow everything?

Lear, scripts alone were using over 26ms, so the simulation frame had to take > 22.2ms, hence there must have been time dilation, right?

I agree that it could be script memory usage.

I stopped in at the sim early this morning to see whether I could get any clues, but it was pretty well-behaved at the time. There was 0 spare time, however, so scripts were using all they could get within the frame, but when I visited there wasn't dilation nor any noticeable lag. There were a surprisingly large number of scripts, however: a bit over 3000, which is kind of enough for a Class 4. (I say "surprising" in that the sim doesn't appear very crowded nor script-intense, at least at ground level, so I'm not sure where all those scripts live.) There is a prefab-rezzer which pushed down FPS when activated, but everything recovered quickly--even while pieces were still rezzing.

Maybe sample again when the sim isn't so happy.

Nope. I'm not exactly sure what the script time figure means. However, we've been told that for each time slice, physics runs first; some number of ms are reserved for scripts, and if physics quits early, scripts get the extra time. But script time never eats into physics time.

If it's thrashing caused by script memory use, you'd see serious time dilation. Sounds to me like you're seeing basic script lag. A typical symptom would be a slow response to touch on a scripted object, without much other kinds of lag (moving, rezzing, etc).

This is what everyone is experiencing on the sim

- Random crashes
- Cant walk, feels like you are walking through quicksand or something
- Sometimes you can not Tp to the sim

(This all doesnt happen all of the time, but VERY VERY often, i`d go as far as to say about 80% of the time)

No one has sent in any complaints about their scripted objects not working. Just complaining about the above, all of which myself has experienced.

Last time I checked in the ms number was 21 and things seemed a bit better.

This is all still sort of over my head but this quote below is exactly the same thing I am wondering:

I believe, if you have region access, you can turn off ALL the scripts. If you really want to see if that is the problem, might want to try that, I don't know if you can set them to running again afterwards, might want to check that out first. But if you do set them all off and you still have the problem, it's obviously not a scripting problem.

Turning off all scripts can cause as many problems as you might fix. Many things won't boot back up right when you turn them back on. best to avoid that except as a last resort...



These, holodecks, and things like that that rez things as needed (not temp rez - but just the act of items being cleared by scripts and not deleting) tends to cause the "Ghost Script" issue to happen more often than a regular delete of a scripted object. If you look through that ghost script bug I linked to above, there is also something going on where script times do report higher than "possible" numbers. So I do think that this is going to be a part of it.

Could be wrong, but that's my guess.

Well, I visited the sim again, and caught it in an unhappy state for a while. And the lag when I was there was almost pure Network time, apparently dragging Script (and sometimes Images) along with it.

What might cause that? Only LL can know for sure. It could possibly still be scripts doing something unnatural to the network (e.g., just how b0rked is XMLRPC these days?) But I suspect it's something more directly associated with network connectivity itself: bad cable or connector? flaky ethernet port? routing problem in the datacenter specific to this IP address? Something dumb like that, I'll bet. And only LL can track it down, and only if they watch what's happening on the sim long enough to see it. Or run some network diagnostics, if you can convince them that it's really a network problem--assuming it really is what it looks like.

Thanks Qie.

Yah I keep getting excessive complaints about the sim. Its REALLY bad now. I have no need for a sick sim, not right of me to rent it to people as a platform to do business on when it no longer is a safe place for their businesses. Then im left with an empty sim, and I cant sell it because its a piece of trash.

Going to contact LL again and hopefully they do something beside pressure me to upgrade to class 5 because something more is going on here, they sold me something faulty and need to get me a working sim.

Im kind of pissed off at all of this.

I appreciate you alls input, far exceeds anything LL has ever given me.

High script time, by itself, does not cause these problems. High script memory use can, because it causes memory cache thrashing. In this case, I believe you'll see time dilation values well below the healthy value of .99.

I suspect that thrashing would cause all the other lag symptoms, including the numbers Qie reports above.

Excellent point. I assume that this would cause the equivalent of resetting all scripts? If so, yes, that would cause problems. For example, if I reset scripts in my house with the doors open, they get borked, because the scripts assume they'll be reset with doors closed.

[EDIT:

Hewee says that this does NOT reset scripts, so it must be a bug of some sort. In the Jira, many report fixing the ghost-script problem by disabling scripts, resetting the sim, and then re-enabling scripts. I saw only one report where it caused problems, with a certain kind of vendor.]

Interesting -- if so, it's relatively new or not universal. I have my own holo type script, which I've used extensively for everything from my living room and bedroom furniture to making gifts for others, and never seen this particular problem. However, I can no longer see it directly because I don't have estate manager access any more. All I can say is that it wasn't a problem most of last year.

I notice that the sim has been upgraded to 1.26. (I guess by end of today everywhere will be.) I don't have Estate tools, so I don't know what it's like, but there should be a way now to see all the top scripts, including those in attachments (identified per agent, anyway).

But there's something happening that I don't understand. The sim, even when healthy, with just nothing going on (events per second barely registering), is still running scripts with every msec it can, leaving "Spare Time" pegged at 0. Do we know what can cause that? Is this what thrashing looks like on sim statistics? (I'd have thought not, but only by analogy to *nix process statistics, which may be quite irrelevant.)

Lear, I agree that script execution by itself should not be able to lag the sim. But again, while I was there earlier, Script Time (whatever that measures) all by itself was larger than the target 22.2ms total frame time. I don't know what that means, but the one thing that consistently was moving with Script Time was Net Time, which seemed the logical place to focus.

Estate Tools don't identify agents. You have to sit on a prim and see what the prim is using to check the scripts. I rcvd a message from Babbage this morning stating that they still have not decided on the UI changes that will show script memory usage. When that becomes available we will also be able to see the resources that an av is consuming script wise.

hmmmm mayhap scripts making outside requests? assets, http, rpc, etc? could it be possible some of those are counted in the script time, even if a portion of it runs on the sim time? might partially demystify why similar script time numbers on different sims have a diffeerent effect

also I think the blame on holo rezzors, might just be frequency bias... they rez and remove more often, so they see the problem more often. at least that's a guess based on the still random nature of it.

stopping all scripts may cause loss of incomming data like new uuids, and since the script doesn't know it was stopped it won't know to check for new data. I think the effect is the same as setting all scripts to not running (which still preserves the stack and heap in there current state)