The SL ervers are not up to the job

First place to look are any scripts making use of long lists that are manipulated. Log into Scheduler Test 1, 2 or 3 in Aditi. This is the server setup for checking if scripts are running over. If a script does not work there then it needs to be changed to less entries.

How long is long? - roughly(ish)

That behaviour exactly mimics what I saw in a sim not too long ago. The culprit turned out to be a permanently temp-rezzed car that had a few hundred prims. Since you've tracked down all the temp rezzers in the sim now, another thing to look for is any shops that may have the non-temp temp rezzer type displays. (I know that description looks stupid, but I'm sure you know what I mean.)

Yes I do know what you mean. I made such a system myself, and used to use it in the store. I think you are suggesting that the highs might be caused by someone using a rezzer like that to see an item. There are only 3 other shops there so I can easily check that out.

Nods. My home sim had horrendous downspikes in performance about a year ago. Culprit turned out to be a (permanent) 280 prim temp-rez space station in pretty low orbit. Seemed to re-rez less than once a minute, so at any one time there may have been two copies rezzed in the same space.

I spent many hours today recompiling everything back to LSL. I'm pretty sure that I got every little bit of script. Early on in the process, there were plenty of lengthy TFTs (Total Frame Times) in the 30+ms range that were attributable to similarly high Scripts Times. As I continued, the frequency of them grew less. I'm now waiting to see if they still occur. I'm not very hopeful that the recompiling will have dealt with it, but it may have.

There's something else too. I've hardly spent any time in the store lately, so I only noticed this yesterday, and then again today. All the poseballs on all the MLP furniture (and there's a lot) had gone. The furniture was fine and the balls came up again on menu clicks, but the poseballs that had been rezzed had gone, including those that were in use by my demo models, which meant that the models were just standing there on the furniture but no longer on poseballs because the balls had gone. It's as though someone had gone round and clicked the STOP button on every piece, but the beds and sofa that the models occupy had the STOP button changed so that nobody can stop them, so nobody went round doing that.

So now I wait to see if it all made any difference.

Maybe I shouldn 't say this yet, but ....

I've been logged in a few hours now and I haven't seen any of the lengthy high Script Times. I've seen high Script Times a couple of times but they immediately followed enormous Net Times, and I put them down to the scripts catching up with things like events and timers. They were very short-lived too.

It's beginning to look like recompiling everything back to LSL has fixed that problem. It still hasn't caused any Spare Time to materialise but at least it's so far so good with the worst problem.

ETA:
Reminder - by high script times, I mean a *lot* higher than the normal 22ms Frame Time - high 20s, 30s and more ms.

No matter how much processing power they put on a server, the users will load it down with scripts until it's intolerable, and then back off until it's tolerable.

If you keep scripts to under 10 msec, things work fine.

The biggest problem that *might* have a solution is that scripts with no events still take 2 microseconds per script. Most scripts are dormant most of the time. So, most of the time the server spends on scripts, it's doing nothing. Seems like there might be a better way.

Admittedly, if the script has a listen posted, it's not nearly this simple.

I don't have any choice about keeping the script times down, Lear. I have so many scripts in the stuff I have for sale that, short of removing a load of stuff, there's nothing I can do about it.

I did have a thought as to why the poseballs disappeared. They self-destruct if they don't receive a message from their host (bed or whatever) every minute. The very high, prolonged, Script Times probably meant that some script or other was going a bit haywire and, even though scripts were using such a lot of time, most of them probably weren't getting a look in for a minute or two, so the MLP systems couldn't send the "stay alive" messages and all the poseballs self-destructed.

Update:
I still haven't seen any of those prolonged high script times. I'm only seeing short-lived ones, following high and lengthy Net Times.

If you hit the memory thrashing problem, I'm pretty sure you should see serious time dilation.

Jesse, some of the things you're saying about Mono and the script memory limits don't jive with my understanding, but I haven't followed all the conversations. Can you point to a Linden saying that Mono's use of more than 64K is a serious contributor to this problem?

That would only be if Mono scripts were run in multiple threads per CPU, which doesn't make sense to me. So, at most, we'd get just 64K per region or 256K extra memory use. My understanding is that they're limiting this extra usage as part of the overall script memory limiting issue, and not that it was the *cause* of the script memory problem.

However, there may be a bug with memory usage by Mono. If Phil is correct that recompiling to Mono caused his sim problems, and recompiling back to LSL fixes it, then there probably is a bug.


PHIL: when you recompiled scripts, did you do it independently in each rezzed object? If so, you're making a big mistake. If you have 100 objects all using script "x", then you need to recompile "x" in one object, take it into memory, and copy it to all other objects using it. This way, they'll all share the same code segment, saving lots of CPU memory.

If you simply use "Tools -> Recompile scripts in selection" for every rezzed object, then every instance of "x" is unique and can't be shared. Mono code takes more space than LSL code, so you'd be using more memory, not less. True that the code would run faster, but unfortunately does NOT reduce the 2 microseconds per script that happens when the script is doing nothing.

Note that 5000 scripts times 2 microseconds per inactive script is 10 milliseconds. If all 5000 scripts are really doing nothing, you're humming along on the edge. If a reasonable portion of those scripts are actually doing something, you're pushing it.

I recompiled in selection, Lear.

I still haven't seen any more of the lengthy very high Script Times so now I'm very hopeful that compiling back to LSL cured whatever was causing it. But I'm still left with scripts using all the time they can get, and never having any Spare Time, so I'm up for anything that might help that situation.

What you said about the difference between copying a recompiled script into all the objects that use it, etc. sounds good to me. It's something I'd never heard of but I'll spend a lot of time doing it to see if I can make any headway into the 'normal' script time that this sim uses - maybe even spot a fleeting moment of Spare Time

Ty for that info, Lear.

Does the sharing of script memory apply only to Mono?

Rolling restarts do not cause the sims to change hosts, but a manual forced restart (as well an a sim crash) will cause sims to change hosts.

I've been tracking regions vs hostnames for over 2 years for my estate and this has always been the case. Scheduled maintenance != new host, manual restarts or crashes = new host.

This has been true for all kinds of region that I have monitored; full prim, original openspace (aka hometead), and new openspace.

-Atashi

Rolling restarts do not usually cause a sim to change hosts, *unless* the server software has to be updated... as in the recent upgrade to 64-bit Linux.

Ah good point. That wasn't a normal rolling restart though, that was a rather unusual situation.

Cheers!

-Atashi

Right, it only applies to Mono. When you set back to LSL, there's no difference between the two ways.

I suggest you take the most common script (or set of scripts) in all your products, recompile just that to Mono (once!) and redistribute it, and then see if you can measure any difference in behavior.

What a PITA, in any case.

Thanks, Lear. I'm going to do it smallish chunks anyway, and wait a while between each chunk, to see if there's any recognisable negative effect, or even when the negative starts to show, if it does at all. It really does take hours to do the lot and the last thing I want is to do it all, find that the problem is back, and have to do it all back to LSL again.

For the information, I've still only seen the super-high Script Times when they've followed super-high Net Times. I do see occasional super-high Image Times and they may give rise to super-high Script Times too - I'm not sure. But they don't really matter because they are so short-lived.

I'm declaring that reverting back to LSL fixed the lengthy high script times problem. I haven't seen a single instance of it today, and I've been watching for it for many hours. The lack of occurances is something that just didn't happen while everything was Mono.

Very interesting, Phil, and not what I would have predicted (so obviously I'm not in command of all the facts). Perhaps it would be best to follow the age old dictum: if it ain't broke, don't fix it!

Thanks for the data point, though.

That thought did occur to me, Lear, but I'm back at the start with Script Time using every millisecond it can get and not even a fleeting glimpse of Spare Time. I'm back to where I would again invite the person over to see that the bots don't cause lag - except that the traffic bots aren't there any more - but the sim is still on the edge, so I'm going to go back to Mono in the way you suggested, and chunk by chunk, to see if it improves the current stats.

What a PITA, though, that the effect of everything you're doing scientifically on this sim could be swamped by thrashing because of something happening in another sim on the same server.

That's assuming that this (recent?) weirdness of hugely inflated script times can be triggered by that thrashing; it's certainly plausible, but I don't know the details of how script time is measured.

(Oh, Atashi and Argent: Thanks! )

It certainly is a PITA, Qie, but an interesting pain just the same.

For today, I've recompiled 38 MLP 1.2 systems (beds) back to Mono, but all using the same scripts from my inventory. I won't change any more today because I already saw something that is slightly concerning. I haven't seen the script time shoot by itself, but I've seen it shoot up following a huge Net Time spike but, unlike yesyerday when the script time caught up in about a second, today I've seen it catch up in 2 or 3 seconds - longer than yesterday. So I'll just watch today.

I'm obviously not staring at the stats bar all the time and I could really do with a stats logger. I might look into that.

Another script!

hehe. Not in this case I'm using the TestClient for it.

I watched the stats for a while in a TestClient, at 1 second intervals, so I could scroll up and see what happened when times shot up. I saw enough to make me decide not to continue changing back to Mono. There were no lengthy high times - the longest was about 20 seconds, the next was about 10 seconds, and the rest were just a very seconds and unimportant. I only saw one occasion when the scripts time shot up without apparent provocation, but at that time, I wasn't displaying the Agent Time and I later found that some serious agent time (5-10ms) will also cause things to back up like serious Net Time does.

I did see a couple of times when something caused the scripts to (presumably) back up. They went high when the cause was over, and they were coming down second by second, when they suddenly shot up very high - 56ms or so - without any apparent reason.

I'll go back to all LSL tomorrow and watch again.

This thread does seem to have gone a bit off-topic

fyi - /54/f7/316012/2.html#post2408887
.