The SL ervers are not up to the job

Thank you, Nika. I hadn't seen the latest on that.

The LL response does indicate the truth of what this thread is about - that the sim servers are not up to the job:-

The content is stated as being the main factor in that sim's problems. The 15,000 prims that are being paid for is too much to use when some of them contain scripts, and there isn't a huge number of scripts in that sim. Actually using what was sold by LL brings the sim to its knees.

The bit about Mono is very interesting:-

... especially as the sim has been running smoothly since the update.

I'll go back to total LSL until the the major updates are on the grid as it seems pointless continuing to put the time into it now.

I don't think it justifies that conclusion at all.

My conclusion is that there's a bug and it's not working correctly. The fact that your region works fine with LSL but not Mono indicates this is likely to be the case.

There's no amount of server power that's not inexhaustible. Regardless of how much power the servers have, it's possible to load them down with scripts and bog the hell out of them, because scripts aren't (currently) limited the way prims are. THAT is likely to change, and no doubt will cause quite a ruckus, but it will invalidate your point. You might not be able to put as many scripted items in your store, though.

This begs the question "what's a reasonable amount of scripts?" Well, if what you had was running fine on LSL, then it's probably reasonable.

No, scripts are MORE limited. There are scheduling problems in Mono that make this less effective, so you will get more predictable behavior from LSO. For example, if a Mono script sits in a tight loop for a long time, its scheduling quantum increases to allow more efficient use of system calls, so that if you follow that up with something like a lot of large list operations you can actually lock the sim (not just the scripting engine) up for multiple frames. I think this is a design flaw... a script that repeatedly exceeds its quantum (whether measured in microseconds or loop counts) SHOULD be lowered in priority at some point rather than having its quantum extended, even if that's technically less efficient.

But Babbage has decided that it's worthwhile to allow a single script to monopolize the resources of a sim so he can run his cool mandelbrot demos, even though having more than a few scripts like THAT in a sim will kill scripting for everyone else.

I agree that server power isn't inexhaustable, and that a Mono bug was the cause of the problem in the other thread, but the Linden said that halving the content in the sim would improve the performance significantly.

Sims are sold to support 15,000 prims and to support scripts which, of course, can't be of an unlimited number. I have over a third of this sim's prims unused and others in the sim are likely to have some unused ones too. There is probably at least 40% of the sim's available prims left unused. So many objects contains scripts these days. Most of what I sell contains scripts. The sim is at or beyond the limit of smooth running, even when there are very few avatars in it, and yet so much of its allocation is unused. Put a fair number of active avs in it, and it will be on its knees. When they bring in the script limitations, it won't be possible to run some business in a whole sim.

It would be fair to say that LL allows so many of this, so many of that and so many of the other, but if you use them all, then the sim will collapse, so it's up to you to mix what's available in a way that suits your needs and that doesn't drive the sim to collapse. But they don't say that. They say you can have up to 40 avs in mainland sim - but you can't - not even if you use none of the other things - not if you want the sim to run smoothly. Sims are fine as long as people don't use what they are sold. I still say that LL crams too much into each box, causing the inability of the box to smoothly run what's asked of it.

I learned something at Andrew's/Simon's office hour yesterday while trying to get some background on this problem. Turns out that the sim "script time" statistic is not just the CPU time that scripts use, but the wall-clock time between when scripts start executing in the frame and when they're stopped... so if they get hung-up waiting for some resource, the script time--and the frame duration--can extend arbitrarily. One such resource can be memory: if the server is thrashing, extended script time is one of the symptoms.

(Probably everyone else had already come to this conclusion, but I was still harboring the illusion that "script time" would be the slice of CPU time dedicated to processing scripts.)

I should have said "No limits on scripts are enforced, as they are with prims."

As I said, the problem isn't that the servers aren't powerful enough. The problem is that it's easy to overwhelm them in any number of ways that SL lets us do, and that there's evidently a problem with Mono (maybe more than one).

Once we find out what kinds of script memory limits must be enforced, we might decide that the servers are too wimpy for the number of scripts we want to run. But that's another discussion entirely.

OK, what's the limit?

There's no enforced limit other than if objects have a limit on inventory size (which may or may not be). There is a practical limit. LL says that it's a good idea to keep a sim under 10 milliseconds of background script time. Since every script takes at least 2 microseconds of script time even if it has nothing to do, that gives us an upper bound of 5000 scripts -- but only if those scripts are doing nothing.

If your point is "Gee, I wish SL servers could support more scripts," then I totally agree.

If your point is "LL misrepresents SL by providing a platform that's useless for the intended purpose," I disagree.

Perhaps WE cram too much into a sim, causing the problem.

Admittedly, SL needs to have limits on scripts to avoid scripts killing the sim, and they're working on that. Just wait until that happens, what an uproar it'll cause, because we won't be able to run super huge and laggy scripts in our sims and on our avatars!

Already we're coming to face the incredible mistake that the resize scripts used in many avatar prim products (where there's one script per prim, and whose only purpose is a false sense of security). I see plenty of avatars with over 1 ms of script time per frame just for the avatar, for scripts that are no longer serving any useful purpose.

So, if LL has made a mistake here, it's not realizing earlier that scripts are too easy to abuse and need to be explicitly limited.

There isn't a fixed limit in terms of numbers. The limit is that a system cannot smoothly run an infinite number of scripts, so it can't be unlimited.

I don't consider my useage of scripts in the sim to be over the top. I'm simply running a business in the sim without going over the top on anything and, even with everything back in LSL, the sim is pushed to the edge of smooth running. Sims are just fine if they don't have much in them, but LL has set them up to accommodate quite a lot and they aren't capable of it if they do accommodate quite a lot.

I do agree that there will be some raised voices when the script limits come out. It could be that I won't be able to have all the items in my store in the same sim, or add to them, in which case mine will be one of the voices.

Anyway, back to the Mono/LSL thing.

I have everything back in LSL now and I'm running the stats. It hasn't been running long but, so far, I haven't seen any "big" frames. It's also not the busiest time of day. Tonight will be better.

Smoothly? Subjective.

Everyday I am in a mainland sim with 40 people and most of the time there are no problems moving. Sometimes there is extreme 'rubber banding', and lag, but the sim almost always has 40 people, sometimes 41!

A sim with 40 people is still usuable.

Our wedding had 64 visitors and we didn't crash.

Its deja-vu all over again! Where have I heard this before - LL is marketing and selling (or leasing with a big upfront buy-in) a product that is maybe not up to the task of supporting all the content that they say it is. If a full-prim sim cannot comfortably support 15,000 prims along with their associated textures, 'average' amount of scripting, plus 40 avatars (and all their prims, textures, and scripts) then what?

Seems like the residents are abusing resources, so LL will have to raise the rates again.

/bitter

I think the problem is that by using prim count as the way they limit server resources -- without explicitly mentioning other resources such as script load -- they have painted themselves into a corner. People know about the prim limit, but without any official, obvious and explicit guidelines about the rest of the resources, the assumption is that there aren't any limits. So people react accordingly and build things without any concern (or understanding) of the practical limitations. Although not really relavent to this thread, the most obvious example I can think of is temp-rezzers. Everyone knows of the prim limit, but with no appearant limit on scripts, what's wrong with using scripts to get around the prim limit? Temp-rez a 200+ prim house, and then complain about the laggy region.

Part of the problem at the moment is that LL themselves don't know what limits to set. They've said that they are analyzing things and gathering information so that they can implement script limits.

But in the meantime, I agree with Phil, in the sense that by 'selling' regions that are advertised to support 15,000 prims, and by not explicitly mentioning any other limitations, they are overselling their product. There is no caveat, instruction, or fine print when you buy a region where they mention that you need to keep the script time at or below X. They don't tell you that you want to keep the number of scripts at or below Y. In fact, there's really no guidance of any kind. Just the fact that it can support 15,000 prims -- and up to 100 avatars (they do suggest 40 is a better number... although I've found things can get laggy past 20...).

-Atashi

@Briana

I don't consider that not crashing is indicative of a smooth running sim

"Extreme rubber banding and lag" is indicative of a sim that isn't running smoothly.

With LSO you get a fixed maximum time (I think 20ms) for scripts per frame, no more. Mono doesn't enforce that reliably, but LSO certainly does.

Mandelbrot plots. LOL. What a geek.

That's interesting. I don't know how to interpret what I see, but it's interesting just the same

The biggest culprit for huge, extented frame times that I see is Net Time. Image Time and Agent Time put in occasional appearances but Net Time is commonly high for a second or three, causing the Frame Time to be extended.

It's a pity I didn't run a stats programme when the sim was having those lengthy high times that lasted one, or two minutes. I'd have liked to have *seen* what was causing them. But since they ended when I went back to all LSL, it's not unreasonable to think that they were something to do with Mono. With only a small part in Mono yesterday, I did see one or two things that were a bit concerning. Today I'm back to all LSL and, although I've seen some high times now - even 10 or more seconds - they have all been due to Net and Image times, with Net Times being the main one. When those two came together, I had a Frame Time of well over 200ms for a short time, and it was slow to get down to normal.

I think that, whatever I see today, I'm staying totally with LSL until that thrashing business is sorted out.

Well, damn. I was with you, since the way the measurement actually works is darned near useless.

Why am I not amazed?
*shakes head*
.

All frame times are based on wall-clock time so they represent simulated real time. Making them CPU time would be more computationally expensive and would also, in most cases, produce incorrect or misleading values. It's not just swapping: caching. disk I/O, or any other operation outside the simulator image will also be directly unaccounted for. Most of the time this will not be a problem, and when it is being able to accurately acocunt fro script *time* is the least of your problems.

If I were to go out and drink a bottle of vodka, and then drive my car at 100 mph on a busy highway, I think we would all agree that something bad would happen.

Now, GM sold me this car. This car is capable of going 100 mph. Furthermore, GM did nothing to prevent this car from being driven by someone who has consumed a bottle of vodka.

Clearly this is GM's fault, your honor, and so I plead not guilty to DWI and vehicular manslaughter.

/me snickers


Seriously. LL is in the process of putting limitations in place to prevent people from abusing script processing and/or memory resources. This should have been in place long ago. However, most (competent) scripters have been well aware of the limitations of what a sim can handle, scriptwise (the current Mono bugs notwithstanding). Loading a sim with scripts (especially if they are inefficiently coded) and then blaming LL for the resulting high script time (aside from the Mono bugs) is quite similar to blaming GM if I choose to drive my car at 100 mph.

Sure, they did not enforce strict limits. However, the limitations were quite well known. As programmers are wont to say (when customers aren't listening): "You can't fix errors that occur between the keyboard and the chair."


As for the issue of 4 regions per server, each server has four processor cores. Thus each (full) region is run on its own processor core. I am not sure that it would be a good idea to attempt to run a region on two separate processing cores. I suppose that you could try to set things up such that each server consisted of a single, dual-core processor, and thus run two regions per server. However, my (uneducated) guess is that this would simply result in a greater load being placed on the database and backbone infrastructure.

BTW, are we talking only about "script time" in control-shift-f1, or are we talking about total script time in estate tools?

The two are quite different. The former isn't very useful. It tells us how much of the recent frame cycle's CPU time was used for scripts.

Scripts are given a guaranteed minimum amount of time (IIRC, something on the order of 5 msec). If physics is taking lots of CPU time per frame, that's the most script time we'd see (barring the current Mono bug that allows it to exceed its share, and also barring thashing, which throws everything off). Seeing ctl-shift-1 script time numbers below this minimum means the sim has very few scripts actually running. This is rarely a problem. You'd have to have less than 2500 scripts present (but not doing anything), and very few of those actually doing anything.

Scrpts are limited to a supposedly guaranteed maximum time of 15 ms, which can also be violated by the Mono bug or by thrashing. Seeing ctl-shift-1 script time numbers much bigger than 15 is a clue that somenthing's going wrong.

Numbers between these two values simply means that physics time is taking priority, and there is enough script work to not finish it all in one frame. This isn't generally a problem, because we generally don't need scripts to respond in 1/45 of a second. No doubt there are cases where it is important, but those would need to be done on a very carefully maintained sim, limited script usage.

Estate tools total script time tells how much CPU time it takes to visit every running script and let it run (if it has something to do) until it yields the CPU (by waiting for an event, delaying, etc.) I believe that this doesn't mean it has nothing else to do: it might have more events waiting in its queue.

That's a very different bit of data, and much more helpful, than the control-shift-1 script time.

Does anyone know whether this number also includes CPU time spent thrashing? Most likely it is, because it's difficult to factor that out in general.

In any case, the control-shift-1 script time doesn't generally tell us much, except that when it goes much over 15, things are borked. Estate total script time is very helpful in figuring out how much script load a sim is carrying. (Unfortunately, as I mentioned above, it's always just a single sample, and you have to do a lot of sampling to get a very clear picture.)

Control-shift-1 Free time happens only if there's not enough total physics, script, and other work for the processor to do to fill a whole frame. By itself, lack of free time is not an indication of a problem. It would only be so if there were good reasons to expect there to be free time for a given sim (few avs, nothing moving, few scripts, etc.)

Being on mainland, I only have access to the ctrl-shift-1 stats. My memory is known to be quite fallible but I'd understood that scripts use what time is available to them if they need what's available, so going over 15ms wouldn't mean that things are borked. The sim I'm in uses 19-20ms for scripts, pretty much all the time. It is briefly pushed down from time to time by things like Net, Image and Agent time - mostly Net time - (Physics time is rarely anything but very low - 0.1 to 0.5). I wouldn't say that the sim is borked. It's in a state in which I would invite people to come to see that it's not laggy. (The only real lag for a person is in and around in the store, which is in the sky, and that's when the person arrives and has to download everything. The store is just above the clouds, so people with long draw distances would also find slowness on the ground while the store's stuff in being downloaded.)

I recalled the limit as being 15 ms, but my memory leaves much to be desired. It's probably 20, based on what you report.

Part of the problem with this stuff is that we have to glean what we learn from what LL says, and they're not always consistent.

Anyway, my understanding is that they are guaranteed a minimum and clipped to a maximum, per frame. If it's going much over the maximum, something's borked.

Assuming the 20ms that Argent thinks, the sim I'm in isn't in bad shape at all, except during the periods when Net, Image or Agent time hit it heavily. (All my stuff is LSO again - is that the LL compiler?). Most occurences are over in a few seconds but I still get the occassional time when those other "time"s hit it, sometimes opushing all other times down to 0, and then scripts sometimes show an enormous time for a few seconds before they get back to normal. But we now know that script time isn't the actual CPU time that scripts use, so those occurences say more about the Net, etc. times, and what happens then, than about scripts. I.e. in the seconds it takes for the script time to come down to normal, following a heavy Net time, for instance, scripts may not even have been doing anything.

I don't trust my 20ms figure, that number was pulled from a hat.

The 20 will be pretty close in actual use. What ends up happening is that if Total Frame Time breaks 1/45th of a second (22.2ms), the sim limits scripts to try to keep it at or near 22.2ms. That's what's going on when you see Time Dilation and Sim FPS drop down. If things get really bad, the sim can't throttle the scripts enough and you get Total Frame Times that shoot way past 22. On a lot of sims, there's isn't much going on for the server to actively do other than process scripts, so script times will often max out near 20.

Personally, I was speaking about ctrl-shift-f1. Agreed, not terribly useful.
.