The SL ervers are not up to the job

A sim restart can only do an individual sim and not the whole box. It isn't possible to find out which other sims share the box from inside SL except by going round all the sims and looking at their numbers, and that's rather a daunting task

The point of this thread is that those things shouldn't matter. If one of the other sims that share the box is causing it, then it's what I said - the sim server system is not up to the task. It cannot function smoothly if what is available is actually used. A sim can accommodate 40 avs, but how would it cope with 40 avs - even if no scripts were running in the sim. What if all 4 sims in the box had 40 avs in them with no scripts in the sims? The sim server system is simply not up to performing what it's sold to perform.

If the problem persists and the Lindens cannot find an obvious reason why, you should request they try moving your sim to another server. Your sim will be down for about 10-20 minutes while the move is completed. There is no guarentee you will end up on a less stressed server though, if that is even the issue.

**Edit** Also Phil I wanted to add that 5000 scripts is a lot. Yes a few bad scripts can put more stress on the sim than 1000 good scripts, but having had a sim for awhile I have noticed once you start hitting 3000 scripts things can start to go south pretty fast, especially when scripted avatars are thrown into the mix. Good luck. I hope you get things running smoothly.

I see what you mean.

Physics is down to normal now but, coincidentally, while we've been writing these posts the script time has gone haywire several times - up to 36ms, and staying high long enough for me to open the Support page to get live chat onto it, but it didn't stay long enough for me to open a live chat. But that's not common - they could have been down to an avatar coming in a few times with goodness knows what script running. The problems I see occur when the script time is normal for this sim.

I could do that.

;0 well okay, kinda 'xactly; i was wondering if that direct-cion had been investigated at all yet... i guess though i think the quest-sion still stands as to whether it's software or hardware/firmware. without knowing what the performance is on the other three sims, it's maybe a tossup as to what's going on. at least that's the way it seems to me.

Last fall I was in the orchestra for a performance of the Fontana Philharmonic. The stage was in one sim and the audience was in an adjoining sim. We had 40 on the stage and 80 in the audience. Everyone was told to remove unneeded attachments, but we were playing instruments that responded to remote chat commands that animated us, sent by a director HUD. Audio was streamed. And everyone in the orchestra wore a HUD that told us what was going on.

The clients were more unstable than the servers. And right at the end we lost comm between the two sims so suddenly one dissapeared from view from the other, but they did not crash. I wish I had thought to look at the performance numbers. But all the avs were sitting down in both sims, so that probably helped.

Apologies if this was already mentioned. LL is working and testing in Aditi right now because in some circumstances scripts can exceed 64K. Enough scripts do it in a sim and it trashes all four simulators.

The easiest test is going to be sending a message to Vektor or maybe Milo and have them put your simulator in Aditi. If it works fine there then you know it is an unfriendly server neighbor.

I have been saying this for a long time. And been complaining to LL about the quality of service lately.
4 Regions on 1 Host . All fighting over 2 gigs of ram. They guarantee you one CPU.. big deal.. its worthless if you don't have the memory to float the processes. They want everyone to only run at 40 users. Which is a crock of crap. It's only because they want to get as much money out of one piece of hardware. Collecting 900 a month just in tier fees off of one host..
I think it got worse lately because they lost 1000nds of Open Space regions. They had a lot of open space hosts that would support 1 busy region and the open spaces have to deal with the lag.
But since they lost all those hosts (They had to consolidate). They don't have a spot to put busy regions on as much as they did before.
When you restart your region it will hop to another host. Well a lot of times now days, the host we hop to wont even hold 20 users. Resulting in Several restarts just to find a decent host the server will perform well on.
Lately has been a massive pain in the ass and the Hosts do not perform equal.
I remember when people had open space sims, they used to complain a lot when they were renting them out as homes.. The ones that were massively laggy were also hosting a busy region on it.. Which sucked all the resources from the open space regions.
Thats the way i see it..
Since they had to consolidate the hosts, it has been terribly bad. I couldn't figure out sometimes why my region would just lock up and stop responding.. My guess is another busy region hit restart and landed on the same host as us.

Really annoying.

Wish they would allow open sim on the grid so i can run my own host :/

:0 lol! yah then i got to wondering afterthought, if the random lsl/mono overhead was at all involved, but lol thought too simple to suggest. i've been dealin' with 'teh random' the past couple days, crowbarring multi-lingualism into my scripts... one day, will test out and work just fine in the field; the next - stack heap lockups. some quick re-engineering and things fine now...

so yah, maybe it's that.

aditi servers are on a different network than Mainland and private sims, of course Linden operative sims are also on that network.

What exactly are the technical specs for the sim servers? Most hosting companies have that pasted everywhere you look, including detailed information of the OS and shared server information?

Actually it is much worse then just a random problem. Due to the bug(actually a design flaw that was introduced with MONO), if you exceed the memory limits the server starts memory swapping between the 4 cores. This is a very bad thing to happen.

Here's a recent article by Gwen LLewelyn on the back-end system:

http://gwynethllewelyn.net/2009/04/01/new-class-6-servers-are-out/

I didn't know we have Class 6 servers.

So much to keep up with.

We'll be running on Macs!

Well...

I removed all the bots and stopped their systems from running for several hours, with no noticeable change. When I was on my own in the sim, the script time was ~20ms and the sim frame time ~22.4ms. There was never a moment when there was any Spare Time in the sim.

There are ~4800 active scripts in the sim (not a huge number as sims go) which never get enough time to run; i.e. they always fill the time available and their time is always in a state of being throttled to allow for other things.

There's a related thread in Scripting Tips from a week ago or so, about a sim running 0 spare time and intermittently huge frame overruns by script time. (The prevailing wisdom there seemed to be that the likely cause was script memory management b0rked by Mono, probably on a different sim sharing the same server.)

One open question I had in that thread is whether a sim restart will cause a regular, full-primmed sim to find a new host assignment, as happens for OpenSpaces and Homesteads. If so, one might expect that any interaction with server-sharing sims would disappear during rolling restarts (for example).

it all the 1.200,000 people who have join second life is what doing it.lol

One thing I didn't mention today, although I mentioned it yesterday, is the periods of time that the sim has a Total Frame Time of over 30ms. The Script Time accounts for almost all of it. During the hours that I had all the bots out, it was easy to see that those periods seemed to occur when someone arrived in the sim - in the store - and they lasted ~20-30 seconds or so. They didn't occur for most arrivals in the sim but they did for a few of them. It could have been coincidence, of course.

A few minutes ago, I watched another of them that lasted about a minute and contained a very brief peak where the Frame Time was over 200ms, with the Script Time just under it - I saw that sort of peak yesterday. I didn't notice if the period coincided with someone arriving in the sim or not, but if they occur when someone arrives (occasionally), what sort of scripts could they bring with them that would cause that?

I read the thread - ty, Qie. In my case, I haven't seen any corelation between high Script Times and high Net Times. A few days ago, a Linden came out but couldn't see any heavy scripts. He restarted the sim but it made no difference. I don't know if it came up on a different server or not, but the stats looked just the same afterwards. During those few hours this morning, Frontier Linden came out. He spotted some temp rezzed prims and gave me a temp prim scanner. It turned out that I had a few temp rezzers operating, from a long time ago, that I'd forgotten about. I cleared those and hoped the high script times might disappear, but they didn't.

As I've been writing this post, I watched another occurence. This time it peaked at only 46ms - only! lol I was writing and didn't notice if someone arrived in the sim, but when I saw it, I could see there was no corelation with the Net Time - only with the Script Time. Net Time does give occasional spikes but not together with the Script Time. Physics Time seems to have stayed below 1.0 all the time today.

If LL is using commercial load balancers, I'd expect a restarted single sim to end up on the same CPU, and a rolling restart to shuffle the deck, so to speak. Phil, did your sim behave in this way both before and after the recent rolling restart?

I don't think 5000 scripts is too many,but I do think the script performance numbers you quote are abysmal, not at all in line with what I see. I spend time on a sim running 16,000 scripts, and it runs fine. (Of course, many of those are mine and thus amazingly well-behaved )

Since LL can't find the problem, I'd request a move to a different server, which they can do. It would rule out the 'borked server' and 'bad mono neighbor' possibilities, with less cost to LL than spending yet more time trying to debug your sim.
.

High-impact scripts would include something like wearing multiple avatar scanners that are set to scan very often (say, a combat system plus an AV radar plus a mystitool, all set to scan 5 times a second or something).

Edit: or even worse, object scanners, which people who do a lot of hunts tend to wear.

Other culprits might include colorchange hair, shoes, and jewelry that for some reason listens on channel 0 instead of putting up a dialog on touch. I know, for example, one prominent jewelry maker who controls color/texture/glow etc. by chat. Obviously he does not release the stuff listening on 0, but you CAN change the channel.
.

(btw, you don't happen to have a weather system on this sim, do you?)
.

Patience, grasshopper. They just upgraded to 64-bit OS, so NOW they are in a position to add more than 2GB of RAM. Upgrading the physical servers to a "class 6" with more than 2GB per server was not even theoretically possible until that was done.

LL is not using commercial load balancers. Load balancers work well for HTTP because most web transactions are effectively stateless and most of the state that DOES exist can be maintained i nthe client via cookies. SL is extremely stateful, with requirements for a high level of permanent concurrent state and real-time updates. There's no place for commercial load balancers... all they would do would be to add latency.

Hi Nika. To save quoting all your posts, I'll write this seperately.

There are no weather systems in the sim at all. I don't own all of it, but I'd know if there were such things there.

The sim had high script times before the last rolling restart, which is why I got a Linden out not many days ago. I was able to work ok though. I only noticed the huge jumps in Frame Times/Script Times yesterday, which was after the last rolling restart. There have always been times when menus took time to arrive, and those high script times may have been there a long time - they probably were. I never examined the reason for the menu delays before.

For some time (weeks or more) I've been sliding instead of walking for short periods (seconds), and been unable to move at all for short periods (seconds), and rubber banding a bit. That sort of deterioration may have started since a recent rolling restart. It didn't used to be like that. It may all be the same problem, but it was only yesterday that I watched the stats bar especially for the reason why menus were delayed so long.

The whole thing is definitely worse than it was not too long ago.

I don't think it's been 2 months since i suggested to someone in a thread that she should come to the sim and see that there's no lag in it due to my bots. I couldn't make such a suggestion now even though there is still no bot-related lag in it.

True! I misspoke - more of a developer than a data center type myself

I was thinking more of a virtualizer that would only be in play during restarts. Obviously, transactions have to be bound to servers.
.

That would be sweet. Lets just hope we don't see another $600 price increase lol

I just remembered something that might point towards the "memory thrashing"?) that was mentioned. I only got around to converting the vast majority of the scripts in the sim to mono about a week ago. These lengthy total frame time highs continue unabated. (Right now I'm watching one in the 30+ms ... in the 40s now ..... finished. That one must have lasted a couple of minutes).

Tomorrow I'll go round and recompile the lot in LSL and see if that stops the highs.