Can something please be done about the old sims?

I'm not sure of who's in charge of the hardware.

But lately, I've heard of people *complaining* that the SimFPS in their sim has dropped from 6800 to 4000. One of my friends was QUITE alarmed that the SimFPS of the sim he just bought land in went all the way down to 900.


Meanwhile, for about the past year, we in Lusk have been dealing with Sim FPS levels somewhere between 35 to 60 on average.

We own 48km2, but we can't hold events on it. We can't have more than a handful of people there. Building is starting to become impossible.

I'm positive that hardware has something to do with this - Before the new series of sims I haven't even SEEN a Sim FPS as high as over 6000!

I understand that scripts and useage effects resources. But, Lindens, PLEASE: Our average Sim FPS is one 100th of what some of the new servers out there are doing!

As we own a good portion of the sim and control our server-side scripts and effects, I really have to conclude, as Lusk was one of the first sims, that we're just talking about outdated hardware.

Hardware that could probably handle the 1.0 population just fine, but 1.4, no way.

I wouldn't be surprised if we were looking at the difference between, say, a 1U P3-800 and the newer sims being 1U P4-3000's.

I *know* that Lusk has some high useage areas that are out of my control, but I do not think that even high useage could bring a sim from almost 7000 simFPS to *35*.

I also know this isn't info that you guys give out. But when these servers get old, do they just... get old, and the people who own the land, oh well, too bad?

Is "server grade" or SimFPS-when-empty going to be a new variable to the price of the land?

We've been suffering for almost a year here, guys -- we tried to talk to the administration about it back in 1.2, but nothing seemed to be done about it.

Please - Any Lindens, just visit Lusk when there are 5-6 people there. Sometimes on a good day we'll see 150 simFPS, but the average is easily 50.

I wouldn't be surprised if it was the slowest sim in all of SL: we're already getting that reputation.

We try to hold an event, and people IM me, "Agh, I don't want to go to Lusk! It's so slow there!"

Please, Lindens, I'm begging you guys to at least consider, or at the very least, give some feedback into hardware differences and whether or not Linden Lab plans to address them?

This might be what's causing some of the older sims to look like barren wastelands. For example, DaBoom is practically empty, but most of the empty land isn't even up for sale. It's just... empty. Like people left and never bothered to come back.

I know my home is routinely running at 70 FPS. Usually because there's a party going on, but to hear people complaining about drops to something below several THOUSAND... yeah, I've heard it too, and it irks me.

To top it off, Lusk crashed tonight, and still hasn't come back. Nothing special was going on, either; a grand total of two people were in it.

I do wish they'd look at replacing some of the original 8-10 servers... This is gettin' kinda nuts. Maybe they figured people wouldn't notice the performance differences -- or maybe they did?

I give it a few weeks until land barons start advertising sim FPS on land they sell.

And yeah, Moleculor. I've been through DaBoom and it was pretty horrible, too...

I'd only assume that at some point, they'll -have- to do some sort of upgrade, but I guess that's not 100% true.

I have zero techie knowledge, but I've always wondered why they don't cycle the newest computers into the core of old sims and move the old ones out to new fairly uninhabited sims.
It seems like you would want the best performance in the most established parts of the world where people regularly go, not in some hideous wasteland populated only by "For Sale" signs.

Some kind of upgrade schedule for older sims would be nice to see, it would be a shame for the sims at the heart of the world to die off.

As Michi says:
"Please, Lindens, I'm begging you guys to at least consider, or at the very least, give some feedback into hardware differences and whether or not Linden Lab plans to address them?"

The older sims tend to get a lot of traffic, too, since they're closer to Ahern. (Which obviously -has- been upgraded.)

I'd figure that LL probably wouldn't want people to walk head first into broken sims as soon as they began exploring out of Welcome/Ahern, either.

But, the logic behind this is anyone's guess. We don't even know if SL is in 'bean counting' mode. I do know, however, that many of the single sim auctions which are won for US$ could pay for three new fast 1U rack servers.

But then we don't really even know their topology. For all we know, all of the new (post 1.2) sims may have their own dedicated 1U P4 3000, and Lusk, DaBoom, and Perry may all share one 1U P3 733 running VMWare, and the very fact that migration to three individual servers being a hassle may be the reason they aren't doing it.

Come to think of it, it kinda seems that way. Those three sims, and maybe a few surrounding it, seem to all share in the SimFPS pain.

Just a theory, but it'd be nice if we heard from someone from THE LAB about this in this thread.

But if history is any indicator, we won't.

IIRC LL once stated that they ran 1GHz P3s. I dont know about the newer sims though.

hm. Chances are slim that the new ones are 1GHZ P3's. I don't think you can even get P3's anymore in much of anything besides laptops. In 2002, sure. Now? neh.

Performance wise, old=1ghz p3 : new=~2-3ghz p4 looks about right :\

We are always working to improve performance.

I stopped by Lusk today and while there were only two of us in there, the SimFPS was closer to 200-250. Most of that is in RunTasks (scripts), some was Physics and some was RunAgents.

For Lusk specifically:
I noticed that you have at least 3 objects that are "Running Globugs". Whenever you start a particle system it sends an object update to everyone nearby. The particle system is designed around the idea that you can start it and let it run, it would download the parameters and run as a clientside effect causing little to no server side lag or bandwith lag. However the script for your glowbugs is restarting the particle system 10 times a second. This means a near constant stream of object updates sent to everyone nearby. If you could reduce these to maybe 1 per second or 1 per 10 seconds, it might help. To see what I'm talking about you can turn on Object Updates from the debug menu. Blue cones are partial updates, red cones are full updates being sent to you client. This tool can be helpful in finding some things that cause lag (though not everything).

More generally:
SimFPS in the 4000+ range is generally from an empty or near empty sim. The SimFPS is just the equation 1 / (sum of all the stats with an ms on it from the alt-1 menu). So with one agent (avatar) in an empty sim maybe the sim is spending 0.1ms handling that avatar and doing nothing else which would give it a simFPS of 10000 (1 / .0001). Add in 3ms of running scripts and you are suddenly at 322 simFPS ( 1 / .0031) with only 1 agent in the sim. Most older well established sims have a lot more content, more scripts, more things happening, and that causes simFPS to drop.

I am sure this doesn't answer all your questions, but maybe it will help.

Kelly, it ends up being a case of 'it doesn't break when the mechanic is around' -- Lusk just crashed, -again-, and now we're at 112.

By the way, this is the 3rd time Lusk has crashed in 24 hours. We aren't doing anything but building.

However it does seem to crash, without fail, whenever a vehicle goes up a slope in the sim. And not the same vehicle, any vehicle.

Eltee says that she did speak to you about the GlowBugs, said that that possibility was eliminated?

We've been seeing Sim FPS values in the double digits now for about 6 months, long, long before the GlowBugs even existed.

Just not sure what to do at this point.

Are you saying however that -- Lusk's hardware is equal to that of the newer sims and -- server speed is not a large determinant of SimFPS?


I agree that I may be misinformed on the mechanics of this; I after all did not write the software or plan out the architecture.

heh well they are more of an aside to the issue as ya saw kelly (turnin them off didn really do anything to help)

my biggest concern would be how dramatically *poorly* lusk seems to 'scale' compared to even an adjacent sim, say perry.

and due to the inverse fraction of simfps i won't even bring that into argument, i'm simply going to talk the number of miliseconds the server spends on a timeslice.

Lusk sees to spend on average about 0.3ms per 'agent' aka if there are 10 agents in/near lusk, the run agents timeslice on the server is about 3ms

in a neighboring sim, perry, the same 10 total agents result in a server timeslie of 0.9ms... leading me to believe perry spends 0.09ms per agent

this would seem to be a rather large anomaly...

im not saying that lusk is necessarily on older hardware but its certianly struggling alot harder to do anything than most of the other sims in the game seem to be

*edit*
(as an addendum i simply deleted the globugs, lusk was still sub 100 fps the entire evening, often dipping into the 40's and skating a bit of time dilation.. with only 5-6 people in it)

(edited because im too tired to craft a technical reply, and failed to do so)

You are quite right, it was an aside, I should have made that clearer. I only wrote about it to put out some of what I know with a concrete example.

The information that is most interesting to me is the type Eltee has given - a specific ms cost per avatar or object or script that differs from sim to sim. I find it odd that such a discrepency would be between perry and lusk. Perhaps the number of child agents, the cost of being next door to the welcome area, has an effect? Or it is possible that something in Lusk is not scaling well with the number of avatars which could possibly be a bug.

My inclination at this point is still content. I will take another look though.

well thanks for lookin at it kelly... im includin child agents in my estimates though (perry actually tends to have more children, with a large shopping mall just past its southern border)

lusk just never seemed to 'scale' well... things that have a concrete say 1ms penalty in another sim seem to have more here.. its hard to plot specifically with so much going on in any given sim but i've made a basic 'lag test' object which has 100 color changers that jus basically run wild (no don't worry i'd NEVER actually leave it out, its jus for testing)... they bring lusk's run tasks from 2.7ms to 11.2 ms.. so one single (linked) object has a roughly 8.5 ms penalty in lusk

in perry the SAME object raises the sim from 1.0ms run taks to 2.7 ms run tasks... or essentially a 1.7ms penalty

i mean *SOMETHING* is not right with that... lusk is 5x more lag induced from the same object than perry?

ooh one other thing i see in lusk that i don't in perry is once one thing starts slown down (like run tasks) all the other 'slices' go up as well in relative proportion.. a 0.2ms slice (say retransmit) is suddenly 1ms and a 0.1 is suddenly 0.5

almost like once somethin starts running, everything else suffers, not even fps wise but jus raw execution time wise, like somethin was overheatin mebbe or clockin down.

I always considered that, that we were getting 'spillover' from Welcome, but Welcome seems to be far away enough that the sim doesn't inherit its children. I think the maximum number of child agents I've seen is 7.

Heh...

want to talk about Sim Slowdown....

I was at one of the more popular clubs last night. I hope they own the whole sim, because they brought it to it's knees!. The sim FPS was down to 3. There were 50 people in the club and when I looked at the map, ALL the green dots in the sim were inside that one building.

And the packet loss... well that little bar in the upper-corner just sat there in the red.

it's my impression that the servers run multiple sims on one machine. Is this load-balanced, so that the busier sims are on servers with fewer sims? Does this make a difference? I'm curious about the whole time-sharing architechture of the system. It would be interesting to hear more about how the system is designed to prevent one sim hogging the resources of one server while another sim suffers.

If that's the case, it makes me wonder what other sims are being hosted on the server that runs Lusk. Perhaps someone else is using up the server's bandwidth?

Each sim has its own computer to run off of. In *theory*, the sim hardware should be the same for every one, from Lusk to the new island sims.

MAybe the older sims need to be taken offline for a day, defragged/formatted/wiped clean, and then restored.

I dunno.

LF

Mocha's been sitting at 5-7fps with just me in it today. I've turned off all my stuff in Mocha.

Attached is a screen-cap of the stats screen. Can anyone tell me what might be the problem? I'm not very sure how to read the stats.

hmm looks like physics and runtasks are both high.. if anythings scripted an physical, or jus stuff bouncing around in general, mebbe impacted vehicles etc that kinda thing... you can turn on the object crosshairs and look around for green's (physics) or oranges (scripted physics)

Thanks Eltee,

Can you tell me what the numbers represent? What is "run tasks"?

I'll check object crosshairs when I get back in world, but I floated around Mocha looking for any obvious culprit, and I couldn't find one.

basically each element is a slice of tasks the server has to run in order to give the clients an update... runtasks is the amount of time spent on script execution, runagents is the amount of time spent updating clients, physics is the amount of time in the physics engine etc.


i'm in mocha now and i'll give a quick breakdown of some basic things.

alot of this was spotted using the object beacons option in the view menu, quick key of that is here:

red: inert object
orange: Object with 'active' scripting (could be more efficient)
yellow: Physics collision (these are trouble)
green: Physical non scripted object (usually inert but lots of them together can cause jitter and deep thinks)

ShowUpdates in the debug menu can be useful too as it will show ya items that are re-synching with the client, something that can be costly, performance wise by giving them a translucent colored 'puff' (either red or blue)

a few puffs here and there aren't bad, something thats STREAMING puffs continuously could probably be improved upon

overall the sim didn't seem too laggy for me.. it was in the 2-300's which is prtty darn good.. i did notice alot of child agents (people logged in 'this' side of an adjacent sim) which if it gets too high can cause problems:

___________________________________________
crashed motorcycle on linden land (3,17
newbie dice (7, 201)
buried stuff *JUST* outside mocha's north edge, at least one is impacting (crimson 27,0)
towers of hanoi (physics and running scripts) (24,244)
Stray Physics Pig (123,193)
Strange over-scripted fountain (9,82)
nother crashed bike, jitterin (9,30)
big physics block (8,23)
Somethin Strange buried under southwest corner (could be a ghost)
The teleport system (not particularly bad, but lotta scripts)
Crashed Jetski (54,93)
somethin buried under ampitheater walkway (193,12
Crashed Flyin Rug (110, 172)

I also saw a bunch of yer leaf rez trees out there too francis, tho they were inactive, an earlier version of those absolutely murdered lusk's performance. (each one alone took about 5ms runtasks time, two of them runnin at once brought lusk from 300 fps at the time down to the 20's, with lots of time dilation)

It looks liek the new ones don have sheets of leaves out, and are prolly much lower density, which should help ease their burden, but ressin lotsa physics stuff in general is pretty rough on servers.

Also jus the general script density of mocha is pretty high, which seems to be increasingly more common in older sims, and its simply noy scaling well. while you might expect 500 scripts to take say 10 times as much time to execute as 50 scripts.. in reality it can be as much as 30-40x as much time to sort through them all... since time spent switching between processing each one is time the server isn't workin on anything else (think opening 500 copies of ms paint... it will jus get slow after awhile)

this is more or less whats happening in lusk.. where one build alone has over 600 active scripts and is bogging the whole sim down pretty badly.

If you haven't payed a land barron for land in a new sim, STFU! Only the new is important. Muhahahahahah!

When we were in Game Dev 1, Xylor worked with our neighbor (Dave Zeeman IIRC) to do some benchmarking and discovered that scripted objects don't get a fixed timeslice. They get less and less as there are more of them in a simulator, with the distribution looking like a curve. This is probably a self-defense mechnaism for keeping scripted objects from killing a sim. So, a script might be running at an average of 100 ops/sec in one sim, but 80 ops/sec in another, with the determinant being mainly how many other scripts were running in each sim.

Anyhow, what has been reported here makes me quirk an eyebrow. It doesn't seem normal.

BTW, what exactly is a child agent?

You're in SimA. They're in SimB, looking into (and seeing data from) SimA. They're a child agent.