Lusk is back down to 57 simFPS -- with *1* agent.

57 simFPS (Sometimes it gets excited and goes to 62)..

This is with 1 agent (me) and 1 child agent.

Since 1.5, Lusk has been running 400-500 simFPS with 8-10 agents... the fastest it has EVER, EVER run.

(Also noticed it went from sim3.agni to sim297.agni - which means most likely, newer hardware.)

We're still on sim297.agni, which means we didn't get knocked back to sim3, but right now i'm seeing nothing more than 57-58 simFPS with just me in it. The thing is, for the most part, unusable.

Physics: 1.2ms, Run tasks 11.5-14.5ms.

Time dilation constantly at ~0.95.

Strangely enough? Sim CPU, 0.50. In the past it's always been 1.00 ...


Now, I can assume that certain script users in the sim besides us saw the available cycles and filled them up with new and exciting scripts... but at this point, I dont know.

If Lusk becomes unusable, with 1 agent in it and even on the new hardware and after the memory leak/degradation fixes, is it pretty much just 'too bad' for us?

I'm getting SO SICK of people coming to Luskwood and saying, "Let's leave, Lusk sucks." or "Don't build here, Lusk is a weak sim." or "Yeah I'd like to hang out here but Lusk is notoriously laggy" ...

We're becoming the butt of everyone's jokes for this. We have the land, but we can't use it. We saw a brief reprieve of a few days, after 1.5, and now, we're at a WORSE condition than we've ever seen.

I noticed that Michael Linden put some new roads in - (i'm not sure why...) - but I can't see anything else that's changed, except that our neighbors have some new spinning device, I believe using a linden spin script. But it isn't physics, and I have no idea if it uses llSetRot or target omega.

Whatever, in any case, Lusk is, once again, dead. And our legacy lives on as a place that's really really nice! and "You guys did a great job with this place!" but... "Too bad you can't do anything here."

With the new SimFPS we were even about to have a 'Luskwood Grand Re-Opening' party in a few days. The first event we could actually hold IN Lusk because it looked like it was going to be able to handle it.

That... I guess, isn't gonna happen.

I looked all over for stray physics objects, big scripted objects, I couldn't find anything.

But whatever it is, is bringing it from 500 to 57. So, who knows.

It looks like the Lindens moved Lusk to newer hardware (the 0.50 sim CPU is most likely an indication of a hyperthreading P4 processor, i.e. newer hardware).

It also looks like the problem never was the hardware (or not to a large extent), but one or more people who are either clueless about or ruthless with their scripts and don't care much about their neighbours (or perhaps doesn't even have any land there).

No single script is causing this. It's probably either alot of small scripts doing something in parallel to accomplish it faster, or something very ineffective, like a linked object using chat to communicate with its link children. Or both.

In any case, the solution would be to find the offensive script and see what can be done about it. The problem is, SL offers no way to track such a script down, so unless you're lucky or it's very obvious and brightly blinking, you'll need Linden help. Probably they don't even have the tools to find it, because it isn't a single script causing this load, it's the sum of many small scripts, so a top(1) like view wouldn't help alot.

*whistles*

Ok, a couple things:

1. Bug report it. Forums are not a good place to report something like this, because A) Lindens don't always read these, and B) You just reported this on a weekend, and this thread will be buried by the time they get back.

2. Contact Live Help. I heard from a Linden a few hours before 1.5 went live that Lindens and Liasons 'n such now have access to in-world tools that can help them track down run-away tasks 'n such. I don't know if it's TRUE yet (or ever), but try anywho.

3. I doubt that the 0.50 CPU thing means it's multithreading. Back in 1.4.11 when Seacliff was hitting 0 SimFPS and the RunTasks were spiking to 280ms, the CPU would fluctuate up and down from 1.00 to 0.01. I don't know what CPU means, but the fact that it's at 0.5 could mean simply that something's wrong with the processor, or there's a memory leak or somesuch, it's most likely NOT a question of bad threading.

Well, Perry's CPU is also 0.5, and its SimFPS is 1667, so I have a feeling that the guess on 0.5 being hyperthreading (it's only seeing useage on one of the virtual processors) being correct, it probably went from a p3 to a p4.

I've bug reported it; I'll try LiveHelp but I kinda have a feeling that it'll be a little like calling tech support.

Might want to wait till some Lindens are on to try it.

I'm wondering about this 1.2 mS physics slice... I don't hear any collisions, but somethin'gs taking that... actually 1.4 mS now... I don't know if that'd bring things down that low, but I also figure i'd be able to find a physics object SOMEWHERE.

No dice though. Object beacons on, show updates on, highlight transparent on, surface patch off.

Not a thing. :\

Kinda sucks that there probably won't be a Linden around until Tuesday. :|

The Lindens have always maintained that the servers were P4s. We haven't gotten confirmation that the 0.50 CPU thing is an indicator of hyperthreading, but I think we can safely assume that all the sims are P4s. Why would they lie?

Well the admission that Lusk used to be on older hardware has already been done. That was never in question. Just what is in question, is what is killing it now.


Land scanners? (Perhaps on purpose? I know there are a lot of these in Lusk, and they run fairly intensively.)
New scripts?
Rotating new prims?
New memory leak in 1.5.1?

I don't know. But the hardware did change with 1.5. New DNS, now indicates hyperthreading, and Andrew did indicate that it'd be happening. So that's really not the issue.

What the issue is, is what's happened between 11:30p PST and 4:30a PST today to tank the sim, to the point where it won't go past 60 simFPS even intermittently.

*sigh* Be right there. I don't have the tools the Lindens do, but I do know how to take a look or two around.

You don't have to do that Moleculor.... Just want to document what's been done, here. I did bug report it, and hopefully I can refer the lindens to this thread...

Just wondering though, the pattern is strange... How difficult would this be to do undetectably and on purpose? (by an angry resident perhaps, or other motives...)

With the old hardware we were at 37-65 ... Now, with the new hardware, and a new server version, after a few days of smooth sailing at 400-500, we're suddenly back to 37-62 ... (it wasn't gradual. It happened some time in the last 5 hours.)

Generally, when I've seen Perry go down this low it's because someone I know rezzed a MASSIVE amount of scripted prims with listens... something that was rather identifiable.

This though, nothing has changed. The sudden-ness of it just kinda bothers me. And the fact that the hardware has been changed out, and now it's back to where it was.

In the past week when the fps was 400-500, we actually had quite a lot of scripted objects out and the worst we saw was ~280-310.

The amount of *work* it would take to get the sim down to 55, and at the same time, not being able to notice anything new ... and it happening so immediately, Idunno.

You'd think the Lindens would have a 'topcpu' style screen for scripts, showing which ones were taking the majority of the CPU time. But from what I've seen and the Lindens I've talked to, that doesn't exist.

Ok, a few things, some are merely observations and not actually related to the SimFPS lag you're describing.

1. The sim FPS may be 50ish, but I don't actually start noticing a slowdown in the way my client behaves until I get onto the Luskwood plot. All those semi-transparent textures might have something to do with it, but my frame rate bottoms out in there, as does my client's response times.

2. There's a car owned by Bird Feaver buried at <0,0,0>. This probably has nothing to do with the lag, but I just thought y'all should know. (Unless the car is attempting to relink and rephysics-ify, at which point I'd say it's probably contributing to the lag.)

3. I can tell you that the Jazzy Jade place there has more scripts than the rest of the sim combined. Whether or not that's the problem, I'm not sure.

4. There's an elevator above Jazzy Jade that's broken. The button to call it, or the elevator itself, is stuck in an infinite loop and it's affecting the other object. They're both sending out updates about five times a second. I'm betting it's the button that is broken.

However, none of that seems to be new (except maybe the elevator), so I don't know what to think.

Honestly, I can't tell a damn thing. *shrug* Users need better tools too. Something to read how much strain an object is putting on the server. Object Beacons isn't enough.

#2, #3, #4 ... are definitely an issue ... but I don't think they're a delta from before, to now... Unless it is the elevator button or the car...

A 'topcpu' sort of thing for our own scripts would be nice indeed.

Anyways. Gonna have to call out the lindens I guess.

Thanks, Moleculor. Helps, just to document this stuff.

BTW, the fact that Jazzy has more scripted objects than anyone isn't that big a deal, since I'm sure they're all vendors and probably don't run code unless someone touches them, but the infinite looped elevator (or a timer set to 0.1 or 0.2) is probably part of the problem.

i have a pretty fun tool i can give youy molecular.. shows 'active' scripts...

every single thing jazzy has rezzed, is active, just about

every one of their hundred vendors has 4 active scripts in it..

they have one of frans' older leaf trees with 200 actively dual-scripted leaves

all their generic stuff is active an running

this DOES cause lag.

this had lusk perma-posted around 3ms runtasks before, and 1.5 ms runtasks right after the update.. the problem *NOW* is somehtin has pushed that to 4.5 ms runtasks and its somethin NEW, or somethin old that turned on, in a real bad way.

Chance does have a land scanner with 16 running scripts in it... thats the only thing i can find that has *THAT* much power.

its not in luskwood, we returned all non owner scripts, jazzy did as well tho in their case thats sorta like spritzing a 9 alarm fire with a plant spray bottle.

no theres somethin else, either in the lusk estates, or hovering over linden land.. and what really bothers me is that theres essentially nothing *NEW*

i've been doin lag scout trips in lusk for months, and nothing really drastically changed anywhere in sim.

i was *IN* the sim at the time it started laggin like that and i was nearly alone, the other 2-3 people were people i know in luskwood and were jus chatting.

i *THINK* the lindens have access now to some sort of 'top' like interface... where they can at least see WHO is eating the top timeslices.

as far as jus general active scripts i have a more than adequate tool for the job, but like i said nothings really new there

Seems to be degrading. We're 35 simfps solid now, Time dilation 0.81 solid...

Not even seeing the 50-60 we were seeing last night. 35-36 simFPS, never going higher than this.

2 agents.


(Actually after running Ezhar's sim stats logger for a while, we're looking at SimFPS (min/avg/max) 27/32/34 )

... nice.

1 agent, 27 simfps. And falling.

Just had a Linden come check it out and he could find nothing.

EDIT: Now, everything in our store is gone. Just gone. We didn't delete or return, land isn't set autoreturn.... and the only people who could have deleted or returned the objects are right here, confused.

I don't know what's going on now.

I'm seeing something similar happening in Amida. Performance has been steadily dropping all day since about 6am EDT; you can watch the simFPS go down on my graph. SimFPS is hovering around 25-35 right now. Time dilation is down to 85-90%. Physics, run tasks/agents timeslices are at 3.5, 5.5, and 7.7ms. This is with 15 agents in the sim and 3 child agents, but the numbers [edit](of agents, that is)[/edit] have been lesser and the performance was just as bad.

I don't know if this is significant or not, but I noticed that since 1.5, in Help->About Second Life, Amida shows as sim5.agni.lindenlab.com; it used to be sim152.

We've been using an in-world grapher, but the pattern looks pretty similar over here.

The beginning time period even looks close to the same. This happened between 11:30p-5a PST this morning, I didn't notice till ~5am, and that looks like just about when Amida took a dive.

I think at this point our minimum has dipped to 10. With... I think, three agents. Even back in the grand old days of pre-1.5 suck, we managed to pack twenty people into the sim before simfps dropped to -10-.

This is just getting kind of ridiculous.

The answer to that question is a little bit mixed.

Yes, it does seem more responsive than a simfps of 30 would usually indicate.

However, it -is- more laggy than it was a few days ago, when were were seeing 400-500 simfps. It was really sailing then. It's not exactly jerking to a standstill now, but it is still noticeably more lagged.

As for how it's manifesting itself - I haven't tried building since it dropped. But I've seen it in the UI being more sluggish in Lusk than elsewhere. Motion is jerky. there's been time dilation, not just in the alt-1 screen but also visibly inworld.

Then there's the additional weirdness of everything in our store vanishing. Not sure if it's connected. But it looks like there's something overall odd going on in Lusk- seemingly in Amida too. If we were still getting the very smooth, fast responsiveness that we were getting a few days ago, I'd be willing to say, 'well, it must be an alt-1 bug'. But we're not.

Any further thoughts on this would be appreciated. I'm not trying to point fingers or whine and complain - just trying to keep the information exchange high so that a solution can be found.

No, Kelly. It does feel slow. Flying in, I know when I hit the Lusk border, between Ahern and Lusk, because my fly speed slowed to exactly what time dilation seems to be reporting: 75%.

Chat dialogue takes time to come up, often, and object rezzes do take longer than usual.

I think the new hardware makes low simFPS a bit more -bearable-, yes, Kelly, but I do think the simFPS truly is slow.

Owing to the nature of hyperthreading, this really appears to be behaving the way an app behaves on a fully throttled hyperthreading processor: surface transactions may be quick, underlying is slow, or the other way around.

It really smacks of HT making processor load more 'tolerable'. And while I'm grateful for that:

Run tasks, run agents and physics *ARE* high. The result IS showing up in world. It might be SMOOTHER than on the old hardware, and 30 simFPS on the new hardware may indeed be more tolerable than 30 simFPS on the old hardware (due to the fact that behind the scenes transactions and perhaps agent updates are happening quicker )

-- but there's something definitely still wrong with the sim, and it IS slow, and it ISN'T just Alt-1.

I agree with you though that the low sFPS rates are indeed not as "evident" as they were pre 1.5.

But 10 avs will probably cause strange things to start happening. Not to mention that strange things already are happening in the sim.

Lusk felt quite responsive on my first visit, and I got a spike of lag or two on my second, but nothing like Seacliff was doing the other day (when with Alt-1 malfunctioning or not, the sim was lagged to hell and back, and Alt-1 said it was too).

I dunno about Lusk, but in Amida, the sim "feels" responsive enough; time dilation is not obscenely low, but simFPS is. Weird things happen that didn't happen when simFPS was higher, though; when unsitting from objects, for example, I start floating (well, 'falling', really, since it plays that animation) upwards rather than being placed on the ground next to the object I sat on (it stops if I float into neighboring Gama, though, or presumably any other sim that's running normally). That isn't a scripted behavior; the objects I've sat on didn't do these things before sim performance started dropping, and some of them aren't scripted.

It doesn't take any longer to rez objects or anything that I can see, though I do see occasional "skips" when flying back and forth across Amida.

I saw this happening in Lusk earlier today with someone who was jumping and trying to use a jetpack. They'd slow-mo "fall" for a while and then "fall" back down.

I really think that with the new hardware, low simFPS is less fatal, yes, but I really want to be careful in letting it be said that "Oh, well, I guess there's nothing wrong, then."

There is definitely something wrong. I'm seeing the same kind of strangeness that Gabriel is describing.

We haven't had any more than 5 people in-sim today. I think that with more than that, things are going to get quite, quite funky...

I just think the degradation is 'different' now. But seriously: anyone who's seen hyperthreading in operation will tell you that that's exactly the way it manifests itself. I think the simFPS is still low, it's just other things are still going through which normally wouldn't.

Alt-1 is not joking about the time dilation though.