severely glitched simulator

Oof... guess I can't be gone over a weekend, can I? ;^)

Still processing all this... Just to cover a route that hasn't been talked about yet: A high Retransmit value has historically been seen when someone in the sim is on dialup and/or has their bandwidth settings too high.

I'm not saying that's a clear-cut solution in this case, but especially since Michi's got some resent packets on that debug view, have you guys tried experimenting with different people leaving the sim as this happens?

It did it even empty lee... and specifically in this instance when auto-return was turned on, on the linden owned property in the sim, 4 objects were returned.. and *IMMEDIATELY* the sim was restored to normal working order... this repeated several times... someone was placing objects and each time the sim would act exactly the same, get one 'pending download' and the times would go nuts, then the object got auto-returned, and the sim returned to normal

i wish i could jeffrey... but we were not able to track this object down.. all we know is someone was repeatedly placing it on the rougly 13k sqm of linden land in the sim.. somewhere.

The sim performance then immediately tanked, we looked for the object but could not find it ourselves... though the plot did indicate there was an object 'owned by other' on the linden plot..

once auto return kicked in and the linden plot had 0 objects 'owned by other' the sim *immediately* revereted to normal status.

Someone then placed this object several more times, though we were not able to directly catch them in the act, we would see a dot on the far corner of the sim, performance would immediately start tanking... the dot would be long gone.. and then 5 minutes later when the auto return kicked in the sim would revert immediately to normal again..

but yes despite several repeated attempts with sensors and beacons etc, we were not able to actually *find* this object, we just would always see it there listed as 'owned by other' on the (large) plots' description

Okay, in a case like that, to nail causality, it'd be best to view the objects before returning them. Any identifying info (script? physical? Um, owner?) would help.

the current problem though in lusk, just to re-iterate.. is somehow the sim is stuck on a .25 cpu slice.. and *EVERYTHING* is running slow, and the timings being reported are actually *good* so the sim ought to be running well... i don think an object is causing *this* directly.. i think there is just something wrong with sim434.agni which is where after repeated crashes over the weekend, lusk ended up

i wish we could have found it lee, but we really looked, i turned the sim inside out coverin just about every square inch and i was never able to track the thing down.. i just knew it was in the lusk estates linden managed infrastructure land... but from ground to 4096m, i was not able to scan/see/anything this object at all.. nor were we ever able to directly catch anyone placing it though dots would appear and pop out intermittently along that edge of the sim

_edit_

you might want to, if possible, boot a 'record' of lusk on an otherwise empty island, taken at some point from friday mid day to saturday morning... when the object was in there.. and poke around... assuming you have a sim state from that time period archived..

(the issue started continuously from pretty late thursday and lasted directly until about 9am saturday morning when nigel set the estates linden land to auto return 5 minute)

specifically look at the 2-4 objects that were on the linden parcels there, as one of them was what was causing this.

Well, Lusk crashed again this morning. Right now I'm the -only- one in the sim: it's just me. And we're dilating from 0.70-0.80, and simCPU is bouncing from 0.50 to 0.40 to 0.25, mostly staying at 0.25: the same behavior that was happening when the sim was on agni435 - now its on agni596 with the same behavior.

Pending downloads keep shooting up to 100 and slowly dying down to 0, then back up to 100 again, repeat. I'm not moving, nobody else is entering the sim;

Seriously: we keep the content monitored like you wouldn't believe Lee; Nothing's been changed or added since this began to happen...

We've been crash-free since before January 05, and now, we seem to crash 2-3 times a day (easily over 10 times this weekend), and when we had a usual performance of 2000-2500 spfps and never saw time dilation, now we're seeing 300-500 empty, and constant dilation in the 0.80's when empty.

This *all* started this weekend. And it started out of the blue, too.

I don't know what to say or do... it seems like we don't even have leads at this point - I've been biting my tongue on saying it's a malicious object being intentionally remotely delivered, but I'm really beginning to get that feeling. And as Eltee knows; I do have a few reasons for that suspicion.

.... I know that there's been a sentiment floating about that: "It's the furry avs, stupid." -- which was kind of part of the allusion to high bandwidth output causing this... I have to say again: this is happening with just me in the sim, and this is the *same av I've been using since 1.1* ... I think I would have noticed a corellation by now.

If a linden could go into the sim.. or look at the logs, see *WHY* lusk is crashing, what is crashing it.. and who owns it... that would probably be your answer lee

Ihave been having the same problem with my sim Isle of Bliss. Ihad a 2 lindens over on Saturday and they couldnt find anything wrong with it. Sunday we had our grand opening and it was running like crap. And we dont have alot scripts running on our sim.

Please Help

Lee -- this is actually the same thing that killed the old Midnight City and forced them to find a new sim. Even if you had 10 agents on dialup with their bandwidth maxed, it doesn't make sense for Retransmit to take nearly 100% cpu time (under ms breakdown, we would see 200+ms spent on Retransmit). If it's even possible for this to occur, that's a huge huge flaw in the sim server.

Well i dont know if you heard why Forest went down, but they tracked it down to a object that rezzed multiple scripted physic prims, someone dropped didnt know what it did, and it spawned so fast database couldnt keep up,.

Server got up it would go belly up after awhile, so they had to revert to a backup to clear it up.

*this WAS a accident*


But still i see sims struggling, i tried something that would freeze up a sim pretty easy (i killed it fast when i noticed) , dropped thousands of objects in a box, = recive msg went over 400%,, also changing texture on loads of prims would cause something like this to,

Well, we just had everyone clear out of Lusk and Perry and are having Jack Linden look at it right now;

He can't find any suspicious objects, but he *does* see the time dilation, sfps, cpu and upload/retransmit oddities; even when empty.

That is, No avatars in the entire sim, and it's still bouncing up and down like a seismograph.

Eltee has been suggesting the following troubleshooting methods but I dont know how much of a bear that'd be to LL: Roll back Lusk one week (when this wasn't happening) - perhaps to an EMPTY server, a spare box, and compare images...

I don't know how possible this would be...

So any word what they think this can be, im in Taco sim right now and its jumping from 600 to 30 sim fps , like crazy,,, and its not the only one, and why always these hours?, this some backup hour or something ?

Sim cpu 0.4-0.1

More troubles with Taco; looks like they migrated from Lusk to Taco, but refuse to get specifically out of our backyard.

Well, the diagnosis is getting recurrant: "it's the attachment avs". I still am positive that this is not the root cause, so we Ruthed Taco.

In other words, everyone went into the default Ruth avatar. No attachments, no adjustments, no -nothing-.

Guess what.

Problem didn't go away.

While attachments in large quantities -do- cause some sim load (we've still had 25 people in such avs in in Lusk before; more in Perry.) - this is not the issue we're seeing. These crashes are new. They didn't used to happen before. Now they're frequent.

Granted, the servers with this problem do not *scale* as well to having a large amount of attachments in the sim, I believe this an -effect- not the cause.

Again, with nothing but Ruths in the sim, the FPS was still fluctuating wildly from 500 or so down to 45, simCPU going from 0.50 to 0.25 and everywhere in between. So the fluctuations were from 45-500 instead of 18-300.

Furthermore, this was never a problem in 1.0. Or in 1.1, 1.2, 1.3, 1.4, 1.5, or the start of 1.6.

This is a -recent- phenomenon. As in, one week recent.

Noticed this sim533.agni.lindenlab.com (Borrowdale)

0.12 sim cpu and dropping,,, randomly going trough
88% Run Tasks
88% Physics
88% Run Agts
9 Active scripts

hey thili i kinda was able to figure out what the problem was.. and LL should be working on it.. i wrote it up here

/111/28/49510/1.html

if you tell lee linden that its happening there, he should know what to do

jus tell him the thing that eltee found out about, with the sim cpu bein all wrong, is happening