severely glitched simulator

Well the sim lusk got attacked and crashed several times in the last few days... and now after the most recent crash/reboot... there is something very wrong with the sim we ended up in.

Essentially its stuck in some kind of retransmit loop... there is one object permanently 'pending download' and retrasmit flicks as high as 50ms, send images also spikes north of 20ms...

I've attached a screenshot of the simulator statistics panel to back up what im saying... but yeah, there is something drastically wrong here, and it doesn seem to have anything to do with whats going on *in* sim.. lusk has been pretty static that way.. for months now really...

It seems like its just stuck in a loop on the backend of the sim process... somehow related to that 1 'forever' pending download

also notice the sim cpu at .21 ... and the fact that all the timeslices together don even come close to adding up to anything resembling a normal percentage

got an even scarier screenshot heh.. just look at the percentages *now*

somethin is *very* wrong in sim land hehe

another, just for good measure.

excactly the same as i seen in The Forest over the last 14? days, it major spikes up Revice Msg , 140-250% , big time lag, been there a few times and noticed it, looked in debug says the same every time

"Kills on unknown object 1 to 4 "

lag gone
goes awhile it happen again,, till server go poof gone.

thili if you can, get some screenshots like we did, with the show gui on, and the sim stats opened so LL can see exactly whats goin on

Yep will do if i encounter it again, waiting for something to break in SL is offcourse things that wont happen when i look for them

*tap finger*

Bottlenecking CPU, (possibly because of too much traffic over ethernet) high retransmit, *27 second sim ping time* - and we are NOT on a low sim: sim 527... that's decent hardware.

Heh, just saw a a Receive Message of > 300% go by... Wouldn't that mean that the sim is *receiving* an amazing amount of incomnig traffic?

And, Lusk just crashed again... We've been stable, no crashes, since mid-1.5, and throughout all of 1.6.

Now in the last 2 days, we've been crashing continuously and repeatedly. We control pretty much all of the sim, and nothing has changed.


... Well, Lusk just came back up on sim 7xx, and now we're at 2300 simfps.

So its' definitely not content. Need to watch if we stay on sim7xx, and go back down to 8.

*Problem* solved for now, mystery *not* solved.

yeah this wasn't content.. this was some strange technical glitch, the never ending pending download of 1 item, and the totally off beat wacky statistics show that

I hate to state the obvious, but this sounds like simple asset server failure for that specific server.

This has happened before. Perfect mention of the problem by Andrew Linden can be found here:
/3/42/33127/1.html

Note the linked thread is from a few months ago, and the asset server crashed hard a week or two ago. The problems seem to be related.

Have anything to add, Lee?

Jiminy appears to be having a similar problem with Pending Downloads. Compare these two ratings:

And this one. It's my personal opinion that after the asset failure a week or two ago, several servers may need to be restarted. That is of course, if this is not a huge script problem this time.

I doubt even a script is potent enough to spike Send Imgs to 110% (missed that in the screenshot). A Linden opinion would be nice.

This is with three agents in the sim.

Here's another shot. The bottleneck appears to be Send Imgs this time.

As of 7 AM EST, I've been told the Lindens are looking into this problem.

From Janus:

My sim is having the same problem. Strange kind of glitch. It seems to culminate in a sudden 'sim-storm' that reduces to the SimFPS and Time Dialation to zero, and then goes away slowly.

I was thinking it might be a physics error or massive communication that switchs on, but the download seems to be the issue here as well.

I do not envy LL trying to track this down...

Noticed a few more sims with the excact same problems, : Pending download : stuck ,, even a very new sim - like Taco has it , lala ,, good luck Lindens

Well, Sim723 that we got assigned to last night, after the crash, is slowly doing the death-spiral again.


What was 2300 simfps when empty, (on sim723.agni last night) is now 400 simfps when empty.

Still that *ONE* constant pending download, that never seems to go away.

1 agent, 1 child agent, 420 simfps: when the same machine (723) - as I posted last night, was doing 2364 with 4 agents and 6 child agents, up until an hour after it came online.

If I'm correct, this will get worse as the day goes on, until we end up in the single digits again and the server will crash.

This is interesting:

In the graph here: from the beginning of the graph to about 15:00, (when it takes a severe nosedive after a crash) - everything's fine - 2000-3000 spfs. At 15:00, we had a crash and came up on sim 527.

Sim527 runs (and slowly degrades) until right before 00:00 where there's a crash (the one mentioned in the posts above) and when it comes up on sim 723, performance shoots up again to about 2500 simfps. Then, at about, oh, 00:30, the performance drops again, but we're still on sim 723.

I can assure you nothing changed content-wise in that one hour, nor did we get a surge of visitors: Lusk was *empty* when that occurred. (We only have 82 active objects at this point and 86 active scripts - a few minutes after the crash, Char Linden came by and reported to me that there were no suspicious actives, either.

The thing that worries me is that this was a *SUDDEN* drop each time, not a slow constant degradation. *AFTER* the big drop, it seems to get worse until the sim crashes, but everything seems to be fine for a while, and then the bottom just drops out.

Checking some sims i come up with

Ataraxis - 5 pending down
Faded Reality - 1 pending down
Forest - 1 pending down
Panther Estates - 1 pending down
Taco - 1 pending down

Ok tooo many have 1 pending down to even list them all, load of sims

Then Panther Estates do this on me , i didnt screen at right time but it peeked at 211% recive msg

I've been seeing this in a completely empty sim Moritz, not only has simfps dropped from the usual 2500+ to about 60, but its been very sticky to move with av's ghosting at the edges, retransmits have been as above, as far as the grid needing rebooting, then i believe when the sims were patched for 1.6.6 on tuesday all the sims would have been restarted.

I've just contacted mick linden about this, he said it was being investigated, as its weekend i doubt there will be any mods on the forum so you may want to im him with your problems and he might take the sim name and add it to the investigation list.

i don't know if its one thats bein purposefully left, or just generally glitched.. but anyway

lusk is split about evenly between our land and a linden managed suburban build area... the infra-structure tho accidentally lost its auto return and had built up 2 objects... nigel came and set it back to auto return... and those two objects were cleared.. and within 60 seconds lusk went from the former shots. to this shot here

yup, over 2000 fps, no pending downloads, and everything is running smooth

so there are objects out there, that when rezed can effectively destroy a sim, with pending download... and chances are good given threats given to some of us, that there are people now using them maliciously

fun.. hope LL can track those down, and pretty well just nuke any account thats been placin them around

Ah! Good!

Why good? Because it means it WAS content and not flawed code! This can be fixed by banning whoever is attampting this and killing the nasty little attack object.

I was afraid it was an actual DESIGN flaw from one of the last upgrades!

I hope this is the case, although i did see this glitch in moritz, which is a 100% linden sim, i didnt see any objects other than the usual snow, trees and road. But the lag was that bad i couldnt really move around to investigate. also im sure most of this sim if not all would be set to autoreturn

Nestor was doing this exact thing from about 2:30am game time, this morning. The readings look exactly the same as those posted here. I slept, woke up and just logged in to find it down.

All it would take would be a single 16m2 section not set autoreturn. After returning all the detritus left on the non-autoreturn Linden land in Lusk, the simfps shot up to 2300. This land is set autoreturn now.

So yeah, in summary, it was a "suspect object" causing the slowdown. Assuming this is an active, (which it probably is, unless it's some sort of invisible interlinked physics) - it likely can be scanned for.

The lindens are going to massacare whoever is doing this if it's malicious.

<sigh>

Some people just have to cross the line.