1.6 And SimFPS

Okay, here's the story behind 1.6 and SimFPS as I understand it.

As part of the 1.6 upgrade, we made several changes to the servers running Second Life, including upgrading the operating system running on many of the machines. This introduced many subtle changes, including changes to the ways that the servers report their own performance.

The important one (as far as SimFPS is concerned) is that the simulator no longer reports as many "empty frames" where it's not actually computing anything. For example, while a new simulator in 1.5 typically came online with a SimFPS of 12,000 or more, the server wasn't actually updating twelve thousand times per second. (For one, it's not a 12GHz processor!)

The end result of this change is that sims now report a lower SimFPS value given the same load. This does not mean that server performance has been reduced (that theory's wrong for several reasons). Instead, it means that a given SimFPS value now represents a lighter load... which, in a high-load (sub-100 FPS) situation, means better performance than it did before.

So, to break it down as simply as possible:

1.5 SimFPS for your sim > 1.6 SimFPS for your sim
500 SimFPS under 1.5 < 500 SimFPS under 1.6
1.5 Sim Performance = 1.6 Sim Performance

And, as always,
SimFPS != Performance

You can always use the other Statistics tools to get a MUCH better indicator of how your sim is performing and what it's spending the most time on. (I personally recommend the listings under Time(ms).) More details here:

http://secondlife.com/tiki/tiki-index.php?page=GuideDebugStats

Hope this helps!

I had no idea that there was a good Wiki about how to view, what it is, and a great description on the Statistics bar. I was just going from past knowledge, and what others had been telling me.

This is a great resource!! Also the knowledge that the SimFPS is now reporting properly whereas before it was not is a good thing to know also.

Thanks Lee Linden!

now if we can just figure out what's causing the low FPS on this end.. hahaha

Thanks Lee, very interresting

actually lee i have seen exactly the opposite... the sim fps numbers are lower to be sure... but now im starting to see *some* time dilation on servers with over 1000 fps, and serious .6-.7 kinda hard to walk or move time dilation in sims with 300-500 fps.. so in actuality it looks alot more like sims are now struggling more than ever before, and even 5-10 agents in a very high number sim will bring on time dilation and usability problems whereas before in a 500-600 series sim even 20 agents didn't task it that badly

Great, now can you explain the 65000-110000 ping sim times after opening inventory?

Lee - could you clarify something for us, please...

Is the purpose of stating the FPS differences between 1.5 and 1.6 in this thread because:


a) Linden Labs believes the "lag" that folks have been complaining about for the past few days is mostly perception.

b) Linden Labs believes there may be a lag issue, however folks are confusing the issue by quantifying that lag in terms of FPS, which no longer means what it used to.

c) Simply explaining the new FPS differences. Recent reports of lag are another matter alltogether.

d) none of the above


Much appreciated if you could respond. Your answer will hopefully drive the discussion in a productive manner.

Thanks!

the lagg i get has nothing to due with any report.

It goes like this. Move ok move ok turn on turn hic.. hic crash...

The purpose of explaining the difference was because I've received several complaints specifically about the lower SimFPS with no mention of actual performance issues.

That said, there are issues people are reporting as "lag". It's difficult for me to diagnose, though, because no two people mean the same thing when they say "lag", and the symptoms that fall under that category cover a wide range of problems, most of which are not server-originated.

Jerky framerate or pauses in the video are a clientside issue. This is typically caused by too much graphics to draw (i.e. 1.6 reset your graphics settings to too-high values), but can be caused by network (i.e. too much bandwidth or too much hard drive caching) in some instances.

Not seeing your inventory, zero cash balance, missing textures inworld, red map, no terrain, the ability to rotate but not move--these are all partial disconnections from Second Life. (You can IM and such because you're still on the user servers, but not the region you're in.) Most of the time, it's because you have some other program installed that blocked Second Life's network access. This is especially true if it happens after a few minutes of logging in.

Chat taking forever to appear, movement not moving for a second or two, then not stopping when you let go, etc. are response time issues. If you're not on cable or DSL, that's probably the cause. If you have wireless on your home network, that may be the cause. Again, if you have internet security software, it's probably a cause.

Server performance issues typically result in slow movement, delays in chat (yep, same as above), script delays, etc. The big difference is that actual server problems are visible in the Statistics bar as problems with Time Dilation, Agent Updates/Sec, and/or Physics FPS. If the primary problem is within Basic FPS, Packet Loss, Ping User/Sim, then it's not so much a server issue (if at all).

I know a lot of people are reporting issues. Just from what I've seen in Support and inworld, pretty much everyone I've talked to who has problems, has internet security software installed. I know most of that software saw 1.6 get installed and, in general, decided to block the "new" Second Life. I know those programs cause a lot of the symptoms people are reporting. I also know it's virtually impossible to convince many people to uninstall their internet security software, to try to troubleshoot SL. (I have enough trouble convincing people to unplug wireless.) So, as far as the forums are concerned, the best I can do in most cases is explain what I know, and hope that's enough.

And, of course, I forward the problems people are having to the QA, dev, and ops folks, to make sure we're fixing the issues that do originate on the servers. But I do try to troubleshoot the primary causes (i.e. internet security software) in the process.

So, to get back on subject: the single most useful part of the Statistics bar for performance is the Time(ms) section. The single least useful value, with the exception of Sim CPU, is SimFPS.

Cristiano: A ping sim/user with more than four digits isn't a real value. The ultra-low bandwidth there suggests that your connection to the server is timing out, as is its connection to you.

Eltee: My first impression is that the two aren't related, because, again, simFPS doesn't equal performance at all. In a case like that, I'd be very curious as to the Time(ms) values; I'd suspect you'd find the actual culprit there.

Lee,

Are there particular brands or versions of Internet security software that are 'known good', and coexist well with SL?

well the timeslices pretty much divide into the sim fps... basically whats happening is the physics fps dropping from its (standard) 45, down to 40, or 35, while the overall sim rate (and time slices) seem basically unafected...

its as if it was just having some form of scheduling issue that was causing it to miss some physics updates... it has the cycles to spare, but they're not being devoted to actually running the physics engine. It creates a palpable movement stutter and makes walking/flying difficult.


btw, is there any way we could go back to the old (1.3) style ms values? the 'new' ones aren't nearly as informative as they only list a subset of the actual times, and the physics ms value is essentially meaningless in the context of the others as its only run according to its own 45 cycle per second timing.

(and while its doing this the physics step itself is actually not any bigger than it used to be, still generally under a milisecond, its just not being run as often as it should it looks like.. even with only 4-5 people in a very fast sim, its already starting to miss physics passes, get 10-15 in, and the newer number sims are performing as badly as the old code did on the slowest sims (not in raw sim fps, but in time dilation, and noticable movement lag etc)

The next time a few people are over i'll actually get some specific numbers to post... basically as it stands now, lusk is runnin smoothly, 2000 sim fps which i know you lindens hate but a few of us with a statistical bent actually don't mind dealing with (as the aggregate inverse of the time it took to render a single pass of the current sim state, but i digress).

The total %'s add up to only about 40%, mostly in run tasks, though there are only 82 active scripts (all of which are old, and known, and don't use physics).

as far as time slicing goes, ya got 0.1 run agents (with 3 agents), a 0.5ms physics step, 0.1ms run tasks, and 0's on the other ms timings.

20 agent updates/sec and 45 physics fps with 1.0 time dilation (aka runnin smooth). In fact its so smooth the cpu is actually sleepin down to abuot a .41 or .42 but thas not really meaningul

the problem comes when agents start to drop by.. as the number of agents increases, and not even by a lot, none of the ms numbers alter much, but the physics fps begins to skip out more'n more causin time dilation, and perceptable lag (for reference purposes this is sim 494, a sim believed to be of the shared dual-opteron class)

Eltee, if you can, let me know the actual ms values during the time when there's problems, as well as the actual number of avatars there when it happens...

Tinker, Norton's been the biggest problem, but I've heard issues with most any type of internet security software if it's not configured properly. (Norton just likes to interfere anyway when it IS configured properly.)

Myself, I block things with the hardware firewall on my router, and keep antivirus running. I've had no problems with that combination.

Nice explinaition Lee Thanks

Cat

P.S;
As I haven't seen any further posts on last nights power outtage as posted by Ian;
"At around 11pm the colocation facility in which SL is housed lost power for almost an hour. We've just now brought everything back online. I hope to have an explaination from them tomorow, which I will post - we're all curious as to how a building with 21 megawatts of generator capacity can go completely dark"

I'm gonna offer up a bit of information on the subject

I would venture to say that SL has a back up generator, not their own power plant. because that would be way to expensive. in the real world a company gets its power from the local electric company in town. alot of companies would have their own back up system incase of a power outage. in that case under normal circumstance during a power outage a relay is tripped when power is lost. that controls a switch. that switch isolates the outside power supply from the generator. the back up genertor would then start up and provide the power to the system. it could be a fault in the switching gear that did not allow the generator to start. it could have been a generator problem itself. it started but there was no output. 9 time out of ten times, if a generator has no output the problem is in the exciter field or the voltage regulator.

Now if LL has there own power plant then there are alot of factors to take into considerator. do they provide there own power and use the local as back up. ( considering LL is in California that may be cheaper). I don't know. most power stations would have a back up capability. so then your looking at different factors. one failure shutting down the others, are they on a automated system. A back up generator came on but failed to synchronise, resulting in a overload condition that shut them down. A phase short to ground will shut it down. there are to many variables. but since they only lost power for a little while then it could not have been to bad. but when all else fails you just tell the boss. " CRAP HAPPENS" .

reseting; most of this time is spent resetting panels isolating the failed generator and checking out displays before bringing the power back up. There are to many unknows to give you an answer. to many electrical components to depend on. I would guess they use local power, this was an isolated inccident and if LL's has there own back up system, and it failed for what ever reason.

LL if you don't have a backup generator I would suggest you get one.

Cat

The servers are located at a co-lo facility and not in the offices of LL. Any co-lo facility worth its weight in used tissue has at least one backup generator. As mentioned in another thread, this facility has "21 megawatts of generator capacity" and that is part of the great mystery as to why the place went dark.

Hi Lee, thanks for your explanations here, they're very good. But what about the above symptoms when you *do* have cable modem, are not on wireless and don't have any Internet security software installed at all? (On a Mac.)

I find response time is waaaaay down from 1.5 to 1.6 in terms of what feels like some kind of buffering problem, i.e., I rotate or do something else, then push the up-arrow key to go foward and nothing happens. It's not a delay, nothing *ever* happens in response to that key-press. Then if I wait a couple of seconds and press it again, the server hears me. This has been a very consistent, reproducible behavior since 1.6 came on, something I never experienced in 1.5, or only very rarely. Another example: I hold down the E key to go up, I go up, let off the E key, push the right arrow to rotate, nothing happens. It feels like some kind of i/o problem to me where the server is ignoring keystrokes while it's doing something else rather than queuing them or whatever it used to do. Again, this has been a very consistent and reproducible behavior since 1.6, and it causes me enough delay and confusion to impact my experience. Though not enough to make me quit or anything

Well we finally got a few people over, and noticed this effect happening... and i've been able to get some numbers on exactly whats going on with the high simfps time dilation...

or better yet, let a picture (or two) speak for me

Basically we have two slices here... both showing some rather perceptable time dilation (and in world movements and such carries this out) but with seeminly little reason for it as according to the numbers, except the physics fps, it should not be happening at all

it seems like its almost just skipping over physics passes it *COULD* have been rendering, but the scheduling, or something else... simply didn't let it...

well here i'll let the pics speak for themselves... ooh an for the record these shots were taken around 8:30pm pst, friday, in lusk as sim631.agni.lindenlab.com though we have seen it on a whole myriad of 500/600 series sims in both lusk, and perry...

it never seems to have happened before with the older system... there i understand the 'raw' sim fps number was a different mnemonic, but you honestly wouldn't see time dilation until the actual timeslices showed that the physics timeslice was large enogh it honestly was not going to get enough cpu time to execute the requisite number of passes per second (45) and still allow the full sim running too (aka it very VERY rarely was ever seen over mebbe 150-200 sim fps, or contra-positively about say 5-7ms total rendering slice (not the 0.5-1ms you see here).. *something* appears to have changed, on the back end, via the sim os upgrade, and it looks like it is not handling loads for the physics passes as gracefully as it once did, causing these strange high sim fps time dilations

jus wanted to post second screen cap in second post so they'd be visible while readin my prior post steadda needin to click links

i just got wireless router here...but had probs with it cause of walls in house lol...so am using the wireless router with wire line that used with old one..netgear is one being used...am having those lag problems mentioned..freeze and very slow..to where cant move at all in sl ....puter is 512 ram, 2.8 ghz radeon, x300 ..dell pentium 4 cpu
with me using the wire to connect to the dsl..why am i still having this really bad lag in SL? never had it bad like this with this puter before......

been a little while.. just wanted to see if you'd seen these screencaps of the stats lee