Speed metrics

At the town hall last night Phil said that LL uses fps and crash numbers as a metric of SL's speed.

Can that statement be expanded please? Do you use sim fps and see how often they're <45? Do you poll everyone's average fps over their session? Something else?

If you're using our session fps how is that an accurate reflection of speed? We have so many ways to change our client's fps - different options, draw distances etc. that it's a meaningless metric - Just by playing with things that don't involve a restart I can vary my client's fps from ~1 to ~20 whilst standing still and changing nothing else.

If it's client fps you're recording how well the servers are doing, but not anything about our subjective experience.

The only really good way I can think of to compare it is to do what some of us try to do anyway when we're used to "home sim fps" - log in somewhere known and static (and maybe unvisitable by others) and see, with the same set up and preferences, what the fps you get are over updates. You might find, and so might we, that things don't change from now on but if you could go back in time you might find that the fps had fallen for you too.

I've not changed my preferences at all, nor my machine. My ISP has INCREASED my bandwidth. My typical home performance has declined by about 30% (measured by client fps) since the late 1.6.x to today - although no decline from 1.8 to 1.9.

The other measure that might be instructive is to tp to a number or points and measure load times - for real measures you'd need the stuff at each location static as well. Again I'd lay odds that the load times have increased since 1.6.x to today, although I've got no hard data for that, just an impression.

=====

EDIT: I've emailed Phil my configuration (and said he can paste it into here too, rather than me starting a new thread if he wants).

Thanks to you both for the speedy response.

I'm still not 100% convinced it's a good metric - you tweak your client performance to get something decent and relog and the % sessions with good numbers doesn't drop appreciably would be my first approximation, but at least it's now clear what Phil was talking about, so thanks again.

I'm going to ask Philip to elaborate on his statements at the meeting, but it might take a little time. Please be patient!

Hi there Eloise! Here is the crash/fps subset of what we are measuring day-to-day trends on a chart which is on a whiteboard in our main development room. Also I've included stats for yesterday to compare to:

Daily % Usage at or above viewer fps threshholds: The % of total user session hours for the day spent at average FPS better than two thresholds: >10FPS = 61.27%, >30FPS = 7.91% (about a 1.5%/0.5% improvement compared to 1/15 for example)

Daily % Usage at or above sim fps threshholds: The % of user session hours on sims that are running better than a certain FPS. >20FPS = 99.12%. >35FPS = 99.93%

% of Daily Users un-affected by Viewer crashes. This is the percentage of people who ran SL during the day with ZERO client crashes: 78.47% (3/14, the day before the release, we won't get accurate data until the server crash bug below is fixed)

% of Daily Users un-affected by simulator crashes. This is the percentage of people who ran SL during the day with ZERO simulator crashes: 83.3% (NOTE: this is typically about 95%, and is a result of the high-byte characters bug which we have mostly fixed)


These stats haven't moved very much over the past several months - we run a daily chart and then look for big changes on the days immediately following big releases. You are very correct that looking at average FPS doesn't take into account all the many different configurations and changes that folks are making, as well as differences in content and location in-world. However, averaging together everyone's FPS and then looking for sudden changes at releases DOES give us a very effective indicator of whether we've screwed something up at a specific release.

Regarding your 1.6 performance drop, can you send me an email with more specifics on your machine configuration? I'd like to see what histories I can dig up around the 1.6->1.7 time for the sort of machine you are running.