Load-Testing 1.7

Hi Vektor,

From previous experience, it seems there are a lot of issues that don't express themselves until they are 'under load'. These kinds of things get missed in the preview grid - unless we schedule some sort of 'lag party' or whatnot

I just wanted to offer it up - The Isabel Sim is nearly 99% commercial, containing a Huge Mall, and a Large club. It is not uncommon to have 500-600 active scripts running, and the number of textures are boggling.

Despite all this - our sim chugs along pretty well most of the time. It was only after the initial release of 1.6.0 and 1.6.1 - that we experienced lag above and beyond expectation.

Possibly it would be worthwhile to include Isabel in your preview grid testing - just from a load test sake.

Travis

Ideally, they should take a snapshot of every sim and test it, logging a bunch of statistics for both versions and seeing if they crash or have significant performance deltas.
A better idea would be to try a random sample of sims, perhaps biased towards those under heavier load, as you suggest.

It occurs to me that if we had scriptable avatars (NPCs) we could create automated load testing suites that could catch a chunk of this stuff before the paying customers have to go on and try to lag the thing to death. Just make a whole ton of NPCs that would wander around, click on things, buy things, babble at each other, run animations, fly vehicles across sim boundaries, etc., and record errors, crash results, and other statistics. It wouldn't replace the need for human testing, of course, but for load testing and repetitive testing of features known to have been fragile in the past, it could be extremely helpful.

In fact, I think this is probably the best justification yet for having scriptable avatars. Even if they are limited to the preview grid at first.

neko

I think it's a little more complex than just automated logging of scripted NPC's, though it's a key component of the automated test system I mention in /142/33/45214/1.html#post486465

Due to the complexity and unpredicatbility of interactions generated in SL you would also need some form of sophisticated logging and pattern analysis/detection to prioritise the targets for the QA team to attack, along with bugs generated by their own hands on testing and bugs generated by user feedback/auto-bug reports.

All these data streams would need to be integrated into a cohesive system to make the most effective use of the resources which have to fix them.

I also think LL should implement a "reward points" system, redeemable for L$ or equivalent, for users submitting bug reports from the auto-crash system.
You might find you get all of them that way

I was going to suggest that only unique bugs identified this way should get rewarded, rather than allowing people to simply keep duplicating the same crash and raking in the L$, but on second thought, I think that would be an excellent incentive to LL to fix reproducible bugs as quickly as possible.

Let's make it L$500 per crash log submitted. That sounds good.

neko

I'd like to quickly address some of the issues that folks have been raising in this thread.

1. We're looking at ways to bring in more sims and estates for certain tests. As 1.7 Beta nears, I will look more deeply at how this may be possible. Incidentally, XML/RPC and email will not be enabled in Preview 1.7; this was how we also did it in 1.6. That will be tested by the Lindens for security reasons. As for load and stress testing, I am all for pushing the software as hard as possible during the preview period!

2. We have the ability to test using something *like* your NPC avatars, that we use for testing load. This was done during preview 1.6. These 'NPC avs' (that's not quite what they are) that we use, however are not as complex as the ones you envision (I'll be dreaming about the ones you described!!), and they are run via the back end thru a difficult interface...they are solely a development test tool, and shall remain so.

3. Stress and load testing was a challenge in 1.6 because the simulators used in preview aren't the snazzy ones we have on the main grid. Many are retired sims. I feel fortunate to have a preview grid, period. Until 1.6, there never was a 'persistent' preview grid here like the one we have now. I am requesting a few additional, newer simulators for 1.7 preview, so that we may get a more accurate look at performance. Cross your fingers!

4. I agree that a broad, integrated aproach to all aspects of testing is the best way. I'm currently sketching out a harness for this, and am taking all of the feedback seriously when designing it. Needless to say, we're dealing with some of the world's most complex software, and I admit that this type of harness/matrix is a challenge. If anyone out there has already created a functional node map, I'd love to see your interpretation of it! I'm also open to any suggestions or input that you desire to have on harness development.

5. A rewards system has been considered, however I feel that the best reward will be an even better SL. What contribution we all make toward the testing effort will have a direct bearing on the SL that we all share post 1.7. I don't like to think that people would care to help because they want money...perhaps I am an idealist, a relic?

6. As for fixing the reproducible crashes asap, the dev team already is on it...however they are as swamped as the rest of us, and sometimes it takes time for turnaround to occur. Did I mention that we're hiring?

I hope that my responses are as good as your questions and suggestions...I love logging into the forums and seeing this type of feedback...you guys are fantastic!

Take care,

Vektor.

A suggestion regarding rewarding / incentive. How about having a plaque added to the WA for a while after launch?

It should contain a dispensed no-mod notecard (or alternatively have it a textrure like the Beta Wall), containing the names of those who actually found and/or helped to fix a bug in the Preview in some way.

It gives a way for Residents to know that they'll have a permanent record of some sort available.

This may be something that should definately be looked at for 2.0, but perhaps could be revived in some small way for 1.7 to see if it will at least work. At the very least it won't hurt.

This is a very tricky area and user contribution with no direct reward runs at the core of self-generating/sustaining user communities and the open source movement.

The current paradigm around engaging user communities as an extended developer network seems to work on the principle that as the user base grows then the fact that there may be a high churn rate within a certain sub-group is really of no consequence because there will always be another keen body willing to join up. Or they use Actor/Network theory and peer status motivators as a proxy reward system.

Or people just do it for ... [insert reason here]

However, the fact is [imo] that the serious bug detection and analysis in such a complex system as SL is *most likely* done by a very small percentage of the overall user population, also *most likely* comprising a percentage, but not all, of the most active experienced developers/builders/designers...who have an active, sometimes vested, interest in eradicating them.
See also: /142/33/45214/1.html#post489053 for more inside LL info on the actual makeup of this group

For their dedication and hard work the whole population gains, and even more specifically the company gains through getting a better product developed - a commercial product with a virtual economy hanging off it.

My actual proposal was "I also think LL should implement a "reward points" system, redeemable for L$ or equivalent".

It's a slightly more broader suggestion than boxing people into a basic transactional "money for bugs" slot, key motivator that it may be for some people.
[which, btw, is sometimes, not always, called a job under the current economic paradigm]

For forward thinking companies it opens the door to thinking about more novel intrinsic and extrinsic rewards that may be implemented so as to cultivate and grow such key user sub-groups for the long term benefits of both the company and the broader user base through delivering "an even better product".

best,
kt

I am thrashing a dead horse, maybe, but a LOT of the people in SL are "buy it and try it" people. These people are not particularly money motivated, rather (speaking for myself), they live a consumer life in SL, not a manufacturers life, and are mainly socially motivated - something to Show Off to My Friends With, for example.

Since these people are a large majority, harnessing thier energy in testing would be a good idea... Especially when they have a party or something in volume, on the test grid., where the number of people on a sim plus flashing lights, plus their scripts, plus them attempting to use stuff they have bought (ported from a previous realease into preview) ... this would give the preview a good shakedown.

Doesn't work if these people have no money, though...

Crazy as it may seem (or disappointing), consumer types Do stress the system, and for money (their own personal reward system - I can Buy Something to Show Off to my Friends), will certainly tell you what went bad, and when.

I wouild suggest a flat amount for coming on the grid (like the sign up money when you join SL ) plus an incident report based fee...

Just a thought....

Yes, for many of us, virtue is its own reward. But not for the volume of people you need if you're going to have a real load test. See Martin Ford's _Motivating Humans_, c. 1992, Sage Publications. Pages 88-89 have a nice overview -- let me know if you'd like me to fax it to you. To sum up, different people are motivated by different things, and you have quite a broad spectrum of users in SL. If you want the functionality that affects all communities to be tested, you need to get all communities involved in the test, and they will not all be motivated by "an even better SL." Some will want fame, some will want L$, some will want some rare item they wouldn't be able to get otherwise, some will want extra votes for their favorite feature proposal. I'm not saying you should give out all or any of these benefits, just that you need to think about how to motivate the users you need to have involved, rather than assuming the required users will be motivated by altruism alone.


As far as I know, most of the LL positions require location in the SF area. Has that changed? Could we hear more about pay scales? Benefits (e.g. health insurance, 401k, tuition reimbursement, etc.)? I might be talked into switching employers, but not on the basis of the lack of information I'm seeing so far.

neko

I was laughing about this myself. If there is any company out there that has a better
setup for telecommuting I can't think of it, and yet they seem to require a lot of their non-techie staff to be right there in the office.

I have a script... hmm, then again maybe Christopher has it. Anyway, it rezzes a giant skyscraper and sets it physical. It really kills a sim, takes ages for everything to fall down.
We used to do it in Olive waaay back in 1.0 - will be interesting to see how Havok2 handles it.

I don't want there to be any impression that there will be no currency to test with in Preview 1.7.

Right now I disabled it because of abuses - people using it to save L$ on their businesses without running any tests at all, coming in in the dead periods between dot releases. I am issuing L$ for testing on a spot basis now in Preview 1.6.x, please IM me if you would like any currency for a test. Someone said it well, "You don't go grab a drink from the volunteer fridge before you do any volunteer work". I have been gladly giving people L$ who have been helping, yet seek to test something of their own.

Currency and stipends will most likely be in full swing for 1.7 Preview testing.

I also want to add that I think that the feedback on this page is great, and I value every word of it.

However I am unable to comment further on a rewards type of system at this point.

*blinks*

Most of the bugs I find are related to projects I'm working on. I come to the preview grid for two reasons: A) test/experiment with a feature, B) work the kinks out of a product and saving on upload costs (aka Product Testing).

While rarely do I come to preview to skirt upload costs for product test, it is a driving factor. Usualy when I come I'll work on on a half dozen projects, I'll chat with folks, play with things and send in bug reports.

Some things you honestly can't test in the preview window. Animations are a good example. There are inconsistancies between the preview window for animations and how the animation will actualy be displayed.

But I see monitary abuses as a minor thing compared to the assholes who come on and shoot people in the name of "testing".

So I have a few questions: How widespread is monitary abuse, how much is being spent on uploads? Besides being annoying, is there another reason to consider this a serous missuse as to restrict it?

I believe this was the key point Vektor made in his post about money abuses in Preview..

The product testing happened between releases when the release on the preview was either at or behind that of the main grid, so useful testing was at a minimum. People entered the preview for the sole purpose of product testing of items like clothing which can be quite expensive if you're a perfectionist, or rather can afford to throw away the preview money on getting that one item just right. They are using a system for their own gain without contributing to it in any way.

That said, a preview without money is not at its full potential. I would hope that people come to preview to test the functionality of their completed projects with new versions. That may require uploads and the use of the money() event.

Baba summed it up well.

I also agree that it isn't at it's full potential without currency, and needs it. I am working on refining how currency is used in Preview during the lull periods.

That said, what did people do before the Preview grid became persistent, to test upload textures and animations? I gather they somehow got by.

If someone has *no concern for contributing to the test process in any way*, yet is coming to Preview solely to do 'Product Testing' that is unrelated to a new feature or bug, (especially during the lulls between releases), then they're not helping. It is a *test grid*, not a cost shelter for those who don't give a darn about testing.

You have done some actual Preview testing, Strife, and thus I have no qualms with you using the Preview grid to some extent for your own stuff.

The one reason (of my many for temporarily turning off L$ in Preview) that I will offer is 'principle'. If 'principle' doesn't have any legs in this day and age, then I'm truly sorry. It should be able to stand well on it's own.

If you, or anyone else who has tested or filed bugs needs currency for legitimate reasons during this period where L$ has been zeroed out, please IM me in the Preview grid and I will be happy to help you out!

Incidentally, if someone is shooting you in Preview, and you haven't permitted them to, the same ToS rules apply.

I don't use the preview grid to test my uploads, so I'm fairly unbiased on this, I think. I suppose limiting $L to avoid letting people use the preview grid for free upload testing makes sense during the lull between versions. But the argument can be made that anyone who logs in to a non-production release is, at the very least, contributing to load testing. And getting people used to logging into the preview grid regularly could help increase participation when there is a release that really needs to be pounded on for a bit before subjecting the entire population to it.

The tricky thing about "principle" is that there are usually many principles that good, sensible people will agree on, even though some of them conflict. The human condition is funny that way.

neko

That's all i heard. I don't live in the SF area though