Is Linden Lab's preview testing process inadequate?

Although I missed much of the 1.7 launch chaos due to a hurricane hitting the same day as the release, I have heard how badly it went, and seen some of the effects myself while using SL this weekend. Between the horrible packet loss that has actually bounced me into other sims while simply walking (for example, walking on my land in Sistiana propelled me to a completely different spot in Grigano) and the slowness of rendering/textures that has actually made sims look completely empty at times until stuff starts to appear, the whole thing begs a major question.

How did several months of testing not catch these issues? There was preview build after preview build, stress testing, and countless release delays to get things right. Major features were removed from this release because they were incomplete. I know an upgrade of this magnitude is obviously complex, but what is the point of the preview and testing if not to catch and resolve the type of stuff that still remains in the release? There was nothing earth shattering in 1.7 that could not have held off a few more weeks. Is the preview grid too small to catch the type of problems that have been occuring live?

Well it seems to me to be inadequate simply because a lot of the problems I am experiencing were not present in the preview grid. If the preview doesn't pick it up, then it surely can't be an ideal test situation?

As Gwyneth so eloquently put it, I think LL doesn't have a concept of feature freeze. Preview worked really well, then it totally broke on launch day, which either means something was changed between the two versions, or that unforeseen scalability issues were exposed by the launch. Or both

What can I say ?

It patently is inadequate. I'm hoping lessons are being learned, I don't want to throw blame around atm.

Preview worked fine, release is well ... um err, not sure I can use the words I feel slipping out here !

I agree, blame is unnecessary - it's not about that. It's about hopefully finding a way to prevent this kind of thing in the future. I know that a tremendous amount of work went into getting the release right. The question is, what happened, and why?

I've been saying since 1.5 that the preview testing as it happens isn't representative of the full grid, and isn't testing new releases in the right way. Most times I've said this in the past I've had the great "well how should it be then, bub" but, that's not my responsibility. Linden Lab should know their software, it's weaknesses, and be able to come up with a preview or test regime that adequately covers it. They don't currently.

Preview really is kinda pointless as it currently happens, other than for the glaringly obvious.

During the update process we were invited to 'go and play in the preview grid'. Some of us did this - and were forced to download an update, which does suggest that some last minute 'improvements' were added...

Anyway, on arrival in the 'preview grid' I was pretty surprised to note a few extra sims present - like the entire 'Linden' area, you know, Ambleside, Waterhead - all the English Lake District area names, along with Duck and Beverly Hills.

I wandered round a bit and found even the 'old' preview grid pretty laggy - especially as it was virtually empty.

Beverly Hills was suffering all the classic 1.7 problems we have come to know and love, it was unusable, so I wandered over to Ambleside. Met up with a few people there, chatted, though chat lag was unbelievable, we all agreed that if this was 1.7 there was something seriously wrong. Movement was all but impossible (I have an enduring image of Siggy Romulus running frantically into the distance - at about one pace every ten seconds...) and textures were taking up to 30 minutes to rez.

At this point it was very clear that there was a SERIOUS problem. So why was the 'upgrade' continued? If any Linden had visited the area, and surely one must have, I mean, why else put those sims online, it must have been crystal clear that there were massive 'issues'. So why press on regardless when, before the main grid went up, it was patently obvious that 1.7 was flawed?

I had been to a Preview stress test gathering where about 70 avatars congregated and although it was a little sluggish, it was quite usable. Nothing like things are on 1.7 today on the main grid. Something was definitely changed along the way and not adequately tested.

Although the size of the main grid may expose some unseen issues on any upgrade, the vast majority of problems occuring now are not, I don't believe, related to that. The performance on my isolated estate is now horrible, with random time dilations, various texture failures, etc. All the same things currently experienced by main grid sims.

There are either fundamental things wrong with the release, which hopefully they can patch quickly, or the mass grief attack caused some other issues we haven't been told about yet, outside the space server. (How's that for a conspiracy theory)

I really just wish they would focus on getting long-promised features out the door and quit messing with bullshit like the behavior of the rotation tool grid, for example. It was fine the way it was, yet they wasted time improving it for improvements sake. I'd rather have Havok-2 then some goofy HUD.

My only observation was that there werent enough people actually TESTING in the preview grid. I'd do my part to try and log onto the preview grid and try and break things so I can bug report them, but at most times I would normally only see maybe a handful of people total on the grid. Maybe it's LL's fault, but maybe it's also our fault, for not logging enough hours on the preview grid helping find the bugs. Linden Labs cant fix bugs they dont know about, and once it hit the main grid, 1.7 now had many times the number of people doing things.

The thing is Rei, *none* of the problems I am experiencing now (and these are major in your face problems) were present on any of my visits to the preview grid.

This is a fair assessment. Andrew has clearly described that the devs work on "pet" projects that get piled into a preview release and then weeded if they can't be sufficiently debugged to not break too many existing features. There are no maps, schedules, teams, or trunk parameters. The dev process operates as a free-for-all hobby shop. That makes it challenging and innovative.

But, he's also claimed that they are organizing a trunk with grafting params as well as teams that are assigned features and deadlines. There is also much hope that the incorporation of such components as the new rendering engine, mono, and speedtree into 2.0 will require a complete rewrite of the server and client. It's very difficult to explain to students the beauty of starting over after hitting a wall but having learned so much. But the devs at LL certainly have the experience to realize how quickly they could duplicate the system, from scratch, in good order, knowing what they already know.

So I voted no. The problem isn't really that they don't test enough. It's that the development model is fundamentally flawed if you're expecting a production product. We're still playing a prototype. People beat me senseless for calling it research. But it is.

Why press on, indeed.

I'm wondering why, with problems of this magnitude, they didn't just pull it right away. Now we are at a week, I think. I haven't logged in today; maybe things are better.

coco

Before the 1.7 release, I would have said the testing was adequate-ish, but would have blamed us residents for not jumping up and down on previews hard enough to really try and break them. Like others have said though, I didn't experience any of the problems from the 1.7 live release in preview, which makes it obvious to me the testing isn't adequate.

As Eggy mentioned, maybe they tweaked something last minute, or maybe it's a scale problem, but whatever it was, something went horribly, painfully wrong with the live release that was not there in preview.

Well I can cause my sim, with only me in it, and no one in the adjoining sims, to drop to 4fps just by panning my camera.

I really think that had THAT sort of thing happened in the preview grid someone would have noticed.

Don't try and blame the previewers. Either the Preview grid wasn't in any way representative of the main grid, or (possibly and too!) the update was modified between last public preview release and the main grid release.

I can't accept any suggestion that the mess we are now in is the fault of the customers! In fact I find this sort of attitude extremely annoying. Development, testing and QC are LL's responsibility - NOT the customers.

The problem is the rollout process.

They need to do a soft rollout to their customers. Any good developer can figure out how to do it.

In brief, the testing process is inadequate because of the "patch once, patch everywhere" structure. There are no formal protocols to speak of, versioning is rare, and the philosophy is typically, "If it broke, tough. Come to the Preview Grid."


That said, Primmies broke under 1.7. The devs have been very supportive of fixing it, but the point isn't prioritizing fixes; it's making sure stuff does not break, for anyone, to begin with. The only exceptions should be acts that have no redeemable value, like global attack scripts, and even then sneaking in a fix at the last minute is just foolish and reactionary.


The only real solution I can think of, standardizing the protocols to let sims themselves be "versioned," was discussed way back. I haven't seen any movement in that direction, though.

Yes it is inadequate. Here's why it is inadequate for me:

1. llEmail is apparently broken. I can't seem to do O2O emailing in the preview grid. I have also not been able to get an answer to questions about why this is (is there a different email addiress to use? if so, that is not acceptable). Some of my more interesting products use email functionality, so i basically can't fully test them in the preview grid.

2. The grid is too small. In the real grid, I have 3 sims worth of land and I have lots of objects in all of them. In the preview grid, everyone an only buy small parcels of land which means they can only rez a small number of objects.

3. There are simply no good tools to measure performance. I have complained about this time and time again, and all we get is more obfuscation. I want to know how much load a single script generates. I want to know how much load an entire object (with all its scripts, moving textures and other garbage) generates. I simply can't do this. In 1.7 they made the CTRL-SHIFT-1 information even more useless. Script IPS doesn't mean anything compared to the old system. How are we to know what is a good number vs. a bad number?

4. Is the hardware comparable? I don't know. This is another black box as far as I can tell. If the hardware running the preview grid is not at least close to what is running on the production grid, it'll be basically impossible to compare performance between two different versions.

WHAT NEEDS TO HAPPEN (to address my issues at least)
1. Preview grid needs to be a mirror of the production grid. Everyone has their same land, money, inventory, etc. Mirroring has other benefits as well (backup anyone?).

2. Hardware running the preview grid needs to be within 90% performance of the production grid.

3. We need better debugging/measurement tools. LL wants to tout SL as a development platform, but then they don't provide develpers the tools they need to make it one. I can't understand the idea behind rearranging buttons on the screen, integrating classified ads and not implementing something that allows me to measure perf impact of object/scripts. Sounds like LL is working on a game, not a platform.

2 worthless cents worth of thought.

This is still Beta software as evidenced by the problems that were evident immediately after the rollout.

Customers should not be expected to pay for full performance of Beta software.

The Preview Grid, at best, represents something other than a true representation of what IS he Main Grid in terms of size, performance and obj load/avatar load at any given time. There simply is not the amount of cross-sim interaction that occurs on the MG when one goes onto the PReview Grid.

With each major rollout I have seen by LL since the "rez date" of my first avatar, I have had decreasing confidence in the ability of LL to adequately test the intended "upgrade". Bugs found in testing have been admitted and yet remain to be fixed - "at some future date". Purely aesthetic features in the U/I have been rolled out while those same bugs remain unaddressed as far as the actual MG is concerned. These two things, along with other actions (such as updates CREATING security issues, breaking functioning scripts in an intentional manner, etc), have led to that decrease in confidence.

Without a deliberate and well-articulated preview testing process, without a detailed plan that is intended to do more than simply hope a bug is found, there seems to be little hope that LL's preview testing process is anywhere near what might euphemistically be called "adequate".

Personally, I think the preview grid should not be tested by "amatuers". I certainly don't think that it is the responsibility of **paying customers** to do so REGARDLESS of any qualifications they might have.

I've long conjectured that not only is this prototype, pre-beta quality code, each release solidifies that opinion and amplfies my belief that literally no one knows how the code works any longer. This is pretty typical of code that started as proof-of-concept and got hacked to pieces over three years.

Not only is it untestable, it is likely unfixable without playing code "whack-a-mole" where one fix breaks one or more other things.

For my sake, it is fortunate that I stopped building and creating after the 1.3 release which was my personal "break everything you've yet made" release. At present, SL is a multi-way chat with a DJ with friends that I would be loathe to part with.

I pity not only the people who try to work with releases this poor, but also the LL staff that has been charged with trying to maintain the unmaintainable.

So it goes.

They should get rid of the preview grid entirely.

They should learn to rollout a feature to a percentage of the user base, and slowly roll it out to to more as it proves to be solid.

It's a bit more complex, but it's a scaleable and very proven development / qa methodology.

Which MMORPG does this currently? (And I only mention RPGs because they run into the same technical issues we do so dont get into that "this is/isnt a game" BS)

It can't ever be an ideal test environment simply because it doesn't have the population, the traffic and the same scripts all running at the same time.

That's been the case with every MMOG game I've ever played. Does anyone actually and reasonablly expect a test environment for a system like SL to be adequate to catch all these types of problems?

See my post above about the 'English Lake District' Linden sims which were transfered to the Preview Grid while the main grid was down for update last week. 6 (SIX) AVs in a sim with the existing content were quite enough to prove to all concerned (except LL) that 1.7 was a seriously flawed. However the update went ahead regardless...

Your first post mentions visiting the Preview Grid "during the update process". I assume this was Monday when 1.7 was already being published? Seems a little late in the game to call it off. Also the preview grid is made up of subpar servers according to a Linden I spoke with the night before the update. In fact this Linden was going around deleting objects with active scripts just to keep things running. I'd almost guess that LL had a real test environment on more uptodate servers that didnt see this performance issue and that our 'preview grid' is piece-meal hardware designed for us to catch more glaring/visual bugs. So any 'performance' related feedback from us would be taken with a huge grain of salt from LL. Either way, lets hope LL finds out what let this patch out the door with these issues going undetected and explains it to us.

At this point explanation is irrelevant.
What they need to do is fix it.
And take steps to ensure it doesn't happen again.
Like it or not SL IS a business, and businesses that repeat errors of this magnitude don't survive in the real world.