LSL Automated Regression Testing Suite

Would any other SL scripters be interested in helping build an LSL automated regression testing suite?

There seems to be alot of concern with undocumented changes to SL breaking scripts with each release. A regression testing suite could be used in future release Preview sims, or upon new SL releases, to validate the LSL scripting platform and expose potential problems.

We could perform alot of the regression tests with in-world scripts and predefined prim objects. Some things may not be capable of being tested in an automated fashion but the majority of the LSL could be.

A large number of tests would be required to really make this worthwhile, given the plethora of library calls provided by LSL and various language capabilities (loops, states, etc). This would best be performed as a collaborative effort. Would anyone be interested in helping contribute to such an effort?

An added benefit of regression tests is that it may help uncover some yet unknown bugs in the scripting platform, and would provide an easy way for the Lindens to reproduce scripting bugs. Reproducable bugs are much more easily fixed.

A more stable scripting platform will attract more developers, ensure the usability of existing scripts, and improve SL as a whole.

Making a valid regression suite will become harder as time goes on.

The problem is that there are more and more sims and scripts and only one version of SL can run at any one time.

One solution to the problem could be to allow the possibility of running multiple SL versions at the same time.

As to how one does that, there are multiple possibilities, and each one has its fair share of advantages and challenges.

Nevertheless, the hypothetical day when there are 5000 sim servers, are we really going to continue to see a one-shot upgrade of all servers and clients across a single shard at one time?

Without access to the underlying architecture I cant go into detail, but in my experience there is generally a way to do phased rollouts for any upgrade. Takes a little extra initial work, but reduces the risk massively.

Azelda

A question from a self-taught, and still learning, programmer:
What is a regression test?

A regression test is a piece of code that has a known output. When testing a new revision of LSL you would run the code, compare the output with the expected and if they differ you know that something broke.

In languages such as perl, regression tests are relatively easy to code because the language has sufficient power to be able to compare the expected result with the actual result. I'm kinda tired now, but some part of my brain thinks this difficult with the LSL API because there are many ways to change the world but far less to read the resultant state back. But I may disagree with myself on this come morning.

An interesting idea, but I think most bugs require human involvement to notice.

For instance, the last revision of SL introduced 2 bugs to my knowledge, namely sounds and particles.

Neither of these two things would be caught by any automated regression suite.

I think the most effective "regression suite" is just to do what we've always done - file bug reports.

Ah, interesting. I can see many things that could've been seen sooner had there been a regression test. One obvious example would be the llGetDate change back in 1.1. It wasn't documented that it went from a MM/DD/YYYY format to a YYYY-DD-MM format.

==Chris

You mean YYYY-MM-DD, right Chris? That's what the documentation says and what it outputs.

I sincerely doubt they would switch from MM-DD-YYYY to an equally stupid format. YYYY-MM-DD is really the only way of writing dates that makes sense. I mean, we don't say "Yeah, SL was just upgraded to 4.3.1," do we?

Look here:
http://www.junit.org/
http://www.nunit.org/

*Edit: Hey, aren't you supposed to be teaching yourself SQL right now?

Gah, SQL suxors, I cant fathom anyone wilfully learning it, I thought the whole point of college was to force down your throat every horribly boring thing no one sane would ever want to learn on their own

Linden Labs had a preview of 1.4 available before 1.4 was released. I am sure that they have internal sims that exist for the purpose of testing new versions. Perhaps they could be convinced to give us a chance to preview 1.4.x releases as well.

Quite frankly, the Lindens are the ones who should be developing a regression suite. I wouldn't be surprised to find out they already have something that they use internally. But given that they probably wouldn't make that available to us, I could definately see the benefit of developing our own set for our use.

Pfft

I'm a database programmer, so sue me.