Rolling Restart for 1.24.10, Dec 22 and 29-30

New *and* surprising?

I guess one out of two isn't bad.

A system as large as SL can be severely set back and/or crashed if 1.25 isn't fully ready for the transfer. I don't know what all that might include for SL but I have seen so many programs (entertainment, economic, medical, ect.) that were put out before proper testing was finished. In the medical profession that could mean someone's death, but even here we are talking about peoples time and money. Not just the overtime that SL would have to pay out to the Dev crew to fix or roll back the problem but time of people who want to be in SL and the lost sales of people who have an on going business with-in SL.

As for me, I don't make money off SL and if SL is down for a day or two it would not bother me a bit. But it sounds like Prospero and crew are doing the right thing for SL and its community.

Prospero, I look forward to 1.25 when it is ready. =)

Lillith

What i am seeing here is that the pilot roll out program is working, it is catching bugs that were not found in qa. This is a very good thing and it is why we volunteered our regions for the program. Sure i would like to see 1.25, but it is better that it be right. Having worked for the last 11 years on a program that could not have down time, we literally changed out the core of it while the system was being used, i know how hard it is to do what Linden Labs are doing. And having to meet advertised deadlines is a must, so i can understand why they are doing a "simple" roll out.

There is no need to flame the messenger.

Um, please point me in the direction of where any limit other than 3750 prims is located. I can't find it.

You were not supposed to use them for events or heavy scripting, unless you happen to be into sailing, in which case it's OK to use them for boat races (but, presumably, not other sailing-related events) that involve bringing lots of people in the sim and require heavy physics and script use.

we can whine and moan all about having to wait until january, i hate to say
it but we may be waiting longer until we even get 1.25, its just a matter of
time b4 something else goes wrong b4 it is completely fixed, but i have to
admit one thing, its kinda better for us to wait longer and have a stable
product than to rush it and have a crappy product,

i would have been a complainer too but in the past year that i have been
scripting, i have ran into a fare share of script errors that seem to occur
one after the other and often times i have thought that one problem was
solved when another "pops up" and i know how frustrating it is to think that
everything had been fixed only to run it and another error shows up, its the
same with the server code and i can understand their pain, i mean don't
get me wrong or any thing, i want the 1.25 server as bad as the next person,
it effects me and other SL weapon makers and maybe more but like i said
before, i rather they have everything fixed before they attempt to release
it again on the main grid.

People who "can't find" a limit unless it's impossible to violate are why it's necessary to have a 1.24.10.

That's cute but you're actually helping me to make my point Maggie. Some of us are not pushing any limits that we know of or can find. And the tools we are given in Region/Estate tabs and in the "view stats" bar often give conflicting information as regards script time and other stats. Being responsible landowners and concerned about performance for our visitors and eventually renters (thanks to the recent price increases we don't have any yet - increases were announced just after we "opened" up our regions", we check them often and have been very careful about object and script load.

And if there are indeed limits beyond 3750 prims and the default agent limit of 40, why should they be buried somewhere in the knowledge base--information about open spaces is item number 72, for example >Land>Information for Landowners. Advising people not to live on them or rent them is a far cry from a TOS limit or restriction. I'm not saying there are not residents out there who have abused the system to make huge profits, but get real! There are bigger management and communication issues at play here.

And the fact of the matter is that LL HAS responded to performance issues on my Open Spaces and even visited them. They've had every opportunity to say - "you're not using the regions as they were intended, we can't help you." When support suggested we "undo" a lot of work that we had already completed to troubleshoot, we complied and the problem persisted. The performance issues, in the end, turned out to be server side and not due to any object or script load we were causing.

So, thanks for your help in pointing me in a direction where I can get some useful information on how to be an even more responsible land owner and resident. I truly appreciate it.

Peace, Keo

"How dare the landlord charge me extra to repair the holes I made in the wall! If he doesn't want holes, the walls should be nail-proof. I don't care what's 'buried' in the lease.

"And another thing! My lights are dimming, and that's the landlord's fault too. He should make allowances for all those extension cords the neighbors ran over here to run the amps in their club. "

I suppose "server side problems" include abuse in the other three OS sims in your server. Well, LL will be nailing down more things so they don't get "lost"...meanwhile we continue to wait for months-old physics fixes. Enjoy your cut-rate, low-calorie sim.

Don't rant at me Maggie! You still have yet to show me the lease where it is spelled out. Perhaps then I will understand better. But not only are the restrictions and limits not buried, they are non-existent in ANY documentation. If they do exist, I ask again, point me in that direction and perhaps I'll better understand Linden Lab's action re: the new policies. And perhaps I will be better equipped to avoid doing anything that will cause the landlord any grief. (Your analogy, by the way, isn't a particularly good one until you can produce this lease, contract or Terms of Service. I do read the fine print. But why should I pay for the holes that someone else put in their walls).

And furthermore, it wasn't my decision for LL to move forward with only revisions and server deploys that don't fix the physics problems ahead of the coding that will serve their new policies. They made that decision on their own.

I have explained that we are doing everything we can given the tools we have to be certain we aren't causing any undo burden on the system. I'm looking for MORE help to further that goal.

And as for enjoying my "cut-rate, low-calorie sim," thank you, I will, right along with the full-rate, high-calorie one I was required to buy in order to get the cut-rate, low calorie ones I also enjoy.

Peace, Keo

They were in the blog, when they were announced. They were contradictory (or else LL had no idea what kind of performance sailing needs) but they were there.

The technical name for this is regression testing.

Strictly speaking you should be change testing everything in the packaged deployment first before you go into regression.

Because of the unstable nature of your test environments (I'll bet limbs that they bear no relation to the current production code) testing things in isolation does nothing much except identify major errors. Until you run it on the proposed production environment you won't see what breaks and since you don't do that you have to wait until it is deployed to see what it really looks like.. Ahhh the joys of agile development.

Since you guys don't document enough or understand the code enough to be able to identify what dependencies/changes there are between the prod version and the final version to be deployed, you're handicapped in ways that make this the equivalent of a whitewater rafting ride from hell. even config mgt doesn't really help here.. as the incremental/iterative nature of your development generates risks that are hard to mitigate with the tools you apparently have available for testing.

There are things you can do to improve your testing process outcomes but to be truly effective they also require changes to the approach your developers take. Your testing has always been a poor relation for your dev teams and it does show, although you have improved by a factor of many in the last couple of years.

Good call for reducing the scope of the deployment to the bare minimum, good luck with the deployment and I do hope you have a solid business contingency plan for the opensims

Please don't feed the trolls.

Maggie, Keokipele, Argent : please stop the discussion over openspace limits. That's not on-topic for this thread, and it's not something that I can really talk about. Yes, we're putting out 1.24.10 and delaying 1.25 because of the announced Homestead transition. But that was a decision of the company, and therefore a constraint place on those of us who are responsible for engineering the releases of Second Life. It was the call of me and a few others (with broad agreement) to delay 1.25 given that we had to get homesteads out by January 5, and I can defend that decision given that constraint. However, I cannot speak to the January 5 decision myself, one way or the other, nor can I speak to support of openspaces. Please. Keep it on topic.

Marie : you are wrong on a couple of things. First of all:



In fact, we run *exactly* the same code in the test environments as we are proposing to run on the development environment. We cannot deploy to the production Second Life environment until QA signs off on the code. Where the environments differ is that we do not (and can not) have a production environment that has 5,000 sim nodes and 50,000+ users online. Load testing is hard. We're slowly getting better at it, but it's hard. Additionally, Second Life is extremely diverse, and extremely flexible. How many other MMOs of any stripe give you as powerful of a scripting language to let you do things that the designers never anticipated in the first place? Nothing anywhere near the size of Second Life. This means that not only load testing, but sometimes "edge cases"-- things that aren't caught in our tests, or things that would have been caught but happen rarely enough that you need tens of thousands of residents to see them-- make it to the production environment before we find out about them.

As such, it's a gigantic effort to fully regression test absolutely everything that there is to test with each release. And, while, yes, I wish we could do that, the simple fact is that we don't have the QA resources to do that. If we were to insist on a literal full regression test, we would not ever be able to deploy a server again. This would mean that, for example, all the various fixes people really want to see that are in 1.25, would never get out. The fixes and improvements we've made in the past would never have gotten out. We have to do the best we can given the resources that we have.

Maggie, Keokipele, Argent : please stop the discussion over openspace limits. That's not on-topic for this thread, and it's not something that I can really talk about. Yes, we're putting out 1.24.10 and delaying 1.25 because of the announced Homestead transition. But that was a decision of the company, and therefore a constraint place on those of us who are responsible for engineering the releases of Second Life. It was the call of me and a few others (with broad agreement) to delay 1.25 given that we had to get homesteads out by January 5, and I can defend that decision given that constraint. However, I cannot speak to the January 5 decision myself, one way or the other, nor can I speak to support of openspaces. Please. Keep it on topic.

Marie : you are wrong on a couple of things. First of all:



In fact, we run *exactly* the same code in the test environments as we are proposing to run on the development environment. We cannot deploy to the production Second Life environment until QA signs off on the code. Where the environments differ is that we do not (and can not) have a test environment that has 5,000 sim nodes and 50,000+ users online. Load testing is hard. We're slowly getting better at it, but it's hard. Additionally, Second Life is extremely diverse, and extremely flexible. How many other MMOs of any stripe give you as powerful of a scripting language to let you do things that the designers never anticipated in the first place? Nothing anywhere near the size of Second Life. This means that not only load testing, but sometimes "edge cases"-- things that aren't caught in our tests, or things that would have been caught but happen rarely enough that you need tens of thousands of residents to see them-- make it to the production environment before we find out about them.

As such, it's a gigantic effort to fully regression test absolutely everything that there is to test with each release. And, while, yes, I wish we could do that, the simple fact is that we don't have the QA resources to do that. If we were to insist on a literal full regression test, we would not ever be able to deploy a server again. This would mean that, for example, all the various fixes people really want to see that are in 1.25, would never get out. The fixes and improvements we've made in the past would never have gotten out. We have to do the best we can given the resources that we have.

@ ARGENT . . . thanks for the hint. I'll look at the blog archives to see if I can find what I'm looking for and I reread your previous post re: sailing. Sorry, I thought you were being sarcastic. Yes, indeed, the information is contradictory and vague and that is my whole point. Residents who truly want to be responsible and not do anything that is against terms of service or any rules are honestly between a rock and a hard place when it comes to deciphering exactly what those are.

@ RALPH . . . I looked up the definition of trolling and quote it here:

Quote: Trolling (a post with an intentionally contrary opinion written with the intent of inciting or getting argumentative opinions)/Quote

Perhaps I've mistaken your response as being directed at me . . . perhaps not. To be clear, I am honestly and sincerely asking for information and so far have only gotten responses that are as vague as what I can find in the knowledge base articles.

@ PROSPERO . . . you have stated that 1.24.10 is to implement agent limits and deal with the new Homestead product as a third type of region. You also stated that "Script limit code will come later as we have better data on just what it should be." Is there a time frame established for collecting the needed data? Has there been any plan discussed as to how much time Estate owners will have to "rein in" scripts if they are not in compliance with the new limits before the limits are actually implemented? Not being all that tech savvy, I am concerned about what happens on a technical/performance level if a limit is implemented and I am over that limit. Mind you, I don't believe we are in any danger of that given our current readings of the stats bar and Estate script debug tool, but from what I understand in these forums and from other sources, those tools may not be all that accurate (and so as not to be accused of trolling or flaming, no, I have no idea how reliable those sources may be, that's why I'm asking).

Sorry, Prospero . . . apparently we were writing at the same time and I submitted to find your request above already posted. If my final questions in my post are not on topic, I apologize. No more discussion from me over openspace limits . . . Peace, Keo

@Lindens

Have they fixed the script related region crash bug that was introduced in the last pilot roll out of 1.25?

The region Quantum was crashing all the time due to this bug. The bug was not present in any of the previous pilot rollouts of 1.25, so it was something new to that last deployment.

The crashing was persistant and agressive when avatars entered the region. There is no way I could use the region while that particular crash is not fixed.

Darling Brody

Darling -- what's the JIRA on the bug you're talking about?

SVC-3519 is the JIRA I made fro Andrew.

https://support.secondlife.com/ics/support/default.asp?deptID=4417&task=knowledge&questionID=5650

Try that link Keokipele.

I just played around in a very busy sandbox running 1.24.10. Built a house, textured it, boxed it, chased some newbies around....everything looks good so far =)

i try and try and cant log in!
and that since 3 days ago..and now i tried for 6 hours and it still doesnt work

Rafaela,
Probably not the right place...and we will need way more information.

Restart everything, routers too. Uninstall, including hidden files, and fresh install the viewer. Firewall?...need more info...

I'm on an Open Space sim and I'm seeing a huge degradation in performance with the 1.24.10. Are you already imposing script limitations Prospero? If not, something has definitely changed!