Rolling Restart for 1.24.10, Dec 22 and 29-30

This thread is to discuss the 1.24.10 rolling restart.

Oh Prospero, and just how much support will there be over Christmas if 1.24.10 causes problems?

Reads announcement properly and sees it will only be Pilot Roll over Christmas. Bad luck for those on the Pilot Roll.

Give up on 1.25 then?

Update 12-16 02:24PM : We are postponing our next attempt at the server 1.25 deploy until after the holiday period, in early January. We will make another blog post then announcing the anticipated schedule. The current tentative dates are on the planned outages calendar.

I don't know where to post this.. But I am going to put it here and pray a wonderful Linden will see it....


For the past two days.. All of my IM's are getting sent to email, every time.
All Transactions are taking time to come through as well.

I don't know if there is a problem I have looked on the Grid status page and no word of this.

I hope its not me.

Thank you for your time

Could maybe you guys do the pilot roll after christmass?

Please elaborate on "This includes the server support code needed for the introduction of Homesteads scheduled for January 5."

/me guesses a first try at script limits, or at least instrumentation to measure bits that they're thinking about limiting. That and removing any full/openspace constants that were lurking in the code in favor of something configurable.

Update shortly before christmas. Very clever ... When will you ever learn ....

I've tried to stay silent and understand the constant delays on 1.25 ...

... but the 'breakage' of avatar collisions that occurred with server 1.22 and has continued for at least 6 months now ... has severely impacted myself and many others in a real financial sense. It's my understanding that a working fix has been waiting since before the 1.24 server-code implementation, but taken out of the initial 1.24 implementation as LL wanted to concentrate on Mono - fine.

... please though, I BEG you, implement the avatar-collision fix that is supposedly just waiting for distribution to the live grid with your 1.24.10 ...

Have script limitations been set for Homestead sims in this deploy or are they yet TBD?

Re: why we're doing 1.24 : Homesteads begin on January 5. We want to do the pilot roll more than a day before the full roll. Dec 24-25 is a holiday, as is Dec. 31-Jan 1. We avoid production changes on weekends and holidays. Put these things together, and we're highly constrained for when we can do server deploys and have them done by Monday, January 5. We basically have no choice but to do the pilot roll next week, and the full roll the week after.

We're still finding the database loading and "missing name" problems in 1.25-- we're well on the track of it now, but we don't have it yet. We yesterday that it was going to take more than yesterday to find and fix those problems, so we knew it was not possible to get 1.25 out with confidence... yet we needed to get the Homestead code out to meet the announced date of January 5. As such, we had to make the decision to put 1.25 on ice for a little while and get that other code out with 1.24. It's the only safe way to do it, since 1.24 is functional and on the production grid right now.

Some server-side changes were needed with homesteads to deal with the fact that we'll go from two types of regions (4 sim/cpu and 1 sim/cpu) to three (openspace, homestead, normal). We put in code to deal with different number of avatar limits that private estate owners could set in each type of region. (Previously, it was hardcoded to 100 for every type of region, which is part of the current openspace problem we have.) Script limit code will come later as we have better data on just what it should be.

We are not going to put *any* other changes in 1.24.10 because that would be risky. Yes, I know that some of you want the avatar collision bug fix. or any number of other things that are in 1.25. Believe me, I really, really, really wanted 1.25 to be happy and stable and on the grid before December 1. It didn't happen. I'm sorry. We're working on it. But, in the mean time, given that we're coming up on two weeks which are not only holiday weeks, but also weeks when more than the average number of people tend to take vacations, we have to take the minimum-risk path to getting the homestead code out by the announced date of January 5.

I do understand that LL wants to prepare code for offering the new Open Space/Homestead/Full Region tier structure ... but please understand what it looks like from a public relations standpoint:

"We're not going to fix any bugs that have been endemic and caused severe problems for users of our virtual environment, but we do have enough time to implement new code that will increase the pricing tier for many of our residents."

That is ... essentially ... what you're saying.

Of course, I'm going to try to be as positive about it as I can ... it's just that I'm struggling with how to phrase this further delay to the 3000 users of my system in a way that will keep them from rioting.


(BTW ... the missing-names issue has been problematic for me on 1.24.9 with RC viewer 1.22.3)

Re: the msising-names problem showing up with 1.24.9 -- it's because it's (almost certainly) cache related. The bug in 1.25 "poisoned" the system in various ways, so that even though the bug isn't there (we think) in 1.24.9, bad data left over from 1.25 being there causes it to persist.

We can't deploy 1.25 until we fix this, and finding it is very difficult. This is the real reason 1.25 is being delayed-- because the bugs in 1.25 are subtle enough that they're hard to see in the first place. Neither the database load nor the missing names problem were visible in any of our test environments, simply because the agni environment is bigger and more diverse than our development environments. We're trying to learn from these things each time we have them, so that we can improve our load testing processes before we hit the regular grid. It's not an easy problem to solve, or we would have done it already....

Yet more breakage because people couldn't frakking respect the limits on OS sims.

I can understand the difficulty of finding a small bug - I run into it often myself. And please understand that I'm not personally angry at LL - I'm just struggling with all of my customers, many of whom will actually blame me for the issue....hehehe (in the past I've even had a customer thank me for fixing the asset server???).

Just pointing out that delays like this really hurt those of us who try to provide services for others in the environment ... and that it's kind of a slap in the face to have the coding for, essentially, higher restrictions at greater cost implmented ... but not the fixes we were looking for. Likely would have looked better from a PR standpoint to put --all-- updates off until January 15th and scale the ratcheting up of pricing/restrictions for Open Space regions and Homesteads back by 1 month.

Pushing out new things to sell (or that are hoped to drive new sales) is prioritized over fixing things already sold.

SVC-2511 has been past qa testing for months according to your own representative. How hard is it to move it into 1.24.10 if it's tested and ready to go?

It also works great on the preview/beta grid.

So what other *crap* has LL Put into 1.25 that isn't working when they put it live that is not on the beta preview and is holding up this fix for something that we were told would be a high priority.

Most real software companies when they call a bug fix a high priority it gets rolled out next opportunity, it doesn't get pushed back repeatedly even after it's been tested and implemented successfully in the beta environment.

SVC-2511 (the collision bug that Kenn speaks about) is tested and done, or did Andrew Linden lie to us? Oh wait, we've tested it with our stuff IN the beta grid. Works great. So what is so hard putting that ONE Fix into 1.24.10?

It's tested and done and works fine.

It's the other crap you're trying to put in that doesn't work...

Stuff that probably wasn't put onto the beta grid, new stuff, not bug fixes.

Thanks for the explanations Prospero.

May we surmise from the quote above that homestead sims will operate 2 sim/cpu?

I just was part of a very hard conversation where we were deciding whether or not to put a very small, very scoped bug fix into 1.24.10 that would fix a problem in our infrastructure that is pretty glaring, and has been broken for a while. It's not something resident-facing that will affect anything you can see, but it does make life hard for our system infrastructure team, and does cost us (not insubstantial) money. We decided not to do this for all the reasons I gave you above, so that we maximize our chances of getting 1.24.10 out during the holiday season-- and this is a fix that we would have considered long before the fix for SVC-2511, because the problem is so glaring and the fix was so small and scoped.

It's not so simple as taking every fix that has passed QA and jamming them all together into the code that gets deployed. There's a difference between "tested" and "ready to go". We have to test the integration of everything. We have to verify that the changes, even when tested in isolation, don't interact in unanticipated ways that cause breakage. *Anything* added into the server software, no matter how innocuous, no matter whether or not it's been tested before, adds risk. We can't afford that risk right now, not these particular two weeks of the year.

No. I do not know what the eventual decision about the sim/cpu ratio for openspaces and homesteads will be. My point was that previously, all 1 sim/cpu regions fit one category (normal regions), and all 4 sim/cpu regions fit another (opensapce regions). Now we have three categories, and what's more those categories may not necessarily be distinguished by sim/cpu ratio.

Nothing that we've put on the production Second Life Environment has not first been on the Preview Grid, and has not first been through testing.

Yes, we don't always find all problems. But you're wrong to imply that we are doing cowboy releasing and sticking stuff willy-nilly on the production environment just for jollies.

Thanks for all the responses Prospero. Please understand that I'm not trying to sew dissention - just trying to indicate how important it is to all of us that a lot of these long-awaited bug fixes be implemented.

I have no doubt that all of you are working as hard as you can on the SL project and that everything is done in as responsible a manner as possible. I also know, however, that sometimes when I'm developing a project I'll not realize just how important a certain portion of it is to my customer base until someone mentions it. I'm just trying to mention that there are quite a few of us for whom SVC-2511 is a major/showstopper type issue and the repeated delays of it's implementation is really wearing.

Is it possible that some of these bugs (the bug you mentioned, SVC-2511, and maybe a few others) could be implemented ahead of 1.25.3 so that we don't have to worry again about whether the new 1.25 code will "stick" or not? (Obviously this ... 1.24.11 we'll call it ... would be after the Christmas holiday season)

There are a lot of things waiting to go out with 1.25, so once we get 1.24.10 out, our focus will go back to making 1.25 work -- hopefully we've found the problem we're looking for. One side effect of this is that we have managed to develop some load testing on test grids that will, with luck, help us avoid some of these problems in the future (fingers crossed). (Of course, SL is so big and diverse that it always finds new and surprising ways to go wrong.)

Quote'd =D