Rolling Restart for 1.24.6, 2008 Sep. 15-17

http://status.secondlifegrid.net/2008/09/15/post246/

As described in the blog post above, there will be a rolling restart this week to deploy a few more bug fixes to the 1.24 server.

Any discussion or questions can go in this forum. Please keep the discussion on topic, and avoid personal attacks.

I see the bug fixes are mostly cleanup for Mono. Anything being incorporated to stabilize the grid in this new version?

Such a waste of our time and resources for a couple of lousy clean up fixes.

Restarts weekly now? Can we have a break?

Those are both pretty serious bugs, Dilbert. I'm sure SVC-2973 breaks some content, too.

If they have serious fixes ready to deploy, I'd rather see them get deployed..

Once mono settles down and LL fixes all the low-hanging fruit, I expect they'll go back to the two week schedule.

Just a minor quibble, really, compared to some of the things...

But if you're going to say "go here to see release notes"....

Please UPDATE the release notes?

The release notes *are* updated. The reason they may not look updated is that the changes that are in 1.24.6 are the same as the changes that were in 1.24.5.96378... but that we ended up not deploying to the grid because 96378 was crashier than 96115.

Zena Juran : these two fixes ARE to stabilize the grid. One fixes broken content, the other fixes a crash bug that's lurking in there. We've been lucky and it hasn't been triggered much yet, but it's there and we know it, and thus we want to roll out the new release to fix it.

Re: schedules of restarts, we don't really have a one-week or two-week schedule. We do try to have at least a week of "laying fallow" time before a new second-number release... meaning that we're not planning on rolling out 1.25 next week. However, when we've done a second-number release, we will roll out patches with bug fixes as we find them and need to patch them, based on whether or not they're content-breaking or stability bugs. That's what we've been doing the last three weeks with 1.24.4, 5, and now 6. (Remember that the first roll of 1.24 that went to the whole grid was 1.24.3.)

you are doeing really a bad work...crashes..lags..region downs..continuous changes..but why dont u try to do a good service..and stop with all this 'inventions'.. iot's crazy!!

perservere, prospero and all the labbers - s'gotten much better already! must keep pushin' on... only the fearless can be truly great...

Keep the rolling restarts rollin'; we're looking forward to fixes that should be in 1.25 to fix stuff broken by fixes to Havok 4 in 1.22 and 1.23 that were held back by Mono/1.24, like SVC-2511 and SVC-2723.

Sooo, what about the -snip- exploit that has been going ever since the beginning? Is this already fixed or why have I never seen a mention of a security fix on the rolling restart mentions?

Nexii -- what script permissions issue are you talking about?

This 3 days of rolling restarts is way way worse than when you used to close down for a day, please test your code and do them in a day. I think if you did a survey you would find most people find this real bad, every time I see a chat about this in-world everyone complains for various reasons, surely you know this. Please make more robust releases and less of them.

In fact, we've received a lot of feedback from people that recognize that having your region down for 10 minutes during a restart is a lot *less* disruptive than having the entire grid down for several hours-- or more if, as often happens, problems are discovered only after the code goes to a Second Life-sized deploy.

I know that rolling restarts are disruptive, but they really are a lot less disruptive than global downtime.

Yes, a ten minute downtime in each region would not be terribly disruptive, but you must admit that that is not the rolling restart experience most of us have had. We're all a bit gunshy about the restarts, many of us lose business even when they go smoothly, and unfortunately too many of them do not go smoothly. Seeing them coming with such frequency is unsettling, literally.

Unfortunately people don't seem to be remembering when the Wednesday 8 hour grid upgrades we used to have instead, would sometimes stretch on to being multiday events. Nobody anywhere in world at all except for Lindens.

Don't think for a moment that your efforts aren't appreciated, Prospero.

And *abso-freaking-lutely*, region-by-region downtime is many orders of magnitude better than the old entire-grid downtime that used to be a regular occurrence. Vive le Het Grid!

If I were to make any sort of minor criticism, its in they way Linden prioritizes downtime: which often places engineering convenience above minimizing customer interruption. As an IT Manager myself though, I understand that's a philosophical decision between having your engineers at peak performance vs. customer satisfaction, and Linden has chosen a position somewhere in the middle.

The fact that I'm awake at this hour on US Eastern time, sitting in a command center for a go-live all night should be testament to which side of that philosophical fence my particular company sits on

Oh, oh, oh, I remember, pick me! Seriously though, things are definitely "better". HetGrid has helped a _lot_.

There's a rumour (emphasis here: RUMOUR) that there's an exploit to turn Mono scripts can be made full perms. I ignored it as I ship all scripts full perms anyway. Version I heard said the last few sim patches were also trying to fix it, and failed, so if you haven't heard about this odds on it is just rumour.

Ready for another upgrade, keep the good work, cleaning and heading perfection of SL!

Seconded!

OK, here's the deal re: the rumored permissions exploit with Mono:

One can recompile No-Mod scripts to run under Mono. Under the old scripting engine, compilation happened on the viewer. This means that the source code has to go down to the viewer. Thus, in principle, under the old scripting engine, recompilation means that somebody using a hack or a customized viewer could grab the source code to no-mod scripts.

However, with Mono, all script compilation happens server side. Some metadata *is* sent from the viewer to the server, but it's not the bytecode of the script itself. The source code for the script is not sent to the viewer. There was a misunderstanding that led some to believe that the ability to recompile scripts meant that script source code could be leaked, but in fact that's not the case.

So, no, there's no permissions exploit with Mono scripts, even though there was a SEC issue filed by a resident based on this misunderstanding. If you hear the rumor, please let them know the actual situation.

(Of course, if you do find another exploit, and can reproduce it (by grabbing actual no-mod source code), please email [email]security@lindenlab.com[/email]!)

UPDATE 2008-09-18 11:17 : clarifying something above. "Thus, in principle, under the old scripting engine, recompilation means that somebody using a hack or a customized viewer could grab the source code to no-mod scripts" **if recompilation of no-mod scripts were allowed**. There is no problem with the source code of no-mod scripts being able to be stolen. Under the old scripting engine, recompilation was not allowed for no-mod scripts. Under Mono, compilation happens server side.

I am "old" enough to recall those Wednesdays when the house finally got cleaned, the dog got walked and the children fed........please let's not go back there!

The Het grid may have its flaws, with added rolling restarts, but it's a huge improvement on NO Sl.

Good luck this time.

Thats really nice all this, but...
why dont fix the platform? i cant log in, and when
finally (after 5-10 mins, aprox.) , my avatar cant move.

Why??

Yeah.. They broke the "fix it" button a couple releases ago.

>In fact, we've received a lot of feedback from people that recognize that having your region down for 10 minutes during a restart is a lot *less* disruptive than having the entire grid down for several hours-- or more if, as often happens, problems are discovered only after the code goes to a Second Life-sized deploy.

>I know that rolling restarts are disruptive, but they really are a lot less disruptive than global downtime.


It is of course much better to do the rolling restarts this way then to close down the grid as we used to, That doesnt mean there isnt a better/more productive/less disruptive, way to manage bugfixes.

I continue to point out that, despite LL's comments to the contrary, the enconomy suffers horrendously during these agressively scheduled weekly/biweekly rollouts. My stores, all 7 of them, are showing 25-35% reductions in sales over the 3 days these rollouts occur (or 7-10 days depending on how many times we end up rolling back the changes).

Whether that is because people walk away once the sim they are in reboots and don't come back, or because there is a load on the infrastructure that is not being accounted for, is unimportant. The effect is the same. I can, and have, correlated most of my "bad days" in the stores with your rollout schedule.

A more thoughtful rollout schedule is whats required, with more debugging, over a longer period of time, during the intial phases, to a larger and more representative selection of regions.

The timer bug you are fixing should have been found and fixed before a complete rollout of the broken code, and rolled out in stages again, the first time. Of course you can't do that if you roll it out to 30000 regions within 3 days with no time for feedback to occur.

Today will cost me personally about $50-60, tomorrow about $60-80, another $60-80 or so the day after, if things go smoothly.

I'm not the only one who runs a business in second life, so multiply that loss by several thousands of us that are paying tiers to do so. Then recognize that non-commercial activity is equally affected whether or not money is involved.

egads, this is frustrating.