Rolling Restart for Server 1.30

Did you check scripts on your estate to see if they were the problem? Did your guests come loaded down with scripts and scripted attachments that contributed to the lag? The practical top load for a sim has been, and continues to be regardless how high you set your estate limit, 50 avatars ... fewer in areas with a lot of shops with poorly written scripts. Toss in a lot of dancing and poseballs and I can see how 45 avatars would lag a sim. Model bots and camped alts also contribute to consuming a sims resources depending on how they're set up. That said I've been at events on many sims, not yours, with 45 avatars and had 20-30 fps steady except when someone TP's into the sim.

Good luck tracking down your lag, because I don't see you or any of us getting any resolution until LL upgrades to cloud servers or from Class 5 to Class Unfreakingbelieveable servers.

It doesn't seem practical to provide an ongoing % of completion during those two days, since we're usually pretty busy making sure that each and every region comes back online after the upgrade. I'd suggest trying to avoid scheduled events at the beginning of the day (~8am Pacific time), because that's when the restarts usually happen.

Rolling Restart, Friday 09/18 8-11AM PDT
Posted by Status Desk on September 17th, 2009 at 04:35 pm PDT
We are having a small additional pilot rolling restart which will follow this schedule:

»Fri. 09/18, 8-11AM PDT : a pilot roll of ~1600 regions

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions will be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

[RESOLVED] Crash issues after rolling restart
Posted by Status Desk on September 17th, 2009 at 02:13 pm PDT
[Sep 17 4:26pm PDT] Our engineers have located an issue, and a fix has been deployed to the grid. We’ve seen a sginificant reduction in region crashes. Please contact support if you are having further crash problems.

[Sep 17 2:16pm PDT] We are currently experiencing a higher-than-normal crash rate after our rolling restart. We are aware of the issue and are working to resolve it. Please check back frequently with the blog for further updates.

There are at least two approaches to scheduling releases that we could use. One is feature-based ("our next release will have features X, Y, and Z", and the other is time-based ("every N weeks, we'll release what's been implemented/fixed and tested so far".

The problem with the first approach is that delays in feature Z mean that features X and Y, which could be ready to deploy, don't get out onto the grid in a timely fashion.

We are using the second approach. We tell all the developers when we'll be doing releases, and if they finish their code and it gets tested in time to make it into the next release, then it can go in. If not, then the code waits until the next release following the completion of that testing. This gets features like X and Y onto the grid at a somewhat regular interval without concern over the fact that feature Z might be one or two months behind. This does mean that the feature list for a release will be more variable than if we were taking the first approach and releasing specifically to get features X, Y, and Z onto the grid.

Are these 2 related and if so what will be the new version number or was the increased crash rate a misconfiguration that needs a new restart?
If unrelated what is the aditional fix being deployed and what version number?

It does seem that "scheduled" releases take more than one attempt to make it to the grid nowadays.

Is the second approach really working? Are you getting quality features out in a timely manner?

can we at least know on what day the odd and even hosts get rolled and in what order?
also can someone confirm that sims are locked to their host between the start of the pilot roll and the end of the final roll?

Yes! Even sims will be rolled first, on Wednesday, and odd sims on Thursday.

During all rolling restarts we do indeed lock all sims to their hosts.

< prospero>
...but until LL does lock the sim to the host it's on, your even/oddness has a 50% chance of changing if the sim restarts. Beware relying on the host number not changing.
< /prospero>

Now you are sounding like LL support... in fact like all support.. its always the customer that is at fault. Truth of the matter is, SL is failing, suffering cascade failures due to the strain of forced expansion it was not designed to handle.

"Log in 40 dancing blingtards via random internet routes (ie not from a rack in the same server room) on a region with 6000 scripts".

Well we got 1.30 yesterday as part of the 1st pilot roll and... EVERYONE has noticed a significant different.. FOR THE BETTER. Still getting a pause when someone enters the sim, but it seems the sim recovers faster (1-3 seconds, vs 10-15). Doesn't SEEM to be degrading over time.. even sim crossings (we have 2 right next to each other, both that got 1.30) before I would literally 'ghost' 1/2 into the next sim before snapping back when crossing.. now.. on 4 out of the 10 tests I did, I just had a small pause going across, the rest I rubber-banded no more than 10m worst case.

Someone did something right?

Well we got 1.30 yesterday as part of the 1st pilot roll and... EVERYONE has noticed a significant different.. FOR THE BETTER. Still getting a pause when someone enters the sim, but it seems the sim recovers faster (1-3 seconds, vs 10-15). Doesn't SEEM to be degrading over time.. even sim crossings (we have 2 right next to each other, both that got 1.30) before I would literally 'ghost' 1/2 into the next sim before snapping back when crossing.. now.. on 4 out of the 10 tests I did, I just had a small pause going across, the rest I rubber-banded no more than 10m worst case.

Someone did something right?

Hey all, in case you missed it, the restarts were pushed closer, to tonight and tomorrow morning, thanks to some fun security fix.

http://status.secondlifegrid.net/2009/09/17/post734/

Thank you Lil

Are you doing just the pilot sims?

We've done the pilot sims, plus a good chunk of the rest. Tomorrow we'll have about 3700 sims to do, mostly odds and the rest of the evens.

Aaaaaannd: we're 66% done. I'll update again when we're finished for the night.