Rolling restarts

This isn't intended as a destructive criticism - more a plea to not do it this way again and why.

We all accept that SL has to be taken for updates, emergencies and the like, but this cycle of rolling restarts has been really, really disruptive to us, the residents.

One minute's notice to quit is a nightmare if you're scripting, mid build etc. It's pretty bad if you're in a meeting too. The "ten minutes then we're closing down for X time" and closing log-ins at the same time is easier to cope with. Ten minutes is enough to stop what you're doing and tidy up, even if not enough to necessarily record your precise train of thought at the time.

Rolling restarts aren't "clear" - if I fly away in some given direction, or tp away, will arrive in a sim that's already restarted, or one that's restarting now, or one that's restarting in a minute's time?

If there were no residents around how long would these restarts have taken? A few hours? So why not slide them into the Wednesday update cycle, or claim an emergency shut down if it's really that urgent and take the grid down last Thursday for a few hours?

No SL for 6 hours isn't fun, but to my mind, and I suspect from what I'm seeing in other fora and hearing from other people, it's LESS disruptive than this rolling restart programme, we know we're out for 6 hours or so, but at least we know a start and end time clearly.

And whilst we're at, this is pretty much idly curiosity I must admit - why so many restarts? Pull the server, switch up, restart.... But every server on each of four days seems a little OTT from the outside.

There have indeed been four rolling restarts in four days.

Two have been rolling updates: last Thursday (1.9.0.22) and today, Tuesday (1.9.0.23). These were necessary to push new code out to the grid -- in both cases, we deployed important fixes that were required immediately.

The other two restarts, on Friday and Monday, were for Operations work. We are in the process of moving some old hardware out of the grid. The timing for these moves could not be avoided due to scheduling demands. I must say that this sort of work is rare. It should be completed early next week.

I am sorry for the disruption to the Second Life community. It is not our intent to cause anyone to have a bad experience. Quite the contrary -- we are working hard to improve every aspect of SL! Unfortunately, we cannot always choose the perfect time for maintenance, code deploys, and other issues to occur.

I look forward to the current Operations work being completed, and for software updates to return to the regular Wednesday time slot. Again, I anticipate this will happen by the middle of next week.

We've had two basic kinds of region restarts over the past few weeks. The rolling restarts, usually handled by Karen, affect all regions and are related to software updates. These happen when necessary to install new software on the sims. The partial region restarts, which are the ones I've posted about in the forums, have occurred in blocks and have been necessary to accommodate hardware upgrades. We're going to have one more small round of these, probably next week. If you care about the technical reasons for the partial region restarts, keep reading. If you just want to know when the next region restart will occur, check back in the forums early next week.

Onto the (slightly) more technical explanation. The grid that powers Second Life is comprised of well over 1000 machines. These machines are divided into classes, where the oldest machines are "class 2" and the newest machines are "class 4". Class 2 machines consume more power and generate much more heat than the class 4 machines. Cooling the bank of class 2 machines has been a ongoing challenge. Sims running on class 2 machines also tend to be slower than their class 4 counterparts. This week, we have completed moving all class 2 regions onto class 4 machines. The move both helps the power/heat issue and provides residents with better sim performance.

While we're retiring the class 2 machines, we're also consolidating some of our hosting footprint. As Second Life has grown, the grid machines have grown out of their original home. To minimize the number of region restarts (and inconvenience to the residents), we're moving machines within the hosting facility into two (instead of four) areas while we also phase out the old hardware. This gives us space to expand as the world continues to grow.

Sorry for the inconvenience caused by these partial region restarts. We've worked to try to minimize the impact, and the end result is a better-performing grid.

~~b