Rolling Restart for 1.27.2

UPDATE @ 2009-08-07 4:30PM: We finished cleaning up from the RR at about 4PM, which, as you might discern from that time, did not go smoothly. I'll put together a description of what went wrong and find the right place to post it.

UPDATE @ 2009-08-07 07:02AM: We're beginning today's rolling restart.

UPDATE @ 2009-08-06 6:40PM: We've completed today's half-grid rolling restart.

I apologize for the late notice, but in order to resolve a security issue, we will be deploying server version 1.27.2 to Second Life starting this afternoon and finishing tomorrow morning. The rolling restart schedule is:

* Thu. 08/06, 2:15PM-8PM PDT: half of Second Life will be restarted
* Fri. 08/07, 7AM-12PM PDT: the remainder of Second Life will be restarted

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

You can see the updated release notes here:

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/1.27

is it related to the so many sim crashes that we have to support from griefing attacks ?

Resolved a security issue. Scary!

Can we get the % done each restart please?

Dante -

Thanks for posting the release notes, my thanks and i'm sure thanks from many noobs who can dig a little bit deeper into the general mystery (to non-technical/new users) which surrounds SL updates.

Cheers

Tristin, I'm not sure what you were asking. Were you wanting to know how much of our half-grid rolling restart we had completed at the time, or what percentage we were going to complete today?

I'm not sure what percentage of today's roll we had completed when you asked, but half the grid is running 1.27.2 now.

RR's used to include 4 or 5 "we're now x% done with what we're doing today"'s.

If LLs estimates on how long a rolling restart is going to take have gotten really good, I guess people could probably figure it out for themselves. Still nice to get a progress poke every now and then, though.

Ya what % is done for each restart day.
Like 25% ect
Just some updates so we can kinda figger out how much longer it might take

Especially when you do a short notice restart like this one, it means people who had planned events or meetings will get caught up short. If you are courting the corporate and educational market like the higher ups say you are, some better info is highly desirable. For example:

"We are restarting simhosts numbered 1 to 3000 today, in numerical order", or
something similar.

The incremental progress updates during a given day is useful info not just on that day, but as a gage on how long it will take the next time you do it.

Sure, in a live environment like this, you cannot guarantee strict following of any given plan. But with info like above, we can get an idea like:

"Hmm, my simhost is 2469, and they do about 1000 an hour, so my restart will be about 2.5 hours after whenever they start, better reschedule that 4pm meeting"

Does this include the fixes for: http://jira.secondlife.com/browse/VWR-6110 and http://jira.secondlife.com/browse/VWR-12525

Andrew Linden found a server side fix for these and thought they would go into the next update which would have been 1.27.2 or 1.30, but I don't see mention of it in the notes.

Has it started yet?

Glad to see the single failed teleport trottle got fixed, luckely only got hit by it once but was annoying non the less

Today, we're doing the remainder of the grid, which should mostly be odd-numbered hosts. We start low and move high. I'll try to remember to make a few status updates here as we progress.

Let me re-state that. We usually start low and move high. Things are not being done in that order today, and I apologize for making this process less deterministic for you all.

"Each region will receive warnings starting 5 minutes before that region is restarted."

Wrong, I got absolutely zero notice prior to the sim going down. Is it any wonder that the users of SL have little confidence in the work you Lindens do when you can't even get the simplest things done right.

goes along with both my regions have shut down today and only one is back up 1 hour later but then thats getting to be every time now too

Dante this is out of the blue but every server rollout for the last 5 has caused that particular region that's still down to be down for 1 hour or longer, maybe something to look at for general reference

You sure it wasn't a sim crash?

well, funny thing is: I got this blue pop up
[8:57] Second Life: Your region will restart in 5 minutes and 0 seconds for maintenance.

I left the sim and parked my avatar at the neigbouring sim which has been restarted yesterday and is already running the new server version. I usually watch as the sim goes off and wait for it to come back online, but today nothing happend, although I waited a long time (one hour).

I checked the server version of the neighbouring sim before going back to my parcels:
[10:02] Anjouli Troncon: You are at 170463.8, 247347.2, 37.4 in N(...) located at sim7807.agni.lindenlab.com (216.82.36.46:13005)
Second Life Server 1.27.2.129782

crossing simborders:
[10:03] The region you have entered is running a different simulator version. Click this message for details.

[10:03] Anjouli Troncon: You are at 170719.5, 247362.5, 22.7 in L(...) located at sim8457.agni.lindenlab.com (216.82.39.16:13000)
Second Life Server 1.27.1.128449

hmm ... how long are five minutes?

Can we get a % of whats done so far ?

So I'm guessing its not done yet?
Why is todays taking so long? Last nites was really fast.

The fault from the 1.27.1 where searching in the categories under Places search gave "None found" as result, is back!! Please, you fixed it once - can you please fix it again soon? It really hurts business

A number of things went wrong during this rolling restart, including that some regions were not warned in advance. I apologize for that. We certainly know what caused it, and it shouldn't happen again (at least, not for that same reason).

That should be fixed (again).

Thanksies (again)!