Rolling Restart to deploy 1.24.5, 2008/09/10-12

We will be doing a rolling restart to deploy server version 1.24.5 this week. More information may be found on the status blog here:

http://status.secondlifegrid.net/2008/09/10/post240/

I know debugging code is not an easy task, but may I suggest something? Hire a newspaper or magazine editor - someone used to finding minor punctuation errors on a deadline - to find and fix syntax errors. He or she doesn't need to know anything about coding as long as a copy of what the code _should_ look like is available.

the compiler already checks that it "looks like" proper code. the devil is in the details of making the code do what you intended, instead of what you wrote.

The Second Life Planned Outages Calendar states 8pm start tonight.

The Second Life Grid Status Reports states starting at 9pm tonight.

Can we have clarification.



T

"We are currently experiencing problems with our Second Life, it is not allowing logins at this time. We are currently looking into this problem, and will keep you updated .... Please refrain from making purchases or rezzing valuable/no copy items. Please monitor this post for updates."

If there's something strange
in the neighbourhood
who you gonna call
ghostbusters

If there's something weird
and it don't look good
who you gonna call
ghostbusters
I'm afraid of no ghost
I'm afraid of no ghost.

i cant view pictures or snapshots that i rez without having to load them in inventory

Can someone help me with this problem

My Sl Name is Revan Contepomi

10-20 minutes, before new login?

i try it now near an hour... great

The deployment obviously did not go well since log-ins were disabled. I can accept that and I am pleased that at at leased the grid staus page acknowledged the fact, although the information as to why is pretty lame and is more a whitewash than anything else. What would be nice is the link that leads to a fuller explanation actually lead to an explanation, preferably with a timetable if possible. There is nothing worse than being kept in the dark unless you are a mushroom, or possibly a vampire. I am neither and I really do like to provided with accurate information. I tm akes me feel like I am really a person and not a fungus.

wow. lol..i didn't recieve warnings before getting kicked off..and hardly could log back in guess this is the rolling restart

yup me neither
just got kicked out

Whatever happened to you guys, it wasn't the rolling restart, as the rolling restart has just begun, and no hosts have yet been restarted as a result of it.

The problems we had before (an hour or two ago) were not due to the rolling restart. A technician in our datacenter hooked up a cable, and somehow a network loop was created. We figured out what was going on pretty quickly, and asked them to disconnect the cable. In the mean time, however, there was a fair amount of chaos. We're not sure exactly how the cable was connected incorrectly.

thanks for the info Prospero.

I noticed after I got back on that none of my sims had been rolled anyway.

hope things go well ^^

My sympathies to the technician in the data center who inadvertantly created the loop. These things happen. I had something happen similarly, once, when I was sent clear across the country (USA) to connect a cable, returned back, and found all kinds of he-- had broken loose becuase of the cable being plugged-in that I had been told to plug in


And it was just an RS-232 lol

I take it the pilot roll has started - thanks for letting us know it began

I won't always update this forum -- but I did update the status blog posting when the pilot roll began, and when it finished.

Hi, tkx for the updates, is it possible to know if today rolling sims will be even or odd ones? Tkx, keep the good work!

In case you didn't see it on the status blog -- we're postponing this morning's rolling restart, as we have one more bug fix that's going to go in. Assuming all goes well, we'll redo the pilot roll later this morning, and then will do a Thursday Night / Friday Morning half-and-half roll of the entire grid.

thank you again for a rolling restart on friday where the germany have their meantime and mostly it hangs

again thank you very much you are the one and only firm who do something like this

Well, it's nice to know that you're okay with it. Because, you know, it's not at all arrogant to expect a US-based company to adjust for a foreign country. And there's also the fact that no mater -when- they run the rolling restart, it's going to affect -some-body in the world.

The problems with globalization at work.

All that being said; I do wish that Linden Lab had waited for this one last bug fix to be shoehorned into 1.24.5 before they started the rolling restarts. I mean, I recognize the effort it takes to maintain the sheer size of the networked computers that make up SL and so only grumble a little bit when part of the infrastructure starts going all wobbly, but still. Better plan the next rolling restart to take into effect bug patches you -really- want in?

So this means my region will have gone through three restarts in two days - is this correct?

only if your region was in the pilot roll then it will go through 2 restarts, otherwise only one restart, offcourse not counting the manual restarts.

Yes, isn't it nice that there are still some people living in good old Europe (not only Germany) and some are even part of the american "coalition of the willing" as the US president called it.

Europe is mostly in one timezone (including the un-willing), give or take an hour and you have Europe (except for Russia) and Africa, you get on top.
IMHO timetables can indeed be adjusted not to affect such a large part of SL (read the statistics) unnecessarily during the local prime-time.

"Globalization at work" means to consider the impact of one's decissions (e.g. timetables) on customers and users whose watches display other times than PST/PDT. And as long as Linden Lab isn't too arrogant to take my Euros for their services, I don't think it arrogant to expect of them to consider my interests, too, even if they are not US-based.

Some more musings:
• I wonder, why I get no poll at the moment at login. I always liked "Has your
experience with SL become better or worse?".
• Is it possible that there wasn't one day since September 4th where we
(and our customers) didn't receive the warning "please refrain from making
purchases or rezzing valuable/no copy items." ?

I...have no idea what this has to do with anything, as Linden Labs isn't a Government owned facility/company.



This is true, but changing the timetable just means someone else gets the short-end of the stick.



Again, someone is always going to get the short-end of the stick, which is what I mean by Globalization at work. Could shift the timescale to a different time, but then...Japan and parts of Australia would get the shortend of the stick, or perhaps the US and Canada will get the shortend of the stick.

And since Linden Lab is a US-based company, operating on the West Coast, their timetable reflects this. I would guess that the rolling restarts are done just after a new 'shift' starts, so they don't have to deal with fresh technicians coming in to work during the middle of it and and barely knowing what's going on. Best to have the same set there from beginning to end, that way they can focus on fixing things when they go wrong, rather then trying to fill someone in at the same time.

And since they are US-based, I would guess that their work schedules for technicians and such follow the patterns that other firms and US-based companies follow.

That all being said, I will make a point to say that rolling restarts get me at the worse times, too. The rolling restarts in the morning, I'm logging in halfway through them. The evening rolling restarts, I'm most of the time doing something that getting disrupted in the middle of it by a region restart is a pain in the ass.

I deal with it by grumbling about it to friends, and then reminding myself how much infrastructure is going into SL to make it even possible for me to grumble to those friends.




* Personally, I miss the RSS feed from the blog/grid status report pages.

* Dunno. They've been having a lot of network problems, it would seem, but then, it's hardly surprising, considering the shear amount of servers that they need to keep up and talking to each other perfectly.

Let's please stop the heated discussion about timetables and time zones and so forth.

We schedule our rolling restarts around two things. First, when those of us who do them are conscious. Second, our concurrency graph. The concurrency (number of people online) reaches its peak mid-afternoon SLT. We avoid doing rolling restarts between noon and 5PM SLT; the current pilot roll is an exception, as we're slopping a bit after noon.

In other words, we *are* basing this on when Second Life is receiving more heavy usage. Yes, ideally, we would do it at something like 2AM SLT, but at the moment the release engineers who manage the rolling restarts would not be in, shall we say, full operating condition at that time. So we do the best we can given those constraints, and avoid the peak concurrency times.

Re: the reason for redoing the pilot roll, the previous pilot roll looked good, but we were able to fix two more bugs that included crash modes; see the release notes page on on the wiki

for more information. These were rare enough crash bugs that it wouldn't have been a disaster if we didn't get the fixes in, and indeed yesterday we made the call to start the pilot roll last night because we didn't think we'd be able to get them in before a roll this week... but we definitely wanted to get the other fixes out there. When we learned this morning that the other two bugs had, in fact, been fixed, it made sense to redo the pilot group, and put off the rest of the rolling restart so we could get those fixes in.

The pilot group regions will have been restarted twice this week, yes, as a result of the rolling restart; that's about 1/10 of Second Life. The rest of the grid will have only been restarted once.

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Beta_Server/1.24