Etchification Rolling Retart 2009/02/17-2009/02/20

If it's any consolation, it totally feels like a Friday to me. Sadly, it's not.

Is there any way to tell if you're in a region that has gone through the restart process?

Two of my regions have reported a crash/restart in the past day and a half, but I'm not sure if that is from the Etchification process or if they just up and crashed. (My sim status device only records that the sim rebooted, doesn't / can't report why.)

When I look at the Help->About Second Life the server stats info does not seem to show anything different from the regions that have not had a recent crash/restart:


Mainly I'm trying to determine if the two that have had crash/reboots are now "safe" for the remainder of the rolling restart process.

Thanks!

-Atashi

A friend asked about this at Andrews office hour last night. Was told (by Neuro, IIRC) that there's no way to tell because etchification doesn't require an update to the sim code - no version that we can see is changing.

Prospero, are these updates going by host number order? Any chance we can get a general schedule of the hosts you're planning to do through the rest of this process?

Unlike normal roling restarts a region can get restarted more then once because this time the sims are not locked to the host, if they where you would have to wait a lot longer for the region to come back up, so even if you could tell if they where restarted becouse of the RR you still can't be sure they won't be restarted again, offcourse with this being the last day the change of the sim being restarted on an updated host is a lot bigger then it being restarted on a host that still needs updating.

Someone at LL needs to look into this ASAP:

http://jira.secondlife.com/browse/SVC-3867

Thanks for the info Meade, Abigail. That's annoying, they could have incremented the server build by one digit or something just to indicate if it was running the old or new distro.

Cheers!

-Atashi

Yep.. I asked and Neuro said there wasn't a sim version change, which basically means non-Lindens can't tell if a host has been done yet..

http://wiki.secondlife.com/wiki/User:Andrew_Linden/Office_Hours/2009_02_19 at 18:03.

It seems that manual restarts on my sim are keeping it offline for about an hour at a time now. Is this going to be the norm?

Should only be for the rolling restart. Since every sim restarted by the RR has to go to a new server, there's a continual backlog of sims waiting for a server to become available.

The time, at the moment, is 08:25 AM.
Entry in the status blog has the following entry:

[UPDATED Feb 25 10.44am PST] The rolling restarts have now completed

Either the person who is making these entries is psychic and is predicting when the RR's are going to be done OR can't tell time.

Either way, it doesn't engender very much confidence that they know what they are doing.

Hey guys -- sorry I've been absent.

Some answers/clarifications.

First, when you see "the rolling restarts have completed" on the status blog, that's just for that night's rolling restarts.

Second: this is not a server software change; it's a change to the underlying operating system. As such, there's no way to tell whether the region you're on is running the old version of the OS or the new version of the OS. Indeed, if your region restarts, it may move from one to the other, as the server software runs on both.

Third : yes, sometimes regions are staying down for a long time. We warned about this in the original post about this particular rolling restart. We're doing our best to avoid that, but because of the nature of this rolling restart sometimes it's impossible to avoid it. This rolling restart is very different from the usual server upgrade rolling restarts we've done. This is the first time we've done an OS upgrade rolling restart like this (last time we did an OS upgrade, we took the whole grid down), and as such we don't have all the experience that we have with server software upgrade rolling restarts. There are a host of additional complications, and as a result we're both showing more warts, and the restarts are going slower than we had hoped. As such, there are still several nights of these restarts left, and they will probably slop into next week.

(And to forestall those who have rose colored past-facing glasses about grid downtime : if we were to take the whole grid down to do the OS upgrade, to avoid having to have couple of weeks of rolling restarts, we'd probably have to take the whole grid down for two or three days, at least, to get them all done. That's not an option in this day and age.)

Dude, it would take more than rose coloured glasses to make "Grid Down Wednesdays" seem like a good idea. You'd have to be completely insane.

It absolutely sucked not being able to log on all day. This rolling restart nonsense is lightyears ahead although I have some sympathy with the view that it's no way to do serious business.

When can we have a sim seamlessly switch to another computer without going down at all? It's just a virtual machine right? Should be able to migrate it live to different hardware.

Copying virtual machines around takes time. And when it's migrated to another machine it wil needs to do some kind of reset to make sure that everything that knows about it knows about its new IP address. So I don't think it would be possible to make it *completely* seamless.

Copying them to a server that isn't available yet... that's REALLY tricky.

several more nights of this? how sweet is that . Wewt. What was wrong with the old operating system?? Too stable?

So annoying

And you've just proved you can't read. They're moving to another OS because Sarge (which they use) is well, no longer supported). Which means security and other underlining software fixes/improvements are no longer being put out for the OS they are replacing.

All of which has already been mentioned in this thread.

Why do i face the same problem after every rolling restart? I own a sim and after the restart IMs to me either from other residents on the sim or from objects, never reach my email box when im off line. IMs reach it when they come from other sims. Looks like my sim stays isolated for IM-> email. The same thing had happened in the previous rolling restart 1 month ago.

Yeah. They should use DOS 1.0 or something else that's really established and stable.

Can you imagine how hard it must be to even apply the simple security upgrades that get pushed through,it's not as simple as clicking the new updates notice on your computer (or more likely pulling up whatever get program these things run) and waiting for it to update and restarting if the update requires it.

Makes me appreciate Windows Updates on my Windows machines, Software Update on my Mac machines and Synaptic and the other assorted apt tools on my *nix based machines.

Wait, whut?

So, you're saying they shouldn't be updating the OS on the servers because it's extra work?

Dude, I appreciate how much work it is. I see it very time they're forced to do a rollback because of unseen bugs creeping into the server. I see it every time they release a new viewer RC.

I'm not exactly sure why it is that you're quoting me or anything, unless you think they shouldn't be updating too. In which case, let's migrate the server to Win3.11. It's solid and still working strong today (I know several people that still use it on toy boxes).

OR they're citing when they fed the instructions to the grid instead of when the tasks finished.

Just somebody making a clerical error - when I see stuff like this, I try to remember that it's still possible to do things like accidentally take out Rackspace's entire data center for three days by ramming a pickup truck into a power pole. This happened in 2007. The fact that the grid has so much up time is really pretty impressive considering how many things can go wrong, and Linden Lab is slowly evolving the chops to maintain it all.

Prior to 2004, a persistent, shared experience like Second Life was a just a theory, and Linden Lab was the first company ever to pull off something this big, so I try to keep things in perspective. All in all, they're doing a pretty fair job of keeping the whole thing running.

uumm wrong.. there were quite a few VR worlds before SL

*hands Prospero a cookie* ... I hope you get a huge bonus this year ... you sure nuf do deserve it. *Smacks hands all over this thread* ...

Hypothetical meetings don't make them actual meetings. If you are an actual business owner, having meetings on an electronic platform ... I'm certain that you would be aware of other type of electronic ways to hold meetings. While SL is a good environment for training, etc., it should not be relied upon SOLELY as the ONLY meeting resource for viable business owners.

And, as business owners, you would recognize the need to upgrade and maintain servers, which takes DOWNTIME.

Now quit yer whining over things you have no control over. Seriously - there are more important issues in this world other than whether or not you can hold your $300L booty contests at a certain date/time without rolling restarts.

*Steps down ... and gets off her soapbox* ... And no, I won't be checking back for more of your whines ... I've had enough for a week just reading this thread alone.

Well, DOS 1.0 was pretty stable and I would say it was established. However I don't think LL is looking for that type of Operating System.

Personally I always liked DOS 2.2


Back on Topic, I hope this rolling restart will be done soon. The non-existing grid issues that accompany it are quite a pain for me.

DOS 2.2 was where they started trying to emulate UNIX instead of CP/M.

Sorry, but you are incorrect. Take a look at VMWare ESX and VMotion. Seamless migration between physical hardware for virtual machines running. Same thing is available for storage systems with storage virtualization. IBM's SVC (SAN Volume Controller) is a good example of that. Move the disk partition (LUN) between physical storage systems while the systems continue to read and write (at fairly high I/Os) without any impact to the system.

Those are two examples that I am intimately familiar with but they are not the only ones. The issue really is one of economics. Does Second Life generate sufficient revenue streams to justify this type of enterprise IT infrastructure.

Regards,
Ty