Etchification Rolling Retart 2009/02/17-2009/02/20
|
Shy Robbiani
Registered User
Join date: 5 Dec 2007
Posts: 2
|
02-18-2009 03:12
From: Atashi Toshihiko I understand that due to the dynamic way regions are assigned to physical hosts, there are some challenges and I accept that *right now* it isn't possible to provide more specific downtime information. It really is something that Linden Lab should be aware of however, particularily if LL is serious about attracting businesses (and educators). I totally agree with Atashi and have addressed this issue before. As long it's unreliable we cannot bring businesses to SecondLife. The problem are not the required restarts and maintenance windows, the problem is that we cannot predict in a timely manner when it happens. Just give us better tools to plan for business events in advance!
|
dino Ferina
Registered User
Join date: 20 Jan 2009
Posts: 2
|
Premium Membership
02-18-2009 04:15
Are Premium Members effected in the same way as Non-Members during these Upgrades and Rolling restarts? If so, what is the difference? If Premium Membership puts you in a group that is handled better it seems it would entice more people to Premium Status. For many Retired, Disabled or shut-in people this is a wonderful tool and a few dollars for membership would be worth it if it made a difference in the status/experience or our on-world time From: Prospero Linden This thread is for discussion of the etchification rolling restart described in the forum post http://status.secondlifegrid.net/2009/02/12/post500/Some additional notes about this rolling restart: * This is to convert our servers from Debian Sarge to Debian Etch. Etch is the current "stable" version of Debian, and will continue to be supported for a while even after the next Debian Stable ("Lenny"  comes out. We need to update so that we can keep current with security updates and the like. * Normally during rolling restarts, we lock regions to hosts. Most of the time when a region restarts, it will migrate to a different host, but they don't do that during rolling restarts. However, this rolling restart is different; because hosts are not just having a Second Life software upgrade, but are being completely wiped and reinstalled, we won't have the freedom to lock regions to hosts. * One implication of that is that some regions will be restarted more than once. There's no way to anticipate exactly which regions will be restarted. * These upgrades take longer than a standard software upgrade. As such, while many regions will be down for a typical rolling restart length of time (a few to 10 minutes), some regions will be down for 30-60 minutes.
|
Neuro Linden
Junior Member
Join date: 21 May 2007
Posts: 4
|
02-18-2009 05:09
From: Jupiter Turbo And when you TP into a sim that has just had its 5 minute warning ??
What, get abruptly logged out 4:50 later since you didnt know it was being restarted ?? Hi Jupiter, when a region is being shutdown or restarted in this manner, warnings are sent every minute from initiation until shutdown, and at 30 seconds too. If you tp into a region where the shutdown/restart command has already been sent, you'll know about it 
|
Neuro Linden
Junior Member
Join date: 21 May 2007
Posts: 4
|
02-18-2009 05:10
From: dino Ferina Are Premium Members effected in the same way as Non-Members during these Upgrades and Rolling restarts? No. This work is being done on a per-server basis, not a per-region or per-resident basis.
|
Neuro Linden
Junior Member
Join date: 21 May 2007
Posts: 4
|
02-18-2009 05:13
From: Deeso Saeed I'm curious about something: Sarge still had a 2.4.27 kernel (gcc 3.3.5), while Etch is a huge leap, having a 2.6.18 (gcc 4.1.1). Assuming you didn't had a custom kernel/toolchain, I wonder if we can expect some performance improvements. You're right, we don't use the stock sarge kernel.
|
Jupiter Turbo
Registered User
Join date: 28 Sep 2008
Posts: 2
|
02-18-2009 06:00
From: Neuro Linden Hi Jupiter, when a region is being shutdown or restarted in this manner, warnings are sent every minute from initiation until shutdown, and at 30 seconds too. If you tp into a region where the shutdown/restart command has already been sent, you'll know about it  Thanks Neuro, that was EXACTLY my point to Meta.. As I finished the previous post: "LL - Don't change the warnings.. There are good as they are..!!!!"
|
Prospero Linden
Linden Lab Employee
Join date: 6 Aug 2007
Posts: 315
|
02-18-2009 07:26
From: Deeso Saeed I'm curious about something: Sarge still had a 2.4.27 kernel (gcc 3.3.5), while Etch is a huge leap, having a 2.6.18 (gcc 4.1.1). Assuming you didn't had a custom kernel/toolchain, I wonder if we can expect some performance improvements I wouldn't expect a lot of performance improvements. Maybe? It's hard to tell, since as always, we can't really test the full-stress Second Life environment until we've got it in the full-stress Second Life environment. QA to this state has focused on making sure that everything continues to work. In any event, though, performance is going to be dominated by the design of Second Life and the communications systems it uses and all of that. We do have people working on server-side performance improvements, but it will be harder than just upgrading the kernel, unfortunately. 
|
Kath McGill
Registered User
Join date: 23 Jul 2006
Posts: 2
|
Tuckborough island is still down...
02-18-2009 08:33
Tuckborough, Just a nice little OS sim off of Eragon that my husband and I own, not alot on it, I tped from the Dreams fair to go onto my island to build at 7:27 am,pst. While i understand the rolling restart can take up to an hour to finish, I have no idea when the island pooffed, and the odds of tping in JUST as the island went poof are pretty astronimical and, its been over an hour and still no sign of the island.
So I sit and wait and watch the water, and people, who, dispite a sign that says Private, do not enter, decided, it doesnt apply to them and they walk past the sign, trying to get onto Tuckborough..(when the island is down)-Ok, I'm easily amused- and they go Bam Bam Bam Bam Bam into the void wall of a sim thats somewhere where it really shouldnt be...
maybe can I hire a linden to help me catch up on the work that I have on the island, just for the hour or so that its taken to do this thing?
When ever ya all get the chance!
Kath McGIll
(going to keep a tally of the time, 8:55 am, still no sign of the island...)
|
Nika Talaj
now you see her ...
Join date: 2 Jan 2007
Posts: 5,449
|
02-18-2009 08:53
From: Kath McGill its been over an hour and still no sign of the island. Hi Kath. Welcome to the forums! The forums are good for a lot of things, but LL has a whole department for product support and they do not read the forums. The Lindens on this thread are not from product support - believe it or not, you'll get pretty quick action on this if you file a support ticket requesting a sim restart. Even if you're not the sim owner, and even if you are a Basic member (as opposed to Premium), if you file a support ticket requesting that they restart an already down sim, it usually gets pretty quick action. *points to the support button on the taskbar at the top of the screen* 
|
Argent Stonecutter
Emergency Mustelid
Join date: 20 Sep 2005
Posts: 20,263
|
02-18-2009 08:55
From: Kath McGill TSo I sit and wait and watch the water, and people, who, dispite a sign that says Private, do not enter, decided, it doesnt apply to them and they walk past the sign, ...which was likely a grey square to them...
|
Kath McGill
Registered User
Join date: 23 Jul 2006
Posts: 2
|
Yes yes,,,
02-18-2009 09:02
we have filed a support ticket, and are still waiting on any answer from them and its been 45 min since the ticket was filed and no answer (so bang goes the theroy the support is fast not saying that its not a reasonable length of time, rather, we didnt even get the thanks for your ticket, we are working on things now be patient, we havent heard anything...), (as well as the islands re appearance, ) Yes, I know what blogs are for, and yes I understand that its unlikely that the lindens would notice my one hey my region is still down note, but for those who are reading this, and their island is looking like mine, then, perhaps they may see that they are not alone.
9:05am and no sign of my island,
sigh
from what my husband says, things look pretty bleak,
Kath
at 9:18 my husband was able to teleport to the island, though, there was a really weird event, it was surrounded by water, and he couldnt see eragon, and from what I saw in my position, the island *still* hadnt shown up.
tping back, tuckborough appeared, but we are still blocked from walking on to the island, its been very very wierd.
at 9:21 tuckborough,,, once again, disapeared.
sigh.
9:30, and a manual restart of both islands, Yea, I have my island back. Hugs to the special linden who helped with this (he knows who he is)
kath
|
Urantia Jewell
Registered User
Join date: 18 Jun 2007
Posts: 22
|
Problem Identified
02-18-2009 09:33
From: Kath McGill ...snip... at 9:18 my husband was able to teleport to the island, though, there was a really weird event, it was surrounded by water, and he couldnt see eragon, and from what I saw in my position, the island *still* hadnt shown up.
tping back, tuckborough appeared, but we are still blocked from walking on to the island, its been very very wierd.
at 9:21 tuckborough,,, once again, disapeared.
sigh. From: Status Desk We are investigating reports of connection issues with a small number of regions. Residents in these regions will not be able to see or cross into neighboring regions, may experience difficulty teleporting out of the region and transactions in affected regions may fail. Please be aware of your surroundings, and if in an affected region refrain from making purchases or transactions.

|
Prospero Linden
Linden Lab Employee
Join date: 6 Aug 2007
Posts: 315
|
02-18-2009 10:05
Folks, I just deleted a few messages-- requests for individual support are not on topic here.
I did leave the last message from the thread, and the response, because there is a problem right now with not being able to see neighboring regions. I'm not sure exactly what caused this, but we are going to be restarted affected hosts to solve the problem over the next hour or so.
|
Scree Raymaker
Registered User
Join date: 17 Jul 2007
Posts: 17
|
02-18-2009 10:42
Not being able to see neighbouring regions isn't related to the rolling restart, as I've experienced the same thing three times over the past four days.
|
Sindy Tsure
Will script for shoes
Join date: 18 Sep 2006
Posts: 4,103
|
02-18-2009 12:22
The forums have seemed really, REALLY slow for the last day. No errors but far slower than usual. Unrelated to the etchification? edit: seems much better now.. never mind. 
|
Meta Starostin
Registered User
Join date: 5 Jul 2007
Posts: 56
|
02-18-2009 12:23
From: Jupiter Turbo And when you TP into a sim that has just had its 5 minute warning ??
What, get abruptly logged out 4:50 later since you didnt know it was being restarted ??
LL - Don't change the warnings.. There are good as they are..!!!! Be realistic, you are making it up. In reality this rarely happens. I have a parcel on the mainland and I know the even or odd number thing. If I TP to a sim that I know is on a private island its highly unlikely that its going to be restarted the same time as my mainland SIM. The 30 second warnings suck and often all arrive at the same time, locking up the viewer. Worse than SPAM. A five minute initial warning is not enough either especially if there is a high level of lag. 10 minutes would be good and no warnings in the last minute. Also if my SIM has not restarted and I TP to a SIM that has, I get the yellow notification telling me that the SIM is running on a different server version. Besides Surfing the Rolling Restarts is a lot of fun.
|
Shawwn Ballinger
Registered User
Join date: 25 Jun 2007
Posts: 5
|
erm. could be...
02-18-2009 12:55
From: Scree Raymaker Not being able to see neighbouring regions isn't related to the rolling restart, as I've experienced the same thing three times over the past four days. Not seeing the neighboring sim, floating off the grid, etc, are generally associated with latency between the affected sims. But when this occurs between regions that are being hosted in the same data center, as has been occurring for us since we were consolidated in dallas last week, then the server code becomes suspect. I have no idea what the affect of the debian upgrade will be on that, likely I will have to ask to be re-consolidated. but the problem is so pervasive now that I regularly cant see more then a couple connected sims at a time. I know this is related to changes that have occurred since the last server upgrade because 2 persistent changes are evident. 1. SLVoice routinely crashes at a sim border now, as never before. 2. When i float off one sim, even into a sim that I can see and interact with (but not navigate in) the location shows me still in the affected sim. Right now this occurs if I rezz in Tropical and try to cross into Prefabs. Volim te, to the south, is not even viewable, neither can I float into its space, prefabs however is visible, i can edit its land, interact with its objects, but i cannot navigate properly inside it and am being reported as being still in tropical at a negative x. you can try this, these are open sims. ooo, now volim te is visible, but cant be crossed into as i can with prefabs. this is wierdness. ok. it rezzed as i was writing this and poofed again. 10 store sims... cant do any business like this. please fix 
|
Alexandra Rucker
Metamorph
Join date: 19 Jul 2006
Posts: 71
|
02-18-2009 20:57
From: Prospero Linden ...I keep saying this, but people keep insisting that they want to view Second Life as being completely unreliable over the entire time from the beginning of the first window to the end of the last window. Why? Were you part of The Lab back when a biweekly - or weekly, depending on how far back - required a mandatory new client each time? And that they were schedueld for 2-3 hours, but often took 6 hours minimum, if not longer on a bad week? *Edit* And often had new and more annoying problems than before the maintenance... */edit* Some of us remember those....and are always afraid a rolling restart is going to do that again. *continues through the thread hoping to see before/after server version numbers...*
|
Ceejay Harvey
Very unhappy customer
Join date: 15 Feb 2007
Posts: 56
|
02-18-2009 21:45
Tp's unreliable, sims loading three times slower, performance sucking monkies, I know the restart isn't over, but if this is anywhere close to the final performance then give us back the pre havock days when performance was good but stability was bad.
you swung the pendulum way to far the other way and there is no signs of this stopping, my opinion is your killing SL with the cure
|
Zena Juran
Registered User
Join date: 21 Jul 2007
Posts: 473
|
02-19-2009 03:26
My sim has yet to be restarted but as the Rolling Restarts are going on, I see a steady 2.0ms increase of Physics Time. This is the "heaviest" Rolling Restart I've ever seen. Sim performance is going to degrade considerably while this is going on. 
|
Ashina Tigerpaw
Registered User
Join date: 30 Jun 2008
Posts: 2
|
02-19-2009 04:32
The new times will be: * Wed 2/18 20:00 through Thu 2/19 10:00 * Thu 2/19 19:00 through Fri 2/20 10:00 * Sun 2/22 20:00 through Mon 2/23 10:00 * Mon 2/23 19:00 through Tue 2/24 10:00 * Tue 2/24 19:00 through Wed 2/25 10:00
Is this right? 15 hours of restarts on 3 of the days and 14 hours of restarts on the other 2 of them?
|
Urantia Jewell
Registered User
Join date: 18 Jun 2007
Posts: 22
|
02-19-2009 07:45
From: Ashina Tigerpaw The new times will be: * Wed 2/18 20:00 through Thu 2/19 10:00 * Thu 2/19 19:00 through Fri 2/20 10:00 * Sun 2/22 20:00 through Mon 2/23 10:00 * Mon 2/23 19:00 through Tue 2/24 10:00 * Tue 2/24 19:00 through Wed 2/25 10:00
Is this right? 15 hours of restarts on 3 of the days and 14 hours of restarts on the other 2 of them? Yes, they are upgrading the Debian operating system from Sarge to Etch on the grid servers. This is an atypical rolling restart. Apparently they had to slow the process down.
|
Triple Peccable
Registered User
Join date: 7 Jul 2007
Posts: 70
|
02-19-2009 09:51
At the moment the status page has conflicting information. The top part says the next rolling restart will start at 19:00 tonight, but the schedule at the bottom says the next one isn't scheduled until Sunday evening at 20:00. Which one is correct?
EDIT: Nevermind. My brain wasn't working.
|
Sedary Raymaker
Registered User
Join date: 11 Mar 2007
Posts: 59
|
02-19-2009 10:16
From: Triple Peccable Which one is correct? It's been saying "* Thu 2/19 19:00 through Fri 2/20 10:00" on the list since it was posted.
|
Triple Peccable
Registered User
Join date: 7 Jul 2007
Posts: 70
|
02-19-2009 10:22
Yes, I see now that I thought the Thursday rolling restart had just completed, when it was the Wednesday one that just completed. My bad. These overnight rolling restarts are hard on my brain! 
|