Rolling restart for 1.24.7 2008 Sep 30-Oct 2

You may want to contact your region owner and have a rollback request made. Usually can have that done within an hour but any new build will disappear.

Evil Vs. Evil

Guess it's a matter of definitions, Prospero was talking about backward rolls, in other words going back to a previus version, that never happened during 1.24, every rollout or partial roleout has been forwards.

You on the other hand sees an interupted rollout that was later redone with a fixed version as part of the same rollout.

to me both are right, they where all going forward but interupted rollouts do count as a failed rollout even if they where not causing to much trouble.

Still I see the current way of handling it as a big improvement and the anouncement that at least at major updates like the upcomming 1.25 will have a longer bake time after the pilot roleout shows that at least Prospero is listening to us.

Thank you for this post. Very well said and explanatory. I feel the same way. And they have the ability to forewarn you if odd or even number regions will be done on a given day. But refuses to tell us until the actual start. Which we have to sit and wait and make room for 3 days of our lives for one 5 Minute restart. Wasting our personal resources. Some of us have been scared by regions being down for 3 to 5 hours more than once. And nothing like running a club and having a restart return all your objects on the region. 5 Minute down time? This is why some of us are so Freaked out about restarts. And not knowing what exact date we will be restarted. You are making us sit around for the full 3 days for a 5 minute restart.
Simple information . Prospero we love you guys and your hard work . We only ask for a clear picture instead of white wash when it comes to being infomred when our regions will be restarted. I can handle taking 3 hours of my week for a restart. But as it stands without the informtion being properly posted i have to take out 9 hours a week from my life and personal obligations to deal with weekly restarts. So as you can see. With being down 1 day a week and on a specific time was less of a strain on some of us than it is now having to take 9 hours a week to deal with updates.

Thanks again for your hard work Prospero. We do appreciate it

Peace

Part of the problem I have with these complaints is that the numbers you throw out do not match reality.

What regions are down 3 to 5 hours? That doesn't happen. As I have said **repeatedly**, if your region is down more than 20 or 30 minutes, contact support. When we hear about them, we deal with them. During a typical rolling restart, this happens to at most 100 or so regions-- out of thirty thousand, which is less than 1% of the grid, meaning that if you're not online, chances are this doesn't happen anyway. (We've recently identified the source of a lot of that and have eliminated it, so that <1% will drop to even less.) Yes, there have been times when one or two regions-- out of *thirty thousand*-- have particular problems that require additional attention, and take a bit longer.

In times past -- many, many months past -- we weren't able to fix regions during rolling restarts, and if one didn't come back we had to wait until after the rolling restart was over. Nowadays, if we get a report about a region being down during a rolling restart, rarely we may say "wait until the end" -- typically if the report comes in when we're within a half hour of the end of the rolling restart. But, as we say in the announcements, contact support if you're having problems. The days of "we can't do anything" during rolling restarts are past, again with the exception of 00.01% of regions that may have special problems.

If you're not scheduling any events on the entire time over which the rolling restart is spread-- you're doing it wrong. All you have to worry about is the specific times when we tell you it's going to happen.

When people complain about rolling restarts because of 3-5 hours of downtime-- it becomes very difficult for me to take those complaints seriously, because they do not match reality. When people talk about having to sit around for 3 days because of a rolling restart, my response is that you look at the times when they're going to happen, and just worry about those times, rather than claiming that they are sitting around and fretting during an integrated 60 hours when there is no planned downtime.

At 06:26, the second-half rolling restart is about 46% complete.

But this is SL, everything is exaggerated here

Maybe mostly-everything but not totally-everything.

Having LL even acknowledge that the way the RR schedule works might cause problems (or serious problems) for some people would help them a lot, IMO.

If LL wants SL to be seen as more of a utility, like phone/electric are and the internet has almost become, and if they want to increase the corporate cash coming in (and it's clear that they do), this is an area that could use some work.

Would you like the electric company to send you a note saying...


I don't know if this is the message you're trying to send Prospero but this is the way it sounds to me, every time. If I was in SL for corporate reasons, this would not make me happy.

/me hopes this is not taken as anything more than it is: constructive criticism. I remember well “upgrade Wednesday” and many things have improved a lot since then! It would be nice to hear more than "this isn't a problem for you, stop complaining" though.

back ya up on this prospero; i'd recently played with a simple sim beacon at my vending locations around the grid, just to kinda see for myself if any sims were having outage/reset probs. all of this rollout's resets have taken no longer than 5:10 for laggy mainland sims; the majority of sims only taking 2-2:45!

"Sim outage, est. duration 0:02:40 (160 seconds) sometime after 8:39:32am 081002. Probably a reset/update. 9:42:12am 081002"

The rolling restarts aren't as big a deal as a few people are making it out to be. Let's look at reality for a second.

LL tells us how long the window is.. on 3 days.


While you may think that this means that you can't schedule any events from Tuesday at 8PM, until Thursday at 9am.. that's just NOT how this works.

Let's say you have an event you'd like to hold on Tuesday. Don't schedule it between 8pm and 10 pm. If you DO have an event that overlaps the restart, remember that you have a 1:10 chance of having your sim shut down for POTENTIALLY half an hour. (should something go heneously wrong). A 10 minute break is something that MOST partiers can adapt to. Those who can't be bothered to come back.. you won't miss too much, because they're the same kind who complains about the quality of freebies, and leaves when a song comes on they don't like. By 10PM, if your sim hasn't restarted.. guess what? you're free and clear for 7 solid hours!

Let's say you had an event you wanted to hold on Wednesday. ASSUMING you didn't get restarted on Tuesday.. Don't schedule it between 5-9am! If you do.. keep in mind that you have a little better than a 1:2 chance of being hit with a restart. If your event is ONE HOUR LONG.. from 5-6 am, you have only a 1:13 chance of a restart. Every hour that your sim goes without a restart, your odds increase. (because all sims WILL eventually be restarted.) So.. by the time that the last hour on Wednesday comes to pass, your odds will be about 1:2.

Now, let's say it's Thursday. you've got an event you want to run then. You check your server version when you log in, and you STILL haven't been restarted. Assuming that the restarted hasn't been postponed, you have a 1:1 chance of being restarted today. BUT.. you have a 1:4 chance in the first hour, 1:3 in the second hour, 1:2 in the third hour, and if you STILL haven't been restarted by 8:am on Thursday, you can pretty much be certain it's going to happen in that hour or so.

But here's the thing. Once the window closes on Tuesday, you can be certain that you won't be restarted until at least 5am on Wednesday. Once the window closes on Wednesday, you can be certain that you won't be restarted before 5am on Thursday.

Regardless of whatever "supercalifragilesystem" you have running that NEEDS to be reset the MOMENT the sim restarts.. (sounds like pretty flaky coding to me).. the reality is that for most people in SL.. and I mean MOST people.. these restarts are pretty seamless.

When that 5 minute warning comes.. save your work, take a copy.. all the stuff that protects you "just in case" something wonky happens like happened to me last time, when an hour of work dissappeared.. But don't act like you don't have 20 hours a day to do things, outside those windows.

This is VERY much like the power, or water company, sending out a notice to your neighborhood, saying that they're going to be working on the water pipes for three days next week. They can't tell you specifically WHEN your water will be shut off.. but they can say a "window" during which they'll be doing those shutdowns each day, and they can tell you that when it IS shut off, YOUR water should only be off for a short amount of time, while they change the fitting... assuming something weird doesn't happen.

The city came by the other day, and did precisely that. They let us know that our water would be off for about an hour and a half.. but they couldn't say with any accuracy WHEN that time would start or stop.

The following assumes "30,000 regions", assumes that the sim in question has NOT been restarted yet, and assumes that the roll is running perfectly on schedule, and finishes niether late nor early.

Tuesday: 3,000 Regions (You have a 10% chance of a restart today)
Hour 1: 1,500 out of 30,000 = 5% chance of a restart.
Hour 2: 1,500 out of 28,500 = 5.263% chance of a restart.

Wednesday: 13,500 regions (Assuming you haven't been restarted yet, you have a 50% chance of a restart today)
Hour 1: 3,375 out of 27,000 = 12.5% chance of a restart.
Hour 2: 3,375 out of 23,625 = 14.286% chance of a restart.
Hour 3: 3,375 out of 20,250 = 16.667% chance of a restart.
Hour 4: 3,375 out of 16,875 = 20% chance of a restart.

Thursday: 13,500 regions (Assuming you haven't been restarted yet, you have a 100% chance of a restart today)
Hour 1: 3,375 out of 13,500 = 25% chance of a restart.
Hour 2: 3,375 out of 10,125 = 33.333% chance of a restart.
Hour 3: 3,375 out of 6,750 = 50% chance of a restart.
Hour 4: 3,375 out of 3,375 = 100% chance of a restart.

In terms of "playing the odds".. the vast likelihood is that your restart will PROBABLY happen BEFORE the last window. The numbers above only consider your chances DURINg that hour, of a restart, assuming you haven't gotten a restart yet. Frankly, the chances are better than 90% that you will get retarted BEFORE the last hour.. it's kind of odd how those numbers work. It's like a bell curve really. You're very unlikely to be restarted on tuesday, but you've got more than a 50% liklihood of being restarted sometime before the window closes on wednesday. (Frankly I think the top of that bell curve is right at hour 3 on wednesday. But of course, once the restarts begin, the odds change.

I'd suggest that it's very probable that a given sim won't be restarted on tuesday, and it's about as probable that it won't be restarted in the last hour on thursday.

TU H1 *
TU H2 *
W H1 **
W H2 ***
W H3 ****
W H4 *****
TH H! ****
TH H2 ***
TH H3 **
TH H4 *

Going into a restart, and planning an event in advance, it seems to me that the worst time to plan an event, is for the last hour on Wednesday's window, or the first hour of Thursday's window.

But there's also 20 hours during the rest of the day.. when you can schedule things in relative safety.

You're saying that the way rolling restarts work should not be a problem for anybody and that there is no issue here? If you're not saying that then I still say that there's work LL could (and should!) do to make it better.

So, let me be the first to admit that the current system isn't perfect, that we know that it's an issue for the reliability of the system, and that it needs to be better. We want it to be better.

We still have some ways to go; we're not there yet.

However, we have made progress since the times when we would have to take the entire grid down for a whole morning, and sometimes longer. We have made progress since the time when if your region went down during a rolling restart, there was nothing we could do about it until the rolling restart was over. Indeed, the fact that rolling restarts are spread over multiple days is progress, even though people complain about it, because it means that if problems are found, (a) we don't have those problems on the whole grid, and (b) we don't have to double-restart the whole grid. (That's what the pilot roll is for. The division of the full roll into two half-rolls also somewhat serves that purpose, but is also partly for practical reasons. We do not want to have a rolling restart extend into a several-hour window around peak concurrency, that happens mid-after pacific time, which makes it difficult to do the entire rollin one day.)

At the moment, practically speaking, I can not tell you when your region will get restarted within the windows provided. If I could, I would. The information I *can* give you is unreliable, and I am going to stand by not giving that unreliable information even though some keep asking for it, because it implies a promise that I'm not able to make.

So where do we go next?

Long term, very long term, we'd like to have code updates work in a smarter fashion, whereby the simulator process figures out if there's nobody in-region, and then updates the region then. Nobody then gets kicked out by a rolling restart. Don't hold your breath for this one, though, as that's hard enough to do and there are enough other active pain points that we probably won't do it right away.

Shorter term, hopefully within then next quarter, I want to have a system in place whereby region owners can ask to be exempted from a restart at certain times. This will probably be initially day-by-day. That is, if the full rolling restart is Tue-Thu, you can pick one day in there where you won't hit by a rolling restart. Later, hopefully we'll be able to give that finer grain, so you can pick one or two 4-hour windows in which your region won't be hit by the rolling restart. This system is not there now, but it is something we've already started talking about.

In other words : we know it's not perfect. We know that there are inconveniences with the system. What I'm objecting to is the overstatement of the nature of those inconveniences. The system we have right now is the best we are able to do at the moment, and is better than what we've had in the past.

Also, Winter, your stats aren't exactly correct... rolling restarts usually don't fill up the four-hour window applied. For instance, this morning the rolling restart began at about 5:20, and was mostly complete 2.5 hours later. I give the 4-hour window so that we'll be sure to remain within that window even if there are some minor hangups.

Good reply, Prospero!

Yes, I absolutely agree that a number of people make it out to be worse than it sounds. Some make it out to be a lot worse than it sounds. Again, I think you guys have made great progress here since I joined. Really. Two thumbs up.

TY for acknowledging that there's still work to be done and that you've been thinking on ways to make it even better.

I do however still think it's more predictable than you say, especially since H4 made unscheduled sim restarts so much less common - I think I read somewhere that it averages around 2 months now.

That's a discussion for a different thread, though. If you'd like to start one on it - you do have your very own Rolling Restart forum here - I'm sure people would like to hear about the process.

edit:

/me points out a mention of the mythical Phoenix colo! LL's area 51!!

This is where they have Magellan locked up. I'm sure of it.

Of course.. this was merely an illustration based on some overly simplified assumptions that people generally make. Obviously rolling restarts sometimes start late, sometimes they end early, and they don't go at a homogenous pace of 3750 regions per hour. But the broader message still holds true, "the longer you go without a restart, the more likely it is that you'll be restarted soon." and looking at it in the broad sense, the best day to avoid scheduling anything on, is the later portion of the middle day... though it is only slightly more likely to occur then, as it is to occur on the first part of the third day.

You don't go into a pilot roll with a 50/50 chance of getting restarted.. the chances are really quite slim on the first day, so most event planning can fairly safely ignore that day. It's like having a bin with 10 straws, one of them short. The first 2 draws, your chances are pretty low, for getting the short straw. If you pull 4 straws out the next day, yes you have a 50% chance of pulling out a short straw, but if you pull them out one at a time, you don't actually reach 50/50 until you're down to 2 straws.

The real lesson is much simpler.. The downtime is only 10 minutes, and even if (gasp) you get logged out and are forced to relog.. you'll be back to work in no time. Though I wouldn't schedule a hundred-hop shopping trip for that 4 hour window.

It's also where one of my sim's is, and TP's to and from it are always slow, and fail more frequently than non-PHX located sims (This is the case with ALL sims in the PHX datacenter.)

This is probably why:
14 ae-11-11.car2.Phoenix1.Level3.net (4.69.133.34) 106.981 ms 107.560 ms 107.173 ms
15 LINDEN-RESE.car2.Phoenix1.Level3.net (63.214.168.5 1311.870 ms 1310.940 ms 1310.973 ms

Um, LL? Phoenix has problems. Big problems. Partly they are due to your ISP of choice (Level3) which is the worst of all major carriers in terms of reliability. Booooo Level 3!!!!

Note that traffic to Phoenix goes through Dallas (possibly even the datacenter) so ping times are WORSE than any other facility.

Yea Exadurated all right sure you are so correct NOT!. But tell me about the day when the bad memory leak was plaguing rolling restarts. My region was down for 15 minutes. I called concierge only to be told i have to wait another 15 minutes before they will do anything. great thats 30 minutes.
Call back they said okay we will look. So they say it still has not restarted and not coming down and they have to let it go thru the cycle. Great 1.5 hours goes by and still no region . I call back and they said we have to wait till they can get to us When it was all said and done it was 3 hours before our region was made available.
I dont complain about something if it doesnt warant it. Im not som ooof that totaly blows things out of prportion. Its truth and i have many witnesses that can back me up on this.
IT HAS HAPPENED

Dallas was like that for a bit when they first brought it up.. Now it's not bad at all.

That tracert excerpt doesn't look very happy, though.

How many times has the rolling restarts been delayed? How many times has a rolling restart started on Time? How many times has it been over looked on telling us if odds or evens being restarted? Many Many Many Many times. Its like pulling teeth from you all just to get a simple little tidbit of info Odds or evens. So on the first half you dont tell us which ones are being restarted we have to sit here the entire 3 hours waiting to see if we get restarted. Pilot rollouts are random still no one knows.
Well after all that obviously sitting on the third day no?

And you don't think scripts need to be reset after a rolling you are wrong. Anyone using dance ball sets have te reset them if someone stays on them when sim goes down or they stay invisible.
Other thing is parcel description gets erased on rolling restarts crashes and manual restarts. So we have to be there to fix that as well. which this has been going on for several months now.
sounds like LL thinks rolling restarts do not require anyone to be there to clean up after it. You are so misinformed. Or just refuse to listen

This makes absolute sense - take a week, take 2 - let that sucker bake till you cant stick a fork in it anymore - GRACIAS!

ok now since its all done can you please tell what three security problems have been fixed?
i was seeing strange things happen in the last days but i dont know if its all good again now

Nothing about it is "BS". An estate owner has to worry about every other occupant of the sim as well. Lets say you got your way there and Leo did you a solid, ignoring any policy and rolling back the sim 5 days for you. Now consider that your neighbor was in the process of a huge build, had worked all week to complete it, had no copies in inventory. Along comes Leo in response to your issue and rolls back the entire sim because you lost stuff on just part of it. *poof* your neighbor loses his weeks worth of work. A rollback is a snapshot at a given time, Items that were there then but arent now will re-appear, Items that werent there then but are now will disappear. If you dont own the entire sim such things have to be very carefully orchestrated by someone who does - or their authorized representative

P.S. A bit of politic please, no need to call someone a liar - perhaps mis-informed, but seriously, why be inflamatory?

Are you still seeing that? Looks ok to me...


9 62 ms 57 ms 68 ms ae-91-91.ebr1.Dallas1.Level3.net [4.69.136.133]
10 80 ms 80 ms 79 ms ae-8-8.car1.Phoenix1.Level3.net [4.69.133.29]
11 81 ms 81 ms 80 ms ae-11-11.car2.Phoenix1.Level3.net [4.69.133.34]
12 81 ms 80 ms 80 ms LINDEN-RESE.car2.Phoenix1.Level3.net [63.214.168.58]

Funny how you left off the time (ms) numbers for the "LINDEN-RESE" server where the problem appeared in your quoted post.

What are you talking about? It's just a different format than the person I was responding to had. The first column is the hop number, the next 3 columns are the min/average/max response times and the rest of the line is the dns name and IP address of the hop.

You can actually tell which ones are times because they say "ms" after them.

edit: and if you quote my post above it uses tabs instead of spaces, making it really, really obvious which ones are the time values. /me looks confused..

edit edit: Sharie, maybe I was just lucky with my tracert.. Doing a longer ping to 63.214.168.58, I get a steady 79-81ms but with a spike to up to +500ms about once a minute or so.. Even a few timeouts..

Ping statistics for 63.214.168.58:
Packets: Sent = 259, Received = 256, Lost = 3 (1% loss),
Approximate round trip times in milli-seconds:
Minimum = 79ms, Maximum = 911ms, Average = 86ms

I'm pretty sure that the ping times come after the router id. Please correct me if I'm wrong.

Therefore, The top ping numbers in the partial trace you have posted are from the previous router, and as such, the ping numbers for the last router in your list are missing. And that is the one, from the earlier post, that showed the higher than normal times.

Again, if I'm wrong and the ping times come before the router id then I apologize.