Rolling restart for 1.24.7 2008 Sep 30-Oct 2

Hey Prospero how many people are on the team when you guys do the rolling restart? and if ya can can you list them here? Like maybe at the start of it or the end...

During the last rolling restart.. something bad happened to my sim, Healy.

The sim went down, and didn't come back up for almost an hour. (the point at which I would have reported it). Rather than TPing away, I simply flew to the adjacent sim.. So I watched as Healy went down, and then as Paw Paw went down.. then I flew to the East, and waited.. and watched as Paw Paw came up.. no Healy.

When Healy FINALLY did come up.. it seemed to have been rolled back about an hour or so from BEFORE the restart. About an hour's worth of work was missing in my workshop. Thankfully, I had the foresight to "Take a copy".. because I had just set up a packaged product, and with the sim rolled back, that package was completely missing.

I'm sure it was just a matter of a corrupted save-state at the time of the restart.. so the automated system went back a "save point" and updated THAT instance of the sim..

While this sort of thing seems to be exceedingly rare, it's important to note that it CAN still happen from time to time.



This sounds like a fairly fundamental change to how llEmail is done.. MANY MANY applications rely on using several (or dozens) of "Email Slave" scripts, to overcome the throttle placed on llEmail.

Why would we want to overcome this throttle? Keep in mind, that llEmail is the *ONLY* method for objects to communicate inworld over distances. Objects can't IM each other. (honestly, isn't it time that this was changed?)

https://jira.secondlife.com/browse/SVC-258
https://jira.secondlife.com/browse/SVC-310

This is something that we REALLY need to have documented. This change may have broken a great many "networked vendor" applications.

This sounds like a fairly fundamental change to how llEmail is done.. MANY MANY applications rely on using several (or dozens) of "Email Slave" scripts, to overcome the throttle placed on llEmail.

Why would we want to overcome this throttle? Keep in mind, that llEmail is the *ONLY* method for objects to communicate inworld over distances. Objects can't IM each other. (honestly, isn't it time that this was changed?)

This is something that we REALLY need to have documented. This change may have broken a great many "networked vendor" applications.

I use llEmail sometimes when I want an object to communicate to me regardless of online status, and I don't want the possibility of losing the message due to IM-capping when IM's get forwarded to email. I find that there are times when the 20-second throttle isn't very helpful.

With that said, if the intention of the throttling is to prevent spamming agents, then I suggest that llEmail be throttled ONLY when the destination is an agent, and not throttled when the destination is an object.

llEmail was throttled when the destination is outside of Second Life. I know that messages within the same region were not throttled (beyond the 20-second delay), and I *believe* that messages between different sims were not throttled (between the 20-second delay).

In any event, the throttle remains quite generous. If you aren't seeing messages dropped, then the throttle hasn't affected you.

Hey Prospero can we get the % complete again?

At 6:04, the first-half rolling restart is 36% done.

At 07:00, the rolling restart is 80% complete.

hey!

my sim has restarted, everything looks fine.

Thanks for the information.

I have to be there and manually reset things on 2 of my SIMS every time they restart. So that utility service window DOES apply. At least for me.

Is there any way to have a notification sent to people who own a whole mainland sim about restarts like a day or two before? I often don't have time to check the status page every day. I've been caught several times in the middle of things with (the sim your on will be restarts in 5 min). Then I have to scrambe to save what I'm working on if building or if it's an event trying to find a place that isn't restarting for everyone to goto.

Honestly, The TP out and wait for your sim to come up doesn't work when every sim you TP to is being rebooted. If there was a way to flag what sims will be rebooted on what day this would help a great deal. Maybe semething like setting the name of the region in the titlebar red, perhaps a new icon up there (I'd suggest a skull and crossbones icons personaly. heh), an indicator on the map, or soemthing. The time we get the info about "even number sims will be rebooted on this day and odd ones on this day" is helpful, but not way to ID those before you are there.

I don't know how the order for rollouts are picked since my main sim has been in all three groups during diffrent "upgrades". Is ther a way to request not to be in the pilot rollout group or anything?

Do rolling restarts allow estate owners to stall the restart for an hour?

The whole "I own a Whole Mainland Sim" being a weird animal in it's own right.. as it's well known that those full-sim owners get kind of short-changed in terms of estate rights.

You at Linden Labs may or may not have realised that LL is organising a major event inworld this week...

An event that is very 'usage' heavy on its servers....

Was this the most intelligent time to be rolling out upgrades?

I mean, I hate to be a kill joy...but you hardly have the greatest reputation of trouble free roll-outs.......

How about simply waiting a week, using a bit of fore-thought and not avoiding setting up a potential melt-down...whether it happens or not...

What's that great phrase...engage brain before taking action?

Again... three security fixes. Isnt't that more important than teleporting out of the sim for 6 minutes?

Sure, If you know of a sim that is not also restarting. That is one of the worst problems. You keep TPing and every single sim you goto is rebooting so you get logged out anyway. The first few times that happnes it's a bit funny, but grows old afte a while.

I know it's not like they can say "here these sims will never be restarted during a rolling reboot" as I don't see a way to shove all 50,000 some logged in users into a few spots. heh

Something like a notice sent to the land owns of the sim that will be rebooted the day before would be nice.

Yes, It is a bit of a weird animal I must admit. A system of being able to "lease" mainland could be nice. I would guess some of the things they could add are left out simple because the solution is to get your own private island.

However, As I don't need 99% of the featues of a private sim for my business ventures it's not worth the extra $100/mo to me. Would I say pay an extra $25 or $50 for some of the features on a mainland sim.... possibly.

I'm pretty sure that since they switched the rolling restarts to go by server number rather than north-to-south, estate owners are no longer able to delay a maintenance restart. It's because they are upgrading all regions on the server at once, so if an estate manager could delay it, then there'd be one region with the wrong software... or something like that.

I just know that I've seen that note for almost every rolling restart lately, that estate owners cannot delay them.

And for what it's worth, I agree with the person upthread who has pointed out that unless we can be told in advance which day out of the three our region(s) will get the restart in, it is a 3 day window. You can rephrase it any way you want, tell us it's only a few hours this day and that, but the day someone says "There'll be restarts over this three-day period" is the day we need to be told which day our region(s) will be hit. Otherwise it's still a three day window. Period.

And comparitively speaking, at work the ISP who provides our T1 line will tell me with 2 weeks notice if there's going to be a 30 minute downtime. Two weeks ahead, they'll tell me what 2-hour window my downtime will happen in. My monthly service fees to LL are double what we pay at work for the T1 line. If only the service level was as high.

-Atashi

My Point Was That When U Do ThIS Stuff BAD DOES HAPPEN And If The Estate Manager Wouldnt Have Accepted My 10000 L's To Do The Rollback Id A Been Out 1600 Us Dollars Because You Guys Screwed Up On The Restart And Rolled It Back More Then 5 Days Worth.
Thats The Point I Was Making.
You Knew U Screwed Up U Did A Roll Back And Lost My Stuff On Ur Own And Then Admit It But Say Opps My Bad But I Cant Fix It Unless A Estate Manager Asks For It Cause This Way If They Dont U Have To Respend All That Money Again In Sl. Is How I See It

Some answers.

First, no, there's no way to delay the restart when you get the warning.

If you want to get notifications of when these rolling restarts are going to happen, I recommend you subscribe to the RSS feed for the status blog.

Re: owning a whole mainland sim, no, you don't have as much control as estate owners do. However, also bear in mind that the tier for a mainland sim is $200, whereas the monthly fee for an island is $300. If you want or need the control you get with a private island, buy a private island.

Re: having to reset everything in your sim after a restart : what needs to be reset? That should not be necessary, and I'd be curious to know what it is that needs to be reset after your region restarts.

And, again, I'm not telling you "there will be restarts over this this three day period." I'm telling you that "there will be restarts during these three few-hour windows."

Re: TPing or flying out to a sim that's not restarting : depending on where in the process it is (because there tend to be more openspace sims, thus more regions per server in PHX), there are typically 100-600 regions down at any one time. Out of a grid of 30,000 regions, I find it very difficult to believe that every region you try to teleport to is down at the same time as the one you just teleported of. That would be extraordinarily bad luck.

The vast majority of regions come back 5-10 minutes after they do down. Sometimes one doesn't. If your region stays down more than 20 or 30 minutes, contact support, because something is wrong.

hey!

i cannot see all the problems with RR
as a simowner i have a look at this forum -> i can see if and when restarts will happen.
i build stuff so i often make backups - my mac can crash also, or my isp can lose the connect, or, or, or, ...
i get the restartmessage, save my work, log out of sl, clear the cache, restart my machine and voila when i log back to sl my sim is updated.

groupmembers can tp to my mainlandparcel if they want it.

- i drive a business and lose "millions of dollars a day" cos of RR?
well its my risk i CAN run a business but i MUST NOT
- my special event gets interrupted cos of RR?
5-10 minutes and people can log back, maybe its a good idea to tell your visitors what to do if something unexpected happens, a sim can crash too or, or, or, ...

i think LL is doing a great job, maybe some mistakes will happen, well there are people working.
but yes - i know im the only godlike one never makin mistakes

many thanks to Prospero Linden for all the informations.

jm2c

Prospero, you're one of my favorite Lindens, but I am having trouble understanding how you can't accept why this is not good enough.

If the "three few-hour windows" happened together, one after another, then maybe that would wash. Like, 'your region(s) will be down from 6-9am, 9-noon, or noon-3pm'. That is three few-hour windows. Of course, if you can't know which window you're in, it's really one 9-hour window. When those windows are spread over 3 days and you cannot tell us which of the three windows will hit us, you're telling us that sometime in the next three days, your server will go down. It's great that we only have to worry for a few hours per day, but it's still three days.

I don't want this to sound condescending but it's going to anyways. Nonetheless, here's the best analogy I can offer.

The guys in accounting only issue paychecks during three few-hour windows. They are Tuesday, Wednesday, and Thursday, between 10am and noon. So, do you only have to wait a maximum of 6 hours to get your paycheck, or do you have to wait a maximum of three days? What if your mortgage is due Thursday morning? Or your car payment? What if you need to book something during those three days which means you might not be there when accounting is ready with your paycheck? Doctor's appointment, whatever. How do you make plans without knowing? You don't. You cannot make plans for the three days.

Yes I know it's not quite the same, an oversimplification, whatever. I know that the regions are *usually* only down for 5 minutes. The problem isn't when everything works perfectly, it's the uncertainty and the not knowing, and it's when things go wrong. When my main region did not come back within the 5 minute window, it was down for 3 hours. Contacting support was useless - they had already decided not to fix broken regions till the roll-out was finished.

If there was some way to know ahead of time, we could at least do some planning, be prepared in some small way. For people planning events, it's a huge deal. For folks just managing a store or rentals, there is at least something they are aware of, some information to pass on to our own customers, if it's needed.

Really, if there's no way whatsoever to narrow it down so that we can be told in advance which *day* our region will be down, why bother announcing it at all? Just let them randomly reboot and have someone say it was another oopsie, the server was unhappy, whatever. I think we're fairly used to that.

Prospero, I apologize for the rant. My exasperation is not directed at you personally. It's just the result of a lot of little (and some not-so-little) things over the last several months, and I'm finding it increasingly difficult to just keep quiet and endure it all.

-Atashi

Is it just me...or did the last re-start cause massive returns of objects to inventory in some sims? I now have a gazillion objects in my Inventory, a bare piece of land, and no house.

Homeless in SL

Welcome to SL I have ben lucky ot not have that happen yet. but I have personaly seen 3 accounts of it. I'd guess you jsut have to spend the hours rerezzing all your stuff.

08-21-2008, 04:17 AM #23
Prospero Linden
Linden Lab Employee

Join Date: Aug 2007
Posts: 110

Note that this thread is about the 1.24 server release; there will be a thread about the new viewer release candidate when that happens.

Old 08-21-2008, 05:55 PM #38
Prospero Linden
Linden Lab Employee

Join Date: Aug 2007
Posts: 110

Hey all -- the deploy is currently paused, as we wrestle with some issues that showed up during the pilot roll.

Prospero Linden
Linden Lab Employee

Join Date: Aug 2007
Posts: 110

Update on paused deploy : we re-deployed 1.24 (now 1.24.1) to the pilot group last night, and it appears that the rare crash bug we encountered on Wednesday is now fixed.

We have some more investigation to do before we move forward, so we won't be going forward today. Most likely we will be continuing the roll early next week.

Seems to me Prospero my region was rolled twice on that server version. Correct me if I am wrong. I can understand your point that the "whole" grid was not rolled for PR purposes.

Renee