Rolling Restart for 1.24.6, 2008 Sep. 15-17

I don't know what's going on with the rolling restart... but avatars are crashing right and left. My friend crashed 4 times within 15 minutes and I crashed twice within 5 minutes (at which time I just stopped logging in. There's always RL).

Form: these are sudden, no warning, "error message being sent to Linden Lab" type crashes.

I hardly ever crash. For me to crash twice within 5 minutes is bad news.

It wouldn't be such a waste of your time if you also had a First Life, and they're not your resources as long as you're on a Linden grid. You might have a valid complaint if this were OpenGrid, but then again, they don't do gridwide restarts, sims restart seemingly at random there.

LL insisted I download a new RC last week. I crashed so many times and so consistently over two days that I had to give up and go back to the regular viewer. Ugh. Lag city.

I am crashing wildly too. I reduced all my settings down to the lowest and I still crashed every ten minutes. I hope this is only a temporary problem...

Has the restart started?

There is no rolling restart in progress-- if you're having crash problems, it's not because of that.

Re: rollouts and rollbacks, you'll notice that we haven't had any of those with the 1.24 rollout sequence. The first 1.24 rollout took a week longer than originally expected, because we saw issues with the initial pilot roll. We repeated that initial pilot roll a couple of times, so that by the time we did roll out to the whole grid, it was a good release.

Since then, each point release has gone fairly smoothly. Yes, there was one case where the pilot regions were updated twice and had to be rolled back once, but we haven't had to do a half-grid or whole-grid reversion to an earlier release.

Re: finding any given bug, it's always easy to say in retrospect that "you should have found that before you rolled it out to the grid". But things that happen intermittently are *very* hard to find, and edge cases that are triggered rarely with a set of scripting behaviors will not necessarily be caught until we have a huge variety of people actually using the code. The bugs that we're fighting with in this latest release were bugs that it took some effort to really reproduce; QA has been working with some residents who were the only ones who seemed to be able to get these bugs to crash the server. Yes, ideally, everything would be perfectly tested by the time we roll it out to the main Second Life Grid. But some things simply can't be tested until the code has a Second Life-sized load and a Second Life-sized variety of uses on it, and that only happens when all of Second Life is running the code....

> Yes, ideally, everything would be perfectly tested by the time we roll it out to the main Second Life Grid. But some things simply can't be tested until the code has a Second Life-sized load and a Second Life-sized variety of uses on it, and that only happens when all of Second Life is running the code....

I would not suggest that you should find every bug before release, god knows there are hundreds hanging around from the beginning of time. The point is that each rollout should be given adequate time to bake - to allow new, important bugs to surfoce and percolate into the jira, before the full size rollout. if you need 5 or 10000 regions, 3 days or 10 days, whatever gets that done.

But you say it went smoothly, and yet we are rolling out to all 30000 yet again to fix 2 bugs. If they aren't major and are affecting just a few people then why not leave them till the next bugfix and include a few more rather then screw up the grid for 3 days to fix a minor bug?

If they are major and are affecting lots of people, then you just make my point simple, you didnt see it in the intial rollout because you didnt give it time. Not because its so hard to spot but because you didnt give the initial release adequate time or breadth to let it be found. Its only hard to spot a major bug when you have a schedule that is completed in 3 days. Everything looks smooth at 200 miles per hour.

Address the real point Im making please, if you can roll it out once and take the time to adress the bugs you saw in the first day then why not make that the norm and give yourself time to see the bugs you didnt see, couldnt see in one day? Give every rollout a week to bake before you complete the rollout, why doesnt that make sense to you?

If you will acknowledge the cost to the economy - or do the digging to calculate it - then you would see that this is far too costly to your customers for it to be worth rushing through like this and having to repeat so often.

Will you post if its odds or evens and % of done this time also?

An update, also posted to http://status.secondlifegrid.net -- the server deploy has been postponed a day, and will start either Thursday evening or Friday morning.

Re: claims that the in-world economy suffers during rolling restarts : the data do not support this claim. Check out the image attached to this post. It is a plot of the total resident-to-resident L$ exchanged, plotted vs. day, for the last three months. Highlighted in red are the times over the last three months when we've had rolling restarts.

A few things to note about this. First, there is a big peak at the beginning of September. While you might say that the rolling restarts on either end of that peak are eating away at the peak, bear in mind that that peak is Labor Day weekend in the US -- also a time when we hit a new concurrency max. The peak is there simply because it was a holiday weekend that had a lot of in-world activity. Indeed, the graph is pretty spiky, and you can see that frequently local peaks correspond with weekends... and there are dips in between them whether or not a rolling restart was ocurring that week.

The data we have on the overall amount of L$ changing hands simply do not support the notion that the in-world economy is systematically suffering during days when we have rolling restarts.

Prospero.. I have a question. And you may not be the right Linden to ask this, but looking at your chart above has brought up a very important question for me. Perhaps you can pass this question on to the correct Linden.

For the last year or so, I have noticed a trend that "generally speaking", sales suck on Tuesdays. Not EVERY Tuesday.. but on the whole, MOST Tuesdays (3/4). Now, I've always assumed that it was just a strange coincidence that demand for my personal store's items, fell off on Tuesdays... That perhaps the sort of people who like the things I make, are busy on Tuesdays. Or perhaps my classified ads or other weekly listings were de-listing on Tuesdays, or maybe I was just imagining it. I don't keep hard records of my sales figures, so I can only say that it "feels" like it's happening.

But looking at your chart, seems to suggest that there is a repeating 7-dot dip in the whole economy, which appears to be on Tuesdays. Is there some "Linden-Side" trend that correlates this observation? I can only say that your chart SEEMS to suggest this... I'm not an expert on charts.

Which leads me to my second question. Is there something going on in the economy on Tuesdays? Is concurrency down on Tuesdays, or is Supply Linden adding funds on Tuesdays (or perhaps Wednesdays?). Is there something about the search system that resets on a weekly basis on Monday night or so??

tuesday dip might be explained on the fact that monday partly falls in the previus weekend for some timeszones.
Another effect is that stippends get payed on tuesday so that after tuesday there is aditonal funds afailable for spending.

Concurrency is always highest on Sunday, and close to that on Saturday. It's not surprising to see weekly trends in things, because so many things in our lives run on weekly cycles.

Why Tuesday? Dunno. I was mostly posting this in response to the notion that rolling restarts correlate with dips-- as that's on topic for this thread I'm probably not the right person to get into deep discussions about all of the trends. However, I suspect that what Abagail mentions is probably part of it.

I've always leaned on the old addage that "The world works on Tuesdays".. Something my dad told me of his experiences as a Salesman.

People take-off Mondays, and many will leave work early on Wednesdays to "break up the week". Some people are off on Fridays, and many is the time that Thursdays are taken off, or people are out-of-office. But if you want to be sure that someone will be in their office, stop by on Tuesday. At the time, I worked in an amusement park, and while we were open every day of the week during summer, the place was eerily dead on Tuesdays. The absolute BEST day to go, if you had a season pass.. because the lines were almost nonexistent.

The idea that SL might obey this rule also, is kind of charming. Of course weekends are hot for sales.. And Mondays make sense, with people doing a little pick-up shopping following the weekend. If people really are "working on tuesdays" perhaps they're staying late, and too tired to get on SL after putting in a full day at the office. But you'd think that that would be reflected in concurrency. The fact that the overall economy seems to support a "tuesday dip" would seem to indicate SOMETHING is going on.

As to the timezone issue.. I live in the Pacific timezone. There is almost no part of the world, in which it's still Sunday, when 12:01 am Monday hits in the Pacific TZ. But at 4am here, on Tuesday, Wednesday starts for some people on the planet.

As to Stipends... that's something I did not know.

I really don't see any problem with rolling restarts. The 5 or 10 minutes that a sim is down is pretty much irrelevant for sales, imo as a large store owner.

It doesn't seem that the rolling restarts are affecting sales, it's the notices sent by the Lindens to not rez or spend any money.

@Winter: In RL, Tuesdays suck in retail. This is just virtual reality mimicking objective reality. People just don't buy as much on Tuesdays. Wednesdays are "hump day", so the buying picks up, as the slide to the weekend begins, and Monday still has momentum left from Monday, and new purchases from going back to work and finding out you need something. But Tuesday is always the doldrums.

The one time I got zhockened on an illicit bottle of wine a co-worker brought to to work at the old bookstore job I had while back in high school was on a Tuesday. I think we had three customers all night. They never knew we both were tanked. (Yeah, I was a lightweight.)

So it's not unreasonable that Tuesdays are lousy for sales in-world. Plan events for Tuesdays, and see if you can boost sales a bit by attracting attention. We tried to put book-signings on Tuesdays so we'd offset the lousy sales.

Prospero, thanks for the reply. No rolling restarts happening? That is most odd. What I mean is... 3 users state they suddenly started crashing uncontrollably, and multiple users on my sim were crasing all on the same day, and no rolling restarts. Makes me wonder what could have done that.

What I did was reset our sim. Once it was reset, crashes stopped. So obviously it was a server problem and obviously something was going on LL side. I have no idea what of course, but whatever it was, it affected more than one region. So my guess is that somewhere in the haystack, there was a central avatar server or something that was tied in to sim status, that was crashing people across multiple regions. Resetting the sim stopped the problem... but an interesting problem for sure.

(Oh to correct prior post. After posting, I decided to give it one more try, logged in a third time, quickly reset the sim, problem vanished).

Again thanks for the reply Prospero... and also the excellent data. I'm enjoying seeing you, Dan, Qarl and others stepping to the front lines to speak with customers. The problems still exist, but you guys do a lot to reassure customers that at least some Lindens are concerned... and are listening.

"This means that the source code has to go down to the viewer. Thus, in principle, under the old scripting engine, recompilation means that somebody using a hack or a customized viewer could grab the source code to no-mod scripts."

I have to admit I was totally unaware of this fact... and I'm totally appalled by this fact. The bytecode was sent to the viewer-- unencrypted??? And I take it this is still unfixed?

No wonder so many "freebies" have suddenly appeared over the past couple of years. Not only are people copying textures and prims... they're copying scripts as well.

Just another nail in the coffin of "bring your RL business to Second Life". Yeah, right. Like people are going to risk their business proceedings with a company that doesn't implement the simplest and most logical of security measures.

Wayfinder -- no, under the old scripting engine, no-mod code was not sent to the viewer in any form. Under the old scripting engine, you could not recompile no-mod code.

You can now recompile scripts under Mono, even if they are no-mod... but there is no need to send the code down to the viewer for this recompilation.

Basically, the source code hast to be where the compilation happens. With the old scripting engine, the compilation happens at the viewer, so you weren't allowed to recompile no-mod scripts. With the Mono scripting engine, compilation happens at the server, so there's no security problem with allowing people to recompile no-mod scripts under Mono.

All is well.

To those keeping track at home -- as I've updated on the status blog post, the rolling restart has been pushed off to next week.

WOOT!!!!!!!!!!!!!!!!!!!!!!!!!!! Didn't notice that till now. This will prevent some very nasty headaches. I'll post it to the scripting forums later this afternoon.

TY for all of the updates and feedback Prospero, they are appreciated by many of us.

Two bits of good news: MONO doesn't have a security problem and it's compiled on the server. That should help bunches. BUT... not sure I understand this line:

"under the old scripting engine, no-mod code was not sent to the viewer in any form. Under the old scripting engine, you could not recompile no-mod code."

If code was compiled on the viewer instead of the server, and no-mod code was not sent to the viewer, how did it work? Was the no-mod code kept in binary format and executed there, or...? I'm still not getting the full picture. I'm sure it's simple, but I havent' had my coffee today.

OK, I must apologize -- I was mistaken.

While compilation of Mono scripts happens on the server, in fact there is no way to recompile no-modify scripts under Mono. Sorry about that, and about the confusion