Rolling Restart for 1.24.6, 2008 Sep. 15-17

Wayfinder -- when you write the script, it's compiled on the viewer, and the bytecode is sent up to the server... under the old scripting engine. Under Mono, the source is sent up to the server and compiled there.

Thereafter, the bytecode for the script exists with the object the script is in, for both engines. If it's rezzed out in a region, it's in the simstate. If it's in inventory, it's on the asset server. When you rez an object from inventory, it is copied from the asset server to the simstate. In no event is the bytecode ever sent down to the viewer. Scripts have always been and continue to *run* server-side.

Basically, yes. After the code is compiled, the bytecode is created as it's own asset in combination with the script as an item and run and the source code is no longer necessary, except for when editing and re-compiling is needed.

No problem and thanks for the clarification!

I was just getting ready to have words with you over this I was /really/ hoping that we could recompile no-mod scripts into mono, and the viewer, as it stands, sort of gives you the impression that you can - you can go through the motions, it just won't list any successfully compiled scripts. However, as it won't list any "no-mod, can't recompile" scripts as errors, the uneducated assume they're happily running Mono.

This just reminded me of a question I have. Now that we have mono, and we set a script to compile as NON-mono, does it run under the old scripting engine? I was under the impression that something about the old scripting engine had changed with the mono deployment.

On the contrary, I see *mostly* sharp drops in your data aligning with the rollouts, on 6/26 for instance, its obvious. Less obvious, but visible if you look at the 7day trendline, is a long slow dip between 7/8 and 7/30 - a period when the rollouts were long and painfully frequent, followed by a general uptick in the "rollout free" period that followed. The only sharp rise that seems to correspond to a rollout is, as you pointed out, labor day weekend.

Im genuinely happy to see that you dug that far. Its nice to have a 2-way conversation on this. I'd be really jazzed if you would dig a little deeper and show me where Im wrong because I think looking at the data for all linden transfers tends to hide the problem.

A few ideas on why your data is a perhaps a bit too general to show the effect on shopping in stores... The largest inworld transactions are not necessarily purchases but transfers, and certainly not store sales. These large transaction wouldnt be affected by shopping habits and would tend to obfuscate that data.

I routinely transfer large amounts of lindens from one account to another for bookeeping purposes, to get them ready for conversion to usd or to transfer them to my partner. Of course this activity isnt affected by a rollout and doesnt correspond in any way with a dip in sales in my stores, its just a bookeeping transfer. Im sure many of the largest transactions are of that sort. It is also always equal to all of the money spent in my stores.

Other types of transactions, planned land deals, group distributions from adfarmers, gamblers, etc. where the exchange of lindens would be large, would likewise not correspond to "shopping".

Perhaps if you ran that spreadsheet excluding transactions over 500 lindens or so you would get a different result, since the average "shopping" transaction size is quite low. (my average is 170L)

What I can attest to personally is that I have every one of the 48262 transactions ever made in my stores catagorized and graphed by day of week, and time of day including 3day, 7day, 14day and 30day trendlines. If I walk through my spreasheet and mark the dips that appear to be irregular they *tend* to tightly correspond to service outages and rolling restarts, as far as i can calculate them from the calendar.

The last 2 days are appearing to be an anomaly to that, my sales yesterday and today are off by about 45% to lows I have not seen in months. I see no explanation in terms of service outtage reports and its happening despite the news that the rollout has been postponed. I've checked the search engine and my placement there doesnt appear to have changed. I would hate to think that just TALKING about a rollout is having that kind of effect. The only thing I can think of to explain this, other then unhappy coincidence and the possible effects of hurricaine Ike, is that perhaps whatever is keeping you from performing the rollout is also affecting store sales? Can you tell us the reason for the postponements? I havent seen it mentioned.


@Winter:

Based on 15 weeks of data with an average of 391 transactions per day and an average transaction size of 170L covering some 40000 or so transactions I can give you this data about daily percentage of total sales...

Mon 14.36%
Tue 13.97%
Wed 12.71%
Thu 13.65%
Fri 13.69%
Sat 15.77%
Sun 15.86%


Doesn't really look like that much of a fluctuation between a low of 12.7% and a high of 15.8% of total sales - but its enough to make an average difference of 11k per day between the worst day for us, wednesday, and the best day, sunday. See how that compares with your experience, and start keeping track, its useful

for hourly changes over 2 weeks or so (5000 transactions)

Hour % Sales By Hour
0 4.08%
1 3.41%
2 3.76%
3 3.31%
4 2.40%
5 5.43%
6 4.21%
7 5.63%
8 4.07%
9 4.80%
10 4.70%
11 5.99%
12 4.53%
13 5.38%
14 5.23%
15 4.13%
16 4.54%
17 3.08%
18 4.08%
19 5.29%
20 4.00%
21 2.83%
22 1.64%
23 3.50%

Showing that sales peak between 5-6am and 7-8pm with a sharp dip between 4-5pm which I put down to a confluence of things, people arriving home from work/school, etc. leaving one location and going to another, the dinner hour on the other coast of the us, etc.

Best time to catch people when they shop: 12pm sl time.

Hope thats helpful. Anyone who wants to start tracking their data gimme a shout in-world and I'll give you a hand.

You'll have to help me out with the utility of this trend chasing you're doing. Having discovered a statistical or numerical trend or difference in traffic or sales between days of the week, just what do you propose to do with that data? Do you think there's anything that can be done to effect a change to the trends? If so, for what advantage? In the case of the restart's effects, restarts "have to happen" so everyone just needs to get past the idea of smoothing any trend related to them, and in the case of the rest of the peaks and valleys, they're probably more aligned with new product releases (peaks), or maturing of available product offerings (valley's), than anything else.

Nice data though.

See, this is what I'm looking at. Primby Block is so busy blaming it all on LL, that (s)he hasn't taken the time to think of any other reason. People don't have money (they tend to not get paid on Wednesdays), or as mentioned above, product maturing, or you're not taking advantage of the Rolling Restarts.

How? Simple. I, myself, have caused impromptu Rolling Restart parties/get togethers to happen. Find a region that has already restarted, and then camp there until the all clear is sounded, pulling my friends there with me, who bring there friends who bring...you get the idea.

Now, onto the the other thing. Yes, it's only two bugs, but they're showstopping bugs. Now, I don't know this for certain, but I would lay money, real money, that there are a lot of vendors losing money because of these bugs. They can't convert their products to mono, or perhaps their vending system is screwed up because of the bugs. Either way, they're not making money. Probably losing more money then you are.

I'm not trying to attack anyone here, just trying to get people to look beyond themselves, and to the other residents that exist in SL (perhaps some of them your customers).

I just hope we get the things sorted out soon. I'm lagged badly since wednesday, and I've lost numerous no-copy items on rezzing since wednesday. Don't know what changed, but something did. I'm in Mapinguari, and while we do have a large number of textures and scripted items, it was not a major problem until Wednesday. Perhaps we got stuck with another overloaded sim with us on the server, or something else...? Anyhow, I'm always for a rolling restart to fix anything.

....And I also have vendorrs which no longer work properly.

PRIMBY, I will second you in applauding Prospero for digging deep into the data to try to find a trend. That is a refreshing difference... and a trend I've seen developing with 2 or 4 Lindens lately... that of getting into the trenches with the customers. Prospero, Dan, Qarl and of course Torley. I also was pleased to see new SrVP Fred nail a bunch of SL problems right on the head, with what looks like more than empty promises to fix them. He seems to have the background and drive to do so. So a pat on the back for all these folks.

But I also appreciated Primby that you just adequately demonstrated what I've told LL for a long time: there are people out here with educations who are quite capable of doing statistical and data analysis. Not all customers are so gullible as to just swallow every corporate line that comes form LL. And that's something LL needs to be aware of. We do tend to blow the lid of corporate BS... so it's best for them to just stop the BS in the first place. (NOT that Prospero was throwing corporate BS at us. Present company excluded Prospero.)


CINCIA. Tracking stats not only has foreseeable benefits, it sometimes has unforseeable ones too. In Primby's case, as a merchant, if he notes a certain trend in falling sales during specific times, he can ramp up advertising or events or specials on those days to offset such falls. That is a very common practice among RL businesses. My local theatre gives away free popcorn on Wednesdays because that is statistically their lowest attendanced day of the week. In addition, tracking statistics and trends is what clued me in 3 years ago to the fact that Linden Lab was stacking sims on the same server, and that lag was primarily due to asset and system servers, not customer creations (contrary to regular LL claims). Keeping accurate statistics and noticing regular trends is what RL businesses are built upon. Sometimes we track data that we have no idea what good it will do... because sometimes that turns out to be the most important information we could have tracked.

Now, I don't spend a LOT of time tracking data because I've worked most of my life as a business consultant and watchdog and any more, I just have a feel for this stuff that proves to be deadly accurate. But if I really wanted to nail things to the wall, if some corporation were paying me to evaluate the effectiveness of Second Life as a business platform... I would be doing exactly what Primby is doing: tracking every piece of data that comes my way.

CHANDRA, no offense but I would bet that Primby is much more acquatined with SL trends and their causes than most other people. I really see no need for insults. People regularly bonk Linden Lab on the head (myself included) and usually, they're right in doing so. To be blunt, I would question where your stats are that allow you to challenge Primby's statements with such dogmatism. Where is your day-by-day sales records-to-restart comparative analysis? Because while I don't keep nearly as thorough records as Primby, I do follow my own sales and yes, rolling restarts are murder on sales.

I don't really understand how anyone would believe otherwise; it's just common sense. Every rolling restart they have done for the past year (that I've seen) has damaged SL functionality instead of improving it. Then Linden Lab spends the next couple of weeks scrambling to fix it. When people can't log in, when 95% of all texture loads take 20 seconds or more to rez and 80% take more than 30 seconds to rez (see yes, we DO have the stats), when people crash regularly due to the latest rolling restart then yes, Chandra, rolling restarts ARE going to damage sales. I can assure anyone who asks that SL has more serious problems going on than it did a year ago. The very fact that LL recently lost 7% of their Premium users should be strong evidence that something is going wrong. With a system like SL, if it were running well the Premium user force should never decline. The fact that it has... at a rate of 1% per month... is really bad news. There is a REASON those people left, and it's not because they were pleased with the results of the rolling restarts. ; )

For the record, I am all for rolling restarts IF they do the job they're supposed to do. Far better than the weekly Wednesday morning half day offline that used to be the case. Problem is, people have come to DREAD rolling restarts. Wonder why?

Primby stated it accurately: Linden Lab should not include land sales and direct L$ transfers in their figures. I regularly transfer thousands of L$ between alts for one reason or another; that has nothing to do with sales. Thirty three percent of all avatars on Second Life are KNOWN by Linden lab to be alts. That means that logically speaking, most likely AT LEAST 50% of the avatars on SL are alts, possibly more. The L$ transactions between those alts do not count as viable economy.

It is possible for LL to track MERCHANT ONLY sales; only track L$ transfers that occur via object. Avatar to avatar transactions should not be counted. Land sales transactions should not be counted (the real estate economy is totally different than the merchant economy and follows different trends). SLexchange systems should likewise be specifically excluded, since they are primarily a "banking" system and hundreds of thousands of L$ go through them every day (millions?) Only an actual sale through SLExcahnge should be registered. Same with Onrez and the other big online L$ transfer systems. Don't count the L$ transactions people pay to an SLexchange broker machine. Only count the resultant merchandise sale.

Such things CAN be coded and observed with reasonable accuracy. It strikes me that one viable way to directly track merchant sales trends is to track L$ transfered via the BUY option of vendors. While that may not track the entire merchant force, I believe it would provide a very accurate sales TREND analysis. One doesn't have to test the entire population to achieve valid results; a significant sample can suffice. Tracking BUY devices would be a quick and easy way of tracking a significant sample of SL merchant sales, perhaps even the majority of them.

Only by creating code to track MERCHANT sales can Linden Lab accurately guage the effect of SL changes upon the MERCHANT population. But I assure that any merchant worth his salt already can predict the result of such tracking without the stats. It's pretty flippin' obvious.

@ Prospero, I would like to expand on my previos post abit, while I did say that in the old days was better, I was not 100% correct, but all the people in world do complain about the new 3 day system as well. I have really talked to people in world and have a couple of suggestions to maybe help with this situation.

1) Create more sandboxes and test the code there first.
2) Create regions that have 512 or 1024 plots and do land grabs as they do in SL Burning life, so people can have a plot for a the duration till the next deployment, you can script them so that plots that don't have enough traffic (the person does not stay long enough in that region) get booted and the plot is put up for grabs again.
3) Run a full beta test program that offers some kind of benifit to people who participate.
4) Try to do the deployment in 2 days.

I am sure if I had a brainstorm session or a test group we can find more suggestions.
But please please do something this 3 day thing is almost half the week of side by side regions not compatible, and alot of other disruptions that run right into the weekend and take another roll out to fix is really not a very good system.

PS please take my post as constructive critisim, I am not against you I am with you, I just want to see things that make people happy and SL grow as most of you working in the labs feel and work towards this goal.

PPS. English is my second language so please forgive my grammer and spelling.

Well I suppose this is getting off-topic now, but the point of keeping my data and analyzing it nine ways to noon is so that I can spot and react to trends in my stores.

Is that water feature still selling just as well after all this time, whats the drop off point?
what sells better landscaping, furniture, prefabs?
What theme is more popular, Oriental, deco, modern, tiki?
Which low cost items should we build more of, lighting, carpets, privacy screens, wall art?
Which type of sim gives a better return on investment?
How do I make a new sim return a profit the first month?
Whats the best time of day for an event? a sale?
Same question - day of week?
Same question - seasonal?
Hows my growth over time and whats driving it (so i can do more of that) ?
How are the various marketing avenues that are available working? should I use classifieds? stalls? billboards? contests?
How do various changes in our parcel ads work to increase profits?
What will be the effect of adding a new sim based on future projections of growth?
How do i increase my average transaction size? Number of transactions per day?
Why is business off the last 3 days and what can I do about it? Do I have a problem?

Its called running a business, and theres alot more you can know. and alot that you can do with the information once you have it - including spot things like how much it hurts business to have uber-frequent rollouts. And what can I do about that? Im doing it here.

I guess alot of people dont notice these things because they have no way to measure. Good luck with that

When I said I walk through my spreadsheet and spot IRREGULAR patterns, that means Ive already accounted for the REGULAR things, like bad wednesday's. I have thousands of products - New products all the time. Nothing to do with it.

Im not blaming SL. I'm trying to point out something they apparently dont see, theres a way to fix it. But they cant fix it if they dont see it as a problem.



Im not trying to have a beer bash, and my friends are not, in general, my store customers. That sort of thing is how you fill a club with people who wish they were someplace else but dont want to hurt your feelings. I don't invite people to shop.



Bringing this up is not about ME. Its about the effect on the economy for all of us who rely on it. As I pointed out in my response to prospero, If the bug was a showstopper it should have been found during an initial rollout if the initial rollout had been given adequate time to bake and let people report their findings. 1 day is not adequate time, obviously, or a showstopper would not get through a full rollout and into another one just to fix it.



If you arent trying to attack anyone you should consider not singling anyone out - just a tip

My business here is a healthy one - If things are running smoothly then, in general, I am thankfully busy doing it, not blogging about it. So perhaps I dont sing the praises often enough. /PRAISEON This is a fabulous platform and has given me an awesome outlet. Prospero rocks; Torley rocks; Spike rocks, and SL, in general, rocks. 3 things dont rock for me here, 1 is these ill managed rollouts. another is a showcase that encourages people to shop in someone elses stores for no good reason while Im paying tiers to run my business here, and the other is LL's policy regarding land sales and driving landowners out of business without comment or courtesy. Other then that, it all rocks - so there - praise a'la Primby

When things are not running smoothly I look for the reason, and, as wayfinder has rightly pointed out, I have the tools and the skills to pinpoint the problem - usually fairly accurately. When that happens I take whatever action seems appropriate to the situation, in this case - bringing the problem to SL's - and your - attention. Hopefully thats helpful to everyone in the end, If not - oh well, more information is never a bad thing. Of course my motives are selfish - I want my stores running as smoothly and profitably as possible. but what affects me is affecting others, and they can only benefit from the extra information.

Okay, so. Because I'm not the best with forums, I'll just do this like this.

First off, I don't have states, no. I don't keep track of that sort of thing, nor can I be bothered to. It's not my strength, but it doesn't mean I can't participate in this, try to put things into a different perspective.

Second off. No, I wasn't attacking anyone. the use of 'you' was a general one. I could have worded it better, sure, BUT, I didn't. That it was taken as an attack, I'm sorry. Not my intent.

I'm not calling all Primby's data wrong. There may be trends. But those trends would be there no matter, simply -because- Rolling Restarts are needed. I was just offering a different look at it, as Primby does seem to have tunnel vision in this area. Not an insult; I have tunnel vision with things too, and need people to point it out.

Perhaps he doesn't have tunnel vision, and everything would be fine if LL did what he wanted. Perhaps not.

ANYway, last point is directed at Primby.

No, you don't run a club, or have beer bashes. That you seem to think that's what I was suggesting is...insulting. You could host a 'poetry day' or an 'open house' or whatever. Advertise that your sim(s) are at the newest server code and people will most likely camp there. The longer they camp, the more chances of them buying stuff. Or just maybe increasing the traffic rating and there by moving you up in the Search listings.

Perhaps I'm wrong there, too. What do I know; I'm not a business major, nor do I try to run a business.

Personally, though; I prefer the three day Rolling Restarts. I don't think I could live with if they go back to what the rumours suggest and shut the place down for a day or two.

Just to make sure we're all on the same page... I don't thing anyone here is against rolling restarts. In fact, I'm all for them. As mentioned in my last post.. far better than a half day or more a week with the system down. Rolling restarts seems to be the only viable alternative. I believe we're all agreed on that.

But I believe what Primby is pointing out (as well as others) is that the restarts aren't working. Most of us have come to dread rolling restart days... not because of the restarts themselves (I can live with our sims being offline for 10 minutes)... the catastrophic situation that generally follows. If the rolling restarts WORKED, I'd look forward to them. In fact, knowing SL, people would probably have biweekly rolling restart events, celebrating something else being fixed on Second Life.

But the truth is, usually the rolling restarts bring in more problems than existed prior. They bring in new bugs and often things work much worse than they did before the restart. Then... Linden Lab starts scrambling to fix the mess of bugs created. But whether they do or not (quite often they don't and those new bugs just remain)... a week later people acclimate to the new problems and try their best to ignore them just like they have all the other problems. At least until the next rolling restart.

It's not the rolling restarts themselves that hurt business. It the mess that follows after. The fix to that is making VERY sure the upgrades are actually upgrades and that they work BEFORE bringing them to the main grid. That requires competent and professional programmers and system designers... not people who make excuses as to why something isn't getting fixed. I'll bet dimes to donuts Google doesn't put up with coder excuses. ; ) And I can't remember the last time I saw Yahoo implement faulty code (I know they have, but such errors are few and far between). I know SL is complex, but it's no more complex than a lot of other networked computer systems out there. And that's what people are griping about: not rolling restarts--- shoddy rolling restarts.

TO be honest I suppose my point is being somewhat obfuscated by the desire to stop having 30k days in my store when they should have disappeared months ago - call it tunnel vision - im after addressing the issues that affect me most.

But let me clarify my position -

I applaud the rolling restart methodology as opposed to the old days of disabling logins, its the way it should have been designed in the beginning, but im pleased as punch we got there.

There is one problem with the way the rolling restarts are implimented that keeps gettting lost in this discussion and thats the only thing im looking to address.

The rolling restart occurs now over 3 days, starting with an initial release to a limited number of regions on day one, progressing to a half-sized rollout on day two and finishing with a full-sized rollout on day three. IF they spot a problem in that tight schedule they will adjust it to address the issue. The problem is that that schedule does not allow them to spot the problems.

That schedule is faulty in one respect. The intial rollout on day one should be followed by a period of "wait-and-see" that allows the bugs which are inevitably being missed on the beta grid to surface during this test phase. 3 days, A week, 10 days, whatever it takes. That is the only real problem I see.

That and the fact that they should not occur so frequently, but as we can all see, there is not agreement that the rollouts affect the economy, even tho i assure you that any informed merchant can attest to it. The problem is that the majority of merchants here are uninformed and will shout down the rest of us as whiners. oh well, we know what we know. And I'll keep pointing it out until someone listens and takes a closer look.

I disagree that this is not more complex then most other networked environments. The client/Server interaction here is indeed complex.

I design networks. I design software that runs on networks. My video encoding platform puts movies in the homes of the majority of 20 million users that subscribe to a major cable company. Its still not as complex as SL because it is not as interactive as SL. People click a button to view a video on demand and our platform delivers it.

In SL everyone has to see the same thing at the same time and that creates many complex issues that mostly have to do with the fact that everything the client sees has to be defined at the server. The intermingling code is indeed complex and bug tracing is very dificult because most of the time there is a client server communcation required and the server is busy doing everything - delivering your textures, filling your inventory, making you look pretty, and, meanwhile, running any number of scripts that are required to make objects work and interact with the client. Amazingly complex stuff.

The bugs are constantly masked by server to server timeouts, server to client timeouts, dropped packets, etc. The coding to deal with all that is very complex, spread across many systems, and, if you look at their source code, they have no easy task to make changes that dont break things.

Pile on top of that the need for LL to expand its business model, increase its userbase, Keep its very vocal clientbase informed, and make their financials look good for investors... NOW, and you have the recipe for many of the issues we see here. Bugfixes go ignored or are ineptly patched because to do it right would require a re-write, and no company can afford that.

So for my part LL is doing a bangup job on many levels. They cant see everything though and where it appears they are missing something I/We are here to bring it to their attention.

1) monday-tuesday-wednesday 'kinda sucks' for sales. needs little explanation.

2) beginning/end of the month 'kinda sucks'. because of bills/rent/etc.

3) there is an overriding three-week cycle that can kinda make for 'superweeks' or 'supersucks' weeks. probably the intermingling of two-week pay cycles crossed with those who receive monthly salaries (on more of a four-and-a-half week basis).

4) none of the above have diddly to do with grid updates! ;0

/me wonders if the people moaning and whining about Linden Lab making sure the roll-out is as good as possible for these major bugs, are the same people moaning about their scripted attachments or the clubs contest board or timed-sex-HUD being broken because of those bugs?
Probably so.

These are major device-breaking bugs being fixed, lay off and let them get it right! It's important.

If no one spoke up when things were broken.. How would LL to EVER know that anything needed fixing?

That's what the issue tracking system is for, JIRA.

Complaining about Linden Lab every time they try to fix something isn't productive or helpful in any way.

No -- both the Mono and old scripting engines are still fully functional. Scripts that have not been compiled under Mono still run under the old scripting engine. You have to recompile it under Mono to get the performance benefits for the server that should come with Mono.

Do you have evidence for this? That statement -- including the "usually" -- is quite an accusation.

I don't deny that we've had rolling restarts that were problematic. We've had bugs introduced where we realize we have to unroll, and we do so. The 1.24 series of deploys has gone pretty smoothly, though. Yes, the initial deploy was delayed over the course of a week or so, because of issues we saw on the pilot regions... but there was none of this global things not working that you're talking about. Yes, there was a bug with Email in one of the roll outs, but we made the call that leaving that on for several days was better than immediately doing another rolling restart.

We know the situation isn't perfect. But the catastrophic picture you paint-- it's simply not correct. You remember the bad times, the problem rollouts, and believe me we remember them too. But please remember that the routine rolling restarts do not fit the description that you describe.

Thats simply not true. The Jira is for tracking specific issues, Commenting on LL policy, or the day to day running of SL, albeit on topic, is exactly what the blog/forums are for. The Jira is a wasteland for the most part.

If you have a complaint about a particular topic and there is a blog entry/forum thread created to discuss it then it is obviously the only appropriate venue to do so.

Commenting on the forums, positively or negatively, doesnt interfere with the process of rolling out the grid in the slightest. It's not like they sit and wait until the thread dies to do their work.

While Primby is correct that the forums are the correct place for general comments and questions about overall operations relevant to the forum topic (rolling restarts here), I want to offer a very contrasting view of the JIRA. The JIRA is very far from a wasteland, and has been extremely useful. Quite a number of the bugs found and fixed during the point releases of 1.24 have been bugs that were first identified in the public JIRA. Having bug reports there, with reproductions, gives us something to get a handle on so that we can actually do something about all of it.