Why the grid is wacked, and why you should put up with it.

This time last year, there were approximately 91,000 users in Second Life, with the average load being around 3- to 4000. Today? 1,814,141 with the user load exceeding 15,000. That is an increase of about 2,000 percent. For every person there was a year ago, there are twenty people now. Think about that.

But Linden Labs should've known! They should've been ready! WHY AREN'T THEY FIXING THE PROBLEM?!!1!one

Linden Labs has... what? around 80 employees? 160 online accounts? And you expect them to be able to fix -every- bug that Second Life has to offer. You expect them to be able to optimize -every- line of code. You expect them to be able to increase -every- bottleneck between where their internet lines stop and your home line begins. Lindens are people, too. They require sleep, they require food, they have families and friends they want to be able to spend quality time with. THEY CANNOT WORK 24/7. Even if they could, they are horribly understaffed, and couldn't do it anyways. But before you go blaming that on them too, consider this: That small staff was all that was needed not too long ago, before the population boom. They have help wanted signs all over this website. They're trying hard to get more people, but you can only hire so fast. THEY ARE WORKING ON FIXING THE PROBLEM.

But... But they always add new bugs!!! They never fix old ones! THEY SHOULD PAY FOR ALL OF THESE PROBLEMS!!!1!

Again, consider what we are talking about. This isn't Everquest, this isn't World of Warcraft. In those MMO games, your choices are limited in what you can do. As such, the amount of coding to go wrong is also limited. In addition all of the content in these games is Developer created.

In Second Life, there are no limits. You can do virtually anything you please, and you can build virtually anything you please. Try making code that can handle every possibility, every outcome, and every instance. Try making code that doesn't just do that, but does it perfectly and on the first try. It is so difficult it can easily be counted as an impossibility. The more open-ended the game is, the more bugs it has, because there is more room for bugs to spawn. Bugs will get squashed, but you have to be patient, and you have to stop shouting bloody murder every time the smallest flaw rears its ugly head.

Oh yeah!? Well Linden Labs OWES ME MONEY for all of the sales I've lost from their CRAPPY coding!!!!!

Linden Labs owes you nothing, in the same sense that the world owes you nothing when the front door gets stuck closed on your shopfront downtown. Linden Labs did not promise you fame and fortune, they promised you an online world with endless possibilities. If you want to try and make money, go right ahead. I wish you the best, and so does Linden Labs. But it's not their problem when you fail. With individual risk, comes individual responsibility.

... Screw you, you Linden Labs loving hippie!!!

Yes, screw me for getting tired of listening to the endless rants about how Linden Labs hates us all and is secretly trying to increase our blood pressure high enough to give us a brain aneurysm. I am growing exceedingly weary by all of the complaints, especially when if a good majority of these complainers would stop and think about all the different reasons WHY the grid is in its current state, they would have very little to complain about. Second Life is growing at an astonishing rate, and Linden Labs is doing all it can to catch up. It will catch up, and when it does things will get better. Until then, you can either suck it up and enjoy the ride, or you can leave. If you want to keep complaining, please, do us all a favor. Exit via stage right.

And with endless possibilities comes the future possibility of more bad days... and good days as well. Problems can be overcome. It's stagnation that is a world-killer.



I've no doubt that many leave quietly, 600,000+ players not active in the last 60 days indicates to me that SL is very much a tourist destination for folks to visit once or a few times before they move on.

It's only the tiny minority of excessively vocal ones that keep feigning an exit, hamming up as much drama over and over, teetering on the edge of dashing from the stage in tears... that ultimately discrediting themselves by not actually following through.

I'm hopeful... and believe that the current problems are difficult, but fixable, and that things will get back on track. If not... well, we'll deal with that in our own ways.

--
"ooo, that really hurt, ahh. ah... " thump kick thump "ooh, ooow, aah, ooo, ooo, oww, aaagh, oow, oww, oo, aagh... " -- Paul Rubens' nearly endless death scene in Buffy the Vampire Slayer (movie)

I agree that this is very difficult, and that they are trying their best and don't deserve too much complaint. But on the other hand, by charging money - quite a lot of money for many people - for participation in SL, they are telling people that they will be able to do these things, even though they are difficult.

If someone tells you "For US$1000, I'll make you able to fly, in the real world, without any machines or aids", and you pay, you don't expect them to end up giving you nothing and say "you should have known it was impossible, but I tried my best, so I'll keep the money". I know that LL is still a relatively small company and needs our support, but they seem to want to pull away from the other side of that model.



The landlord of the store could well be held responsible for this - and in the case of a database failure, Linden Labs is in the position of the store landlord.

I do have sympathy for Linden Labs and I know they have a very difficult job. But equally, if I had just bought an island (especially in the US$195 rush) only to have my business, or participation, suddenly be dramatically reduced because I couldn't tell people that I'd moved, I would be very upset too.

LL is charging for Second Life and they are failing to deliver a working product. That is inexcusable, especially considering the kind of real money that is now flowing through SL.

Highly irrelevant. Think of something else instead:

the number of users right before last Friday was pretty much the same it is now. Before the last patch the servers were capable to deal with search queries. Post the patch, all sort of emergency measures had to be put in order to keep the grid barely functional.

To answer your original question, the grid is wacked because changes made in last patch (either client or server-side, or both combined) broke it. And there is zero reason why "you should put up with it". The same programmers who broke it, should fix it. Or be dismissed and replaced with ones that can do better work.

Kezz actually forgets two important things.

1) the population boom that is causing so many troubles is started with a decision LL made letting people make free accounts.

2) LL is charging people for a product but having problems delivering it.

Sooooo...let me get this straight...in Secondlife there are no limits?...I can do whatever I want?...build whatever I want?
hey that sounds great!





So, wait...I thought you said I could do anything? Doesn't LL give me a world where I can do anything?
Oh, that's right...just not now....I understand.

Seems to me that they're not providing the service that they advertise. I think THATS why many people are angry.



Here's one way to solve some of the lag and poor sim performance. Get rid of the freaking camping chairs. Those have gotta be one of the stupidest things I've ever seen.

NOBODY likes a camper! They're zombies!! wasting our bandwidth and killing everybody elses performance.
And for what? a measely few lindens.....its pathetic.

Club and shop owners should be ashamed of themselves. They should be creative and think of other ways to attract customers instead of these silly camping chairs and dance pads.

Yes, that's correct.

If you're going to split hairs, realize that the words "could" and "can" were used. No where is the success of your endeavors guaranteed. What matters is that you have the potential to do whatever you want - if you're clever enough. That limitation is coming from you, not Linden Labs.

I do however totally agree with you on the removal of camping chairs. Nothing but leaches bogging down the system with no useful input.

1 want to make a 2d sprite based video game

now i cant becuase the animation system sucks the prim movements are too delayed, colission is hit or miss, theres no extended keyboard support ect ect ect

i would tend to agree with the op, but this time 2005 i was standing in a dead empty sim waiting upwards of 10 min to save a script unable to move

guess what i did last night

so i guess things have improved, they have increased the load, while delevering the same crappy performance

I don't think he's talking about long-term things here, though, Fenix.

Over this week, I've had several occasions where sims crashed and rolled back - on one occasion, some work I'd done on a script was wiped out. That's the kind of service I think he was talking about. I hope you aren't really arguing that trying to save my script was an "endeavour" for which success wasn't guaranteed

I think aboutthat every time I get online.

The question is, how many more paying users are there. compared to this time last year? 5*? 2*? It's not 20 times.

Linden Labs has increased the load on the grid 4-5 times, and increased the cost of running the grid similarly, but have they increased their income by a similar amount?

Think about that.

There's a lot of stuff in SL that is unavoidable. The low frame rate, for example, is due to the unique content in SL. But the load on the grid is mostly do to avoidable mistakes LL has made.

* The user base (as noted) is artificially inflated by non-paying users.

* They don't seem to consider database load in feature design.

The new friends features require more access to the presence and user databases. These are slow (look how long it takes to update map visibility in the friends list). A simpler design that didn't make map visiblity a "per friend" option, but just let you make yourself mappable or not globally, would have been easier to implement, cause less load, and allowed people to hide when privacy was required.

I'm the first to defend them when they're attacked for things they can't do anything about, but the grid load is not in that category.

You are correct Yumi

Of course not Yumi, looks like I misunderstood. My mistake.

As a matter of fact, I've lost track of how many time I've lost work due to sim crashes/roll backs.

Argent- I gotta quibble with you a little.

Load on the grid is generated by online players, not offline players. Last year we were at what, around 8k online, 9k? Now we're at 15k to 16k. not quite double, not 4-5 times.

And I don't think it's the number of paying customers that matter (though I'm certain that number has doubled since last year, just look at how much the grid has grown!), but it's also how much those that do pay are paying. There's more landlords than ever... even if there isn't 2x the number of premium accounts as last year.... I'm quite certain that the major land barons are paying more than 2x what they did a year ago.

More customers certainly increases the storage requirements, but with the architecture that LL uses, that's fairly trivial.


Also, keep in mind those 300k or so players active within the last 60 days... but no active in the last 30 days. These are alts, tourists, people that really don't add load. And, 30 days later, every one of those 300k "inactive 30 to 60 days" avatars is a completely different avatar. After 2 months you have 600k of basically 'dead' tourist and alt accounts.

What counts is how many people are online... and how many people are invested enough in SL (premium or renters) to be able to leave persistent objects in the world.

As well... LL sells via the LindeX...

So... has LL's income (increased) in proportion or better:
to the rise in online avatars? Absolutely.
to the number of land owners? Absolutely.
to the number of 'regular accounts'. Ah... Dunno. We didn't have population stats last year that accurately reflected 'players active in the last 2 weeks.' (anyone inactive >2 weeks isn't a "regular" in my book. They might be when they get back, but not at the moment).

Has LL's costs increased in proprotion to the growth?
Absolutely no doubt. But there were positive mumblings of approaching profitability not that long ago.

So... Do they owe us a perfect world because we pay them? Nice to think so. (I don't think we were ever promised 'extended keyboard support' though.


The game is "spin the bottleneck", performance chokepoints will be discovered any time there's growth. Good design may anticipate and avoid several... or make it easier to quick-fix the system when unexpected bottlenecks appear. But new chokepoints will always become visible in a growing system... fixing some can require actions not unlike breaking a badly healed bone in order to set it right. Our current issue may be one such example.

This won't fix the grid load, merely the lag in individual sims. The open door policy is what's killed the grid. We can't even search PEOPLE anymore, gee, I wonder why, maybe because it has to search, oh, 1.9 MILLION names.

It's search places and classifieds that have been borked.

Searching for people works just fine... and has been fine as far as I've seen.

The percentage of unverifieds that are posting place listings... or classifieds ads is... oh... probably VERY low.

Find another scape goat.

[edit: Ha.. okay I'll read the blog first before spouting off next time. However I stand by my claim that the quantity of names is NOT the problem.]

eh its all database load at some point

Aw come on and think about this Argent. Not 20 times? hmm, our land mass has now tripled because of all of the new islands that have been purchased in the last month. Look at all of the money being spent by everyone, every single day now compared to last year. This money isn't being spent by just the same ones as last year. Yes we do have unverified accounts, but once they seen the game and are interested in sticking around they do usually become verified to have more options. The money trail is the one that tells the real story. The money says that SL has literally exploded with new verified accounts spending a great deal of cash. Look also and you will see that there are one heck of a lot more buisiness owners as opposed to last year. As opposed to the market being diluted by all of the new competition, everyone is making much more then they did last year.

If you read the thread you'll see I *do* know this. I'm going by the figures the OP quoted (3-4000 up to 15000, or 4-5 times), with the total (offline) players increasing 20-fold.

And the the load... the total number of transactions... in any directed network go up more than linearly with the number of agents in the network. In a small network each agent may have interactions with 3-4 other agents. Double the network size, you generally don't double the number of interactions, but you do increase them. Let's assume I'm not horribly atypical, and use me as an example. Now, even thouh I do cull it from time to time the sheer number of people I run into has meant that my friends list went from 20 to 100 people (5x... much less than the increase in the total database, but definitely bigger), there's 2x the number of active agents (I'll use your figures, it doesn't matter what the exact numbers are), so now the user database is getting 10x the traffic even though the number of active users has "only" doubled. No wonder the presence server is bogging down.

There's what, 5x the number of sims, so again, every time you search for sims or look at the world map you're causing 5x the traffic.

They've made improvements, so where 40 people in a sim used to bog you down completely, now you can actually move around. But those improvements have cost money, AND there's now more places where there's that kind of local population. And, yes, "looking at another avatar" is an interaction.

You'd think so, but the problems they've had growing the asset and presence servers: at one point a single asset server going down took the rest down with them. Maybe they have scaling problems, and maybe it's simply superlinear growth in load.

Not so recently. That was before they opened up unverified accounts adn triggered the massive growth they're currently dealing with.

Whether they "owe" us any more than we're getting or not is beside the point. There's a difference between responsibility and accountability... if they are the ones whose actions caused the problem, then they are responsible... whether we can "hold them accountable" to that or not.

And it's not just "spin the bottleneck", not when the same bottlenecks keep coming up again and again.

I should know better than to contradict you... You're quite right about the more than 2x incidental loads that occur because of a 2x population increase...

Though with regards to map loads, however, it SEEMS that LL has made some changes since last year in how the map image data is transfered, (terrain/builds, not icons/markers), in order to reduce the total volume of data sent when looking at the whole map.

And to split a hair... when I mentioned the increased storage requirements being relatively trivial, I didn't mean to imply that that meant retrival performance as well. There's a lot of inactive or low-use data stored along with SL. Sims that are usually empty, huge player inventories that belong to people that don't log in often, etc.

I have absolutely no insight into what's going on behind the scenes, but I'm guessing that what started off as a single monolithic asset server has now been broken up over several... and likely the same for the inventory database. (those transitions being the 'bones that need to be broken to be to re-set properly' I was thinking of)

I agree with all your other points though. Certainly the load associated with a 2x online population growth is greater than just 2x. Opening the doors to unverifieds definitely increased the challenges associated with running SL, I'm not sure how much that actually increased the costs of running SL though.

Billing support for unverifieds is non-existent.
More AR's/live-help calls to process, certainly.
Grid growth isn't driven by verifieds, not unverifieds,
but database performance and replication is driven by both,
better to tackle those issues now than wait.
I'm not sure what LL's fees for bandwidth are, but
Philip has been quoted as saying it's "very cheap".
(Heck, how does youtube pay for their bandwidth?)

I would like to support my lindens, but so far I have received no response to my application for one of the bilingual positions. And I more than meet all the requirements. Even made sure they received it... nothing... and they haven't hired anyone either...

HERE I AM!!! I WANT TO HELP!!! MAKE ME A SPANISH SPEAKING LINDEN!!! PLEASE!!!!!

Support our Lindens .. I must have missed the Blog post where Linden Labs announced they had became a registered charity ?

ok from a buisness POV? SL screwed up.

when You advertise as heavily as Philip has been doing, about this virtual world, people expect you to deliver. LL knew with all the potential new residents comin, that they should have had everythign in place, a whole lot sooner. You can sit there and pontificate on how we should tolerate bad customer service, crabby and sometimes rude Live Helpers, and the constant nickle and diming paying users, for services that should be included.

so on one level i can see your point, but on a buisness level? th whole lot would have been pinked slipped long before now.

Annoucements about new support people .. new live helpers .. liasons .. are all very well , but thats sticking plasters for the broken stuff . When LL announce they have a new product manager head hunted from somewhere impressive , then we should really get excited. That will demonstate they mean business and want to get it right first time . Someone who wont respect someone just because they were Philips buddy in the old days when SL was still not much more than someones bedroom programming project..

(aside . This perhaps needs moving to advocacy I guess .. I dont know if such a forum exists , or I would have done so)