Concurrency, Landmass, Grid Size... any limit?

Okay, this is a general question and I'm trying to see if I 'missed any research' out there in the blogosphere or press. Perhaps someone can inform or point me somewhere...

* * * * *

Once upon a time, I recall reading a statement that said that a grid concurrency of 100 thousand was a 'target' - a design factor for our grid.

Of course, such things are moving targets, but I'd imagine re-engineering the grid for orders of magnitude greater would be a Big Deal.



Some "stupid math", give or take:

20k regions, 60k concurrency,
30k regions, 90k concurrency,
40k regions, 120k concurrency?

Now of course openspace regions will throw that off some.

First question: are there some known hard limits ahead, and if so, what is the current strategy to mitigate?

Second question - presuming eventual growth, are we headed toward a grid limitation that may halt land rollout for a time, putting us into a supply side market until the limit can be punched through?



Edit: '100 thousand' concurrency, I had typed '100 million'

I recently heard that the existing server code would barf if we ever hit 65535 concurrency. That would be the first thing to fix, I reckon.

Reference: http://www.sluniverse.com/php/vb/general-sl-discussion/12798-report-stolen-content.html#post274039

65,535 is a major number, but I can't imagine they wrote it with that limit built in.

I thought I had seen it at 67K once. Is there a graph of peak concurrency anywhere?

Without some major re-enginnering of the central databases, the math goes:

give or take:

20k regions, 60k concurrency,
30k regions, 60k concurrency,
40k regions, 60k concurrency,
400k regions, 60k concurrency


Second Answer
LL can continue to create continents and PIs al long as people continue to buy them- even if that means 10 sims to every active avatar.

I think you'd have to speak to somone like Meta or maybe Zero to get an answer to this but according to the blogsphere we've had concurrecy of over 65,000 this year, lately problems have been arising at around 60,000.

There's a graph here:

http://taterunino.net/peakmonthlyconcurrency.jpg

I have no idea, but I do believe we've hit peak concurrency for sex-related threads here in the forums today.

Was it just over a year ago or so when Cory was talking about things relating to scaling the grid.. "backbone", more scaleable messaging protocols, heterogeneous client and server "capabilities" and all that stuff? For some reason the 100k seems vaguely familiar, but I can't be certain. It was all a hot topic for a very short time, right around the time Cory did the last townhall, and then it all sortof died down and we never really heard where it was going.

Possibly alot of this stuff is being hashed out in more detail within the AWG, as most of the work involved in scaling to that sort of concurrency is going to start at the architecture level.

Good replies so far. Yes, what did happen with all of this?

It's not like I am expecting some kind of Y2K-like apocalypse - but perhaps a 'hard bump' worth planning for.

Sort of like storing a little bit of food and water if you live in earthquake country.

More info welcome - in the mid term, I'm wondering if we'll hit any technical bumps during say, winter 08/09. Planning purposes and all that.

I am not aware of any hard limits as such.
LL is currently desperately trying to untangle the central database (not the asset cluster), which is the current bottleneck, being essentially single-point.
Once that is done, I think we can see a fairly steady scaling until either the group system becomes *entirely* intolerable, or the asset cluster overheats again.
The asset cluster *should* be largely a question of shoveling enough hardware into the server room by now, but the group system doesn't scale.
Sheer bandwidth may actually become an issue as well, though I don't know how close we are to redlining that and needing a new datacenter. The clients pull a *lot* of texture data. While a fairly big thing, this is, however, something which can comparatively easily be predicted and planned for, "only" costing money to establish.

The "decentralization" Mitch Kapor spoke about may be part of a long-term plan to farm out grid servers even more distributed, like the experiments IBM is running, but it's still all surprisingly hush-hush.

This is not any official information; just the vibe as I have picked it up.

I forgot who wrote this, but the anecdote goes: The system begins to get extremely slow at about 20 users, so management goes to the system folks and asks them to find the place in the code that says "20" and change it to "40".

The point being is that the slowdown at 20 is an emergent property of the system and lies distributed throughout the it.

The current problems with SL stem - to the best of my understanding - from some design limitations inherent in the system. For example, transactions (money for objects) should have been implemented with transactional atomicity (the money gets debited and the object delivered or both are rolled back). But API calls like llGiveInventory don't return status or error indications. I expect that they never expected an inventory database entry to fail so they made no provisions for when it did. This assumption worked fine for the first couple of years until the transaction rates got so high that the database would silently drop requests.

There are other places where heuristics dictated limits: since sims tend to fall over with more that 40 avatars, they put hard limits on that number. A sim can handle more, you just don't want to be there with 50 of your best lagged to death friends.

Fortunately for us, but not for them, I think SL has come close to the point of "nobody goes there anymore, it's too crowded" - Mitch Kapor's slide showing SL at a population inflection point notwithstanding.

Yesterday's peak was 65,732.
http://www.massively.com/2008/07/07/second-life-daily-news/

when i joined in Feb 2007, the highest peak was around 25k and it was BIG news at the time. who new?....

Hahahaa. N00b.

When I joined, there were only 40,000 members in all of SL and we rarely hit 5,000 online at once.

P2

but out of all the members from day one to now, i'm the only one that matters.

On Malachi's point on transaction logging and rollback, that was also something Cory talked about, I think at the same time he was talking about hetgrid and all the other stuff.. this was on the table, at least on a piece of paper on the table, or maybe a napkin with scribbles. Maybe somebody spilled coffee on it and it was forgotten.

Anyone know were to get concurrency data for the earliest years?

Seems like I remember being in SL when there was only 30 people online.

There were more than that in Dec 2003 just about always. I remember maybe 20 in one place for the 'show and tell' things (those still happen?) ...but it sure did feel empty sometimes.

What really made it more 'community' feeling was the events announced by official staff every now and again. Oddly enough Caledon (and I'm sure all the other communities) seems to kind of have that small-grid feel in a way, where most people kinda know each other.

Aaaanyway!!! Back to topic - any evidence, anywhere... of any hard (or at least lumpy) limits to grid expansion in the near future?

Is it just a myth?

Is the central database something like 'peak oil'? Or will we just sort of meander into the vastness of cyberspace a just few crashes and borked teleports at a time, without really noticing? Okay, I ramble. Someone set me straight!

To get more than 60k online smoothly we all need to dump our pretty viewers, clothes, buildings and just run bots

Desmond, here is the proposed redesign of the architecture that they claim should lead to true scalability in the future:

http://wiki.secondlife.com/wiki/Structural_Design_Overview

Fascinating - I'd been reading snippets of this in the past... I presume the IBM interoperability proof of concept was an element of this working perhaps?

While not exactly ignorant of all this (EE background, designed networked systems myself) - I must confess I have *no* clue of how easy the road is 'from here to there' considering they can't just roll the grid back to 2002 and press reset.

Any techie-wikinistas able to summarise the big obstacles ahead?

I think in a nutshell they have to decentralise all currently central databases and servers, make more of them to handle the load for each "grid area" and distribute them to handle parts of the grid. So to visualise this the current grid might just be one "grid area" with an asset server, login server etc which is interconnected with another grid area that also has an asset server etc.

Big hurdles would include handing data off to other grid areas, this would pretty much include almost everything that you would have to transfer to say another grid like an OpenSim when your avatar crosses over into it. Assets, money, state etc.

Perhaps the recent announcement shows progress in this architecture, or if not at least that work is being done to create these big grid areas.

Thats my understanding of the issues anyway.