SL Scalability -- Issues to address.

Cory moved the technical discussions from his weblog back to the forum since this is a far better medium for that --- very sensible, I think. I'll pull my post on SL scalability out of Philip's weblog over to here for the same reason, and then follow up about other areas of non-scalability. Repost follows:

That's actually incorrect though. SL's grid is based on a statically tiled resource architecture, which scales perfectly for static resources and doesn't scale at all for dynamic or mobile ones. To put it in non-CompSci terms, the SL grid only scales if all its players remain at home.

If we assume that the envisioned future is one supporting popular communal events and not one where players are forced to stay at home, then the static tiled architecture will have to change to a dynamic one which moves computing resources to where they are needed.

It's totally senseless in engineering terms to have a large number of zone servers doing almost nothing when a large proportion of their population leaves to watch a popular sports match in a different zone, while that particular zone is rendered virtually unusable through lack of computing resources.

The above scenario is pretty much upon us already, and it can only get worse. After all, if an event attracts N people to it today then it is statistically likely that it would attract 10*N people if the population of SL were 10 times larger.

So, it's completely inaccurate to say that SL is already scalable for growth to millions of users by architecture and design --- you can't achieve that with static tiling. Its most basic structure would have to change.

(And this doesn't even touch upon other barriers to scaling, of which there are several.)

Originally posted in blogs on October 18, 2004 05:33 PM

There's a useful paragraph in the programming section of Cory's weblog which deals with the scalability problems in the asset server:
This sounds excellent, even though it doesn't mention the approach that is being taken (there are many).

We handled database non-scalability in a multi-million customer ISP by placing a cluster of replication boxes in front of the main database, and forward-replicating to this cluster any tables that gave us read-loading issues on scale-up. One can do something similar to write-loading problems using a separate cluster as a write decoupling point, and then merge the updates at leisure into the main database with a reduction in latency.

But... Morgie...
Haven't I explained to you already that the servers can hold much more than the current limit, and the true impediment to scalability is client-side FPS?
"Scaling" servers and clients to even more people / prims is simply a matter of getting more powerful hardware.
LL keeps statistics on how many FPS everyone is getting, on average, and they raise prim and av limits when they see that we can handle it.
If everyone upgrades their computer this christmas, you can probably expect this to occur.
I could go on forever. For instance, inter-sim message passing is much slower than intra-sim message passing for the obvious reason that networks are an order of magnitude slower than your average northbridge.
Try raising land while standing on your plot, and then raising land when standing across a sim border. You'll see what I mean.
We used to have town halls on a 4-sim corner. But we gave up on it because we weren't really gaining anything other than message passing overhead and extreme amounts of lag.
Never mind the fact that hardly anyone is hitting the sim limit these days, and that you can easily get around it with teleport offers.
Or the fact that large gatherings of people are simply not a critically important feature for LL to support. How many people do you realistically expect to be able to chat with?
This isn't everquest. This isn't about getting 200 people together to slay the Greater Demon of Golgotha. This is about quietly building in your little corner of the world, chatting with a couple of friends, or setting up a shop that a few people will visit now and then.
The only reason why you see giant gatherings of 40+ people is because a few players are gaming the dwell system.

I was at a uhm, place, last night there were approx 30 people and the server fps was 2.

thats only partially the case eggy.

It only takes one really bad person with one land for landless plot to essentially render an entire sim useless. And a single careless build can pretty much ruin client performance for an entire area as well (such as making HUGE numbers of large lights)

So while its true in the low use sense sims could hold much more, object wise... *script* and physics wise they are in many cases pushed rather near, and often quite past their current limitations. (aka a server could easily support 30,000 normal prims, but even with just the 15000 they have now, they can't really sustain even 1000 active scripts... and yes there are some individual builds i've seen with that many, or more)

No Eggy, scaling from improvements in hardware are entirely orthogonal to scaling resulting from using a scalable design. Any improvements in hardware are a bonus, but the major improvements to scalability have to come from design --- basic lesson taught in Eng/CompSci 101.

Try to convince Philip that he can only double the size of his SL population when his hardware gets faster x2, hehe. I think he might respond with a rude word or two.

The main problems of scalability are architectural and algorithmic, in all systems everywhere, and it's extremely true in SL because there are so many points that give scaling problems both at the server and client ends. This thread is about dealing with them, as we see Cory doing with the asset server.

I disagree, Eggy... Events that could attract and, more importantly, support hundreds of AV's in the same general vicinity would be awesome. A 200+ AV drum circle would be an incredible experience for everyone. Or perhaps we could have real competitive sporting events that draw large crowds. I'd love to see boxing competitions, or a regular NASCAR-type racing circuit, or someone scripting a team sport like soccer (football for all you non-Americans).

If SL could scale to larger gatherings (whether the problem is client-side or server-side, it doesn't matter), I guarantee that the creators of SL would create Events that attracted such crowds.

- Ace

Indeed, Ace.

Furthermore, that is almost certainly where SL is heading, despite Eggy thinking that Philip wants SL to be just a 2-bit chat client where most players stay at home or chat with a small circle of friends.

You know why I think that? Because I doubt that Philip wants SL to fail, and as soon as there is competition in this area then the inability to hold major events would be end of business.

It's just a guess, but I tend to doubt that Philip's vision is anything like that limited.

I agree with you, Morgaine. In fact, as a former all time There player (I do log in sometimes, but not all that much anymore) one of the things that There thrives in is actually that, being able to host events with alot of people, alot of action (full scale buggy or bike racing, for example) and without having to worry about borders or zones or all that. Of course, you can argue that both have different architectures and all, but when people use a service, they don't expect to understand or even have to care about the internal works.. they just want it to work as they expect it to. Reasons to why it doesn't are just reasons to make them (the customers) leave all the quicker.

EDIT: I know I might get flamed at by first saying I don't play There much anymore, and then saying that one of There's qualities beats SL current capacities. But then again, that's just an example of something that can be done, and who knows? Maybe some new "BRIMSTONE LABS" popps up with some venture capital and starts building a metaverse where you can build, script, and host full scale, large, active and scalable events. If we are not prepared, then God (or who/whatever we believe in, if we do believe in someone/thing) help Linden Labs and SL.

I'd not even heard of There until I'd subscribed to Second Life a few months ago ... and it was only a couple of days after subscribing, because ex-There people seem to be everywhere in SL, they come out of the woodwork every few minutes.

Seriously, all systems have their good points. A truly good system is one which is prepared to learn from those which came before and from any changes in those currently around, and to take on the best features from absolutely anything.

This thread is about scaling, and I appreciate your comments about our deficiencies in that area here, because one only improves after recognizing current areas of poor design. Don't let that restrict you though: any other improvements that SL could make to be as good as There in specific areas where it may currently lag would be invaluable.

Non-Disclaimer: I don't have any shares in Linden Labs ... I just want to have fun here, and to be at the leading edge in 3D persistent worlds

LOL... I'm actually in SL because, even though There has those qualities I described earlier, SL has properties that are more valuable to me than them. And the fact that SL is crawling with ex-Therians means that that is something the population seems to like and need.

My post goes in the direction of pointing out how the only organisms, companies or services that survive over time are the ones that are continuosly evolving, fixing inefficient aspects and introducing new features to themselves. Today SL might be the next best thing.. but I hope I don't have to see myself posting in another game's forum some time in the future, talking about how "I used to play SL, not in there much anymore because of lagg getting everyone pissed off". And it's not that I'd be a traitor by doing so... I'm just a customer searching for the best value on my money!!

Right on! And ditto.

The only thing I'll add to what you've said is that I'm also a systems developer in a whole pile of areas, so I'm not accepting the blinkers being pulled over my eyes about what can and cannot be done on a technical level.

Let the progress continue. Detractors can go back to playing space invaders.

It's worth noting another benefit of virtualizing the grid and moving the sim boxes into a dynamic role, namely the removal of zone lines and all their associated problems. Once the grid is no longer implemented as a static assignment of machines to land areas, the zone boundaries become no more than lines of environmental property change.

While those concerned with the scalability of SL probably see elimination of zoning issues as secondary to allowing the system to expand to Philip's millions, from a player's perspective the perceived advantages might be slanted in a different way.

Those who are currently struggling with their vehicles dozens of times each day on passing from one tile to the next are having a pretty miserable time at the moment. While the bugs are almost certainly fixable, even better would be to eliminate the boundaries which give rise to them altogether.

I do like discussing scalability of SL, as it involves the very center of the vision of the Metaverse, and while LL clearly has many things in mind, it doesn't hurt to provide our perspective.

But before you assume your knowledge of Internet scalability dwarfs that of Linden Lab, perhaps you can consider:
...
1. Philip said "similar" not exactly like.
2. Philip likely simplified his statement, since his audience was a town hall meeting with many less-technical people attending.
3. Just like a sim can be overrun by too many users, so can any regular Internet webserver(s), or the bandwidth provided to the webserver(s), or a router in the path to the webserver(s). (Buffer overflow being a leading cause of lost packets).
4. There already are sims in SL that are not "perfectly scaled" 1:1 serverim - the empty space filler sims. There's no reason why multiple servers couldn't host a single sim, either.
5. Philip went into great detail about how Moore's Law will allow sims to be more and more powerful via sheer computing power. While you correctly state that this is only one aspect needed for scalability, the main crux of your thoughts involve saying the grid won't support the proper people. This is simply not the case.
6. Let's assume the Internet's "topology" is IP addresses or domain names. Unless a webmaster is masochistic and enjoys constantly changing bandwidth provider / domain name, these will remain fairly static. This is thus similar to the sim-topology of SL. In fact SL allows more flexibility since you can up and move your store / home / etc.

I find the biggest thing needed for scalability is organization. If all of the Metaverse is suburban sprawl, it won't take off. Linden Lab can build this into the design of SL and I believe it will accelerate the growth of SL, or we can wait for large commercial companies to do it, and then we are at their will.

Oh, I agree Hiro.

The only key point I took from Philip's words at the Town Hall was the desire to scale to millions of customers. Like yourself, I didn't take his reference to the Internet as anything more than a vague analogy. Indeed, the audience was largely non-technical as you say, and I'm sure that his choice of words reflected that.

I believe you need to keep in mind that scaling to millions of users will happen over a period of 10 years or so. They have plenty of time to worry and overcome any shortcomings you throw at them.

Eggy, agreed.


I didn't think it was a vague analogy at all; my post was showing how SL's grid actually DOES scale as the Internet can.

SL will replace the Internet.

It's the static tiling that doesn't scale for mobile objects --- that's a direct property of the architecture.

You actually stated that "There's no reason why multiple servers couldn't host a single sim, either", which means that either the load handling becomes totally dynamic as I've suggested or else the zone is subdivided into a progressively finer (but still static) tiling.

You didn't say which you meant, but I assumed the former (which is why I agreed with you overall), since the second approach has the same terrible scaling properties as the current grid.

And I said the tiling wasn't "static". LL already has shown the architecture is not static by the fact that we have the empty filler sims - where one server servers 4 sims.

So your assumption that "LL cant' scale SL because the grid is static" is false because the grid is not static.

Hiro, did you see the words mobile objects everywhere in this thread, right from my first post all the way to my last? You seem to be discussing some completely different issue.

If a box is assigned to zone A then when that zone's home avatars and other objects like vehicles and attachments move to zone B for some event, then however much processing power you put in zone A it isn't going to help overload in zone B.

That's where "static" comes in, supporting mobile objects, avs in particular. Ie. regardless of how clever you want to be in zone A, it won't help in zone B unless you break the static assignment in the grid and go dynamic.
So that nice box currently serving me in Kamba will help reduce the load in Gibson when I go there, will it?

I think not. It's statically assigned to a zone.

I see what you're saying, but there's no reason we can't just look at a sim as one interface to what amounts to a server farm.

What's the bottleneck? It's likely not hard drive space or memory.

Network is a potential bottleneck - too many people accessing one set of objects. Since I would guess the overwhelming interactions are read and not write, a RAID array could effectively address such an issue.

But moreover I don't think it's network - yet - I think instead it's sheer CPU speed. It's clear that the more people, the more lag. The more shops and scripts and such, the more lag. CPU speed can be farmed out to multiple machines.

In the end, perhaps there still needs to be a "master" sim computer, but it seems scalable nonetheless. And the problems mentioned are still the same problems any Internet web site may face.

Yes, that's exactly where this has to go, ie. server farms organized into load balanced clusters each handling one particular domain that needs decent scaling, like an object management cluster, an asset server cluster (which they seem to be working on), a login cluster, and so on.

It's what we do on large corporate sites that need to grow in a flexible way, not just for their public-facing websites but also internally to cater for many different kinds of company requirements. There are a large number of load balancing products on the market which can help system designers do that rapidly, if one doesn't want to configure such a system oneself out of Linux boxes. Load balancers are a huge growth area in the market.

Before you can do that though, you need to break the static assignment of server-to-zone, and that's the basis of this thread.

Right then. And I'm saying that the static assignment is clearly not static since there already is an example that is not server-to-zone.

Unless you mean, for instance, having the "floater" servers available for any zone, rather than just backup for busy zones? I had sort of assumed that say - for instance you have Da Boom, which is very busy. You'd have a backup server that'd be just for Da Boom

As for having "floater" servers available for any sim in SL - for Linden Lab that might work okay, but I suppose I'm making a big assumption:

-- eventually SL will essentially be HTTP protocol, where companies will have their own servers that dole out the data. In that case, it'd be impossible to have shared "floating" server farms.

Your example of filler sims is the wrong way around: it illustrates a static mapping of one server to multiple zones. In no way does it address the issue of multiple machines all working together to handle the requirements of one common zone. Using one box for multiple zones decreases the resource per zone, it doesn't increase it. And it certainly does nothing for mobile objects moving elsewhere in the grid.
If you mean backup in case a live server goes down, that doesn't address the issue of multiple machines sharing the workload for any given zone. The backup server isn't doing anything, it's just waiting.
If by "floater" you mean generic unassigned standby server then again, it doesn't address the issue of multiple machines sharing the workload for any given zone. While unassigned hot backups within cluster pools are very useful to reduce the impact of faults, they don't contribute anything towards load reduction.
I have no idea what that sentence is supposed to mean in the context of SL. Outside of the context of SL, HTTP happens to be an extremely easy protocol to handle in a distributed fashion in server farms, and there are umpteen thousands of sites doing exactly that today so I guess what they're doing isn't "impossible". I don't see any relevance of this to the topic under discussion though.

Nah, more like I'm saying - clearly they have a logical translation from server to sim that isn't a 1:1 ratio - if that's true then I assert 1sim:X servers should be very possible.


No, I also mean something on the order of the server farm, just reinforcing one sim only - used on busy sims. Though redundancy would be nice too


What I meant was that, not that we would use HTTP, but that like HTTP, the SL server software could/will evolve into a protocol that is open source, so that anyone can run a sim server and just need space allocated on the grid for it and the proper registration, just like DMS.