Grid architecture for a million concurrent users?

What sort of grid architecture will satisfactorily support a million concurrent users?

Something like the WWW?

Runescape has over 10 million members, gets closes to 1 million online but the graphics they use isn't as complex as SL.
They use multiple servers and have cap on how many can be there.

Well, perhaps someone should send that idea to Philip and Cory et al and save them the trouble of devising a new grid architecture to sustain a million concurrent users.

The WWW isn't a DRIVE (Distributed Realtime Interactive Virtual Environment).

I just made that acronym up yesterday. (I'm sure it's been used before; just let me have my little moment of joy before citing passages from acronymepedia.com) .

Ha Ha,

DRIVE I like it.

Just curious, how many concurrent users does WOW get at peak periods?

If I understand correctly, WoW is sharded, so concurrency of WoW users doesn't entail the same problems as it does for Second (All Hail The One True Grid) Life.

To answer your question directly, I don't know.

They have told us that at peak times they stream 6GBPS at 34000 Users. If we use this number, we can guess that with 1,000,000 users they would use 180GBPS. That is just an insane number right there. The backbone would have to be located in many locations all acting as a single item. Many co-locations all over the world would be a good start.

yeap that was Runescape does. They have servers almost everywhere in the world.
They also have more active players then I have ever seen then in this place too.

Maybe, but I don't think they have the bandwidth requirement the SL has.

The problem with comparing most "Games" to SL is that most games are fairly static in comparison.

There are no players putting objects up, tearing them down, changing the land etc.

Designing a grid for 1 million is an interesting idea, and personally I hope they have already designed and implemented it, because if not it's going to be a nightmare migrating. (Yes I know there are already scaling issues, but is that due to the grid design or the lack of horsepower driving the grid?!?)

I've often wondered what the grid would be like running on some big iron (mainframe), there are a number of big iron computers out there that are capable of running Linux in partitions (Think VMWare only done better)

Some of these systems are capable of running thousands of virtual machines at once at very good speeds, AND some of them have nifty virtual networking/storage capabilities. (Basically the Linux VM thinks it is doing things normally, but in reality the host computer is doing some grunt work, inter Linux VM communication never actually enters the ethernet, file systems are shared etc etc.

There are other benefits too, Linux VM crashes, no problems you can "rez" a new copy and start it up.

I'm sure LL know all about this stuff, and if they aren't doing it there is probably a very good technical/financial reason for it. (It might even be that there is something they need that isn't available doing it this way)

Agreed. And Fd, I play Runescape every once in awhile, they rarely get over 220k concurent users (there's only enough servers to support a maximum of roughly 275k users)

This has been rehashed many times before elsewhere, but the general problem is that the caching system in the SL client totally sucks, and the cache generally is useless for anything more than temporary storage of your last hour of use or your last location within 100 meters of your current SL location.

Anything outside that time frame or position is redownloaded over and over and over and over again. The caching system in the SL client is sufficiently useless that there might as well not even be a cache system at all.

Hopefully with the open source client this will finally start to get fixed by the userbase itself, though LL seems to have seen the light somewhat with the new HTTP texture cache in the First Look viewer.

-Javik

If they can get the cache problem out of the way, then the real issue is updates of new objects, and making it so that when you move around your moves are reflected across to possibly hundreds of people around you.

For a world with a million concurrent users, a limit of 40 users per sim is just not going to cut it. Imagine someone building a stadium and there are literally 10,000 virtual people in the stadium, and a view distance of 1000 meters. If each user only needs 256 bytes per second to keep you up to date on their position or whatever they are doing, that is 2,560,000 bytes/sec, or 20.48 megabits/sec incoming for each user.

Your jumping up and down or waving your virtual team pendant in turn requires those 10,000 to also be updated to see your jumping or flag waving.


If server power does not increase significantly to handle this, then it may be necessary for LL to shrink the sim size to permit more sims to be visible at once. Imagine a sim only 10 meters by 10 meters. With a view distance of 128 meters you can see approximately 100 sims around you and 40 users per sim permits 4000 people to be visible and interacting with you.

If they shrink a sim further yet to just 5x5 meters, then in your 128 meter view distance you can see about 400 sims and 16000 users. Note this requires 128 CPU cores, and about $160,000 worth of 8-way datacenter servers using current technology.


One way to reduce network traffic for such a high density grid is to include 3D object occlusions into the network bandwidth consideration. If someone is occluded from your view, then do you really need to receive network updates from them? No, you don't.

So this way if you are in a building and someone is outside, if you cannot be seen then you literally are not in their world at all. Imagine an apartment complex 20 stories tall with 400 residents actively occupying the various apartments on each floor. If they cannot see each other then they are not part of the network traffic of any other apartment resident.

Similarly text chat and sounds could be occluded or severely muted by walls and objects just like 3D object occlusions. In this world you could talk with someone in your apartment and people in a neighboring apartment 3 meters away behind a wall cannot see your chat text or hear any music you play.

So there's lots of ways to grow SL to permit a high-density user environment, but it'll take time to work all this out.

-Javik

Heh, group chat in a group with, oh, 100,000 members.

I doubt that a million is a sensible limit. Why only a million?

How many concurrent users does the internet have?

About caching, that may be an idea to lower the work pressure to the main servers, making local "chaching proxies" where people connect to get access on the main grid, a bit like some providers do with the common Web, doing a local cache of some website called recently by users, so other users requiring that GIFs and JPEGs get them a lot faster and from local proxy instead from the original server. Guess anyway most of the bandwidth is textures anyway.

As I see it there are two main differences between the way SL works and the way the 'www' works, generally speaking.

The first has already been mentioned - the fact that every user's movements, actions, clothes, etc. must be viewable by every other user in the vicinity. This alone is substantially different to how the WWW acts.

The second though is, I believe, the all-mighty asset server. When you go to surf the web, there is no single centralized server dishing up pages, not even a single centralized DNS repository. I saw an interview once with one of the guys who started or helped start the www and he said that they had to make a huge decision right at the start - there would be no centralized index. They knew that a centralized index could not scale and therefore they would not have one. The tradeoff was that they knew that you would end up getting 404 page not found errors. Because data moves, gets deleted, etc.

I don't know of course how SL really works internally but I see a parallel between the asset server, and the centralized index that the www doesn't have. The web, with a centralized index, would be slower and fail under lots of concurent use, but you wouldn't get 404 errors. SL, with distributed asset service, would (theoretically) be quicker, but you run the risk of not being able to access stuff if a server is out, or if the data has moved or been deleted but the links haven't been updated. So, people or prims with missing texture.

Just my L$2, some speculation.

-Atashi

Pardon my innocence in asking this question but I read somewhere the very big search enginces, (Google, Yahoo, MSN) do effectivly cache vast amounts of data on the web

I'd love to see SL scale up to 1 million concurrent users and beyond. There does seem to be some scaling problems as regards the asset servers. I hope the back-room boys and girls at Linden Labs tackle that problem soon.They'll need to soon otherwise SL will be brought to its knees by it's rate of growth.

I use to play all the time until recently.
There is 10mil members.
There are days were 1 million are logged. I have been there and the lag is pretty bad.
They only allow lower number of players per server.
They don't put everyone on one server.
They don't have animations, graphics, or complex inventory we have.
When I did Activeworlds there were very few people on.
Building in Activeworld is whole lot harder because you have to remember codes, you don't upload graphics, you link them from your website when I was a member.
Activeworlds host only certain shapes and doesn't host textures.

If there were a million concurrent users, on a unitary grid like the one we have now, only many more sims, what would it be like looking at the event list?

Would we be able to have a map zoomed to show the entire grid? (Well, we can't really do that as things are now.)

Yep. I know how Activeworlds works. Activeworlds isn't dependent on a central server. This is why it would work with a billion people.

That's an interesting point: it is how the client works but it isn't how it need to.

What your client really needs to know is what is in your view. Rather than doing the more complex work of determining this server-side, they decided to throw every object in your view distance at you and let your GPU figure out what is viewable. Two simple examples come to mind: you are playing *INGO and you are in a chair with a fixed view, why is SL sending you information about objects 128m behind you? The second example is that while you are in a windowless club, the servers are dutifully sending you everything outside the walls which your client will never see.

Every once in a while, someone tries the idea of pre-loading all the links on your currently browsed page with the expectation that you'll probably move from the current page through a link. These pre-loaders are dreadfully inefficient because the degree of linking is too high and prediction of the browser's next move is impossible. This is not unlike how SL works currently.

I could easily see some sort of distributed 'asset server' where people pull asset info from whatever data warehouse is nearest them...

The real problem with scaling for global use and million+ concurrency isn't the asset server, as far as I can tell it's more of a simulator and inventory db problem.

Continents could be made more 'regional' in nature. For example, the "old central mainland" hosted on the west-coast of the U.S. near LL, southern mainland east-coast U.S. the southern continent in the west pacific area, the northernmost continent in the UK or Europe somewhere...

People could still explore continents hosted much further away geographically... but probably have to put up with a little more lag in doing so.

The real problem, as I see it, is going to be finding some way to handle more people per sim... not more people per continent.

I wonder what the effects of hardware improvements such as
[quote]update SAN FRANCISCO--Intel has built a prototype of a processor with 80 cores that can perform a trillion floating-point operations per second.

CEO Paul Otellini held up a silicon wafer with the prototype chips before several thousand attendees at the Intel Developer Forum here Tuesday. The chips are capable of exchanging data at a terabyte a second, Otellini said during a keynote speech. The company hopes to have these chips ready for commercial production within a five-year window.[/quote]
will have on systems like Second Life.

If one core can run a sim, that would be continent on a chip, eh? Sort of maybe?