Using grid computing to solve game asset storage

Just for fun, I thought I'd propose a way to centralize game assets (avatar info / textures / notecards / ?) across a grid rather than on a central asset server:

Assets should be stored on one or more sims rather than a central server.

A very very fast, *in memory* and very *basic* lookup server would be required to respond to key lookup requests as to which is the home sims(s) for that server. And, potentially, which is the least loaded with requests.

First Advantage: the whole world is not held hostage by a single point of failure (no matter how 'replicated' that point of failure is).

Well, it would be held hostage by the very fast lookup server, much in the way the internet is held hostage by DNS.

Second Advantage: ability to decentralize servers and allow an anyone can host style of service, since they don't have to clog up your asset server with their assets.

Disadvantage: lag on a sim can cause localized problems.
Advantage: localized problems can be a much better problem than a global problem.


For fun, rather than criticising this idea, please propose your own.

It's a nice idea.

Course rez times would be longer. Also sims with high load on them could offload there assets to other sims. Spreading the load to the other sims. I like this idea.

/me wants to start poisoning the DNS cache.


Now if we could just get Trimming to agree it's a good idea the forums would be much more peaceful.

The current asset server is a central server and each sim has a cache serving it (squid for people intrested) this seems to be the fastest setup once you have rezed an object once but has its down falls for grid reboots. Again its another system LL is looking to improve.

Squid is very very cool. Interestingy enough, it would work just as well with the layout described above.

the keys could be hashed to values that indicated what servers they were on. that'd eliminate the lookup point of failure. in fact, the software could essentially raid itself by hashing each key to two different servers. so the client could access the "backup" copy if the "main" hash wa down.

I believe....
Only problem with that is an object is defined as a list of other prims. Each prim with it's own key. So it's possible that a large object could be spread across all the servers. Downloading a single object could result in 264 connections (9 for textures).

I think I sugested some thing simlar, just I called it p2p network located inside and only for ll.

Well, since my name was mentioned, I'll go ahead and opine here.

Grid computing is very good for some classes of problems, but not so good for others. I'm not entirely sure that this data model would be easy to break up across multiple servers; I imagine they have probably already done as much of that as they can comfortably/easily do.

Also, you wouldn't want the databases running on the sims, they're busy running other code! Any time they spend running a database is time they can't spend on your code.

A partitioned database approach could work, in that you could distribute load based on object keys. But there's a whole new class of bugs that could arise from this kind of distribution. Consider me transferring you a 6-prim object with a lot of inventory... most of my scripted objects end up looking like that. If I am just transferring you a copy, that's no big deal, my client can go talk to all the different databases. But what happens if you then want to transfer the locked, no-copy item to someone else? How do you handle it as a transaction? All the individual pieces must transfer, or none can. How do you handle a rollback if one server is down but all the rest of the objects think they transferred? What happens if a SECOND server goes down during the rollback process? And how do you guarantee being able to do all these things reliably, *wihout race conditions*, across an arbitrarily large number of machines?

The engineering involved in getting this working RIGHT would be extraordinary, far harder than creating the entire existing client has been. In other words, it would probably cost them AT LEAST ten times as much to develop this kind of system as to just buy a big-iron system from an existing vendor.

It might make sense for SOMEONE to develop this technology, if it's doable. But it doesn't look sensible for LL to try to develop this while doing all the other things they're also trying to do.

I would love to see SL running some sort of freenet-like dynamic replication/relocation model. If you think about it, it makes sense

Well, one possibility would be that an avatar is completely homed on one sim rather than split across multiple sims, so the atomic transaction could occur that way and another process would be responsible for replicating it to its backup sim.

Okay, I guess this is only tangentially related, but...

I'd like to see them open up the data-fetching architecture... so I could run my own caching proxy locally.

Then, if I wished to, I dedicate 50gb on my (usually fairly idle) linux box to caching. With some good indexing. There are many things in the world that change fairly infrequently -- the textures for all your close neighbors' houses, that club you go to every couple of days, all your friends' custom animations and sound clips -- but they can fall out of the simple 1gb max SL cache before you need them again.

This could really improve rez times in places that I go frequently. Also, this would be a tremendous benefit for when my girlfriend and I are both on SL at the same time -- the proxy can fetch the data once and give it to both of us, instead of having to download it all twice.

And if enough people did it, it could reduce the load on the asset servers. And, there'd be no reason that LL couldn't colo a caching proxy on the east coast, or europe, distributing the load even more.

Pedro, I assume that would be quite doable... SL is already caching, it must have an algorithm all done for that. Just move the caching to another machine, let it be much larger, and share it. Not a bad idea at all.

Simply allowing individual caches to be larger would, in many cases, allow the same thing, and probably would only require a few lines of code be changed.

blaze, you could home an avatar on one sim, but what about his/her inventory? An inventory is just a set of pointers to objects in the data store. If the data store is scattered across many sims, then my inventory is still scattered across many sims, even though all the pointers might be local to the current sim.

What if I transfer something to someone who's in another sim? Who's responsible for the transfer, my sim, or his? And what if something fails, how do you roll the transaction back?

What if I move from sim to sim? Does the entire database for me get sent over the wire? What happens if I move back before the transfer is complete? What happens if I go into a third or a fourth server before the second server has all my data?

We already have big, big problems at corners as it is, with the tiny bits of handoff data going from sim to sim. Transferring an entire inventory at every sim crossing is going to add a vast amount of load, and I don't see that it would give any particular benefit.

*must... not...mention... the...product...that...already...uses...a distributed...data....architecture *

They use Squid for cache? Are the sims talking with the asset server through HTTP requests or XML RPC O_O ?
That sounds very unefficient.

Does the use of Squid imply that its HTTP between sim and asset server? And if so, doesn't that imply a lot of TCP session creation/destruction as HTTP links are established?

Not that I'm brazen enough to be telling the Lindens how to build their grid.

- Ace

I agree Trimming, acidity is an issue and not one easily overcome. Something I probably neglected to consider as a con in my original email.

However, I still believe the distributed architecture outweighs the centralized one. The internet is a pretty good model for what this world should become, but with just a little more segue from site to site.

Hopefully they are migrating that way..

Trimming, you could have a central inventory server. One that just handles inventory lists & transactions. Since it doesn't actualy have to have the assets it shouldn't have to much trouble with the capacity. This would also eliminate the need to develope code to handle bad transactions on a multi system level. There really can't be more then 80GB devoted to inventory lists at present.