Only 8,300 people logged in right now and the asset server is almost worthless.

it was laggy today and we were on an Empty sim with next to no Prims on it.

The speed of MySQL is very overrated. It does very well on trivial queries like "SELECT * FROM foo WHERE id = 25", but chokes on a serious amount of joins.

Last time I checked, it also had a very crappy locking model. MySQL isn't alone in this though, many databases like MS SQL Server use the same method, and it's got really crappy performance in some cases.

You don't have to go as far as Oracle to get something better, there's PostgreSQL which is completely free. I'm currently using it for SL related purposes.

Slashdot and the asset server are completely different things, though. First, Slashdot is an application with a very massive amount of reads and few writes, which is something MySQL can deal with pretty well (it would die on a workload with lots of reads and writes though).

Second, Slashdot uses massive amounts of caching. They pregenerate comment pages and serve them statically, without hitting the database. This provides a very good performance improvement, but I'm not sure this can be applied to the asset server.

Now, finally, now that I think about it, what the asset server does doesn't really require a relational database anyway. It sounds a lot more like a fileserver to me.

Thanks, Kelly, for keeping us informed. It really does help to know that you are aware of these problems and hopefully doing something about them.

You should. Definitely and as frequently as you can find time to.
A bit more communication and information flow will go a long way to convincing customers that LL is on the job. Something that isn't often obvious.

Hearing about 'transparency' from his highness is all very well - actually having it demonstrated by a Linden who knows what they are talking about is a real breath of fresh air. Thanks Kelly.

hmmmm.... LOL

To my understanding, the Asset System IS basically a big caching system. So to optimize it, is more a question of improving the caching mechanism or distributing the data using a multi tier cache, for example the way akamai allows content delivery on demand. (It's what all the big guys use for asset management/content delivery)

I talked to one of the Lindens at SIGGRAPH this week about the asset server problems, though we didn't talk about the specific database product being used. He acknowledged that the single asset server is a major choke point, that it needs to be re-architected, and that work is in progress at LL to do that.

He also said that they had bought some sort of very high-end server hardware for the asset server (the name of which I have forgotten), and that they had found the performance of it disappointing -- if it had lived up to the vendor's promises, SL wouldn't be having any trouble at the current level of server load. My guess is that they went in that direction to buy time for the re-architecting of the asset system, but didn't get as much time as they had hoped for.

Well, what I was trying to say there is that Slashdot and the asset server are very different things. That slashdot manages to use MySQL has absolutely no relevance towards a completely different application.

But, now I remember seeing screenshots of a folder full of files named by asset key, which I think were from the asset system. This suggests that the asset server is pretty much a fileserver: A big disk with a file named after the key, containing whatever is inside.

Such a system should scale wonderfully. Objects never change, so all you need is get an asset is to get say, http://server/key. You can easily add more of them and retrieve the data from a random server.

Some crappy filesystems like FAT and ext2 deal badly with folders containing thousands of files (linear lookup when searching for a file in a directory makes things slow), but things like ReiserFS provide a great improvement.

I'd love to find out how it actually works (I'm just guessing here) and what exactly is going wrong. Maybe LL is short on bandwidth?

Thinking of expanding to a SAN/NAS based solution with multiple (say 10ish) asset servers plugged into a either one or two EMC/NETAPP's with variable mount points serving various either parts of the grid?

Thats what I would do, granted it would get a bit more compex than that when it comes to setting up the LUNs, and its performance would depend on if your network is teh rox with the fiber or if its teh sux with the ethernet.

Yes, that's what they use now. A high end NAS solution (isilon ) and it's clearly not up to the job.

Basically, the thing is to improve access time to the data. There are several scenarios for that:

1. Improve global access time for any asset (The Eve Online way)
Eve runs their asset system completely from solid state memory. It's bloody expensive, not really scaleable, but does the job. The problem is, at a certain level, the logic to find a specific asset in a huge storage area will eventually take up too much time, even if the actual retrieval is blindingly fast. Also, the networking component will be a bottleneck when there's a lot of access through a single point.

2. Improve access time for those assets mostly used. (Caching)
Caching is normally done by any filesystem as it is. But a filesystem cache can't have the specific intelligence that is required for a specific job. So we have job specific caches tagged in front of the filesystem cache, like squid proxies for web, index caches for databases etc. SL needs a specific cache, tuned for the specific way assets are being accessed (and written). To improve networking speed, caches should be distributed over several caching servers. The biggest example of that is the mentioned Akamai Edge network, which uses over 20,000 highly intelligent caching servers all over the world, to both reduce access times and network hops.

a) Dang
and b) If I was right, I really can't move across the country
Love anyways for working on it and giving us some much needed dev feedback.

Very pleased to see your informative reply, but I'm a bit confused now. According to an interview with Ian Wilkes here (http://dev.mysql.com/tech-resources/interviews/ian-wilkes-linden-lab.html)

"Lenz: Can you elaborate on what kind of data is stored in MySQL?

Ian: In MySQL, we have all the user accounts: who you are, what you’re wearing, what you have in your inventory, etc – your transaction history. Also, presence information goes in there, which includes what sims are running on what hosts, various things we need to know to keep the grid running."

That bit about inventory certainly sounds like the Asset Server, or at least some part of it. Is the confusion between items already rezzed (being cached outside a RDBMS) and items in inventory (stored in MySQL)?

I've noticed the transaction history slowing down, which is understandable after it was greatly enhanced a short time ago. Happy to see that work is underway on all this, but I am definitely in the 'buy a commercial RDBMS' club. And for the record, my human works for a large RDBMS company, but he's a technician, not a sales person, so he wants to see you move to something scalable that makes the app function better, not line his pocket with commission money <g>.

Regards,
Taras

As i think:
No, the inventory servers (SQL databases) are used to store all metadata, and are meant for complex queries. (Who owns what items, which items are comprised of which prims, how are they modified etc.)
The asset servers store all raw data (textures, sounds, bvh animations etc. etc.) and are basically a big lookup table... input an UUID, get back the asset.

Ah, now I understand. Makes sense, but I would have had the cache DHCP modulators configured through a load balancing protocol running off of a 2300 chipset working within the .net framework but outside of spam filters and "computers".

If I may ask, what does it cache? I'm quite out of my area of knowledge here, but I'm feeling very curious about the whole thing I don't think the simulator needs things like textures, so maybe it caches scripts?

Also, wouldn't a large cache between the asset server and simulators result in a lower load against the asset server than having each sim have its own cache? It seems to me that independent caches would result in multiple sims requesting the same asset to cache it.

I think I must be missing something here, because the task seems like it should be very scalable, but still it's not running well. I'd love it if somebody could enlighten me about that.

That makes no sense at all

Sims maintain a local cache of their textures and objects. This cache is emptied upon grid restart. Sims have to "recache" after grid restart. I've asked a Linden about this caching and been told that sim resources aren't recached until someone visits the sim and looks at things!

Of COURSE it, er, doesn't....

Right - someone get that bloody Panda back in it's cage - now please!

And here I thought that asset server == database server == running MySQL/Some DB server
Now that I think about it, I remember a screenshot of a web page on the asset server (it was durring the first part of the Mono integration thing), and it makes sense now.

And somehow, I understand why it's completely falling over like a house of cards on trampoline on a regular basis.

What kind of servers are you guys using?

isilons. They're huge NAS clusters.

Yeah but whats the scalability of isilon? I know with an EMC you can pile on the datamovers and in essense exponentially increase your number of connections et al.

For instance I can add two more datamovers to the two overworked ones I have downstairs at my work, move the luns over to the new datamovers (simplified I know) and I double my efficiency (to a point, the more mount points you get the faster it will get, a hard lesson learned when it comes to EMC).

Also how many servers in the cluster do they actually have going into the NAS solution? If you have reached the process limit its time to expand....

Moving on to your points:

Item 1: Moving to solid state memory, I can see how this would work for Eve, but SL is a whollllle different animal. This would be prohibitively expensive considering the sheer number of items in the world et al.

Item 2: Caching, this would be another expensive one and I'm not sure entirely the best way to do it, cache say just the textures loaded into the world in the last 5 days (i.e. viewed) with textures from items unseen older moved to the back of the pile as it were... now if you were going to do THIS with solid state memory it would still be insanely expensive, but less than going full out with it ^.~

At any rate we're talking major architecture upgrades here...

SQUID can be configured to keep hot items in RAM. Let's hope the SL SQUIDS are doing that.

And on the NAS comment, wouldn't SL's implementation be more like a SAN?

Why are assets not stored in a database? Are they just stored as files? Wouldn't manipulation of assets be easier if they were stored in a database? Or is the asset collection not suited to database storage? The Isilon webpage says that Isilon is adept at storing "unstructured "data. Is the asset collection considered to be "unstructured?"

I love this stuff. I'm such a geek.