Asset Server Problems - Everyone Kicked Out

Already exists. There's a layer of caching called the 'squids' infront of the primary asshat server, unfortunately, for saving - this doesnt come into play.

-Adam

For your info:

A lot of people keep referring to the asset server as some form of database. I asked a techie Linden once how their back end works, and he gave me this overview of asset control in Second Life:

Each 'asset' is a text file containing all details of an asset. When needed, these text files are called off the asset server via Apache.

The SQUID http proxy (look it up on google) caches web pages or resources, and caches the 'asset' text files that can be requested, and speeds up the whole process of getting asset text files to our SL clients.

The asset server itself is:

"The asset servers are nothing fancy. Dual opterons, I believe. Most of the actual asset query load is handled by the squid servers in front of them, though." --> Quote from the Linden.

I believe the contents of the asset server is probably backed up by a second fallover machine, with both of these standing behind the collection of SQUID http proxies.

*** In case you want the raw conversation, here it is, names removed ***

Linden: Heh. We have three raid5 systems set up in a chained replication system.
Jsecure Hanks: oh the asset server is a chained replication system?
Jsecure Hanks: cool, that's a smart idea
Linden: It's not actually a database, but we use a similar method to replicate the assets.
Jsecure Hanks: it's not? How do you store all the data?
Linden: It's all done via http. PUT/GET.
Jsecure Hanks: So you have a mysql system, which talks via apache in HTTP PUT and GET?

Linden: No, it's jsut storing the assets in a hashed directory structure.
Jsecure Hanks: ok so how does one of those work, you have a file for each asset? (I'm learning a LOAD tonight

Linden: Yep.
Jsecure Hanks: it's quite fast really....
Jsecure Hanks: So every time you fetch an asset it skims a file for the details and sends that back via apache

Jsecure Hanks: so what kind of machine do you need to deal with all the requests SL generates?

Linden: It just sends the file. It's that simple. You want that texture? Ask the asset server for it, and it grabs it and sends it to you.

Jsecure Hanks: oh I see
Linden: The asset servers are nothing fancy. Dual opterons, I believe. Most of the actual asset query load is handled by the squid servers in front of them, though.

Interesting to see that they're actually using a replication system. So why is this not making the asshat subsystem adequately fault-tolerant? I mean, I dont have to tell any of you that a main goal of these systems is to transparently spread the added load when any given replica fails. You dont see google going down, do you... They just pop a box off the rack and pop another back in.
No company should have its core business brought down by a single hard-drive failure or some weird file-system error. It's just silly. This is the 21st century, people.