Ideas for repairing / enhancing the Second Life cache?

The local cache that Second Life uses has many shortcomings that affect all of us; the main thing being, well, it doesn't work and hasn't since about v1.5. The problem this causes is un-necessary load on SL's servers by forcing people to download, over and over, textures and other asset information rather than storing it locally on a hard drive. I've started this thread for us to share some ideas on how this can be improved, and hopefully fixed.

Idea 1: De-centralize the cache on the client machine's hard drive.

Currently, on PCs at least, the Second Life cache is stored on the hard drive in the folder \Documents and Settings\<Your User Name>\Application Data\Second Life\cache

Within this directory, there are files for caching the avatar(s)' inventory/ies (a GZIP'd file named as <avatar key>.inv.gz), several SLC files which appear to be object data, and files for indexing the cache and the data itself... oh, and the name.cache file with avatar name/key pairs.

I'd like to see sub-directories under this directory; let the file system do the indexing and heavy lifting! Under the cache directory, create a directory for each sim, and "other" for assets that aren't in world, for example, inventory.

cache\Indigo
cache\Ahern
cache\inventory
cache\textures
cache\other

This natural indexing method would also make the 1 gigabyte SL cache limit a thing of the past; its ridiculous we have the same limit for cache that we had when there were only 40 sims when now there are well over 1000!

Idea 2: Allow user management of cache

It would also be great if under "About Land" (or somewhere similar) we could choose which sims (or specific land plots) we wanted to locally store ALL asset data from: textures, object data, land height map, and so forth. Perhaps this could be done in conjunction with a setting to NEVER download anything from a specific plot. (hello, Impeach Bush solution!)

For my home sim, I would definitely choose to permanently cache all my sim's data. If it could be done on a per-land plot basis, I'd block out Nexus's plywood in Olive until he finishes his build up!

I'd also like to have a local cache of my entire inventory.

Then Second Life would only have to check a time-stamp to see when an object / asset was last modified; if it hasn't been modified since it was downloaded, no need to re-download the asset (texture/object/clothing/notecard/whatever). The current system should handle things in this fashion, but it clearly does not; I see textures reload in my home sim all the time, and not from the hard drive; they start blurry, and become more clear in several iterations, which connotes a download from the SL sim server bank.

Idea 3: Use Reversable Encryption

If this cache data was stored with reversable encryption, this could also act as a legitimate backup copy of data. This does open up the possible danger of someone who knows the "key" being able to steal objects from the hard drives of SL users. A kill switch should be built in which would cause everyone's cache to be deleted upon their next log in case the "master key" needs to be changed. The client could then rebuild the cache through downloading from the appropriate sims in the background. This would also allow LL to rebuild a user's inventory from their local copy on the hard drive instead of saying, "Oops, sorry, you're screwed." Additional security measures would also have to be considered, but using reversal encryption - at least on the local cache of one's inventory - would be fantastic.

Anyone else have ideas on improvement that could be made? If we're going to have a draw distance of 512 meters in the 2.0 renderer, we should really be considering a much better local storage cache first. Imagine trying to download 512 meters of objects all at once!

Regards,

-Flip

Flip, my impression is that the cache mostly works - I have a fairly slow internet connection and objects definitely show up faster than they would if my client re-downloaded all the textures repeatedly. What seems to be updated each time is the object positions (also geometry?) and I don't know what you can do about that - some checking is necessary due to SL's dynamic nature.

What I think would help is forcing LL developers to use the same kind of connection (at least like one week in month) people from e.g. europe have - I'm 25+ hops away from a sim and my ping is rarely under 250-300ms. It's easy to do the kind of "update everything on the fly over the network" development LL does when you sit directly "on the grid", have 10ms latency and your packets don't get delayed or lost.

I like the idea, but could you then set the size of your SL cache to vary with the size of your hard drive? I mean would it make sense to be able to case two or three sims? I could easily see my self buying a couple hundred gigs of deadicated hard drive to SL if it would make a difference.

I was under the impression it was not the cache causing problems at all, but a bottleneck in the interest list mentioned a few weeks ago?

It's the Interest List and the Cache loading problems together that have to create all these problems. Hopefully Linden is not like "what are they talking about?" with these issues because I see them everyday now here on the boards.

I still like the idea of P2P, sharing textures of a sim from other users. I wrote in another topic about using a system only in slight relation to BitTorrent where if you are in an area long enough your interest list becomes "seeded" with everyone elses and that becomes a stream that is downloaded straight into your cache, bypassing LL altogether so you are downloading only world data from Liden and textures, AV's, sound, from other players in that sim.

But then you run into organization issues with the cache and keeping the clients and servers "synched" together but without knowing even the most basic stuff about LL's setup and how those clusters are organized (assuming they even use clusters, I just guessed). Would it be impossible for us to get a picture from inside the grid room LL? Just a picture! Don't have to tell us anything specific if you don't wanna

When I comment on the possibility of there being some value in increasing the cache size I get told that a larger cache would slow things down because the time required to determine if the information is available locally is greater than the time required to just go ahead and get it off the servers.

Also I get told that the cache doesn't work, and that it does work, and that the problems we see of redownloading something that was in the cache ten seconds ago are due to problems in the interest list.

It seems to me that a relatively simple change - doubling or tripling the cache size - could be tried out on the preview grid just to determine from real world experience if that might make the system work better.

Surely there are some constants defined that would make increasing the cache size fairly simple.

No point in trying that, I suppose, till one is sure the interest list and the cache code are both working fairly well.

It sure is aggravating having a totally empty 250 gig hard drive crying out for SL data to be stored on it.

Sorry, I don't understand how idea 1 would be useful. Also, surely the encryption must be reversible - what would be the point in caching something if you can't get it out.

The cache certainly needs improving though, at least in regards of the textures. Not sure if checking when things were modified would help the way things are currently, but it might if better (user controlled?) instancing was introduced.

Actually, they can't get us a picture. When a bunch of us went to visit LL in July, we were lucky enough to be taken by the Lindens - headed up by Ian - to see the LL data center. I can confirm, with my very own eyes, that the asset server is NOT a Commodore Vic-20.

The hosting facility did not allowed photography of any kind and we were escorted through the entire building. It was kind of funny; we were seeing the data center for the first time, as was Philip! He had never been before.



I've heard the same things as you. If it takes longer to check a time stamp on a texture than to find said texture from an index, then the problem is clearly with the design of the index! Time to fix that. Some of my ideas above were exactly from hearing these kinds of things.

I like your idea or testing on the preview grid.

Candide: The interest list not working directly affects the cache. While I'd agree it does partially work, there are other times I bring up the texture info debug screen, and see if loading texture I view every day!

Jake: I couldn't agree more. Hard drive space is so damn cheap these days, its a shame we can't get a better indexed caching system to utilize it. I have 60 gigs just sitting on my laptop begging for use! And that's on my LAPTOP!

Regards,

-Flip

What organisation/synchronisation issues are you talking about? IIRC textures can't be changed and always keep the same UUID.

Right now, the cache system writes some very large files (like the windows paging file) to reserve space to that directory. I'm proposing that using the default OS file system as part of your indexing plan might be more efficient that trying to index and constant change a giant file. In my experience, its been better to let the operating system do what it does well. I'm by no means and expert in the field of caching, however.

My commentary on reversable encryption could have been better stated; I was really trying to say that instead of just using it for the cache, it should be extended to be able to be reverse through the SL app to restore lost data on the asset server.

Regards,

-Flip

Ah. Okay, that makes sense now.

Does having all the assets in one server strike anyone else as odd?

Excerpted from SecondLife/app_settings/tree.xml:I think this points out an interesting property of how SL assumes all content is dynamic even when it isn't at all.

The tree "Pine 1" texture isn't necessarilty cast in stone, as editing this file shows. How many players change the texture_id in practice? I'm guessing none. How many times has LL changed that texture_id? I would guess at most once.

However, every single player will at some point ask the asset system for UUID 0187... at one point, possibly multiple times during a session, and certainly every time the player clears the local cache.

Since the texture for "Pine 1" is effectively immutable, why is that texture not part of the installation? How many asset requests would not ever need to happen if just the tree textures came pre-loaded? Sure it would break the symmetry of all textures coming from the same source, but at a time when LL is doing everything they can think of to keep the asset system from falling over, ought they not think a little "smarter"?

As for the proposals above, they are all sensible, but I would caution against them trying something tricky like asset swarming until they can first nail down what they broke in the comparatively simple cache and/or interest list.

To my understanding, somewhere in the last year or so, LL added a second tier cache (proxy cache?) to the sims themselves so that it goes asset server => sim cache => client cache => display card memory. This was a sensible approach to a system designed around a central asset repository that is more static than not. However, as the old maxim goes: code that doen't exist is bugless, adding the sim layer cache added the potential for defects there and increased the complexity overall.

The one thing they have going for them is that cache invalidation is not as difficult as it might otherwise be if major changes to an object did not cause a new UUID to be instantiated.

As far as the "oddness" of a central repository goes, this is a problem in all database oriented designs; at some point you have to have an authoritative instance of an element. In clustered databases, this is accomplished by complicated row-locking and coherency mechanisms. As expediency was one of the prime SL design constraints and - as a start-up - they had no idea if the thing would actually fly, this was not a wholly unreasonable choice. It is quite true that it does not scale well which we are all now seeing the effects of.

Well, just the fact that they are using what I believe to be a "hard-coded" grid meaning they add/remove sections manually and don't really have much control over where the power "goes" (meaning it's locked in that grid) but they do control how it's sliced up in terms of priority like sending textures might have a higher priority over sound.

The future is in being able to host large events and being able to reverse the effects of lag. If you were around more people (sharing data with them textures, sounds, etc.) your lag would be less, and more when you are by yourself and only pulling from LL servers.

If they had one million users pulling from the servers all at once? I would not want to pay the membership fee to run all those computers. The way things are working now they will make it to expensive to operate.

Okay, I came back from the pub drunk and started writing this. I turned into a rant that has very little to do with what we're talking about but I thought I'd post it anyway in case anyone can dredge anything useful out of it. With any luck I'll get the ideas in my head into a more coherent form tomorrow.

<drunken rant>
The thing is, Philip has said he sees Sl kinda like the web. Leaving off what that implies for the long term business model for now, it does imply that a centralised server model isn't going to work once the service becomes popular.

For initial uses a single asset server makes sense because the development is easier. Once it starts scaling, though, it's necessary. We can hope that 2.0 addresses these issues, at least in part.

Another thing Philip has said is that Snow Crash was an influence in the creation of SL. In that book the metaverse is controlled by the protocols governing it. I could totally see the majority of SL being an arbitary framework to develop those kinds of protocols under, limiting the data transferred and the latency between nodes until the hardware catches up with the vision.
</drunken rant>

Am I understanding properly that individual textures each get a UUID, and can't be changed, and could thus all be cached on our local hard drives and never removed except as needed to keep from filling up our hard drives? Not just the Linden stock ones, but everyone's custom ones as well?

That's what I've always understood.

This seems to be contrary to the "everything is dynamic, therefore everything must be streamed" line of thought one hears so often.

Of course it's just not true that everything is very dynamic, that is, changing all the time in SL.

Lots of structures are stable for months.

----

The advantage of Flip's idea 1 would be that if you were in a sim it would check the cache for just that sim (or that sim and just the adjacent sims) instead of having to check a cache that holds data for all the sims. Thus it would find things faster and be able to leave things in longer. Is that the idea?

We can't change textures once they're uploaded, so really they're not dynamic. Perhaps in the future they will be (maybe coming with HTML on a prim) but for the moment I see no reason they can't be cached (semi?)permanantly.

If there was decent instancing the same could be true (well, after a time modified check) to builds as well.

I hope not, that could be done without using the file system.

AJ, I enjoyed your drunken rant - that is some good long term thinking, heh.

Textures can not be modified once they're uploaded. They're assigned a key and left on the server. Another cool feature would be to allow a common texture bank, but it would require a change to permissions. Instead of having the texture bank in the library just be a few, why not have a place where all textures can be browsed that are set to "next owner can copy/modify/transfer"? A fourth checkbox would have to be created, naturally, to set it to be publically available. This would cut down on new uploads by allowing people access to the tens of thousands of textures already there. I'd share several hundred myself that I've gotten from "Free texture" web sites.

This raises another question: when a texture is downloaded is one sim to the cache, is it saved as part of that sim, or just generally so it can be retrieved if the same texture is used in another sim? I'd say it should be saved generally so it can be retrieved for any sim, but what should be done isn't always done.

Regards,

-Flip