"Permacache" feature

I think so. First, my hat is off to the great strides that SQLite has made since I looked at it last, but I think this is saying what I mean better than I did (from the SQLite FAQ):

" Is SQLite threadsafe?

...

"Threadsafe" in the previous paragraph means that two or more threads can run SQLite at the same time on different "sqlite3" structures returned from separate calls to sqlite3_open(). It is never safe to use the same sqlite3 structure pointer in two or more threads.

..."

See www.sqlite.org/faq.html#q8 for the text I've ellipsed out (ellipsed -- is that a word)?
~

P.S. I would be happy to be corrected on this! It applies directly to some platform validation we're doing at work.

the cache is located oin you system drive (normall c\ Documents and Settings\<USER NAME>\Application Data\SecondLife\cache


There's 1 file of approxmatly the cache size you've set (I have it set to 1G, windows has reported it from 983M to 1.03G soemtimes there's a simularly named file around 600K. then there's a name.cache file, which is a plain text UUID, time since epoch (presumably when last seen, avatar's first and last name

all space seperated. I haven't looked for the mapping of texture UUIDs


note that while SL is running, many many more files are stored in there

You can have more than one DB structure pointer (handle) open at a time on a DB.

I think what you mean is that it isn't good for multi-user / multi-process use, which, yes, you can't do.

Of course, it's much easier to watch what goes on when you have a Mac (since it's really an Unix box underneath ).

Textures are sent just as regular, streamed files, with their UUID as filename — but they're RIFF-encoded (ie. audio ) files, so I guess Philip has simply re-used his clever code from his RealAudio days I expect that you can even play them on a RealAudio player, to listen to some screeching noise They're probably encrypted, though (like the name.cache, which, despite being plain text, the UUIDs are slightly encrypted), so it won't be easy to decrypt the textures out of a RIFF file.

When SL shuts down, as well as when it fires up (because of crashes), the textures are put into a DB2 database, and an index for that is created. Since every filename is clearly unique (thus the use of a MySQL UUID), hashing it should be absolutely trivial. Actually, when LL's developers talk about the asset server having "a proprietary filesystem", what they mean is that all assets are saved on disk with the UUID as a filename (and the contents of those files tell the system from where — ie. which sim — you should load it). Can't beat that in terms of performance And, of course, this is KISS technology at its best Also, it makes local, sim caches (using Squid) incredibly efficient — absolutely no need to use "databases" for that, it's just files.

Inventory is saved to your disk as a very simply-formatted file and then gzip'ed. It's basically a dump from the asset server, with references to UUIDs, and in a nice, human-readable format. Don't worry, you can change it but you can't break it, SL is clever enough to understand you've tampered with it

So, why does the local disk cache not help much? Well, just take a look at the average size of textures: these days, most are about 512 x 512, with 24 or 32 bits, so around 1 MByte each, uncompressed. Compressed you'd see perhaps a 10x factor in space saving. This means that a 1 GByte cache on your disk can hold around 10,000 textures or so.

Now, since each sim can have 15,000 prims, and each prim has an average of 7 faces (don't ask me why... I'm quoting this from another thread), this means that you'll have 100,000 textures per sim or so, on average. Well of course a few will be duplicates, but these days, most people put unique textures on most faces. But since sadly your disk can hold only around 10% of that. So, this means that when you travel across a sim, you'll be constantly loading and discarding textures, all the time.

IMHO the problem is not a "bad" caching system, or a "bad" cache design. It's just a limitation (1 GB instead of a more reasonable 100 GB to hold, say, 10 sims' worth of textures...) on the SL client's disk cache size. There must be a reason why the cache is "only" 1 GB, but I fail to understand it. It's definitely not due to "performance" issues. As mentioned on this thread, using an index to look for a texture in a DB2 file is a logarithmic operation, and searching for a texture in 1 GB or 100 GB is just marginally slower. The "new" textures are not incorporated into the index DB2 file, but simply written to disk while SL runs — so, they work just like the asset server, ie. a simple filename search, and you can't beat performance on that.

All in all, this is one of the best and cleverest caching systems I've ever seen and the one with the best performance, no matter if you think they should use another database type. Nothing beats simply looking up filenames on a disk

Well, a sql db has a much faster lookup than NTFS which is why bill gates is so gung ho on migrating in future version of windows.

But thatnks for the write up, very illuminating

Still, in the end, why are we limited to 1GB??

technically the SL cache format can support 2GB; if you were motivated you could edit the exe to support it. Lookup time is based on how many assets are in the cache and how the dirty the index is. When an asset is deleted the entry in the index isn't deleted it is just set to null. Every now and then the cache needs to be cleaned up. As to the mechanics of the cache i am not at liberty to discuss.

So searching the disk cache is a linear function (unless they keep a sorted optimized version of the index in memory, which in that case it be log).

For images SL uses JPEG2000, which is a most excellent format, it can handle massive data corruption very well and still produce an image. When you enter an area and the images are only half visible, that is because they have only been partially downloaded. If parts of the image are missing the image can still be rendered.

SL uses Ogg-Vorbis for it's sound format, when an image is played it is decoded to RIFF-Wave and written to the cache folder.

ok so you're saying the index is limited to 1GB because they don't have a sorted list of hashes?

That's daft, when what they're looking up is effectively a 128 bit random number. Simply using an n-ary tree on the bits on the UUID would be better than a linear search.

I think it would be better to cache the prims separately and keep them longer. This would reduce the number of (oops I flew into a building) issues. Also pre-download the surrounding prims that are larger than 5 meters since these are usually the main parts of buildings.