How much cache would you set up if you could...

That's about how often you'd need to clear this kind of cache, because about the only thing that can corrupt it is actual file system corruption.

You download a file to something like "$cache/03/e4/3e/03e43e....j2p.incomplete.$pid". When the file has finished downloading you rename it to remove the ".incomplete.$pid" suffix. If you never finish downloading it it will get deleted the next time the background cache scrub process sees it with a process ID that isn't live. Unless the file system screws up, you're never looking at an incompletely downloaded file that you're not actively downloading.

I'd like to do it by time. Cache everything I've seen in the last week, or month, or whatever.

That's what caches do, only without the granularity. When it needs room for something, it tosses the least recently used stuff.

Aside from the current broken caching system and having to reload textures in my home sim repeatedly...and stalled texture loads...

From what I understand the current 1GB cache can hold about 4 sims worth of textures, considering I frequent about a dozen sims regularly each week 3GB would probably be fine for me. Thinking ahead, 10GB seems like a better amount to work with, 40 sims worth of textures would cover even the most TP happy people

Simple is almost always better, especially with coding.

Cache is one of those things that works well only if it is the "right" size. Yes, you can have too much of it and it can actually hurt performance if you do. Cache only helps performance if you re-reference what you put in it. Otherwise, it actually hurts performance. The bigger it gets the longer it takes to search it.

In this case, "searching" the cache involves one file open operation, with a well distributed name space and no directory having more than 256 entries. You will need to have an awfully slow local file system for that to be slower than fetching the same texture from the sim... which is going to look the file up in exactly the same cache structure (SL uses squid internally) before hitting the asset server.

When I had a 12 drive array at home for ..testing, I set up a 500 GB cache directory for SL. It actually worked pretty good until I got some bad textures in it, and had to clear. Took almost 30 minutes for SL to clear the cache directory.

For very large caches, having a raided drive array is really needed for best and fast access.

The current cache is very very fragile, with part of large objects in the VFS and part in the overflow file (the "texture cache", and no purging of incomplete downloads. That's part of wht my JIRA is trying to address.

For someone on the kind of Internet connection I have, or someone with a small cap, you don't need RAID-10 to win.

Thanks for the link to Weird Al's "Spam". Such a nice link needs no reason.

Without doing too much work clearing disk space I could devote a hundred gigs to an SL cache.

Would a system such as you suggest help the lengthy and frequent gray textures problem and perhaps the deformed till loaded sculpties problem?

Well, I believe it would. There would likely be a balance point where the time the OS spends looking up and opening files exceeds the time downloading textures from the sim, but I think it's worthwhile to allow people to find that balance themselves... particularly given the wide variety of network connections people use (or suffer from).

Of course individual users would need to limit their cache size somewhere along the line but isn't this something that the client could help with at installation as it currently does?

A workable cache of 300GB might be extremely rare but would an unweildy cache of any size have a negative impact other than on the user? The only problem I envisage is that someone who does not know what he is doing might ramp up the cache in spite of the configuration suggested at installation, which problem is easily fixed by emptying it once again and clicking the 'Recommended Settings' button.

My internet connection is not capped by my ISP but very mobile residents must be hampered by the status quo if their ISP is limiting downloads.

Note: I don't understand the relationship between animations and cache: bvh files are not downloaded into it so why is the information on frequently viewed animations erased when cache is cleared?

I'm not familiar with squid internals, though I know it's been around for a long time and is fairly popular. I'm inferring stuff from the Jira entry, and trying to understand this.

The mechanism seems to be mapping the UUID directly into a relative file path at a fixed depth (either 3 or 4). I'm not sure how you can guarantee that the bottommost directory won't have more than 256 entries, though perhaps that's not as important as the size of the higher level directories. My days of worrying about the number of open file descriptors are long behind me, but can you really keep 65K directories open at once? Or is that unimportant?

Also, will the performance characteristics carry over to MS platforms? While my understanding of UNIX file systems is just a hazy memory, my understanding of FAT and NTFS internals is non-existent. For that matter, are there important performance differences between running this on a server configuration (multiple high speed hard drives) versus a typical desktop/laptop (single hard drive at the slow end of the spectrum)?

Please don't take this as criticism or even skepticism. I'm all for simplicity, and nothing I know about the current SL client cache would make me bet on it being better than this. This is purely technical curiousity on my part.

I don't think I've run into caps, although I'm sure my ISP is capping some people somewhere. But I am in a rural area, near the end of a cable run. I do see a noticeable drop in available bandwidth during the early evenings. My guess is that I'm seeing contention from all the neighbors who get on at that time, while during the day, there's much less overall traffic on my particular piece of cable.

Regardless of whether my speculation is correct, I think there are other reasons besides ISP caps that may cause people to suffer from slower than average networks.

No. And even a cache much larger than optimal wouldn't have a significant effect on the user other than the missing disk space.

You wouldn't need to empty it, just reduce the assigned size and let the cache sweeper trim it down.

If it turns out to be a problem, if you get more than 256 entries you delete the oldest file in the directory when you create a new one, or it may be that you would change to 4 levels instead of 3 if the cache is more than (say) 40G... however it turns out.
Directories are never "kept open", so no, that's not really relevant. And files wouldn't be kept open longer than it takes to read and write them, as they are now.
Squid runs on Windows too.

Does the texture cache actually, you know, work now?

Even just flitting around the empty region, it seems like (according to the texture monitor) I am downloading the same textures over and over.

Did they ever fix the stupid texture download bug?

Like Jed Clampett's old truck. It need replacing.

A few have mentioned it on this thread so far. Also, I think my ISP does something similar to yours, which has the essentially the same effect.

That aside, it's not my style to go haring around taking in the wild and wonderful sights of SL but quite a number of residents do. I can easily imagine that a big, workable cache would be an asset in that case.

I might even consider getting out and about more.

If anyone has a rough answer, what things activities etc uses up download the most and least.

All Aussies are on capped downloads. When I have no gig left and am on dial up speeds, I can't TP at all I load up really slow, I have lag, but oddly I can build fine once it is all loaded...

Going to a new location, causing SL to download all the new textures, is probably the biggest bandwidth user.

To use the least, stay in the same spot, where the objects don't change much. They can rotate -- most rotation is done client side -- but if they're actively moving like a robot, that causes object update messages to be sent to each client. And of course, with as few as possible other avatars, with them doing as little as possible (moving, changing animations, etc). Chat and IMs are inexpensive, so chat all you want.

I just looked at my computer and hey, I've got 293 GB free after owning it for 2 years, so I voted for 100 GB, as long as it can find the files easily. Going to vote on the JIRA now.

Squid's cache system is meant to handle huge (talking terrabyte territory) caches with minimal interruption (ie, deliberate human interference). And it lives up to that... I have a squid cache here that's been running for ~3 years without being "cleaned." There's ways to detect and get rid of invalid objects from a cache that don't involve throwing the whole cache out; if programmers can produce code that does this for free, the guys who produce code for pay should be wondering what they're getting paid for on this one...