Q: How many Lindens does it take to make a local cache work?
|
|
Ravenous Dingo
Registered User
Join date: 25 Feb 2005
Posts: 78
|
02-24-2006 15:20
From: Mack Echegaray From: Adam Zaius I've always wondered why LL just doesnt harness the local filesystem. It's designed /exactly/ for this kind of storage and retrieval, and in most cases, usually does a good job. Filesystems like NTFS/FAT/ext3 are incredibly inefficient at handling large numbers of small files. Unbelievably so in fact. This is why so many programs invent their own mini filesystem-in-a-file (eg OLE Structured Storage) .... because the regular FS just can't cope with say 10,000 files of say a kb or two each being handled at once. It's this sort of problem that motivated Hans Reiser to design ReiserFS, which has as one of its unique selling points the ability to handle huge numbers of very small files, very efficiently ( http://www.namesys.com/). If it's actually true that the cache contains when it's full, something like 26,000 assets then it doesn't surprise me at all that this causes huge problems. Designing a filing system that can handle such large numbers of small files with high performance is extremely hard. You can't just use the operating system because OS filing systems can't cope with it. So you have to invent your own. It *can* be solved, Reiser4 managed it pretty well, but it's difficult. Web browsers like Firefox default the cache size to only 50mb, and though I use it all the time my Firefox cache only has 5mb of data in it - but I've seen people say they have a SL cache up to a gigabyte in size! Oh, something else that confuses me .... the original post said "I have a theory why it is slow when full and fast when empty" .... this doesn't sound like hard facts about how the cache works. The guy himself said it was a theory. Maybe he's right! None of us know. But jumping to conclusions based on incomplete evidence is bad. Final thing, am I the only one who is a bit tired of Introvert Petunias constant negativity? OK, the software isn't working well for you, we get the picture. Hanging around and flaming the 'Lab probably won't magically make their team more productive. If I were you, I'd relax and go do something else for a few months. Come back in the summer and see if the new versions have better performance - if not, too bad, if so then great! pwned. |
|
Introvert Petunia
over 2 billion posts
Join date: 11 Sep 2004
Posts: 2,065
|
02-24-2006 15:25
From: someone Defragmentation? That's something you used to have to worry about back in the '80s, isn't it? If only that were so. I've seen the SL cache file span 1500+ segments on an NTFS volume.  The more I learn about Windows the more poorly designed it seems (e.g. paging activity while there is still available RAM, the disk buffer cache declining to use available RAM, etc.) It kinda gives me the willies. |
|
Introvert Petunia
over 2 billion posts
Join date: 11 Sep 2004
Posts: 2,065
|
02-24-2006 15:30
From: Ravenous Dingo pwned. Wow, that was like the most useful post ever! Thanks for contributing. |
|
Mack Echegaray
Registered Snoozer
Join date: 15 Dec 2005
Posts: 145
|
02-24-2006 15:47
From: Introvert Petunia Have you any reason to believe this other than the index.db2* and data.db2* file names? I just checked and they do not appear to be in any Berkely DB format from Berkely DB v2 to v4.3. Berkely DB allows for AES encryption keyed via a password, and to stop trivial ripping of textures they're probably using it. But no, you're right, I don't know for sure .... just a guess based on the file extension and the fact that using it makes a lot of sense. They also seem to use some other formats like for the inventory and the objects_whatever_.slc files. But for me, my cache is about 500mb data.db2.x |
|
Ravenous Dingo
Registered User
Join date: 25 Feb 2005
Posts: 78
|
02-24-2006 16:49
From: Introvert Petunia Wow, that was like the most useful post ever! Thanks for contributing. YW! |
|
SuezanneC Baskerville
Forums Rock!
Join date: 22 Dec 2003
Posts: 14,229
|
02-24-2006 17:08
From: Introvert Petunia Ithe disk buffer cache declining to use available RAM Can someone point me to an authoritative source (preferably Microsoft) that says that in comprehensible language and also to somewhere that explains why this would be?
_____________________
-
So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.
I can be found on the web by searching for "SuezanneC Baskerville", or go to
http://www.google.com/profiles/suezanne
-
http://lindenlab.tribe.net/ created on 11/19/03.
Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan
-
|
|
Strife Onizuka
Moonchild
Join date: 3 Mar 2004
Posts: 5,887
|
02-24-2006 17:20
From: Starax Statosky I realized he was just filling a role. And my role is ResMod. And this is a PG forums. And your head is sticking up a little to high out of the trenches.  SL's cache hasn't changed since... forever. In retrospect, they probably keep an intermediate hash table ontop of the cache index.
_____________________
Truth is a river that is always splitting up into arms that reunite. Islanded between the arms, the inhabitants argue for a lifetime as to which is the main river.
- Cyril Connolly
Without the political will to find common ground, the continual friction of tactic and counter tactic, only creates suspicion and hatred and vengeance, and perpetuates the cycle of violence.
- James Nachtwey
|
|
Introvert Petunia
over 2 billion posts
Join date: 11 Sep 2004
Posts: 2,065
|
02-24-2006 17:31
From: Mack Echegaray Berkely DB allows for AES encryption keyed via a password, and to stop trivial ripping of textures they're probably using it. But no, you're right, I don't know for sure .... just a guess based on the file extension and the fact that using it makes a lot of sense. They also seem to use some other formats like for the inventory and the objects_whatever_.slc files. But for me, my cache is about 500mb data.db2.x First, thanks for being cool after our little spat, I'd say that in PM but you seem to have not enabled that. So, thanks. I don't think that AES encryption is applied to the "magic number" at the head of the DB file as that would give you a known plaintext/cyphertext pair which would sort of dent the encryption and make the file unidentifiable as the magic number would have less magic. Furthermore, if the 2 in "db2" is meaningful at all, it would suggest Berkely DB is version 2 and AES didn't enter until v4. So I don't think that these actually are Berkely DB files. Although I need to get to a real machine to test my guess about the "magic number" in a BDB v4 with AES. I was using the Linux file(1) command to provide the data for this, all the rest is the conjecture posted here. |
|
Starax Statosky
Unregistered User
Join date: 23 Dec 2003
Posts: 1,099
|
02-24-2006 18:00
From: Strife Onizuka And my role is ResMod. And this is a PG forums. And your head is sticking up a little to high out of the trenches. SL's cache hasn't changed since... forever. In retrospect, they probably keep an intermediate hash table ontop of the cache index. Shoot it Strife!!!! Pleeeeeeeeaase!!! I can't take anymore! I've hit the fucking forum refresh button 200 times today!! Please! somebody!!!! SHOOT ME!!! |
|
Kim Anubis
The Magician
Join date: 3 Jun 2004
Posts: 921
|
02-24-2006 18:40
*shoots Starax*
*picks up wand*
*runs*
_____________________
http://www.TheMagicians.us 
|
|
Strife Onizuka
Moonchild
Join date: 3 Mar 2004
Posts: 5,887
|
02-24-2006 20:52
*shrug* sorry, i don't carry guns since my Seburo jammed. Just atracks unwanted attention.
*pokes the corpse with a stick to see if he is dead; then rifles though his pockets*
_____________________
Truth is a river that is always splitting up into arms that reunite. Islanded between the arms, the inhabitants argue for a lifetime as to which is the main river.
- Cyril Connolly
Without the political will to find common ground, the continual friction of tactic and counter tactic, only creates suspicion and hatred and vengeance, and perpetuates the cycle of violence.
- James Nachtwey
|
|
Introvert Petunia
over 2 billion posts
Join date: 11 Sep 2004
Posts: 2,065
|
non-session persistent cache
02-27-2006 06:17
Yeah, okay, it is only alpha code, but the Linux client throws away the cache data files upon exit. I use that as a rough metric as to how valuable that data is thought to be by the guy working on the client.
They throw em away hard too, I tried making hard links to em so that I could do a post mortem on data files, but those links got zapped too. Regardless, they do not appear to be a Berkely standard format or any db file that Linux file(1) knows about.
|