cache the damn world

is there any way possible, instead of streaming every damn thing every damn time u log in ... just store it on the pc, place's ppl have already visited, so we don't have to load it every damn time we go there. Just have the client and server communicate with each other to find out what is different or new, and then LOAD just that, not 50,000 prims all at once ... I dunno ... having the place's u been to stored on your pc, seeing as the client is so small would just make thing faster and run smoother for the user's. I can't stand having to load thing's everytime I go somewhere, that's prolly part of the lag issue, YOUR STREAMING THE WORLD EVERYTIME WE GO SOMEWHERE .... take that load off the server's and our bandwith by caching the damn thing on our pc's ...

this has been addressed in the past i think the response was something in the order of because the world changes so often its not possiable.

not to mention that the world is probably about... hmmm... everytime i do the math i seem to get about 25GB...

I asumed there were about 100,000 users and about 10,000 groups. That for all the sims the average number of prims in them is about 5000 and there being about 380 sims. That the average user including inventory takes up about 64k. The average object requires 4k. The average group requires about 4k. Then for images we assume there are about 1000 unique images per sim. With the average image taking about 32k. Then for land RAWs it takes 2k per sim.

so...

100,000 * 64 + 1000 * 380 * 32 + 10,000 * 4 + 380 * 5000 * 4 + 2 * 380

6400000 + 12160000 + 40000 + 7600000 + 760 = 26200760 kb

24.98 gb

it's sounding like if you hacked the cache code for the client you could download probably 1/4 the asset server.

I pulled most of my numbers out of the air so please correct (user count should be about correct).

I would be happy to devote 25GB to SL

wait the land requirements are wrong it should be about 6k per sim as you need to specify rollback, land permissions, stream url and access lists.

which makes it still about 25gb but 26202280 kb

then if you consider cloathing you end up with probably another 1-2 gb

so 27 GB

still very reasonable...

This is nothing but gut feeling, but I think by the time client and server walked through a difference analysis between server state and client cache at log-in, customers could be looking at a much longer start-up time than it takes just to send down a fresh point-of-view.

- yoshi

No no no, you don't store the whole lot, just the parts that you need most often. It's meant to be a cache after all, standard Least Recently Used algorithm applies to bump old stuff out of the cache when you're out of room. And caches work fine regardless of the total size of the world, just as long as your own working set fits into the cache, which it does trivially.

Alas, at the moment caching seems to be almost totally broken. You leave one spot, move just a short distance away, return and it's reloading everything again. That should never ever happen. It may be caching, but it's certainly not implementing LRU. In fact, it behaves rather like they've implemented the exact opposite of LRU, so it's chucking out whatever it last loaded.

Obviously that wouldn't be intentional, so what we're looking at here is a bug. Probably the largest bug in the game.

It's no surprise that James called for caching and wasn't aware that there was a cache already, because after all it's repeatedly reloading recently loaded items. So for all intents and purposes, the caching simply isn't there.

Oh, yeah, that; that's annoying as all get-out, and a definite regression over NewView's behavior when I first signed up.

But I had thought the OP was tackling a slightly different issue, which is the client initialization during login...it just struck me (again, gut feeling is all) that type spent double-checking inventory with the asset server and going through a checklist of changed objects within the initial POV could easily exceed the time it takes NewView to push a baked-from-scratch POV down to the client, which is what I thought it does now...

-yosh

Uhhhherrrrgghhh... uhm... Have you checked your settings recently? See the option for cache size? Yeah. SL already does that, thank you very much for your suggestion.

Well, as Morgaine Dinova pointed out, it doesn't work now does it, and I have it set all the way up, still loading everythig everytime I move in a diff direction ... thank you very much for your reply

sound
i forgot to count sound ~_~ add another 10 GB.

hmmm didn't think about it to much but yeah since a sim is less then a GB then it should basicly all fit into my 1GB cache.

guess caching is foobar; as suggested above.
guess i'll start compaining about that now.

<compaining>
caching is broken.
please fix it

</compaining>

lol jk

take your time do it right

Considering, since the beginning of February, the number of sims in SL have increased about 12-15 fold, that means by Strife's assessment, by next Spring caching all of SL will probably require over 200gb. By the end of 2005, likely over a terrabyte.

Even if SL stayed 25gb, the fact is that it changes every day. There'd just be so much to recache anyway that the bandwidth from the servers would be insane.

Add to that the fact that sorting out so much data is time consuming. It's actually easier just to download new things at a certain point. If I remember from when I first brought this up as a question when I was a noob, the rationale then was that already 1gig of cache was cumbersome to the sorting algorithms.

Well, as Morgaine says you dont have to cache the whole world, just the bits of the world you saw recently.

There's probably someone here who's an expert on caching algorithms, but just off the top of my head, why not do something like:

- on the server, store the time an object was last modified
- objects not modified in over 24 hours are considered "suitable for caching"
- when you tell the client about an object, it sends a RecommendCaching flag = True/False with the object
- client only caches objects with this flag set to True

Next, divide the world into sims, or possibly quarter sims, for finer granularity.

When you enter a quarter-sim, you tell the server the cache-sequence identifier of the last cacheable object you received
- server sends you cacheable objects for that quartersim in sequence, starting at this number
- any cacheable objects that were modified/deleted (and hence became non-cacheable for 24 hours) are sent by the server as a cached-object-deletion command within the cacheable objects update sequence.
- non-cacheable objects are sent normally, using whatever the current algo is

Cacheable objects are cached on a quarter-sim granularity. If you run out of cache space, quarter-sim caches are deleted atomically, starting with the oldest (least recently updated).

There are probably standard algos available for doing something like this.

Of course, there's quite a lot of work to implement efficient caching, and it's hard to measure the result objectively. It's something that's hard to market compared to say, oh I dunno, the ability to make animated NPCs.

Azelda

Strife; the Asset server has 1.2TB of data currently on it; I'm certain 25GB may be a little short.

-Adam

Yep, that's the right kind of approach, Azelda. Your splitting of a sim region into four is heuristically spot on, since the X and Y size of a quadrant is roughly the same as the default visibility distance. I think I'd generalize the algorithm for allocating focus areas a little, so that one sim might divide its region into 4 quadrants, another might maintain only an overall list plus hotspot lists at arbitrary centers of activity, and so on. Let the sim decide, since it knows what's going on in its region and how best to deal with it. The allocation doesn't have to be regularly tiled, as long as the server can always determine the relevant focus list, which is easy.

Btw, someone might be asking themselves, why are we creating these small focus lists? It's because they are optimized access methods into the sim's object cache. Each sim needs only one LRU cache for all its objects, but that's inherently organized in LRU time order. Since a player's range of visibility in the XY plane is less than a whole sim's worth, this implies that looking for updates relevant to a particular person in the main cache would require scanning over many cache entries that aren't relevant for the player's current location, and that is inefficient. The localized focus areas provide an optimization, a presort of the main LRU cache by location.

In the general case, a client will be obtaining updates not only from one single focus list but from several, and indeed not necessarily from just a single sim. Players located near a corner of their region would typically be receiving updates from up to 4 sims. That happens now of course.

The other thing that should be said about update focus lists is that although they provide locality-sorted access to full objects, this doesn't mean that all the data of an object will be streamed down at the same time. Some types of data are hugely more important than others, for example bounding boxes and barrier properties (phantom) must get update priority over full shape data, and shape data must get priority over area colouring and that over texturing information, and so on.

What this means then is that the client caching algorithm uses the server focus lists to get rapid access to relevant objects, but internally it builds its own lists of object data to be requested, probably one list per type of data.

This is important, because in a system like SL where you can teleport into a brand new region, you can't afford to be loading irrelevancies. Indeed, the technical reason for the grey-out on teleport is because the key structural data of the target area is missing. (There is also a "gaming design" reason for grey-out, but that's a separate issue.) The client's sort of the download list by data type ensures that the technical barrier to immediate teleport is reduced to a minimum.

Caching is not borked. Log on. See how quickly everything appears? Now fly to a part of the world you've never been to. See how slowly everything appears?

SL still has to check the server to see if anything has updated since you cached it. That's why it looks like it's still loading things. It's not. It's just checking timestamps (or whatever).

Then go back to the place from which you just came. See how slowly everything appears again? It shouldn't.

There will almost certainly be zero sim deltas since a few seconds ago when you were last in that part of that region and therefore everything in your cache should be instantly displayable after the server has confirmed zero delta, namely just milliseconds. Now do you understand why caching is borked, bigtime? It may be storing the data, but it just isn't using it, or not properly. It's either bugged or just not designed right.

We only just arrived from there, so that location's cache delta list is empty, or nearly so --- there's nothing to search! It doesn't do a whole database scan you know. The sim object deltas are held in the order of most recently-updated first, assuming commonsense. By definition all that the server has to do is to return the last delta (12 bytes max) for the place to which you just returned, and that's enough for the client to start using all its cached data. It should be nearly instantaneous even with just a single unoptimized delta list per sim, whereas with some subdivision to reflect the limited range of visibility, it could be even quicker.

And that's without optimizations. I can think of a ton that can be done, and most of them certainly are done already because the LL devs are clever chappies. But it's just not working.

There's a range of problems in this area, and it's hard to know whether to ascribe them to bad design or just to bugs. Another very common one is loss of texture information as you back away from an object. The visible texture resolution decreases naturally with distance, but then when you approach the object again there should be no delay in redisplaying the texture at its highest resolution. After all, it was cached, in theory. Something about caching is badly wrong, possibly just the way it's used. The catchall term for this is borked.

I'd believe it; If i hadn't done some math 1TB would have been the number i would have given (if someone had asked). Consequently i didn't consider script memory. Which I assume is also stored on the Asset server.

Do you know the correct numbers?

So it's about 3GB per sim (since land mass is tied to user numbers)? hmm thats a bunch.

And it doesn't. Not for me at least. Unless they've changed something in 1.5.5, since I haven't been on to explore much in about a week.

I believe the cache is flushed every time you reinstall SL.

Case in point - I log in at my home land and everything reloads.

In any event, the bottleneck does not appear to be the caching algoritm, it appears to be bandwidth and server memory / power.

I'd like to see, as a solution, to allow prims to be marked as "draw first" that would give them preference in the rez queue. This way, I could mark the walls and roof of my shop, and when someone flies in, these would rez first, not the contents of the shop inside... it seems like a more natural transition.

Earlier this year, someone from LL mentioned they were at about 100GB on the asset server. I believe it was around the time that the huge southern expanse was first introduced (Abbots and the other California natural area-named sims, but before the sims named after moons, like Tethys, and way before the snow sims) - but before it was all settled. As of today, I think the southern expanse is about three and a half times larger, taking the snow sims into account. Then you have the huge eastern expanse (sims named after places in Hawaii and Japan etc.) These are all filling up pretty quick. That, plus the huge amount of new users, who are uploading textures and sounds and things, would make >=1TB pretty believable.

No, my point was that Morgaine's description of what was happening is something that -isn't- happening to me. i.e. Caching is working perfectly fine for me. If I leave an area and come back to it later, it loads quicker than areas I've never been to before.

What about bullets? Or vehicles? You'd have to stream all those seperately..

I suspect LL knows full well that greater caching is a part of the solution to scalability, but is it #1 on the list of priorities?

Tough call..

Yes, truly a lot of variables. I bet since caching is limited, we'll have to rely on a truly dynamic technology to deliver the ever-updating content.

You know, like streaming data.

*wry grin* It'd be a great asset if Second Life were based on streaming technology, or even better, if the CEO had experience with a major streaming company, like Real Networks. That'd really help. *snickers*

It's easy to dismiss people's comment with a snigger, Hiro. Much harder is to try to analyse the current extremely poor level of performance for anything like rapid interactive sport and to spot things that aren't working. We all know they aren't, even after you've been in an area for hours so there's nothing left to load. Only someone that hasn't experienced modern MMOGs could argue that SL is even close to optimum.

So stop being a fanboy and help us figure out the problems. If you respond to everything with excuses about streaming and key people's backgrounds in Real then you're just putting on the blinkers and appealing to authority. Help us. Only fanboys hinder on that basis.