cache the damn world

I think just that fact that the designers of SL were able to make SL, post dot com crash, located in the most expensive cities to live in in the nation, is just amazing. A viable dot com, and a successful game. Any short comings in the protocol can be ignored in light. And Real has been for years fighting to keep a hold of it's empire by suing the pants of others for patent infringement. We should be proud that LL isn't in bed with Real.

LL has been wise not to get involved with the streaming companies. They are just barely better then the record companies. With the exception of Nullsoft (before they were bought out by AOL) Last I heard Justin was out of work.

Speaking of caching and loading of stuff, I wonder why stuff directly in front of you -- in your line of vision -- doesn't seem to rez more quickly? It seems to take awhile to get refined from pixellated bitmaps. I wonder if there is a way to automatically priority load, like when you're in a shop with lots of items pictured on the walls, so that they get refined first and then whatever else follows, follows.

I'm told that scripted and physical objects load faster. I'm not sure what determines this, but if its the same criteria as sensors, you can drop an empty script on a prim and it will be marked as scripted. Do this for all of your wall prims and you will in theory get what you want.

G'damnit, SPEAK FOR YOURSELF.

As I've said several times in this thread, CACHING IS WORKING FINE FOR ME. The symptoms you describe DO NOT EXIST FOR ME. Don't say "we all know" when WE DON'T.

Yes, SL could use some serious optimization in some areas, and caching may need help, but it WORKS right now. Don't claim it doesn't.

Moleculor, I've just tested it so that we can be constructive and analytical about this.

I picked two spots at telehubs in reasonably well built up areas so that the effect will be pronounced. They happened to be Montara telehub and Daikoku telehub, for no reason at all other than that they appear to be in areas of roughly the same prim density but without any significant number of avatars present at this time (in fact, most of the time none present at all); also, this was done with no music streams playing (I first checked that the telehubs had no local music, and then I turned music off altogether in my client, to eliminate variables). I specifically avoided a place near to my region, just in case there is some special optimization going on near one's home.

I then went to one telehub and remained stationary for 15 minutes for everything to load. It didn't need that long, obviously, but just in case. In any event, the bandwidth usage had died down to just the quiescent background trickle. Then I went to the second telehub and did exactly the same. Finally, I teleported repeatedly between these two telehubs. After the first two ports, if the caching were working properly then I should have seen almost no download activity, since very few objects if any would have altered since I was last there. The object delta lists of each sim would have been empty for my range of visibility.

What did I observe? On every single teleport where my cache should have been near perfectly up-to-date and no (significant) downloads required, my bandwidth usage shot right up to near the peak of my 300kbps setting, and remained there for up to half a minute. (For the record, I'm on a 512kbps ADSL on a business-grade 20:1 ISP, and this is after business hours, aka "a better than average home link", but this test is not conditional on good links at all anyway since it should have virtually nothing to download starting from visit #3). One very important observation: the local geometry and textures at the telehubs did stay in the cache and were reused on teleport, because they appeared instantly despite the huge network traffic. Btw, I did not move the 3D camera away from the avatar, as this could have polluted the cache. So what's all the extra traffic? (Not everything in visible range would appear instantly though --- eg. the Shop Centerpoint building 40 yards from Daikoku hub would take up to 15 seconds, and The SOMA in Montara which is about 80 yards away would take up to a minute.)

I conclude that the cache is either not caching all the downloads from the initial visit to each telehub, or else that it is caching everything but that some of the cached data is not being used, or else it is being used exceedingly slowly for some reason. Or both, or all three.

The key point here is that if there were zero updates to make since my preceding visit, then something like a trivial 12-byte transfer would be sufficient to let the client know that it was still up to date. The traffic that I am seeing is dozens of thousands of times greater than 12 bytes. Some of this is undoubtedly due to small local updates in between my visits every few minutes, but surely not a lot.

If the really large bouts of traffic that I am seeing do not represent cache refills, then what exactly are they? In the absence of alternative suggestions, the caching looks suspect (ie. it's only partial), because caching should by definition eliminate the downloads for all objects that haven't changed, and what else is there to chatter about so intensively on the link?

OMG. I thought of one possible explanation. Each sim may not be maintaining an object delta list at all but merely recording the time of last modification of each object in the database, and then might be trying to chat to the client about every damn object within visibility range. The huge traffic might not be downloads at all but time queries.

OMG, no way. That would be the ultimate in non-scalable cache implementation design borks.

Thanks Eggmeister, goo goo ka choo

When you teleport into a new area, you get a full update ("red smoke" of every object there, even if it is cached. Turn off the far plane and fly away from an area for a little bit and then fly back. You will see objects disappear and then re-rez.

i agree that things could be cached better, maybe even if only a ground map (yes i know people can terraform, but in most sims its limited enough that changes are relatively minimal and it doesn't happen that often). it would give the client one less thing to worry about, have a sim wide modify date and put updating the terrain at the bottom of the priority list. it would be nice if things loaded in such a way to make flying from a telehub to wherever smoother, even more important now that there are such large areas without telehubs.
also a larger cache size wouldn't hurt, 100 megs isn't all that much. i wouldn't mind giving SL a gig or more.

Ah, max cache size is 1000 megs (a gig). But I have heard from those who have experimented that if you set a cache too large, it may actually slow you down as it scans your HD for the necessary information. So there's some sort of balance.

If things load faster in subsequent visits, caching is working. That's what caching IS. The fact that your client is communicating with the grid does not cancel out the fact that things ARE loading faster. If you want to make this a thread about the possibility of unnecessary network traffic to/from SL upon loading an area you've been to before, go right ahead, but don't leap the the quite probably erroneous conclusion that it MUST be caching because some newbie came in here ignorant of the fact that SL has a cache.



*sigh*

100 megs ain't even a choice for cache size. It's 50MB, 200MB, half a gig, and a gig.

Newbies. *sigh*

Not a "dismissal", Morgaine, but an acknowledgement. blaze mentioned two scripted things that needed streaming, and I made a comment to show that Linden Lab's approach made that a central issue. I don't see how this could be classified as a dismissal at all.

I "sniggered" because blaze also has a reputation for being extremely negative towards Linden Lab in his posts. While his concerns are legit, he could also read the white papers and found out the same thing.

That's twice you've been overly harsh on me, Morgaine. Is calling me names is being productive? Just because I don't choose to aid your speculation about the algorithms used for caching doesn't mean that I am ignoring or dismissing them. It means that I will let you come up with your own theories, and the pieces of information that I can add that aren't being brought up in the discussion, I will.

Furthermore, I find it absolutely unbelievable that Second Life can be dubbed "extremely poor" in quality of performance. I honestly can not believe you have the gall to say this. Consider:
1. No other MMO has such a robust object creation, scripting, and physics that is all dynamic and rapidly updating.
2. No other MMO provides entirely user-made content. Others have professionally made stuff that users are allowed to combine like Lego blocks (TSO, crafted items in RPGs, etc). The most we have is Linden trees.
3. Most people seem to be having fun.
4. Consider the variety of problems Linden Lab must face. Everything from denial of service attacks on a mass scale to individual complaints, legal issues, player suggestions, bugs (which EVERY MMO has), growth, scalability... let alone new features they want to implement. I think questioning which priority is "#1" is kind of naieve, frankly. We can look at problems purely as engineers but that is a tiny fraction of the big picture of developing software, let alone dynamic software that promises to let users do just about anything.

This you guise in trying to improve things. This is fine as long as your discussion remains in technical issues, and does not try and extrapolate mandates for how the company behind the software should act. Oh, and then there's the huge fact that nowhere have I seen the detailed algorithms and source code posted on how Linden Lab does caching. These discussions are useful, true, but logical guesses and interesting suggestions, at best.

And then to accuse people of sabotaging your interpretations and "not helping solve problems" of these mandates because I'm a "fanboy" and not harshly cynical as your criticism here comes accross? Come on, now.

Hehe, thanks for your post Hiro. It was accurate.

Righto, I missed the nuances there, Hiro, and I wasn't aware of any such past history. I grant you that, and about the harshness. I'm not the world's best diplomat, but I do know my stuff technically so I'm trying to put that to use here. I totally love a line of argument being pointed out as flawed by logic or by fact (even when it was my own argument), but I just can't stand the process of reasoning itself being derailed. Sometimes making social comments like sniggering can derail it, so I reacted badly to that. Thanks for pointing out that it's not what you had in mind, I misinterpreted it, and I'm always happy to acknowledge mistakes and say I'm sorry. Which I am --- sorry!

Oh but that's totally true, Hiro!!!

The performance of SL even with a modern-day CPU, the latest GPU and good quality broadband is much much worse than say EverQuest or Anarchy Online using an obsolete Pentium 3 with a 2nd generation graphics card and a narrowband link. The difference in performance levels is immense. This is plain fact. I am a very great fan of SL (this is where I live by choice after all, it's the best world around) and I praise it most highly. That does not make me blind to faults however. ("Praise without criticism" is pretty much a definition of fanboy, and I'm not.) I report things how they are, and I'm critical of specific problems so that these things get addressed, and in most cases I'm trying to point out solutions too. You can't provide solutions if you don't highlight the problems.

None of those things contradict the observation that current oerformance is extremely low compared to other online worlds. All you are doing is pointing out that it is harder to achieve in a dynamic world. Well that's beyond doubt!

But SL's dynamic world is just a largely static world where most things CAN change all the time BUT DO NOT, and you can make use of that relative invariance. Once everything is loaded, there is no strong reason why a dynamic system cannot be exactly as efficient as one with static objects. It just takes good design.

And that, btw, is one big reason why I have no time for comments that others have made along the lines of "it can't be optimized because it's all dynamic". That's just nonsense, it simply illustrates a lack of understanding of the technology.

Yes, all those things are true, but there are always solutions to such issues when there is a will to solve them. Lack of resource can be combatted by harnessing the power of the community --- there is no shortage of open-source developers both on the net and right here in SL, and as Philip Linden pointed out, SL already uses open-source code anyway. And design insights are not limited to any particular category of people like paid Linden staff alone, so SL can certainly be improved by implementing more of the suggestions that people make here.

We are now beyond that point in the analysis, Moleculor. It's not a simple yes/no situation.

Some objects are definitely being cached and used from the cache, we know that for certain owing to the immediate appearance of hub objects and textures after teleport #3.

Not all objects are appearing immediately though, despite having been fully loaded (we assume) on the earlier very long cache prefill visit. We do not know whether this means that they are not cached but downloaded afresh, or whether it means that they are cached but not reused immediately.

If one assumes that the undisplayed data is indeed in the cache but unused, then it seems likely that the observed high traffic for 15-30 seconds after subsequent teleports might mean that high-volume chatter between server and client is needed before cached objects can be used --- ie. use of cached data is being delayed. And, I have at least one hypothesis for why this might be so, which needs testing.

That's the current state of analysis, I'm still trying to figure out ways of getting more data. Constructive issues of fact and insights into possible causes are always welcome. Feel free to help us make our world better.

Hey, Morgaine, it's alright, no harm done to me.

As for the performance comparison, there just is none. The number of objects in EQ is far lower. The textures can be professionally compressed, reused, local lighting can be worked into alpha layers in textures... a WHOLE bunch of tweaking can be done.
And Everquest zones are static. The only thing that changes are mobs, and those are finite with specific rules.
And after playing 3 years of it, I can say for sure that even despite these great limitations, it STILL was very buggy - weird spawns, broken quests, etc etc.
EQ was static. SL is constantly changing. AVs in EQ were static. Ours are fully customizable. There's no control over user content, whereas any other game, ANY other game out there will have the professional graphics gurus tweak the heck out of it to make it run super efficiently.

For more info on this, I recommend you read SL's white papers, listed on the website - specifically the one on why SL needs streaming data.
Hence, SL is pretty damn impressive.

Well it's always possible that the SL client code doesn't do things correctly, we can't tell. In principle though, cache size does not have that effect. There is never a scan involved.

Objects in caches are organized in a variety of ways, and the access method is not always the same thing as the storage method --- in fact, there are frequently multiple access methods into any given storage base. Think of the access method as an accelerator only.

The two most important access methods for any cache are those concerned with access to stored data by a key of some kind, and the LRU algorithm for flushing out dead wood rapidly when the cache is full. Keyed access is a well-honed technology, and the vast majority of modern storage systems (not just caches) can do it in O(1) time, which means regardless of the size of storage as long as no underlying transitions occur, such as unexpected overflow into tertiary storage. There is often a sizeable fixed latency which doesn't contradict O(1), eg. sometimes a hashing algorithm is poorly chosen, but a good system will generally encode helper information in a key to reduce that.

Some systems will compromise slightly and provide access in log time instead of constant time, which can save on storage overheads, but generally that's no biggie --- the log time grows very slowly both by definition and in practice. For the small cache sizes relevant to SL though (just a few million objects and not too many gig of storage say), linear access tables in memory are absolutely no problem so O(1) is a piece of cake. With SL as it is today, the cache should be able to grow by at least at order of magnitude without slowdown, and probably two orders.

That said, let's go back a step. We don't actually know that cache size is in any way an issue at present --- we would have to do teleport tests similar to mine but over much wider territory to determine that, eg. prefill the cache at say 25 different hubs first, and then go back to the first one and see if the data is still cached.

All those things are true, but you're not addressing the small but key point I made earlier about precisely that:

To add to this, the only scenario under which no speed improvement is possible at all is one in which all objects and all agents and the environment change all their properties during every video frame. That is not the situation in SL, it's not that dynamic. Everything is static and can have the living daylights optimized out of it once it is downloaded; if something changes you just flush it out, start again, and redo your optimizations.

This applies to EVERYTHING, Hiro. Technically it's not actually hard either, so to answer each of your points:

The number of objects is not a limitation here --- see my reply to Torley about cache and access table sizes. On the fly, once an object is loaded and cached, its textures can be professionally compressed just like EQ static ones (there's no secret about professional-quality algorithms), and reused, and local lighting can be worked into alpha layers in textures in SL too. Nobody said that the client has to display the texture you put on an object directly after all, so factoring in the lighting to the texture before you send it to the card is very sensible. (And the Cg which LL use can handle that directly anyway --- I love these new GPUs.)

A whole bunch of tweaking can be done, as you phrased it, because once an object is loaded it is effectively static until it changes. I can't think of any major static tweaks that cannot be applied to dynamic data once it is loaded. It's all just data, regardless of whether the artist has just dumped it on your table to optimize or whether it has been streamed down from a sim. The only thing that's slightly different in SL compared to a fully static world is that we need to periodically throw out our earlier optimizations, reload and start again, that's all. It can be done. Mutable objects are no barrier.

Hiro, I write papers like that. You know that appealing to authority doesn't advance a logical argument. The LL people are admittedly great, but they're just engineers like me. And like you yourself too if you are one as well. Everyone can have relevant insights if they have the technical knowledge to underpin them, not only those who are paid by LL.

And for a developer, it makes no difference whether data is streaming to the client from a sim or streaming onto your desk from the graphics department for optimization into a static world. The same work needs to be done, whether it's done interactively or not.

Damn right! And it can be made fast too, and even more impressive.

Can anyone help me to think up a test to separate out the effect of SCRIPTED object updates from all others in those tests that I've been doing, namely jumping back and forth between two telehubs after having prefilled my cache for 15 mins at each one?

I don't have the prims nor land area to be able to set up my own test environment for this, but maybe there is a large highly primmed site somewhere that is totally script-less on principle? Yeah, I know it seems unlikely, but worth asking I guess.

The problem is that there has been some indication that *ALL* scripted objects within the range of visibility get updated to the client pretty much continually, so in effect the cache helps very little or not at all for them. If this is indeed so then that's the source of 99% of SL's perceived lag at the client end in the absence of server or network congestion.

The current interpretation is that a mere 60 seconds between telehub switches is long enough to accumulate 15 seconds worth of changes in scripted objects at the last telehub, and hence is responsible for making the client thousands of times slower at using its cached information than it should be.

(It should take just a few milliseconds to obtain the the object deltas from the sim of the region reentered after such a short absence because the delta list should be empty, assuming no agents have been around and that there are no wandering mobiles.)

In case anyone's wondering why this is important, it's because if the region of object delta capture of a player is larger than her range of visibility then you achieve speculative idle-time cache preloading under most conditions of limited local movement, and the effect of that is instantaneous use of all data from cache with near-zero updates across the network. In contrast, if all scripted objects get reloaded continually then in effect the client never enters fully-updated+idle state at all in most places, so speculative preloads need to fight for bandwidth. The result is a perceptual sluggishness orders of magnitude worse than necessary.

Interesting comments from Philip on the issues contributing to lag:
As he says, there is no network capacity bottleneck as such, in the sense that neither the Internet backbone provider's bandwidth nor the local broadband provider's bandwidth are being saturated.

When he says that the servers are struggling to get data ready to stream, I wonder if he means "struggling on their own" (ie. a CPU and asset cache problem), or "struggling in their dialogue with the client"?

Since he mentions low FPS in the viewer in this context, he must be referring to the single-thread issue, but I wonder if this was intended to suggest that the struggles of the server are related to this bottleneck, in that the client is not able to hold the cache-time-query dialogues with the server fast enough because it is busy rendering?

I remember reading somewhere a linden saying how the client-side network code and the rendering code were in the same thread. But as to when or who said it i just don't remember.

Aye Strife, Phoenix Linden said it here.

I'm always hungry for even the tiniest scraps of hope, so when I read ...I try to focus on the word "current". Tomorrow is another day.