Slow Texture loading in recent viewers (1.21 and later)

OK, I just went to extrovirtual, stood in front of the tiny vendor wall, cleared cache, relogged. As soon as the boxes appeared I focussed on the tiny tan hedgehog. It loaded first. I cleared cache and relogged again, focussed on the tiny black hedgehog. It loaded first. Texture priority is happening.

Whatever problems you're seeing, they're somewhere else.

Argent, with respect, I think you're reading too quickly, and possibly from a predisposed viewpoint. Otherwise, you'd have read:





I disagree. First, it takes a certain amount of time to focus on multiple textures... during which time other textures have already begun downloading. It's difficult to track more than one texture at once, and just complicates the random rez factor by multiplying the number of items that are possible to rez.



Argent, is there any evidence to indicate that there is a specific order of texture loading? Not in my experience. As a friend of mine used to pointedly ask people, "Why do you believe that?"

Your statement is taking it for granted that there is an established, precise order in which such textures rez. That's likely not the case, due to several variables: the precise angle in which you are standing, the precise field of view, the status of the sim at the moment of your rezzing, whether you're using ALT view, and much more. That means that every time you log on, the factors that influence texture rez order are going to be different. Therefore, the results will be different. Even slight differences can have major effect on what texture gets cued to load first-- not to mention the fact that there may be absolutely no order to texture rezzing to begin with. The system could conceivably be programmed to just take whatever textures it comes to first and load them... in totally random order.

Evidence: Whenever I have logged in and waited for gray items to rez, it certainly appears to be in total random order. This impression is echoed by hundreds of people who have commented on this issue.

Regarding your Extrovirtual test: if I may, you ran one test, in the same spot, in the same region, and report positive results. That has SOME significance, but it's by no means a population test in multiple environments.

In addition-again- you are testing under non-valid conditions: that of turning your bandwidth down to 50. No one is going to run their bandwidth at 50 just to get textures to prioritize. Run your bandwidth at the default of 500. Then see if you are finding prioritization under normal user conditions.

Again: supposed prioritization that does not function under normal user conditions-- is not coded to the needs of the system. It's like builiding a network search engine that works unless more than one person is using it. It doesn't meet the needs of the prescribed environment. In order for there to be valid prioritization in place, it has to work for the great majority of users under normal operative conditions-- otherwise, for all intents and purposes it doesn't exist. It's like a car that hasn't run for five years. Doesn't really suit the purpose, does it?

I said this before: I don't think it should come as too much of a surprise that texture prioritization doesn't work. You stated this yourself: apparently very little (if anything) in the texture system is working correctly right now. Even the "speed increase" of 8-11 seconds isn't "working"... it's just working less poorly. So even if there is a prioritization system... there's little or no evidence of such. If it only works at random times and only for a few people... then it's really as non-functional as the rest of the texture system.

All I know at this point is this: I have run multiple tests in differing forms in different environments, and I noticed no observable, consistent prioritization of textures. I have been trained Argent, to run unbiased and observational tests. In addition, I know my system and feed. I can assure you... it's nothing "local" to my system, nor external to Linden Lab. I have the best and the fastest and like I've told LL in the past-- it's not the customers-- it's THEIR system.

The way the interest list is ordered is known. It does not vary... it is in the order of the sim's local ID for the object. If you clear cache and relog your position in the sim will be the same each time. You can experimentally verify this. I have done so.
That's because the local sim ID for an object is based on the order objects were first rezzed or most recently relinked, which over time will appear random. But it's predictable.
It doesn't matter. It's an absolutely valid test for the point that I have been trying to make... which is that texture priority *is* implemented. I didn't say it was perfect, or that there's not other things breaking it, I am ONLY responding to someone's claim (was it yours? I don't recall at this point) that it didn't happen and had never happened.

If you want to argue about anything else, be my guest, but do it with someone else.

Argent, I'm sorry you consider this "arguing". I considered it discussing a valid issue. If that's how you feel then you're correct... I'm discussing this with the wrong person.

Since I'm examining solely the data without emotional bias... you'll understand I'll have to stick with my findings. Summarized: evidence indicates there appears to be no functionally CONSISTENT or even common prioritization of textures under standard user conditions.

I respect your right to disagree, but considering the extensive nature of the tests run and the resultant data-- I stick by my findings. You either respect and recognize my findings... or you don't. Totally your choice. I'm sorry that you feel your limited tests are valid and mine aren't... but nothing I can really do about that. I ran extensive tests; I reported my findings. Whether you personally agree or disagree is your priviledge. I certainly have no need or desire to "argue" this or any other matter with you.

You're missing the point.

I am not attempting to dispute your findings.

I am responding to a claim made earlier in this thread that texture priority was not implemented and had not been implemented.

This isn't about "how I feel", or about me taking you seriously, or not taking you seriously. I'm responding to a specific statement of fact that I believe to be erroneous.

Fair enough. At this time, I can't personally say one way or another whether there is ANY texture prioritization or not. It is possible that some kind of prioritization code is in place that under certain circumstances, kicks into place (as I stated prior... there IS some significance to your findings).

So some people could indeed be experiencing a degree of texture prioritization at times. It's even conceivably possible that for some people it works ALL the time (unlikely but possible... just as some people claim to "never" lag). I have no idea why such things would be the case, but it can't be discounted as a possibility. I know LL techs never seem to lag or experience texture delay, sooo... LOL

All I know is that in my tests, and in comments I've heard from many, many people, texture prioritization either doesn't exist, it's totally broken, or it simply isn't up to the environment. My belief is most likely it's broken or not up to the task, because LL has said it IS encoded. But we know the reliablity level of their code... so it's not too far a stretch to consider such malfunctioning.

So in your basic statement that you're disagreeing with statements that there is "no prioritization" of textures... I'll agree (even if I've unwittingly implied that myself in past messages). I think your statement is a fair one-- supported by your own test at Extrovirtual. I would still find however, that under normal user conditions, such findings may prove out different.

So I'd say at this point... we have no disagreement in your basic premise as stated above. I don't think anyone can say for a certainty there is NO texture prioritization or that it isn't experience by anyone from time to time. Since we have no data on a wide population experience from reliably observant users.. we can't even know to what degree it's performing or not performing. That would be interesting to find out-- but I doubt you or I either one have the time or would be willing to perform such an extensive test... especially since that's probably what paid LL analysts should be doing at this point.

You can reproduce my experiment.
It's more likely that there is *always* texture prioritization going on, but other problems are masking the effect. It's just too unlikely that the texture prioritizaton code would get turned off sometimes. That would take a deliberate decision to disable working code.

There was a massive reduction in performance when they made the last set of security fixes, and that's also when texture prioritization became unreliable. I think that figuring out what they broke back then is more likely to be effective than trying to analyze the problem by assuming there's a problem in prioritization... because we DO know that there are problems elsewhere that are big enough to screw ANYTHING up.

LOL, thinking on it, you may be exactly right.

In truth, could be either way... that everything else is messing up prioritization, or that prioritization is itself borked (or both). I guess we'll find that out once (if) they get everything else fixed. But after 5 years... and seeing the very limited progress of the last 6 months, I don't think it wise to hold our breath.

I have tried 1.22 RC6.
The experience is exactly as i had prognosed. Just a iniwinilittle better, but hardly enough to play Second Life with that ! Geeez
I totally stand on my earlier post regarding this issue.

Stop diddling and fiddling.

1.20 is very ok, make that the mandatory viewer again, and call that 1.21B
From there... redevelop to 1.22.
If you have to roll back to earlier server code because of that, bad luck. You can also do that quickly we have seen.

Don't say: "but we are getting there...", i know that kind of talk from developers.

Instead of playing around 6 months and cripple everything, restart the iteration process from the 1.20 state.

I have now sold all my land in sl btw finally; am i going to reinvest? NOPE !
The declinement of inworld spendings is too much correlated with this issue.

You wouldn't want me as a CEO ! (lol).

here is a question... i'm having the same issues, what is everyone running off? i'm running on Vista

I run XP sp2. I detest Vista (and yes, I have it on a laptop, unfortunately, so I can back that up.)

Part of the problem as I understand it is that when you start loading textures for an area, you only get part of them in the initial burst of data. The Server simply doesn't send all the data on the textures requested, or malforms it so it's not all utilized by the viewer. And the viewer then has timeout/retrys for every individual texture, which it one by one requests from the server and asset again. So textures appear garbled for a minute, then sharpen as the retry list in the viewer fetchs them more slowly again.

The facts are clear on the textures problem.

Before the move to the new network and the recent upgrade. Textures worked well. You laoded them once and did not ahve to load them again unless your cache got full or you cleared it.

Now you have to load textures each time you go into a sim. Tp out and then back into the the exact same spot 10 minute later and they have to laod again.

Before the upgrades i had 30 fps in my main sim. Now i ahve 13 fps.

This morning we did extensive testing. We basically shut down the sim, turned off scripts, etc. FPS never improved one bit. This was tested with 10 avatars tping in and out.

Go to another sim and come back, textures reloaded everytime.

Further we discovered tonight that script performnace has improved and etc. But that does use no good since the textures are lagging everyone to death.

The problems with texture load issues are identifyable and established.

Summarized:

1) Linden Lab is using a poor-choice texture protocol (namely UDP instead of HTTP). Reportedly, their current system instead of transferring textures in packets, transfers them in several "rez sets". If anything goes wrong during that transfer, instead of simply resending a short packet, the UPD system pauses for 10 seconds, then attempts to resend the entire texture again. That's why texture loads almost always take increments of 10, 20, 30, 40 or more seconds. It's an extremely cumbersome and inefficient way to transfer textures.

2) There is a serious cache deficiency as well as bad algorythms which fail to use cache properly and load textures by user proximity.

3) There is a denial-of-service attack caused by Linden Lab's own software, on its own servers, that has resulted in textures being redundantly loaded.

The most common way to experience this repetitive texture reload is to wear a new skin or piece of clothing. How many times do you see that texture become crisp, then fuzz out, then become crisp again, then fuzz out... That is (apparently) one example of the same texture loading over and over again.

Detailed information is available at the following web site:



<a href="http://elfclan.ning.com/profiles/blogs/major-linden-lab-coding-snafu">LINDEN LAB CODING SNAFU</A>

As I understand it, the sim has a copy of all textures on it, so the viewer polls the sim for textures now. If the texture isn't on the sim, then the sim gets it from asset and passes it to the viewer.

The viewer then is supposed to cache all these textures and keep local copies to reduce traffic.

What I have seen is that the cache on the viewer may as well not exist, because not once have I seen a texture pull from it. Even if it's a place I was at only 5 minutes ago, the viewer takes its sweet time loading from the sim again. I sure hope the sim's caching of textures works better than the viewer.

Seems to me that fixing this texture cache problem should be job #1 as it kills everything - user experience, viewer bandwidth, sim to asset bandwidth. I'm a bit surprised since a b-tree of UUIDs in ram that link to filenames on the drive is child's play to implement. Just write that b-tree to the drive every 5 minutes or so to ensure you still have them in a crash or a quit. Simple, fast, and effective. Protecting the contents of the file can be done by simply XORing the first 16 bytes of the file with a known pattern. No, it isn't crypto. It doesn't have to be - just a fast scheme that prevents most users from getting at the contents easily.

I totally agree Shockwave. But I've been harping on this issue with Linden Lab literally for YEARS. When Balpien Hammerer and I discovered the hard-core truth about texture loading last year and brought it to Dan Linden's attention, the resulting song and dance resulted in two things: me (and several others) being banned from JIRA for doing no more than stating facts about this matter (the archival history is here for anyone to read).

Since this problem is creating a Denial-of-Service attack on their own servers and since this is a relatively simple coding concept (changing UDP to HTTP and getting the cache to work right)... and since this problem has gone on literally for years (and increased significantly around mid 200, I have to believe at this point there is something that just isn't right with Linden Management. I have no idea what's going on behind the scenes, but I know that no amount of user begging, pleading or confrontation has budged Linden Lab an inch in this matter.

It was just announced this month that someone at the Open Sim project just increased texture performance by 200%. This tells me one thing: if OpenSim could do it... so could Linden Lab. So I have to wonder why, after all these years, they haven't.

Poor texture handling is undeniably the #1, topmost show-stopper critical error currently on Second Life. It drains the servers and asset servers performance-wise, it damages user experience, costs merchants thousands and thousands of dollars in potential sales (people get tired of trying to get vendor textures to rez and just leave). So when a problem is this bad and it doesn't get fixed... users eventually get to the point that they just start writing Linden Lab off and wait for the eventual crash and burn.

Now while Linden Lab may not like to hear such statements, these are nothing more than realistic observations. The fact is, Linden Lab has pretty much ignored this texture issue for years. So I seriously don't expect them to get any more concerned about it now. At this point it becomes transparently obvious there's some reason they haven't fixed this major, major issue... and they're not letting us in on what it is.

Any programmer that has been around sl for a while know the root of all the problems. Textures, lag, scripts, sculpties etc.

You take a server and viewer program, combine them and you get twice the problems.

You take those two and place bandaid patches on them, then heep on added features, then more bandaids, then more features, then more graphics, then more bandaids, then more. At some point the bandaids start to dry out and peal. Your end up with a house of cards to some extent. One peice fails and a whole section fails.

Not to mention your own people are frustrated by the fact they can't take advantage of many new advances because your software will not support it or the bandaid did not work right.

Till Linden Labs takes a core group of programmers and recodes both the servers and viewer from scratch, we can only expect more such problems to crop up. This is a 6 or 8 year old program, still on version 1.xx.

It is time to stop adding to that program and recode to version 2.01. meaning start from scratch, take what you have learned, take from the new technologies what you want and/or need and build a new program.

So many of LL's problems are simply solved if they used newer technology and systems that were not available when the program was first written.

I personally have not seen one recent upgrade that has not given and taken. Given something new and taken away something we had.

Textures are the big crash now. We have tested extensively and script performance is up, actually really up this time. But does not matter, fps as measured against outside constants to ensure it is only sl changes show, fps are down to 1/3 what they were before the most recent network change and the latest server version. Viewers have had little effect on it while there was a drastic change when the new network came in.

Not sure who the techs think they are folling when they say nothing changed. Clearly the way textures load has changed. Clearly something change to force textures to reload everytime you come into a sim. Shut down your scripts, physics, colliosion and then watch. Nothing really changes in fps. Go to the edge of your sim and face in to the center, wait for everything to load. Now turn and face out to the linden ocean and wait. Watch FPS shoot up. Now go back to facing the center and watch them dive again and watch some textures reload even though you have not left the sim.

Since we ping response times in several servers accross the US to make sure internet responce times are constant. We also have graphics tests on three of them. One happens to be right outside SF.

Fact is everything is almost the same, constant, the only changes are when sl is loaded. Performance there is almost exactly 1/3 of what it was.

Now unless SL has reduced bandwitdh on thier new network or throttled texture bandwidth, or reduced graphic resourcez on the sim servers, etc, then the problem clearly has to be the server and to a lessor degree the viewer software.

You take out the scripts, collissions, physics, and your left with the graphics and the textures.

And forget the testing. We have long time users, dozens of them coming to the sims, and now crashing out when these people never used to ahve any problems. I have had user after user tell me their fps is down by over half in every sim they visit. Yet their performance outside sl is the same as it has always been.

LIndens love to point to the client and say it is your computer's problem, your ips, your city. Fact is, when it si thousands and thousands of users experiencing the same drop in fps, it is not us, it us within sl.

Time to look in house for the causes and fix them. First step is to recode the core programs and rebuild a server version that takes in all the advances out there.

samatha

You know.. with all the blog posts by the Lindens showing us charts and graphs on how booming SL is and profits are up across the board, there should be no trouble using some of that increased revenue to upgrade and improve our SL experience! That IS what they always say is their top priority ..... right?

What a fantastic answer. Even I understood what you were saying and I am very impressed by your deductions. I don't come to the forums every day, but, since I am here I will put in my two cents that I have also noticed how slow the textures are in displaying. It has really affected my ability to shop because I have to stand there forever and wait on the picture to appear. No matter how many times I try to focus, the problem is still there. Sometimes, I can get one item to display by just clicking on the gray box and chosing 'buy' then when I cancel that request, I can see the item clearly. But, the rest of the items in the room are still gray.

BT

There are several things that need to be done in parallel.

First and foremost, I'd fix the broken caching system. If it doesn't work, then the code isn't done yet. Period.

Second, I would split the viewer into two seperate threads - one for IO, another for graphic generation. All you need is a shared memory and a circular buffer of commands going to the graphic thread to accomplish this. This is almost paramount as nearly all new computers will have at minimum two cores in a year or two. If they don't do this within a year, their nascent competitors surely will.

Third would be to move to a Mesh system for body, clothing, and as an alternative to the sculpty for objects. If they can do it in RealXtend, SL can too. There will just be two types of skin and clothing - original and Mesh.

Fourth needs to be a different model of the sim. There need to be two types of prim - static and dynamic. Dynamic is a prim that is either physical, or has a script running in it. It's silly that the physics engine has to look at every prim in a place, even if they are motionless homes that never budge. This simple fix would allow more prims to be available and improve simulator performance. (Any prims in EDIT are temporarily Dynamic).

While we are on the topic of scripts, Fifth would be a Script leveling system that only limits processor time to resource-hog scripts while leaving others to operate normally. This too would prevent sim lag, and keep one user on a sim from unfairly depressing performance for other people elsewhere.

Sixth would be a straightforward 2dimensional indexed array ability in scripting. So much scripting time is burnt up just getting around this limitation in LSL - put it in, and watch the script execution times plummet as people don't have to stride Lists anymore.

Seventh would be allowing mega prims up to 100x100x100 to be made by people who are either verified or Premium. Every argument against them can easily be made for 10x10x10 as well, so let's give them to people who can use them and have no interest in getting banned for misusing them.

Shockwave, there's an interesting thread on the SLdev list right now about revamping the cache system and you might be interested in copying this and posting it there as well since at least at first look separating static and dynamic objects looks like a great idea.

Havok already separates static and dynamic objects in the physical simulation.

but I think the definitions of static and dynamic used in Havok might be a bit different from what would be good for dealing with updating textures and such

I don't think there's any need to worry about the details of objects once they get the frikken texture cache fixed.

I'm not an expert by any means on the technical details of the texture cache but the point I seem to be getting from the sldev conversation si that it can't just be fixed it has to be entirely revamped which is what those ideas are about.

Caches are EASY, if you're not trying to use the complexity of the cache design as some kind of piss-poor excuse for DRM. They could write a better texture cache in a day, if they wanted to. They could gank the cache code from Squid, which they're already using, and have done with it. It's NOT rocket science.

There's no reason to try and come up with some kind of complex static-versus-dynamic cache scheme. Just make a three level hashed directory tree with a background LRU cleaner process, and if you can't trust the OS last-access-time just touch the files once a session and use last-modify-time. This is what they're already using internally on the sims and what 99% of the HTTP proxy caches in the world do. It's easy, simple, reliable, and efficient.