ATI rendering issues

Hi, all -

I understand that there's quite a bit of frustration over our ATI related rendering issues, and we haven't recently posted any information about the prognosis and cause of these bugs. I'm going to explain a bit below about how our rendering engine works (as compared to most other OpenGL rendering engines), and update the status of our top few ATI bugs.

Since this is going to be a somewhat lengthy and technical post, I'll just state a few things up front.

We are fully aware of all of the bugs, and it is a very high priority that these bugs get fixed. We are currently in contact with ATI about these issues, and have been for quite a while - they are aware of them, and we have been working together to resolve them, either through fixes to our code or to their drivers. Finally, while we have done a marketing deal with nVidia in the past, all of the major issues that we have been having with ATI graphics cards appear to be the result of issues with our core rendering engine, and not necessarily with vendor-specific optimizations.


Now, for an explanation of how the Second Life rendering engine is different from others (and how that exposes lots of bugs) -

Right now, almost all of the other 3D engines on the market (Quake, Unreal, etc.) are designed to do one thing particularly well - render a single level at a time, with a fixed geometry and texture set, and some moving characters and special effects. This means that most of these rendering engines use techniques such as BSP trees and pre-baked textures for rendering. This means that when you load a level, it loads all of the textures and geometry for that level, and then very little of that data changes until the next level, where all of that data gets swapped out, and the data for the new level gets swapped in.

The Second Life rendering engine has a much trickier problem to deal with. A lot of this comes from the unique nature of Second Life - the fact that all data is streaming, and the fact that we allow users to create and modify texture and geometry data at any time.

One issue that we have to deal with is that at any time textures may be streaming down from the server. This means that we use the progressively streaming texture to generate a lower resolution texture first, and then can increase the resolution of the texture as we get more data.

A second issue that we have is that our texture set is essentially infinite. At any single location in the world, we have FAR more texture data available for us to use than can fit comfortably in the world. Most FPS games deal with these issues by very carefully limiting amount of textures in the level design. We, however, have to deal with this by dynamically reallocating our texture memory based on our best guess as to which textures are more important for us to have at a high resolution. This means that we are constantly swapping textures in and out of texture memory while you fly around. Geometry data is handled in a similar way - as you move around the world, we increase the level of detail of some geometry, and decrease it for others.

Basically, this means the driver load for Second Life is substantially different that that for any OpenGL game. Most games use dozens of textures, which don't change during a level. Second Life uses thousands of textures, which are constantly changing resolution, and being uploaded and downloaded. We also use a much larger amount of dynamic geometry than any other game, since all of our geometry can be changed at any time, and we're unable to pre-build it since it's streamed from the server.


Fortunately for us, nVidia's OpenGL driver has worked pretty much flawlessly with our unusual driver load. Unfortunately, ATI's hasn't - we've exposed a few issues with it. These do not appear to be bugs that are due to vendor specific code - they all appear to be occurring in areas where we are either using standard features of the OpenGL API or non-vendor specific (ARB) extensions.

Here's the outline of our top few ATI specific bugs, and an explanation of what's going on with them. We are currently working with ATI in attempting to resolve these issues - as someone has mentioned before on a thread on this forum, you will occasionally see them in world doing testing. Please be friendly and supportive! As these bugs get fixed, or we find out more about them (such as ETA's for driver releases that will fix them, or ways to avoid them), we will let you know.


Black screen when FSAA is on
----------------------------
Symptom:
When full screen antialiasing is on, almost immediately after the login sequence finishes, the screen turns black. Afterwards, the screen flashes occasionally with the rendered world, but otherwise is black.

Explanation:
This appears to be a bug in how the GL driver implements selection. GL has a special mode which can be used for selection - picking which object is immediately under the mouse. Unfortunately, when we use that mode when the driver is running in FSAA mode, it appears to break some state in the GL driver, possibly getting it confused over which buffer is supposed to be swapped to the front buffer.


Texture corruption
------------------
There are two forms of this bug. On Radeon 8500-based cards you will see corruption on objects at medium distance, that disappear when you move close to them (low res mip-level corruption). On Radeon 9700-based cards, you will see texture corruption sometimes as you zoom in very close to objects with high-resolution textures (high res mip-level corruption). It will most likely occur when you are moving in an area in the world near a large number of high-resolution textures.

This appears to be a result of the interaction between our texture management system (as described above), and the ATI driver's texture management system when there are lots of texture transfers happening at once. We also get occasional crash reports that ATI has stated are likely to be in their texture management system, these may be related.

You can "fix" these problems temporarily without logging out by switching between full screen and windowed mode - either to or from is fine - this will cause us to destroy the GL context and all resources associated with it (including textures), and rebuild it again.


Avatar turning red when bump mapping is turned on
-------------------------------------------------
This appears to be an issue with the driver incorrectly performing our vertex program on the second pass used for bump-mapping. An ATI engineer has confirmed that the problem goes away when the vertex program is being rendered in software emulation mode, instead of running the program on the card itself.


Reflection map only on west side
--------------------------------
On Radeon 9700 cards, the sky is only reflected on the west face of the cube map on the water and on shiny objects. We have not confirmed definitively at this point whether this is due to a driver issue or due to a bug in our engine.

I haven't seen a detailed post like that about nagging problems in my entire time doing external software testing. Good news for the ATI owners.

Now, let me know when you can reply like that to my post below.

Once those problems are fixed, it'll be nice to know that I can actually have some choice in video cards when I go back out to get one in a few months.

If you could get the medium-mip texture corruption fixed, I would worship you all like tiny gods.

Another temporary fix that seems to work for me is to:

a) set video card memory to at least half what you really have... my radeon 9000 has 128 MB, so I set mine to 64)

b) turn off AGP acceleration

c) turn your texture cache size to less than 512 MB ram <--- works the best

LF

giddy ep! thx for the tips LF

Thanks for writing that nice report Doug. It's good to get some feedback on this issue. I'm not even an ATI user and up until now I'd begun to think they were forsaken...

thanks a million for the update.. some of us really were feeling abandoned...

one thing i really would prioritize is the fsaa black screen with the selection mode.... it *REALLY* is hard on people who enjoy fsaa in games (since by defauly drivers don do per game settings) and its totally lethal for ati equipped new players, as once they login to a black screen that they can't get rid of... they're not likely to log in again...

the various corruption related issues can be annoying.. but they're not *terminal* like the fsaa one is



dunno if it helps but i've stated in the past with that fsaa corruption thing that clicking the mouse within the selection area (3d space) does seem to give you a brief flicker of the actual draw buffer back... in fact clicking it often enough basically restores the scene entirely, so long as you can keep clicking...

theres something that happens immediately after making a mouse click, some slight difference in the code branch thats executed for the following little tiny piece of time that sorta kicks the engine into working properly again, but just for 2-3 frames before it reverts back

and its not the selection setup itself i dont' think.. as the actual engine works perfectly in general... its something in the engine, or in the driver, that happens in response to the actual mouse movement, some little thing thats being triggered in that segment of code, thats breaking it. so long as you *never* bring the mouse over some of the 3d space... the second the cursor does move over the space, *then* it goes black... after about 2-3 more 'good' frames

(its also weird that it doesn't actually happen when selecting any of the 2d objects in the game, aka menu's, maps, script edit windows etc.. so long as they touch an edge, and you move onto them from the edge, the screen stays totally clear with FSAA.

theres a couple more odd effects i've noticed with the screen blanking in fsaa mode...

each time you click the mouse *down* you get one good frame.. followed by a black frame.. followed by one more good frame.. followed by total blackout. *if* after the first 'good' flash, you move the mouse cursor over a 2d screen element however... the second frame does *not* actually flash in and it stays black as well

while moving the mouse around the 3d space within the SL window... it stays black.. *until* you stop moving the mouse entirely for the time it takes to render two consecutive frames... at which point (either holding the left button down, or not holding down, doesn seem to make a difference) you are given exactly one good frame, and then back to blackness. Again, if you do this over to a 2d object space instead and stop moving... you do not get that good frame, it all stays black

also the same applies for releasing the mouse... if you release the mouse it renders mabye 2 more black frames, and then a good frame. When releasing the mouse button over a 2d screen element, the screen stays totally black again


i don't know if any of this particularly *helps* but it can't hurt... at least you can mabye pinpoint down the exact parts of the running code are letting the good frames through and hopefully figure out whats changed between those and when its all black


btw im runnin xp w/ a 9800 pro, cat 4.2's, for i guess verificatoin purposes if you wanted to try this stuff on yer end

(ps, this weird mouse based frames peekin through thing is why i have so strongly beleived this problem is caused somewhere within the SL rendering/selection code and not actually the ati drivers... as i would seriously think if it was the drivers it would be an ALWAYS black in fsaa, you'd never see frames comin through or rely on mouse movement stuff kinda deal... especially that weird one where leavin the mouse still on two consecutive frames allows the second frame to come though clearly, but then it goes back to black after... somethins jus weird with that)

A status update on the ATI bugs...

We've verified that the issue with full-screen anti-aliasing is the result of us using a part of the GL API which ATI hasn't implemented (yet) when FSAA is turned on - we're looking into some workarounds - hopefully we'll have it in in the next minor release, but no guarantees.

The mip-map corruption issue has been verified to be a driver bug (at least for Radeon 9500+) - it's fixed in one of their current internal builds. It's also possible that this bug could also be the cause of various crashes in the ATI driver, as it would potentially have resulted in some nasty memory overwrites.

The problem with reflections working incorrectly has also been fixed in one of their internal driver builds - in a later version than the above.

Still no word on the avatar turning red when bump-mapping is turned on - I believe that ATI is either looking at it right now, or will be shortly.

We'll let you know when a driver fixing these bugs becomes available. It appears that it's a fairly major release, though - so no promises on when it'll be available. Current indications from ATI are that the mipmap fix will be out in a release in April, and the reflection map fix in a release immediately afterwards.

- Doug

w00t! Im definately happy to hear that. Just to know that you guys are trying your best to get our problems fixed makes me all warm and fuzzy inside ... Just enough to wait two months for a driver fix

Thank you Doug for providing updates on this, I greatly appreciate it.

damn good work guys... can't wait ta play with fsaa!

It work. I change the Texture Cache Size to smaller than 512MB, it work on my ATI All-In-Wonder Radeon 8500DV. Thanks guys, but hopefully I can make use of my Radeon 8500 full features, catch up guys/gals

Told you ATI can't write a driver to save their life. Grr. So glad I don't have one of their cards any more.

ATI has just released new Catylist drivers for their Radeon cards. I haven't been able to find a README or CHANGES file, so I've no indication of what, if any, improvements were made in regards to the SecondLife client until I get home.

Here's hoping!

They don't even mention SL in the ''Known issues' section of the 'Release notes. Makes you wonder...

Well i was feeling adventerous and decided to load up preview, with full fsaa on... guess what... whatever in the mouse selection code was causin the buffer blackout has been fixed because i was runnin around with full fsaa on and having a ball

Also the selection for ati in general seems *VASTLY* improved... no more 'hiccup' lowball slow frames when ya click or move the mouse... the thing stayed totally smooth... and at a *MUCH* higher frame-rate than equivalent areas ran in the full game (we're talking goin from 20ish fps with lots of pauses and hiccups to a decently smooth 40-50ish fps *WITH* 6x fsaa on)

jus wanted to thank the lindens for takin the time to address some issues that have been lingering for months an hopefully makin SL a much prettier place for all of us ATI users

What preview are you talking about?

[Post Edited to remove a huge chunk of techno babble]

Here is the short version:

While I got bored and was messing around with my profiler, I ran NEWVIEW.EXE through it (ATI CARD)

Results: 80% of time spent in video driver was the video driver waiting for data from the game.

The game spent most of its time in the following functions:

stateSort (while video card was idle, starved for data)
cleanDeadObjects (ditto)
updateCull (ditto).

Basically, the video card could be doing some culling, and perhaps the fact that just about EVERY texture is different means that state sorting is actually slowing it down instead of speeding it up (ie, its busy sorting and culling when the vid card could be rendering and culling on the hardware instead)

Of course, everything here could be completely wrong, but I did previously have the percenteges and the exact function call traces.

Do the lindens know that their EXE file is instrumented with function names, offsets, and call traces? It certainly helped to see where all the CPU time is going. Its going to the game engine (by an order of magnitude), not the video card. The ATI card is -idle, waiting for data- most of the time.

it would appear *ALOT* of that is fixed with the preview... if you haven't tried it i strongly reccomend it... you can immediately notice how much faster the culling and selection are (aka 512 draw in the release version on ati yer lucky for 1-2 fps even if yer lookin straight up into the sky aka its rendering ALOT of stuff ya can't see

with the preview it seems to only render SPECIFICALLY whats on screen aka with 512 draw when lookin at the SKY i was gettin 40ish fps... it lowerd abit when lookin at land, but stayed well over 10 in every case

well they didn't really detail which of the ati problems was affected so i'll give it a shot myself:

1) fsaa

*BANG* slaughtered

kudo's to the lindens on this one, its the one that i've been the most frustrated with, and the one i knew was the most likely to be client side... and i can tell you i'm locking that display tab to 6x and never lookin back... its hard for me to describe how much better this makes the game look... but then now all you fellow ati people can set this up and see for yourselves, so i won't have to ^.^ (as an aside one of the primary reasons movie and tv computer effects look so good is they are routinely rendered using insanely high levels of anti-ailasing

2) selection problems in general

again seems they have all been sorted out... probably related to why fsaa's working now with no formal announcement (as whatever was causing the hiccup in the selection seems to have been what was blanking the fsaa output) aka no more hangs or pauses or 'lowball' frames (for those who like watching the statistics bar) every time ya click or move then stop the mouse

3) texture corruption

well shes back.. and shes badder than ever.

not only are corrupted textures more prevalent than they appeared before.. they now seem to violate the 1024x1024 and above rules from 1.2 era clients...

I'm routinely seeing the 'default' particle corrupt now, which is new.. as well as the reflection map, which actually sometimes is *better* than the 90% generic grey alternative... but can be gistracting

4) random crashes

alot more common as well... potentially related to the corruption as well... i'm getting alot of 'application errors' on this client that i haven't seen before.. aka like a friendly little window telling me sl is about to crash as opposed to generally just tanking to desktop or bluescreening the pc


5) shaders

basically wholly unaffected... turning on the vertex shaders still gives far off people 'black' or flickering heads... and up close they still get color corrupted around the avatar mesh

thats not wholly unexpected as i have a feeling this may be a shader precision issue, effectively the shaders simply may not be compatable with ati... which isn't wholly suprising since i think they were developed with the help of the nvidia .cg toolkit.. which obviously doesn't rank full ati interoperability among its primary features



6) overall 'client' leak... probably not specifically ati related but seems to bog down the client more and more... envntually leading to a situation where you get 3-5 fps standing in a place.... where you relog and pull 30-40 easily

uh.

i am so computer ignorant its laughable.

i know this
i have a 512 ram motherboard.
i have a 9600 ati pro.

now. can anyone tell me how to adjust settings on my card.
and, what settings in game, should give me best performance? i have tried to read all that is said in this thread, but frankly a lot of what is said assume a person knows how to adjust the card settings, and in game settings.
i just dont know how to do either. i have the latest driver from ati installed that one released 3/10.

i seem to have okay performance, but at times its pretty crappy too. thats about as tech as i can get, sorry.

well we're glad to have you marilyn but this *is* the tech forum heh so its generally gonna be a bit over the top techhy from time to time


as to general settings the most important thing you can do to speed up your SL experience is going to be reducing the draw distance.. aka how far out you see.. and how many crazy things end up all on your screen at the same time.

the game defaults it to 128 meters.. which is a decent distance for exploring... but i find when performance really matters... a setting of 96 meters does smooth things out a little bit more.

Its in the preferences under the options tab i believe (actually the one *after* display settings) kinda near the bottom.

as to driver settings... there isn't a whole lot that will speed up your game experience... the driver generally comes tweaked for speed from the get-go... what you *can* do is under the display propterties for your computer... go to the settings tab.. and click advanced

This opens up the window dealing with most of the 3d cards settings... and one thing that i think might enhance yer second life game experience would be to go to the 3d tab, pick openGL, and set the quality slide bar one notch higher than the default.

That actually shouldn't impact your performance that heavily.. and will give you a rather significant boost in the games' clarity and vibrance... textures will look abit sharper and the objects people build will be smoother and more realistic around the edges instead of lookin like yer staring through a screen door

tyvm eltee, i am doing those suggestions, much appreciated, being so new here, its nice to have someone respond so quickly.

Forced FSAA still locks up my system (Radeon 9200 with Omega 4.3). But, yeah, it looks great for the 3 minutes or so that it works.