"Pulse Lag"

Hello all you technical people.

For the longest time the EXACT same thing happened to me. Everytime my view moves it gets stuttery in pulses and sometimes when my view does not move. Mysterious. The Fast Timer thing shows a big huge spike of blue every time a "pulse" happens. Off the screen huge.

Last night on a whim (love those whims!) I asked my friend to take off her hair. It has a million curls but whether they are torii or cylinders...uh... ANYWAY she took it off and suddenly SL was smooth as silky greased butter with no pulse lag and my fps doubled too. w00t! When she put it back on everything went back to bad.

This is just personal experience but shaving my friend's head fixed pulse lag for me.



Little followup... From the Torleypic it looks like it was the blue "World Update" that spiked so hard. I do not know if anything else spiked because World Update spiked all the way off the screen so nothing else showed. I need a bigger screen!! )

Hi guys,

Here's my understanding of things, hope it helps.



A "colo" is a co-location facility, basically a business that hosts many servers in one location. Linden Labs own offices don't contain the grid, instead running it is subcontracted to a dedicated company that owns the (specialist) buildings, routers, racks, air coolers and so on. LL then pay the colo for bandwidth, electricity and rent.

LL currently seem to host all the sims in a single facility in San Francisco. So, which sim you're in won't make any difference - a fully distributed grid that actually has sims which work better depending on physical location may one day arrive, but it doesn't exist today.



Ping times *do* affect performance, but it's not directly related to viewer FPS. Essentially the world is being simulated almost entirely on the sim, so if ping times are high then the world won't update quickly enough to feel real. This is different from not being *drawn* quickly enough, which is viewer FPS (vs sim fps).

In order to offset the fact that the world is running a fraction of a second behind what you see, the viewer does physics prediction and attempts to figure out what is going to happen ahead of time. This is why you can walk around and it'll respond instantly (when working properly) even though there's maybe half a seconds delay between you and the other avatars.

If ping times are very high, like >1sec, then the physics that the client predicted gets increasingly out of whack with the physics that actually happened. This causes you to start jerking all over the landscape and things start stuttering and "rubber banding" as the viewer fixes its own view of the world to be in sync with the grids view.

This can also happen if sim FPS starts dropping for some reason (eg the sim has started swapping or is doing an auto-save).



Hmm, the labels seem fairly self explanatory to me but I've done game programming before. Would anybody benefit from a quick explanation of what each part means?



I'm not a Linden and this is just a guess, but I'd expect "ping sim" to be the time in milliseconds it takes for a message to go from your computer to the sim and back again. "Ping user" would be the time it takes for a message to from your computer to another players computer and back again (relayed via the grid). I don't remember what sort of numbers I get here but I'd expect ping user to usually be higher than ping sim.



Router faults occur all the time, and part of the service colo facilities provide is that they have dedicated staff that monitor network performance in the aggregate. It may be that LLs colo company is not very good

The main problem is that _any_ router fault _anywhere_ along the path from you to San Francisco can seriously screw up the SL viewer because it's so different to, eg, web browsing or file downloads. The internet has a kind of "weather" and there are often localised bits of turbulance that come and go as things break or get overloaded and engineers from the networking companies get on scene to repair it. You don't tend to notice this with traditional internet apps because the computers automatically compensate for it.

The main problem is todays internet is mostly used for bulk transfer of data that isn't time-sensitive, eg, file downloads, web browsing, instant messaging and email. Even audio streaming isn't that time sensitive because a large buffer is used to smooth out the kinks. Like with graphics cards, SL uses things in a way that isn't very common and suffers a bit from it. As time goes by I'd expect to see these problems get fixed (eg by the quality-of-service internet protocol extensions).

OK, er, hope that is clear!

I just started getting this today as well.

Being a reasonably technically minded person, I attempted to look further into it.

Since I've got an nvidia graphics card and a little experiance, I locked the card to display in wireframe mode only.

I noticed something.... interesting.
(unfortunately, I am unable to get a screenshot of it, I'll try to grab a copy of FRAPS later.)

I hang around the sim Hemaris most of the time.

We've got an usual server framerate of 45, physics 44.8, time dialation 0.9, and a script performance usually between 10000-20000. Decent amount of empty land, mostly simple structures.

For some reason, the polygon culling code is not working properly -- Objects hidden by a wall or other impassable non-transparent objects are STILL BEING RENDERED, and then the wall is rendered over the top of it, and so on and so forth. Combined with the screwy cache system, it really does a number on my system. Since there's nothing I can do about the culling, I skipped over to the cache. Shut down SL, emptied the cache folder manually (deleted everything in C:\Documents and Settings\UserName\Application Data\SecondLife\cache) and started SL back up. Boom. Smooth. It took a minute to recache the textures in my home area, but after they rezzed, I got *no lag* *AT ALL*.
My system can usually do 13-18FPS -- after emptying the cache, it was rendering between 25FPS and 28FPS.

Not sure why flushing the cache did it, perhaps it had cached too many complex objects.

Hope that helps someone else. I think I'm gonna start emptying my cache every week or so now.

i forget what the extension was, but there's a way to change the target in your shortcut to clear cache every time you login. i agree clear cache and clearing often helps. i noticed the culling thing also the first day i started sl (this was over a year ago, so it's been "not working" for quite some time). i've always assumed this was by design, and there must be a logical reason for them doing it this way; something about the special nature of content here, though honestly i have no idea what that reason would be. anyone know? or is it just "easier" to not write that code in your 3d engine?

(if someone has a list of run -options for the viewer, could you link me? i know there are several other special options like -multiple, -login name password, -colo, -port #, etc. what was -noprobe?)

Kamilion I don't think occlusion is even in, so it's not unusual.

There might be two seperate issues in this thread, but Dizzy's sounds identical to mine. I'm pretty sure it's related to avatar attachment LOD due to reasons I mentioned earlier, I expect some peoples attachments will be very high poly due to lots of prims but this seems totally over the top considering I get a stable framerate after the freezing.

Here

I'd like to add my name to the list of people this happens to. Like you, Dianne, I started noticing this about 1.5 and it kept getting worse ever since.

Clearing the cache will relieve the problem anywhere from minutes to hours. Sometimes I've been able to keep going an entire session without trouble, other times it happens quickly.

The common denominators I've noticed is that it's primarily laptop users reporting this, and that things grind to a halt much faster if I'm doing anything that involves a view of multiple twisted torii or other complex shapes.

What I suspect is that there's a bug about one or more of the primitive options that were newly included in version 1.4. Whatever it is, certain graphics cards (like mine - Radeon Mobility 9000) are unable to process the shapes properly, and build up errors.

I think it took a while to show up and kept getting worse and worse simply because there weren't that many of these complex shapes around at first.

----

My more cynical thought is that one of the newer primitives might not be supported to begin with by certain older cards, but there's so few of us that LL just wrote us off.

There is an ugly little truth about laptops, most laptops us shared memory for both video and main memory that will cause problems. Some advertise they have "dedicated" video memory but in truth they have system memory specifically allocated to video.. main memory is slow slow slow compared to dedicated, on the video card memory. If you have 512m system memory and 64 megs is dedicated to video then you only have 448m for your system to use and windows is going to use a significant potion of that just for the computer to run (may be close to 200 meg last I read) Which doesn't leave SL much to work with and you run out of resourses real quick. That's when you start using the hard drive as virtual memory and in laptops that means you pull up a chair, pour a drink with an umbrella in it and wait.

I am pretty sure SL does not clip hidden objects "by design", as the standard algorithms for figuring out what can and cannot be seen from a given location usually assume a static world.

That said, I'm sure it can be fixed somehow!

Okay.

Data (a tiny bit)...

I was in my workshop this evening and got hit by what appeared to be the same kind of pulse lag, so I opened up the FAST TIMERS!

Everything seemed okay (or at least low numbers on all the readings) with the exception of the "RenderPool number which was higher than all the rest and was between 45 and 50 most of the time.

I walked around my workshop and tried looking at what few tori I had around, zooming in etc. What I experienced was the same pulsing staccato freeze-lag. The rather nice Depeche Mode song I am listening to is stuttering in time with the pulses.

The Render Pool number is high and *sort of* peaks a tiny bit higher with each pulse, but the pulses themselves seem to have no correlation to anything I am looking at or any particular direction I am facing at the time. The most complicated object I have right now is a chair leg made from three tori, but zooming in on it has no effect on the problem at all.

So yes the Render Pool number is high throughout the phenomenon and a bit higher on the individual lag spikes, but seems not related to any actual rendering task. After about ten to fifteen minutes of these pulses, it all stops "like a door closing" (shades of Star Trek), and the render pool number goes back down to the low teens (about 12 most of the time) all the other numbers stay low (as they were before), and everything is smooth as glass.

Throughout all of this, my FPS has stayed respectable, but my ping was fluctuating wildly with the user ping and the sim ping usually being at opposite ends of the scale from each other. Sometimes one ping is below a hundred while the other ping is in the thousands, then they quickly trade places, and trade back again. I think there was packet loss, but I am not sure. Now that it's over, the ping is back down to the usual 50 or 60 or so and both user and sim pings are "back in sync."

The only other correlation of note was that Anshe Chung was in the sim and left (suspiciously?) just before it started.

*raises eyebrow*

I find the closest thing I have to this is typically a 'Sky Update' event... big blue spike, every few moments... fairly regularly.

Fortunately it doesn't seem to affect me too often, I can't narrow it down to a specific place or combination of factors. It's there or not, and persists for a while when it is. It's even noticible when typing and infuriatingly so

Not very helpful... sorry.. just adding my 2 bits

for what it's worth, i'm not on a laptop, and use pretty standard hardware. pentium 4, geforce 6600 AGP, 2.5GB RAM (got paging off now, though it made little difference in the offending spikes of lagtasticism. the one thing i do have that may be odd is hyperthreading. i read in an old post that someone suspected buggy code in that. so for the hell of it i set sl's affinity to just one processor. no dice. same problem.

today's much more successful experiment has been using the "-purge" option to clear my cache on every login. it's annoying having to recache my home every time i log for a sec or crash(and slower thanks to the dozen or more afk mall dwell campers in my sim), BUT i haven't "felt the pulse" all day, at home or elsewhere, which is promising. if you wanna try the same (on windows) right click your SL desktop shortcut, go to properties, and add "-purge" at the end of the "Target:" line, so it looks like:

"C:\Program Files\SecondLife\SecondLife.exe" -purge

then start up sl as normal and you should login to a fresh empty cache every time. you could also just do this manually by going to the network tab under preferences in game. i dunno if this is the magic fix most of you are looking for, but it seems to have worked for kami and myself. i'll post back soon after i've had more time to judge the difference for myself. clearing your cache on every login may also be a tad excessive (and if everyone started doing it, might make asset server angry). once every few days or weeks (whenever you feel the lag coming back) is probably enough, depending on how much you're on and travel around and such.

i think it was somewhere around 1.6 that they stopped clearing everyone's cache on every client update to save the asset servers from exploding, and i'd say that's about the time i first noticed the pulse lag, which makes a bit of sense now that i think of it. now that our caches aren't cleared automatically for us every update, i think it's time some of us do a little cache spring cleaning it does build up and get unhappy after a while!

thanks logan for the options list exactly what i was looking for. apparently i didn't look hard enough on the wikki before!

so no! cache clearing was sexy, but as i suspected, still not the solution to my problems! i do however this time have sexy pictures to share with torley and whoever else may be interrested.

pic 1: (i log in and let everything rez in around me. everything's smooth. then i fool around a bit (specifically, i alt-centered my camera on a bunch of things, walked around in circles in my house, and moved a single prim up and down for a few seconds). the lag cometh for me! centering camera on things (even long after all textures in my visual range have completely loaded in and rotating my camera a tiny bit) does it pretty reliably almost every time. look at all that green bar action! (i forgot to further expand the render section for more detailed color coding on that bit. if i had, you'd see 99% of that green bar going to renderPool)


pic 2: i don't touch anything. leave camera in same location. hit alt enter twice to change to fullscreen and back. woosh! lag is gone! much less green bar! smooth as silk :3 check the difference in fps on the right-hand debug window! note that k-tris/frame is the same in both shots, and texture memory doesn't seem to have drastically changed. also in my case, i don't seem to be getting the wild ping fluctuations others reported here. I will remain smooth like this for a few minutes, or seconds, depending on what i'm up to in world before it goes back to hell and i have to repeat this step.


i hope this helps somehow

This is exactly what is happening to me. The little bar at the top claims no packet loss. At first, I thought it WAS stuttering fps, it looked exactly the same to me... except that the the debug console tells me that is not so (there is a very slight drop in fps, but not much, and it goes right back up after every pulse).

This isn't caused by weird torii, unless every single sim has them; teleporting to a different place does not help. The only thing that makes it go away is, like Dianne said, time... or restarting SL.

I should say there is NO, I repeat, NO hard drive activity when it is doing this. It is NOT just a simple matter of page file thrashing. I have a large amount of ram (1.5GB), which seems sufficient most of the time.

Everything is turned off except for shiney, detail sliders to minimum, distance at 64... you name it, it still happens.

I see frequent lag, too. According to the fast timers, it's due to a "sky update". When I switch off the Sky render option in the debug menu, the lag is gone, but some strange effects show up where the sky has been before. I included screenshot of the timer line in question here.



I hope this helps a little.

*bump* this is an important issue to me and i'm not letting it die yet

I have the same issue, I have a 7800GTX, 10k RPM WD Raptor hard drive, virtual memory disabled, 2 gigs of 2-3-2-5 1T timing ram, 3500+ A64 and get surges of lag, always in a crowded room, sometimes all alone, I have no clue what causes it but it has been doing it since I got the computer in August. I have learned to just ignore it and maybe one day it will go away, I do find it odd that my friends computer with a Radeon 9200 doesn't do it when mine does lol. Oh well.

Occlusion-culling experimental code has been in SL for a few versions now, but it's disabled. You can CTRL-ALT-O to 'see what you could have won' if it actually did anything.

I know very little about this stuff, but thought I would add my own data to the mix in case it helps.

I was using the stock Dell PCI video card (reported by SL as ATIx300) and getting 5-10 FPS, but the were smooth. Yesterday I went out and bought a 6600GT PCI-E card and installed it. Now I get 15-25 FPS, but now I am also getting the "turning stutter" as discussed here.

I tried different combinations of graphic settings to see if any one of them was the source and it had no effect. I tried to figure out what was different in the field of view when it stuttered, and it didn't seem to be effected one way or the other by av's, water, or anything specific.

It's annoying though. I'd almost prefer a smooth 5-10 FPS over "pulse laggy" 15-25 FPS.

I hope this isn't too obvious, but I thought I'd check: Are you running any other active programs? Especially intermittantly-updating little window manager / panel applet eyecandy sort of things, which can cause tight blips of CPU/X activity.

Well, I do have a few things that run in the background, however they were there before with the old crappy video card as well and it wasn't an issue.

You're not going to notice 50ms pauses at 5fps that will be obvious at 20fps.

Your Dell doesn't have the duel core CPU does it? This has been a long standing problem with duel AMD processors and SL (plus other games). If that isn't the problem it may be network problems or another process stealing CPU time, hit control+alt+delete and open the task manager and watch your open processes while SL is running and see if there is any undue activity that coincides with the lag you are experiencing.

I think I finally got a screen capture of the kind of data Torley was saying the techines need.

It woudl really help to get someone working on this thing because as far as I can remember I have *always* had this lag, across different machines and through many releases. I just dont see it being a client side problem or even related to the particular users hardware.

Just as a refresher:

FPS is generally good and ping is also good (for me this is about 65 or so on both user and sim), no packet loss, no time dilation, and then out of the blue....

"Waves" of lag, usually every two or three seconds, lasting about a second where the entire world freezes, the music strea stutters, typing stops etc. It sometimes happens when you initiate a key movement to turn, but that could be just when you notice it cause you are expecting to move and you dont.

The only symptom noticeable is (for me anyway) the ping races up and down the scale usually with the user ping and the sim ping being at odds with each other. One rises into the thousands while the other stays in two digits or even goes down, then they reverse places sometimes.

It happened to me last night and I opened the fast timers and went up to almost 800 metres to get away from any objects, textures etc. to make sure I wasn't seeing anything due to the local geometry or texturing. When I did this the first time the lag spikes were composed of geometry updates, even though I was 100's of metres away from anything. Unfortunately I forgot to take pictures.

When I logged back on later, the exact same thing was happening, so this time I took pictures, but instead of geometry the spikes are composed of image updates it looks like. This, hundreds of metres in the air away from anything. In between the pulses, everything is running smooth and all the bars are tiny and right over on the left side, just as you would expect hanging in the air away from everything in an empty sim.

In the top right corner you can also see both the pings sliding up and down the scale over the course of the three pictures being taken (less than 30 seconds elapsed time).

Pictures follow: (3)

Edit: I *would* post pictures but I guess attachements are screwed up again.

That's SL for you. Lagging for almost a year, never been fixed, (never even been believed by tech support that it happens at all ), then finally I do all the investigative work myself and get ready to give the info to the techies, and you cant f*cking post attachments!!!!

Good work LL!

send them to me in world or fa_nyak@hotmail and i'll host them for you if you need :3
btw not being a big hardware person, could my p4 with hyperthreading be simmilar to the dual core amd's you speak of, and hence an issue, or is that something entirely different? and i also have a geforce 6600 (mine's agp, not pcie), so either or.