EURIKA! (Packet Loss/Performance/Missing Items SOLVED)

After getting another e-mail from [email]service@lindenlabs.com[/email] telling me that the disconnects, lost items, inability to move, inability to stop moving, items not rezing for 20 mins was the result of my graphics renderer... I decided to finally once and for all stop this BS and find out what was really going on.

My first step was to verify that my graphics renderer and card really did have nothing to do with it. I first upgraded my nVidia driver, which didnt help anything. I then borrowed a newer, high-performance vid card from my roomie and installed that. Also, not helpful, as I knew it wouldn't be.

Since all of these problems seem to be related to packet loss (which the SL Client is reporting) the next step was to debunk what people here have been saying ... "It's your ISP" ... I looked in the logs from SL and determined that sim957.agni.lindenlab.com was the server my computer was talking to. Traceroute tells me I am 10 hops from that IP address.... I did a standard ICMP ping with the -t option. (Makes it keep pinging 1 ICMP packet every second until you stop it.) The result:

Ping statistics for 72.5.12.107:
Packets: Sent = 244, Received = 244, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 43ms, Maximum = 75ms, Average = 43ms

Nope, NO NETWORK PROBLEM.

The next step was to log into SL and see what it did. Same thing as before. I decided to pull up the ctl-shift-1 frame rate window (which I normally dont) and immediately I noticed that the number of incomming packets was off the scale.



I also noticed that the packet loss was related to the number of incomming packets.... (and yes, that says 18 % packet loss which is pretty normal for me the last several weeks... I cant imagine how I managed to write a complex script the other day with that... just boggles the mind.)

It dawned on me pretty fast that LL only expected a max of about 50-60 packets per second based on the debug screen only going up that high, and that perhaps 200 packets per second was a bit much for it. Then I started wondering about this... after all, SL is not the only thing running on my computer or using the network buffers.... I mean, it's WIN-DOZE... there are 40 back end API's

Looking in my hardware setup at the parameters for my network interface card, I discovered it was set to have 64 recieve buffers. Which is probbably the default on most computers (I dont think ive ever even looked at this before now on this particular computer)

Increasing the network recieve buffers to the maximum of 128 buffers had the following effect on SL:



EURIKA!!!

For the first time in 8 days, I was at 0% packet loss at least part of the time, and was able to see and interact with the SL world somewhat. There was still some packet loss when the number of packets went over about 150 but it wasnt the 10%-25% I had been seeing before, not by a LONG SHOT.

I actually saw a couple of moments where it got close to 300 packets per second...

I dont think thats the only problem though, because at one point I encountered a total stop of all network data which lasted about 4 seconds



Now the bad news... While some users might be able to change their buffer settings for their NIC, I dont recomend telling everyone to do it. Anyone who isnt comfortable changing settings on their computer SHOULDNT even TRY this. (The stunt you have just witnessed was performed by a trained computer professional. Death or Serious injury to your computer can result in attempting this stunt....)

My recomendation to Linden Labs is add a PACKETS throttle slider right next to the network bandwidth one in the options.

The other piece of bad news (for LL anyway) is that I really AM a computer professional. I charge $25/hr for consulting work and just blew out 6 hours DOING YOUR JOB. I expect compensation.

And If I ever hear a LL employee tell me again that a network problem is being caused by my graphics card....

**Carefully, but nonchilantly examines his claws**

w00t!

I'll test it myself when I get home... the NIC on my laptop has a maximum buffer of 64 :/ ... unless there's another place to manually change it (I have a drop-down thing in my hardware settings)? I've never had this type of problem before, so never had to mess around with the default settings.

One other thing - is it even possible to change this setting with wireless? I use ONLY wireless at home, but reciently went wired again to see if that was the problem with SL... which it wasn't, though I didn't try the buffer thing. Everything's always 'just worked' with me setting things up the way it makes sense (even in Linux, oddly enough... but I was lucky enough to get a card natively supported in Linux, completely by accident), so know next to nothing about tweaking settings there.

If this works, expect a little L$ from me, at least. I'm broke as far as real money goes, so this is the least I can do

Pay up, LL!

Cool post, thank you.

My NIC (Intel PRO/100 M) doesn't appear to have such a setting. However, it does have a Receive Descriptors setting:

The default is 48... and it has a maximum value of whopping 1,024. Is the Recieve Buffers setting a registry hack?

Heh, I'm game because I think you're on exactly the right track. I am becoming increasingly convinced that many of SL's performance problems are down to the connection choking - others have already commented that this could be down to the lack of flow control on UDP traffic.

LL remain, as we've so sadly come to expect, deathly silent on such issues.

Have you tried to upgrade the network drivers and see if that helps?

I'm affraid I have never played with wireless... if you look over left at my picture, it should be aparent that I would only use CAT-5 wired networks.

No, It (should be) part of the config for your Network Interface card... At least it was for me. MyComputer > Properties > Hardware > DeviceManager ... Then find your NIC in the Network Adapters section and go into it's properties.... then advanced... lol... At least thats where I found it for my VIA "Rhine II Fast Ethernet Adapter"

But based on what you posted from the "Help Text" I would say the setting you found is the right one to set the size of the Network card buffer

Another related question might be do routers have packet buffers (pretty sure they do) and how big are they. I'm betting the how big is actually an onboard memory limitation and so not setable. It may not be standard across all routers though.

good call though, I've played with a lot of settings but I don't think that's one of them. I don't have significant packet loss but I do occasionally get that drop in connection for a few seconds :/.

I've also recently been getting a wierd effect which seems to me like my outbound data isn't being recieved by the region. Controls sticking, and chat being recieved but not being sent in a timely or orderly fashion. No indication on the debug panel about packet loss and bandwidth isn't 0 nor is it very high. typically about 30kb/s. I still seem to recieve chat durring these momentary hickups too. All definitely new since 1.9. Considered the possibility that it's my connection but I can't find any reason it should be. No traceroute failures, ping time outs, or inconsistant packet turn around times. it's got me fairly stumped.

You must be a Linden in Disguise for saying that....
** sharpens his claws on your life preserver .. throughly shreading it **

It's a VIA NIC, on a VIA motherboard. It was upgraded when I upgraded the VIA motherboard drivers.

But it wouldnt have mattered anyway if I had or hadn't... because the upgraded network driver would still have the setting to only have 64 buffers by default....

The SL servers are sending 250 to 300 packets per second to a program that was only designed to handle 50 per second.... (and somehow amazingly manages 200) .... You do the math.

Even increasing the number of buffers on the user end is only a temporary hack, and quite like putting a piece of celo tape on the titanic. As LL programmers add more "features" it will increase the problem as more and more packets must be sent....

** spits out pieces of shreaded life preserver **



I know that they do because Ive seen security reports of buffer ..... (thinks) overflow..... (thinks_more) ... vunlerabilities..... (lightbulb comes on... THAT would be why some of us are seeing our routers LOCKUP!!) .... but I dont think Ive ever seen a setting on a home router so you are probbably right that it is whatever memory is installed is the buffer

I take back what I said about the VIA network driver, The VIA Rhine series NIC's (and a few others) are aparently NOT included in the standard VIA motherboard drivers and ARE seperate... however the Fast Ethernet driver from VIA hasnt been updated since November of last year... and makes no difference in the test results either.

No matter how much any LL employee or supporter wishes, it isnt anything other than a design flaw (or bug) in the Second Life Client/Server pair.

Jitar, out of curiousity, what's your ping to server and server to client?

Eyes zero in on Router Lockup...

Whaaaa?

My Netgear router is 1 year old. About once a week, it 'locks up' and has to be powered off and reset. I've been thinking its a firmware issue, and have been hoping for Netgear to release something newer - as I already have the latest.

Interestingly, each and every time the router locks, I've been playing SL.

Jitar - have you posted to the Answers forum on this - or called Linden Lab's support at the 800 number directly on this? It'd sure be extremely interesting to hear a Linden perspective and/or acknowledgement on this one.

The pings shown in the 1st message in this thread are typical. General network tools will report 45-ish ms with an occasional rare blip to maybe 80-90 ms... SL however reports times about 3x higher than that and I often (probbably once or twice per minuite) see SL report 1000+ ms for a second or two... sometimes near 2000 ms.

Changing the number of buffers dosent seem to have changed that.



The door you just opened is booby-trapped ... for my trouble I have been threatened by a Linden with a Ban/Suspension. DON'T EVER call the 800 number, or email to support if told to by one of the in-world or Forum Lindens. It is a MAJOR mistake and will only cause your blood pressure to go through the roof.

Thier perspective is that It must be something wrong with my computer or it's my ISP's fault....

After posting to answers, I was basically told to submit a bug report and email to [email]support@lindenlabs.com[/email]....

Two of the e-mails I got back say it's my graphics Renderer causing the problems and tell me my Video card is not supported. (and send me a link to the minimum specs page... which says the gForce-2 card (which I have) **IS** the minimum required. Which tells me they have never read that page or dont really know what video card I have and are just saying "Graphics Renderer" because they have no clue what else to say.) .. which is why I spent time proving that WRONG before doing anything else.

One Linden tells me that the ping program I have is no good and I should use some ping program in specific, on mailing the results from that (which show exactly the same thing as the basic windows ping)... then another tells me THAT one is no good because it uses TCP packets and SL uses UDP packets so to use another one... (which also reports the same thing as the 1st two programs) ... I reply back telling that person what ICMP is and suggesting that they just fix the bug I reported and leave me alone... where upon I am told that the email address is only for customer support and not bugs so they are closing the support ticket.

After being dicked around like this in circles for a couple of months... they wonder WHY I'm so pissed off at them and have been ready to close my SL account for the past month. The one that kinda tipped the balance was this gem I got yesterday morning, in reference to my gForce-2 video card supposedly being unsupported.. and I quote: "because your computer does not meet the minimum system requirements I am required to NOT give you techinical support in the first place, yet have been." ... EXCUSE ME?!? This one prompted a phone call... which basically ended up as "My department doesnt handle that." ...

I immediately offered my land for sale I will be either canceling or downgrading my account before my next premium membership is due. If I'm not going to get premium service (or even BASIC service) I'm certainly not going to pay for a premium membership to this POS... Aparently, Linden Labs dosent want my business. (or more precisely they want my money but dont want to do anything to earn it.) .... No Deal.

At least my radio listeners got a good laugh last night from some of the songs I played in response to all this...

Give them a screen shot of the page with the minimum requirements too >

On another note, the NIC in my computer at home doesn't have a place to change the buffer size at all. :/

On yet ANOTHER note, my second life connection last night was the worst it's EVER been. I sat there with the ping stats up, and the scenario was pretty much thus:

'ZOMG BANDWIDTH!!!!' (kps shoots up to max, everything around me loads at once)

I take a step forward...

'ZOMG PACKETLOSS!!!!' (kps shoots down to zero, packet loss shoots up to 30 % with 2000+ms times, I fall through the floor)

'ZOMG BANDWIDTH!!!!' (I stop falling through the floor, start back at where I was standing before I took a step forward, kps shoots back up to max, everything around me loads back up at once)

I take a step forward... kps immediately goes to about mid-way down the bandwidth bar... I take another step forward... holding steady... I go for broke and click on a vendor...

'ZOMG PACKETLOSS!!!!'

And so on, and so forth, for a good solid hour before I dubbed the tedious experiment over and got offline.

I have a Geforce5500fx 256MB... when stuff is actually *loading*, I'm fine... fps is not the fastest, but very usable, and everything is pretty and everything. It isn't until I see packet loss that things start going to pot.

I need to borrow someone's high-end machine for a few hours and see if this same thing happes, or someone here with the same problems and a high-end machine needs to get with Jitar on how exactly to word everything so they can stop getting stuk on 'it's your ISP' and 'It's your graphics card'

Shame shame... Looks like im going to need to buy new network cards... None of the ones i have on my computers go beyond Recieve Buffer: 64K. :/

EDIT: i usually buy whats cheap. >.>

Really, that must be some other way. Yesterday I had like 80% packet loss that´s awfull, I lost more than 45 minutes building an AV and when I logged back everything was like lost? Everything was back to the first stage, all modifictions lost WTH?

Wouldn't lowering your bandwidth have the same effect anyway? I set mine low to about 200kbps and that alone will cause my lag meter and PL meter to both go red initially at login since it's downloading everything in the parcel I'm in. After a about 2 mins everything calms down, objects are loaded up, and it's all kosher. If I max it to 1000, then everything appears nearly instantly right after login.

Also the client wouldn't be able to access your nic at a hardware level to change what you're talking about. I mean you don't even mention how exactly you change this anyway, so it sounds like this setting is very different depending on the make and model of your NIC. Besides if you're also behind a router you have to deal with it's limitations as well, and other local network traffic.

The problem I see is not the number of bytes (bandwidth) but the number of PACKETS (the number of pieces) which come all at once.

Correct. The SL Client, like every network client on your computer gets it's data through a network card driver and an API which sorts the data by port number (known commonly as "Sockets" ... THAT is where the buffer setting I am talking about is, in the network interface card driver settings.

When ANY program accessing the network can't handle all of the data comming to it at once, the driver has to store that data until the program is ready to recieve it. The driver will "buffer" the data packets until either the program needing it asks for it, or until an event handler can be fired to handle it.

If the buffer is already FULL when a packet comes in... that new packet is lost.

A Year ago, the SL Client could make it's program loops to handle all of the packets. But with all the new features, and the increased number of objects in the world that means more packets ... At this point, the SL client cant loop through the packet events fast enough to keep up... More and more packets have to be buffered and it's reaching the point that the average buffer setting for many users is no longer enough to handle the overflow.

Also correct. Every NIC has it's own driver from the manufacturer and every manufacturer has different ideas of where to put the settings and what to call them.

Like I said, I found my buffer setting in the hardware profile for my NIC, but your's could be anywhere depending on what the maker of your NIC felt like doing. .... Which is why I said the average computer user probbably COULD NOT do this and that it would be up to Linden Labs to find a solution.

Some NIC makers dont even have a setting for it, on the assumption that the user wouldnt know what to do with it. (And they are probbably right)

Routers handle data the same basic way as a NIC Driver. When there is an overflow of data, it gets buffered.

Some of us are having our routers lock while playing SL, and I suspect the reason is that those routers cant handle MASSIVE_OVERFLOW... when they run out of buffer space they keep trying to put the overflow data in the buffer anyway, causing a crash of the firmware program.





All of this is why I said my fix was like putting a piece of celophane tape on the titanic... LL is going to have to make some major changes to the way they send data because this CAN'T be fixed on the user end. Even increasing the buffers on the user computer only helps a little...