SecondLife client can't handle bad data?

The first rule of defensive programming: Assume all input is intended to crash your program, and design accordingly. Apparently, the SecondLife programmers have never heard of this.

SecondLife has crashed far too often in the past few days. I'm posting the last few lines from the SecondLife.log file from each of the crashes in the past 24 hours. Note that the last line is almost always a WARNING or ERROR about bad data:

2004-02-12T03:19:56Z WARNING BAD vorbis decode in decodeSection.
2004-02-12T03:19:56Z WARNING 104974e3-dfda-428b-99ee-b0d4e748d3a3 has invalid vorbis data, aborting decode

2004-02-12T03:24:35Z INFO Preloading af5a54e2-fb30-3bfc-a414-332fd6c7242f queue length: 0
2004-02-12T03:24:36Z C:\src\linden-branch-1-2\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect!

2004-02-12T03:50:51Z INFO Attaching object (4) item_id=20db7495-635a-2a94-956e-62d10bdb918f task_id=919992e9-46fb-048c-2448-97572409557cto Right Shoulder
2004-02-12T03:50:52Z C:\src\linden-branch-1-2\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect!

2004-02-12T03:53:57Z INFO 32 msgs decoded, net 0 msgs queued this frame!
2004-02-12T03:53:57Z C:\src\linden-branch-1-2\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect!

2004-02-12T03:56:26Z INFO Initialization complete
2004-02-12T03:56:26Z C:\src\linden-branch-1-2\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect!

2004-02-13T01:57:58Z WARNING BAD vorbis decode in decodeSection.
2004-02-13T01:57:58Z WARNING 31a228aa-90ad-5338-bd8e-e2fd9db99b23 has invalid vorbis data, aborting decode

2004-02-13T02:02:40Z WARNING BAD vorbis decode in decodeSection.
2004-02-13T02:02:40Z WARNING b072e8fa-6d02-6f53-24d7-70be1471e79b has invalid vorbis data, aborting decode

2004-02-13T03:52:48Z WARNING Message #244 received but not registered!
2004-02-13T03:52:48Z C:\branch-1-2\linden\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect! patches per edge 16 i 17 j 3 dc_offset 1.19909e-036 range 34746 quant_wbits 250 patchids 547

2004-02-13T03:55:43Z INFO 163 msgs decoded, net 0 msgs queued this frame!
2004-02-13T03:55:43Z C:\branch-1-2\linden\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect! patches per edge 16 i 28 j 3 dc_offset -5.49392e+024 range 33701 quant_wbits 63 patchids 899

2004-02-13T04:12:54Z INFO Transmitting sessions stats
2004-02-13T04:12:57Z C:\branch-1-2\linden\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect! patches per edge 16 i 7 j 20 dc_offset -1.14296e-021 range 128 quant_wbits 0 patchids 244

2004-02-13T04:15:22Z INFO 127 msgs decoded, net 0 msgs queued this frame!
2004-02-13T04:15:22Z C:\branch-1-2\linden\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect! patches per edge 16 i 25 j 31 dc_offset -5.25351e+020 range 32803 quant_wbits 53 patchids 831

2004-02-13T04:18:08Z INFO simulator_enable() Enabling 63.211.151.91:12035 with code 1809
2004-02-13T04:18:08Z C:\branch-1-2\linden\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect! patches per edge 16 i 26 j 29 dc_offset 8.74026e-040 range 20224 quant_wbits 128 patchids 861

2004-02-13T04:25:49Z INFO LLAvatarTracker:rocessOnlineNotification()
2004-02-13T04:25:49Z C:\branch-1-2\linden\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect! patches per edge 16 i 25 j 21 dc_offset 9.16657e+013 range 49184 quant_wbits 224 patchids 821

2004-02-13T04:28:16Z WARNING Unknown token in inventory file '}'
2004-02-13T04:28:16Z WARNING Bad UUID string: 25d85epe

2004-02-13T04:31:15Z WARNING Bad UUID string: d62b74c7-ccd5-8265-e687-e5e9e42aret_id
2004-02-13T04:31:15Z WARNING unknown keyword 'pe9c2k' in inventory import of item 00000000-0000-0000-0000-000000000000

2004-02-13T04:37:38Z WARNING BAD vorbis decode in decodeSection.
2004-02-13T04:37:38Z WARNING 6a24baf3-4d94-b105-4779-49e0d98ef258 has invalid vorbis data, aborting decode

2004-02-13T04:41:43Z INFO Discarding duplicate resend from 63.211.151.91:12035
2004-02-13T04:41:45Z C:\branch-1-2\linden\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect! patches per edge 16 i 20 j 2 dc_offset 7.78777e-037 range 33408 quant_wbits 0 patchids 642

2004-02-13T04:44:16Z INFO simulator_enable() Enabling 63.211.151.47:12035 with code 1896
2004-02-13T04:44:16Z C:\branch-1-2\linden\indra\newview\llsurface.cpp(701):ERROR Received invalid terrain packet - patch header patch ID incorrect! patches per edge 16 i 24 j 17 dc_offset 4.26903e-022 range 24580 quant_wbits 56 patchids 785

2004-02-13T04:46:57Z WARNING unknown keyword '-1daat-b8n00000-0000-0000-4ce983' in inventory import of item 00000000-0000-0000-0000-000000000000
2004-02-13T04:46:57Z WARNING unknown keyword 'pWWal21-breahout-based' in inventory import of item 00000000-0000-0000-0000-000000000000

2004-02-13T04:49:37Z WARNING Unknown token in inventory file '{em_ihh'
2004-02-13T04:49:37Z INFO LLInventoryModel::buildParentChildMap()
2004-02-13T04:49:37Z INFO complete inventory update

2004-02-13T04:52:53Z WARNING unknown keyword 'inv_item' in inventory import of item 120a9d56-7351-d71b-f5e6-d5246de3eec3
2004-02-13T04:52:53Z WARNING Bad UUID string: 0e7dfion_date-7fd0-721f-6c0000ion_date

2004-02-13T04:56:03Z WARNING unknown keyword 'b0' in inventory import of item 00000000-0000-0000-0000-000000000000
2004-02-13T04:56:03Z WARNING unknown keyword 'ecf' in inventory import of item 00000000-0000-0000-0000-000000000000
2004-02-13T04:56:03Z WARNING unknown keyword 'naup_mask' in inventory import of item 00000000-0000-0000-0000-000000000000

2004-02-13T05:06:38Z WARNING BAD vorbis decode in decodeSection.
2004-02-13T05:06:38Z WARNING 15cbbfd3-29b0-88ab-e2b4-e10ef9e7750b has invalid vorbis data, aborting decode

I was just helping out someone yesterday with this kind of problem. He was getting crashes all over the place caused by what I'm guessing is a bad data stream.

I agree that the system should not crash when bad data is given, but as a programmer, I know sometimes it's hard to catch all the possibilities, and when you do so, you often do it at a cost of performance.

I'd rather lose a tenth of a second per frame to verifying data than spend 45 minutes out of each hour rebooting after the SecondLife client crashes.

Yes, it's that bad. If you look at the timestamps on the log entries, you'll see that I'm averaging one crash every four minutes. And each crash necessitates a reboot, otherwise my computer will go into a hard freeze the next time I start SL.

Yeah, this is the same problem I was having and still am. It went away for a while, possibly due to fortunate events in the dark lands of the network powers that be. But now, it's back.

It doesn't help the fact that my roommate and I both like to come on here, and in the last half hour -both- of us have crashed at least 8 times in a row. After all these crashes, the client isn't picking up that there was a crash and so isn't requesting any reported log files.

It would be nice for LL to come up with a seperate program that harvests crash data from the client when it doesn't report automatically.

We've been working on this problem for a while now - we're really uncertain why we've been having these issues - they appear to have started happening a couple of weeks ago, and as far as we can tell they don't appear to be related to any particular patches that were made in releases around that time.

At this point, one of our current hypotheses as to what is happening is that some piece of firewall or NAT software is rewritiing our packets incorrectly.

If anybody who's been crashing a lot could give me information about their network configuration, I would appreciate it.

In particular:
Are you sharing your internet connection with any other Second Life users?

What firewall are you using? Is it hardware/software?

Are you using Internet Connection sharing?


The only other suggestion that I would make is that after this kind of network data corruption happens, you should probably clear out all of your cache data files.

As for the crash logger, the new version of the crash logger actually sends us information silently without a dialog in the case of a "normal" crash - the reason why we changed this was so we had a better chance of getting data in the case of something that hosed the display drivers (sometime switching out of full screen or popping up a window causes display driver crashes).

- Doug

Internet connection: Comcast cable internet using an RCA DCM-315 cable modem. The modem is connected to a Gigafast EE400-R router (hardware version 2.01, firmware 1.78a). The router is in turn connected to: my web server (always on, with TCP ports 80, 113, 1000-1022, and 8081, and UDP ports 113 and 1000-1022 forwarded to it), my main computer (either a homebuilt PC or a 15" Powerbook G4), and my uncle's computer (usually off). I'm seeing this sort of crash on both my PC and Mac, but it seems more common on the PC.

My main computer isn't running a firewall. The web server is running an old version of AtGuard.

No one else on the connection plays SecondLife.

I clean out the cache so often, I've got a batch file to do just that sitting next to the SecondLife icon on my desktop.

Want any more information?

Ooh! Ooh! Finally I know I'm not alone!

Okay, here's a little more info for LL regarding this issue... I've been on the phone with Comcast tech support numerous times regarding this. As difficult as it is dealing with most of their drone-like support reps who only read from their knowledge base, I have made some progress in finding out exactly what is happening.

When I start SL and start moving from sim to sim, the entire cable NODE I am on drops completely offline for a few minutes... that's 170 customers. This started a few weeks ago, just like was mentioned. Note the problem is most prevalent when I move between sims or look into other sims. If I am standing in one place looking at the ground or something, everything works fine.

While packet sniffing to see more detail, during normal behavior there is the expected UDP stream between multiple servers... different sims. When this problem arises I see a lot of the UDP stream is fragmented packets and other weird junk shows up that I think SL is accepting as proper data.... then the connection dies and there is nothing for a couple minutes.

Perhaps packets to/from SL are rebooting my node's router? heh. It sure acts like it.

I think the only difference here is that my connection completely cuts out before the client crashes most times. I've talked with other people in-world and they've said that they have been experiencing similar issues.

Or maybe I have a completely unrelated issue. Who knows. It's suspicious that Carnildo and I are both using Comcast though.

ISP: Comcast cable
Modem: RCA DCM-315 cable modem
Home router: Netgear something-or-other (4 port switched cable/dsl router)
Home network includes one always-on web server running FreeBSD 5.something.

I use 2 machines for SL, both have the same problem.

One is a toshiba satellite laptop connected via Netgear wi-fi card and Netgear ME102 wireless access point. The laptop ends up being double-NATed, but never caused any problems like I thought it might).

The other is a home-built pc with nvidia nforce2 motherboard using on-board LAN port.

Comcast has been under increasing fire for putting invisible bandwith caps on high bandwidth "abusers". It would not surprise me at all if this was very poorly implemented on their routers and strains relatively unstressed parts of the router software - thus the suspected reboots.

Shared media networks - e.g. cable IP - have proven to be a terrible means of connectivity over the last 100 years; witness the death of telephone party lines. At least IP over electrical lines never caught on...

We'll I've "tried" to have both of us at my RL home connect to SL at the same time, but have never been able to for more than a few minutes. We keep getting logged out of SL if both of us are on at the same time.

Lately, the client seems to "feeze" for a few milliseconds, work for a few seconds, freeze, work, freeze, work, etc. If I close down SL, the freezing stops (i.e. when it does freeze, the entire computer "locks-up" for a little bit, mouse cursor included).

I'm also finding that in the past couple of weeks, my entire network connection is taken down every few hours(and every computer that shares the same connection), necessitating a complete reboot of the cable modem and firewall/router. Not sure if it's SL doing it, or something else.

Router/Firewall: Linksys BEFSX41 (latest release firmware and also tried latest beta firmware)
Connection: Cable Modem through Road Runner/BrightHouse Networks
Operating System: Windows XP Home on all systems

Hank

I'm curious. Do you have only 1 IP address? Both comps can sign on but are crippled with one IP addy. As Higbee and myself discovered some time ago on a call to LL, you have to have a separate IP addy for each computer running SL. You may already know this but I've been surprised by the number of people I have spoken to that did not know.

As far as the other problems, at home I run:
Windows XP
Motorola Cable Modem (Roadrunner)
Realtek Ethernet Card
NO ROUTER

When I go to Higs we run:
4 computers with 4 separate IP addresses
2 run to a hub which runs to a linksys router
2 connected to the router
Motorola Cable Modem (Roadrunner)

No problems have been experienced using either of these setups. I don't know if this helps but I thought I'd throw it out there. Peas.

I'm not crashing, but my cable connection does drop more frequently whilst playing SL, regardless of the bandwidth limits I personally set. I have two computers that can play SL together with no problem (Mac & PC) and each is allocated 1Mbit under the SL settings. The connection drops are pretty much exactly at the same frequency they were before, when each was limited to 300kbps. (The connection has a total of 3Mbit to work with). The frequency of connection loss is the same whether or not both computers are playing at the same time.

I *do* on occasion have a drop in connection when I'm not playing SL, but it's far more frequent in SL. Also, it doesn't generally disrupt my playing. The connection drops, and when it comes back, I'm still in SL and updates start coming through again.

ISP: Comcast
Router: D-Link Wireless (DI-624)
PC: Pentium IV
Mac: Powerbook G4
1 IP address, shared

Also, my router has psychotic episodes every Sunday night at 5:30... I'm pretty sure this has nothing to do with SL, but it's an odd, baffling factor that I thought I'd mention.

Hey, guys -

Thanks for the debugging info, keep it coming! We're looking in to different ways that we can get more information about what's happening. We'll probably try and contact some of you guys individually to try and get some more information, and get you guys to try some debugging options.

On a related note, Second Life should be able to manage multiple users running on a NAT in most circumstancese. However, we've found that there are some routers/firewalls (especially older ones) that will not behave properly in this case, with issues ranging from connections dropping randomly, to misdelivery of packets from one user to another, eventually causing crashes.

If you're having problems running multiple connections, you'll want to try upgrading the firmware on your router. Also, it would be great if you could contact support so we could add the particular router into a hardware compatibility table.

- Doug

Any estimate on when the client will be able to deal with the immediate cause of the crashes: the bad data that's reaching it? Given the choice between crashing sixteen times in the hour I'm trying to run an event, and excessive ghosting or high lag, I'll take the ghosting and lag.

Is there any information available about what percentage of users are experiencing these problems? It is a truism that people who are satisfied with a product or service tend not to send compliments as often as those who are dissatisfied send complaints. I don't seem to have any problems with rendering, even though I use an ATI video card and run with some of the "troublesome" settings such as AGP Acceleratin and Avatar Vertex Rendering turned on. I don't crash very often - maybe once every 20-30 hours in game (hard to estimate since it is so rare) even though my system is not that much above the minimum requirements. The few crashes I do experience seem to be related to the resizing issue that Doug mentioned, and only manifest upon exiting the program.

Granted, there are enough problems being reported that SL almost certainly has software flaws. However, if 90% of all users experience virtually no problems, maybe those who do have problems can look elsewhere to reduce these problems. Over the years I have found that things like buying the cheapest hardware components (especially motherboards and memory) can lead to seemingly random software problems. These problems can be general, or specific to certain programs. Also, there is a lot of shareware and freeware out there that, while they themselves run well, can cause problems with other programs. Finally, although I have not had this problem so much with WindowsXP, it was common knowledge that you should reboot Windows at least once a day to avoid memory corruption and leaking problems. I can only think that I do not have so many problems because I bought quality components when I build my computer, and I don't run Kazaa, or shareware games/antivirus/spam filters, or stuff like that.

The only hardware in my setup that might be considered "cheap" are the network card and the cable modem, and the modem isn't a part I can replace -- it's leased from the ISP. However, both of those have been in use since before I started using SL, and neither has given me any problems with other software.

bump

This issue still exists with a LOT of people, not that it's really LL's fault.