Bizarre Memory Leak

I'm experiencing a memory leak in a script. What's bizarre about it is that it is inconsistent. There are no variable factors between runs, the script is using the same data set, same everything each time, yet it crashes at different points, or even occasionally makes it all the way through the run. I've gone over the script with a fine-tooth comb and cannot see any place where data might be accumulating, and even if there were, it should be the same with every run, yet it's not.

Another oddity I noticed is that in the initial memory report at the start of the run, there is a 32-byte difference between re-compiling the script and resetting it. I'm not sure why that would be, or if it makes a difference, but there does seem to be a higher success rate if the script is recompiled rather than reset.

Any thoughts, hints, ideas, suggestions?

We scripters are not allowed information about the way sims work. The information I know of are from test runs made by scripters
Is the behavior the same whether compiled in MONO or plain LSL?
How much data do you have? is it a big application?

It's too big to compile under LSL. When first compiled, it reports 32908 bytes free on startup; when reset it reports 32940 bytes. The full data set I'm testing with (object inventory names, types and perms) is about 18k, processed in chunks <= 5k.

Is this a bug in SL?

I can't post the script here, as it's proprietary, but if someone thinks they may be able to shed some light on it, I can give you a copy in-world.

well I can shed some light on one aspect.... the difference between compile and reset memory reports is consistent throughout MONO... maybe consistent isn't the right word... but it is observed behavior with all mono scripts, and you can pretty safely ignore it as a cause.

as for the random period crashes.... that sounds like something buggy in the mono VM (there are still some segments of code it just doesn't like) but without seeing code, it's impossible to tell, but it's unlikely to be anything basic if it crashes at random points....

you'll definitely want to confirm that it's not crashing at the same execution point. a simple ownersay + counter increment sprinkled through the script should do that... if the numbers are the same, the error is in a specific section of code, (hang time before crash may vary with sim conditions), if the numbers are not the same through consecutive crashes, then there's not likel anything you can do about it but try to guess which section it's having random issues with and write that differently.

Could it be the garbage collector? Since we can't control, or know, when that will run it may, sometimes, be recovering memory for you at different times during the execution and in different amounts.

If, for instance, it came along near the beginning of a run there might not be much to recover and your script will crash early. If it came 'just in time' it could delay the crash for a lot longer. With all the variations in-between?

Yeah, I put some debugging code throughout the script and it appears to be crashing at the same point in the code, just at different points in the run. The script is just reading the object contents and POSTing that to a PHP script; the crash occurs on return from the little function I have that makes the HTTP call.

I reduced the chunk size to 4K, and the leak seems to stabilize at a tolerable level through a dozen runs, so I'm going to go with that for now as I'm incredibly swamped (isn't that when these things always come up?)

When I have some time I'll see if I can extract just that section from the production script and come up with a stripped-down repro and post it to the JIRA.

sounds like an encoding problem perhaps.

How so?

different encodings have different byte lengths, which could vastly expand the memory size of the data being sent/received. they aren't necessarily linear, since some characters get expanded, and others don't.

if it's possible to clear variables before you get out, that may help, because I'm betting there is some doubling occurring somewhere in there along the encodings, and that's what's causing the memory spike and collision.