Lists and Memory Management

I have a couple of projects involving lists which collect data over time, either through other objects or avatars interacting with the object, or, collecting data off of a timer event. In either case, new data is being appended to the lists over time, E.G. a visitor tracker that records visits to an area or clicks to recieve a notecard (name, time of day). My question concerns ways to manage the lists so that I don't run in to out of memory errors or stack heap collissions. I am curious to find out how people manage their memory with open ended lists.

(1) Do lists have internal memory management, e.g. similar to how sensors only detect 16 (or is it 10, i forget) objects or avatars at a time and silently drop the rest?

(2) What are the most efficient methods for managing memory in lists -
(a) what data types (string, integer, float, vector, key) are the most efficient for storing data in lists;
(b) is it more efficient to store a few variables as a single string in a normal list, or, store the variables separate from each other in a strided list;
(c) are there significant memory savings only storing an object/agent's key as opposed to their name (and getting their name at runtime through llKey2Name);

(3) What are ways people employ to cap or limit lists from growing so big they overflow script memory, or, does using up the available free memory by continually adding new list items even cause stack heap collision errors

The most efficient way I can think of to manage memory in an open-ended list is to include llGetFreeMemory() in a timer event and delete the first index entry if the detected memory exceeds some threshold value (which you would determine on a case by case basis depending on the size of each list item). Are there other, more memory efficient ways to truncate a list besides this technique?

Another option might be to have one script contain the bulk of the code and pipe list data to another script that just stores list data through llLinkMessage.

Of course, I can store list data in an external server, but, for the purposes of this discussion, I am trying to avoid the need for an external server.

I've also heard that adding data to a list using MyList += NewListItem is less memory effective than by MyList += ([]) - MyList + NewListItem.

I'm really curious about ways others have dealt with this issue. If you have sample script fragments to share I would truly appreciate it.

Try this thread...offers some of the best advice around:

/54/c1/88658/1.html

-2fast

Nope.


See this page for a guide to the amount of memory consumed by the various datatypes.


According to the page above, a string as a list element incurs an overhead of 12 bytes, plus 1 byte per character. So encoding mixed data into a CSV string will almost certainly take up less than they would as individual list elements. However, remember that in order to access any data in a list, you have to pass a copy of that list to one of the llList2* functions, so you always have to keep as much memory free as is occupied by the list.

For that reason, when dealing with mixed data, instead of a single long strided list, I prefer the "poor man's struct" of several lists (or for that matter, strings... see below), each of which store a different piece of data, and are read "in parallel." While each list (or string) incurs a separate 21b overhead, they can be read individually, so I only need keep enough memory free to read the largest of them.


A key is essentially just a 36-character string, so unless the name longer than that, the key will consume more mem. Also remember, llKey2Name will return an empty string if the agent in question isn't in the sim at the time it's called, so for something like a visitor list in which the keys are only converted to names when the list is read, this method would be virtually useless.


Which would probably work great... if llGetFreeMemory() returned the actual amount o f free stack space, instead of just the "low-water mark." The only way something like this might work is if the value returned by llGetFreeMemory() were less than a pre-determined value, dump the list contents to e-mail, then reset the script, since that's the only reliable way to reset the free mem value.


The syntax you're looking for is

MyList = ( MyList = [] ) + MyList + NewListItem;

This is more efficient because using += keeps two copies of the list in memory for a time, the original contents of MyList and the copy of those contents onto which NewListItem is being appended. Putting ( Mylist = [] ) + before that erases the contents of MyList after it's read, since concatenation operations are evaluated right-to-left.

I also use this trick when passing a copy of a list to one of the list manipulation functions, if the return will end up replacing the original value, e.g.

MyList = llListInsertList( ( MyList = [] ) + MyList, [ NewListItem ], 2 );

The only way I can think of to prevent s/h collisions is to track memory usage manually. At the very beginning of default state_entry(), assign the return value of llGetFreeMemory() to a global integer, call it FreeMem. Every time you add an element to a list, add the amount of memory it takes up to another global integer, call it ListSize, and subtract that amount from FreeMem. (Do the reverse when subtracting an element, obviously) If ListSize > FreeMem, the stack and heap will collide if you try reading a value from that list, so perform a check before adding to a list to make sure this condition never occurs. It gets a little more complicated if multiple lists are used.

Personally, I've all but sworn off lists for large-scale data storage. The per-element overheads are just too expensive, when you consider you can encode an integer value as 2-3 string characters (at a cost of 2-3 bytes) as opposed to the 15 bytes it takes to store an int in a list.

(Yes, Virginia, LSL does have arrays... they're called strings )

For extremely weak values of "array", sure. =)

Maybe I'm showing my age, but I like to think: C does have strings.. they're called arrays.

Back on point: the answer to this question, ultimately is: dyndns.org, and llHTTPRequest().

LSL has some really severe annoyances, and this neatly sidesteps all of them for one not-completely-annoying up-front annoyance. Most importantly, you have all the memory and storage you need. And asynchronous computing power. And your choice of languages to fit your need.

Maybe CLR will give us the latter. I'm not holding my breath. In the mean time, I can make do with web services.

LOL @ "poor man's struct"

LOL @ necropost.

I've never tried this myself, but I think that if you have a script that checks to see how many pieces there are in it's list, when it goes over a certain number, it'll tell the next "node" script inside it's inventory to start writing down to it's list, rather than the list from the node that was reaching full. In this way you could literally have the capacity of 10 scripts for your list. And when they fill up (if you've really got that much to copy down) you can just add another.

Only problem with this way is that'd it probably get real complicated once you start getting a lot of stored lists, and I'm not quite sure how to have, for example, the avatar tracker to check the other scripts to make sure that they haven't been there already w/o storing that list. This is more of just if you want to store all the keys/names of people who visit your store, and then be able to say a command to have all the scripts say back who visited in chat or something of that sort.

This thread is from March and is dead. When you post to a really old thread it is called "necroposting" which is what Tyken was referring to. In some cases it is alright, such as when you personally want clarification on the points raised, but not when you want to add comments for the thread/OP.

Please say NO to necroposting

LOL @ pre-fix post:

------------------------------------------------------------------------

Hello pi Mip,

Tyken Hightower has just replied to a thread you have subscribed to entitled - Lists and Memory Management - in the Scripting Tips forum of SL Forums.

This thread is located at:
/54/27/174060/1.html

Here is the message that has just been posted:
***************
LO @ necropost.
***************

------------------------------------------------------------------------

Must be nice to perfect... oh wait.

Blast, my ninja-editing is no match for automated trickery!