OK, I knew lists were bad, but...

list entries;

default
{
    state_entry()
    {
        string message = 
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##";
    
        llOwnerSay("String length is " + (string)llStringLength(message));
        
        integer i = 0;
        
        integer oldLWM = 0;
        integer newLWM = 0;
        
        while (TRUE)
        {
            newLWM = llGetFreeMemory();
            llOwnerSay((string)i + " entries, memory low water " + (string)newLWM + ", delta " + (string)(oldLWM - newLWM));
            
            entries += [message];
            i++;
            oldLWM = newLWM;
        }
    }
}                

Sheesh. So now I have to go re-design my entire database strategy. It's essentially a 2D table, and I was hoping to store it as a list of strings, with each string being a delimited list. These are per avatar, and I'll distribute this across 'memory server' scripts based on the key (a really simple hash, which is basically the first character of the key, which was found to produce a good random distribution). But still... that 400 character string is on the low side of what I expect to be storing.

Anyway... and it's my fault for not checking this until I'm almost at the end of the project But at least I have somewhat clean (I think) interfaces into the "database" from the rest of the scripts, so re-writing this shouldn't require changes to the rest of the code *crosses fingers*

So... suggestions? I think I'll try the "store lists as strings" code in the examples forum, which pops entries out from the front, checks/operates on them, and sticks them back at the end. It'll be way more compute intensive (in my code anyway, but I'm not so sure that llListFindList is efficiently implemented any more...), but...

One brute-force solution would be to just increase the number of scripts, and use a different hash. 1st and 2nd character of the key, but that needs 256 threads

Any other suggestions?

And this just blows my mind. Storing 400 bytes requires an extra 400 or 800 bytes of overhead. Amazing. Maybe there's an error in the math in that script, and if there is, someone please point it out to me, it'll make me a lot happier

It's actually very simple: your script contains the 400 bytes (the definitions of the strings, hardcoded into the script) already, and then copies it all into message, using another 400 bytes.

But the 400 bytes that goes into the hardcoded definition should only be counted once, right? Why does it take off 800 bytes every time I add it? I'm probably missing something really simple here. I'll run another test where I define the string in a different script and send it in via link messages.

I'm also wondering if this is because of Unicode, and each character is taking 2 bytes. If that's the case, then the list overhead isn't that much. Which is also borne out by this script:

string entries;

default
{
    state_entry()
    {
        string message = 
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##";
    
        llOwnerSay("String length is " + (string)llStringLength(message));
        
        integer i = 0;
        
        integer oldLWM = 0;
        integer newLWM = 0;
        
        while (TRUE)
        {
            newLWM = llGetFreeMemory();
            llOwnerSay((string)i + " entries, memory low water " + (string)newLWM + ", delta " + (string)(oldLWM - newLWM));
            
            entries += message;
            i++;
            oldLWM = newLWM;
        }
    }
}                

Same thing, the list is replaced by a string. But surprisingly:



So... (1) strings appear to be grown in chunks, not by the size requested by a concatenation. Kind of like how I understand C++ STL vector/containers to work. And (2) This actually fits less entries than the list did.

OK, test 3, with lists:

string message;
    
default
{
    state_entry()
    {
        message = 
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##" +
            "Test String Name 1" + "##" + "Test String Name 2" + "##";
            
        llSetTimerEvent(1);
    }

    timer()
    {
        llMessageLinked(LINK_SET, 0, message, NULL_KEY);
    }
}

list entries;

integer i = 0;
integer oldLWM = 0;
integer newLWM = 0;

default
{
    link_message(integer sender, integer num, string str, key id)
    {
        newLWM = llGetFreeMemory();
        llOwnerSay((string)i + " entries, memory low water " + (string)newLWM + ", delta " + (string)(oldLWM - newLWM));
        
        entries += [str];
        i++;
        oldLWM = newLWM;
    }
}                

And the same thing with strings (the sender script is the same)...

string entries;

integer i = 0;
integer oldLWM = 0;
integer newLWM = 0;

default
{
    link_message(integer sender, integer num, string str, key id)
    {
        newLWM = llGetFreeMemory();
        llOwnerSay((string)i + " entries, memory low water " + (string)newLWM + ", delta " + (string)(oldLWM - newLWM));
        
        entries += str;
        i++;
        oldLWM = newLWM;
    }
}                

So if it is Unicode and 2 bytes per character, then each entry is 800 bytes, and 800 * 15 = 12000, and the script starts out with 15000 free, so that actually lines up, kinda.

So there's no memory benefit to storing data as a string, instead of as a list? That totally doesn't sound right. OK, so maybe it's because I have a hard-coded string. But how does that get counted twice for every addition? And with link messages?

Edited for typo. Also, I thought UTF-8 only used 1 byte per character as long as you stayed within the ASCII set.

SL uses UTF8 encoding for storing unicode characters so it can be anywhere between 1-4(?) bytes per character

I thought the higher sizes only came into play when you stepped outside the ASCII character set though. I guess that's not true.

No, it's true - basic ASCII chars are represented with one byte in utf8

I had a similar problem come up, but I came up with a much easier solution, even though it won't work for the vast majority of people.

My database is nearly always static, after it is complete. Only ever once in awhile will it change, but those changes need to be propigated to all objects utilizing that database.

So I took the easy way out, hehe. I set up a server that, when emailed with a query, returns the UUID of a notecard inside the server. That notecard is the database. The notecard is split up into rows and columns, delimited by pipes '|'. This makes it easy to return certain lookups, because my objects almost always know what row to look up based on an integer.

The problem is, it's relatively slow, of course. Even with the timers turned up, it takes a few seconds for the email to go back and forth, and then another second or two (on a good day) for the dataserver to return the notecard line. And if I actually need to search the database for, say, a string, god forbid, it might take quite awhile, depending on which row the string is located on.

But at least that makes this possible... and it really doesn't use hardly any memory at all in the calling script.

Well, this information is highly dynamic, so notecards are out, unfortunately.

Well... you could always... (and believe me, I'm hating this even as I say it)...

Create a secure interface to an external database (like mysql) using php and XML-RPC...

Now if that isn't overkill for something that really should be a kind of simple scripting project, I don't know what is.

Dear god, we need more than 16Kb. X_X

I'm actually considering that It would increase the complexity somewhat, woudl require a serious rewrite of the DB scripts and maybe the access scripts too. And it makes the whole thing dependent on network lag issues. My chatbot AI pig uses email out/XML-RPC in, and it can get pretty slow. And of course, this requires an outside server.

I'm not so worried about security, because although this is a vendor-like thing, there's no real money riding on the contents of these lists. And it's kinda meannt to be semi-public information anyway, so the main thing is the storage/organization/presentation.

Well, anyway, thanks for all the help, everyone. I'm still puzzled by why increasing a string by 400 characters uses up 800 additional bytes.

Well, I think the unicode answer is pretty likely to be right, although it is still somewhat annoying that basic-ascii characters take up 2 bytes. I kind of wish we could utilize these two bytes, like, store a 16-bit integer in each character...

It has to do with how LSL works and de-allocates memory. Basicly LSL doesn't deallocate the memory that entries uses till you store data into it.
The memory will look something like..
[message, entries, entries copy, message copy, entries + message]
LSL likes it's memory contiguous.
Now i think about it, it sounds like a memory leak.

list entries;

default
{
    state_entry()
    {
        string message =
            "Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##";
        llOwnerSay("String length is " + (string)llStringLength(message));
        
        integer i = 0;
        
        integer oldLWM = 0;
        integer newLWM = 0;
        
        @loop;
        {
            llOwnerSay((string)i++ + " entries, memory low water " + (string)(newLWM) + ", delta " + (string)(-(oldLWM = newLWM = llGetFreeMemory()) + oldLWM));
            entries = message + (entries = []) + entries;
        }
        jump loop;
    }
}

Object: 0 entries, memory low water 15039, delta -15039
Object: 1 entries, memory low water 14641, delta 398
Object: 2 entries, memory low water 14218, delta 423
Object: 3 entries, memory low water 13810, delta 408
Object: 4 entries, memory low water 13402, delta 408
Object: 5 entries, memory low water 12993, delta 409
Object: 6 entries, memory low water 12581, delta 412
Object: 7 entries, memory low water 12169, delta 412
Object: 8 entries, memory low water 11757, delta 412
Object: 9 entries, memory low water 11345, delta 412
Object: 10 entries, memory low water 10933, delta 412
Object: 11 entries, memory low water 10521, delta 412
Object: 12 entries, memory low water 10109, delta 412
Object: 13 entries, memory low water 9697, delta 412
Object: 14 entries, memory low water 9285, delta 412
Object: 15 entries, memory low water 8873, delta 412
Object: 16 entries, memory low water 8461, delta 412
Object: 17 entries, memory low water 8049, delta 412
Object: 18 entries, memory low water 7637, delta 412
Object: 19 entries, memory low water 7225, delta 412
Object: 20 entries, memory low water 6813, delta 412
Object: 21 entries, memory low water 6401, delta 412
Object: 22 entries, memory low water 5989, delta 412
Object: 23 entries, memory low water 5577, delta 412
Object: 24 entries, memory low water 5165, delta 412
Object: 25 entries, memory low water 4753, delta 412
Object: 26 entries, memory low water 4341, delta 412
Object: 27 entries, memory low water 3929, delta 412
Object: 28 entries, memory low water 3517, delta 412
Object: 29 entries, memory low water 3105, delta 412
Object: 30 entries, memory low water 2693, delta 412
Object: 31 entries, memory low water 2281, delta 412
Object: 32 entries, memory low water 1869, delta 412
Object: 33 entries, memory low water 1457, delta 412
Object: 34 entries, memory low water 1045, delta 412
Object: 35 entries, memory low water 633, delta 412
Object: 36 entries, memory low water 221, delta 412

string entries;

default
{
    state_entry()
    {
        string message =
            "Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##Test String Name 1##Test String Name 2##";
        llOwnerSay("String length is " + (string)llStringLength(message));
        
        integer i = 0;
        
        integer oldLWM = 0;
        integer newLWM = 0;
        
        @loop;
        {
            llOwnerSay((string)i++ + " entries, memory low water " + (string)(newLWM) + ", delta " + (string)(-(oldLWM = newLWM = llGetFreeMemory()) + oldLWM));
            entries = message + (entries = "") + entries;
        }
        jump loop;
    }
}

Object: 0 entries, memory low water 15052, delta -15052
Object: 1 entries, memory low water 14672, delta 380
Object: 2 entries, memory low water 14264, delta 408
Object: 3 entries, memory low water 13872, delta 392
Object: 4 entries, memory low water 13464, delta 408
Object: 5 entries, memory low water 13072, delta 392
Object: 6 entries, memory low water 12664, delta 408
Object: 7 entries, memory low water 12272, delta 392
Object: 8 entries, memory low water 11864, delta 408
Object: 9 entries, memory low water 11472, delta 392
Object: 10 entries, memory low water 11064, delta 408
Object: 11 entries, memory low water 10672, delta 392
Object: 12 entries, memory low water 10264, delta 408
Object: 13 entries, memory low water 9872, delta 392
Object: 14 entries, memory low water 9464, delta 408
Object: 15 entries, memory low water 9072, delta 392
Object: 16 entries, memory low water 8664, delta 408
Object: 17 entries, memory low water 8272, delta 392
Object: 18 entries, memory low water 7864, delta 408
Object: 19 entries, memory low water 7472, delta 392
Object: 20 entries, memory low water 7064, delta 408
Object: 21 entries, memory low water 6672, delta 392
Object: 22 entries, memory low water 6264, delta 408
Object: 23 entries, memory low water 5872, delta 392
Object: 24 entries, memory low water 5464, delta 408
Object: 25 entries, memory low water 5072, delta 392
Object: 26 entries, memory low water 4664, delta 408
Object: 27 entries, memory low water 4272, delta 392
Object: 28 entries, memory low water 3864, delta 408
Object: 29 entries, memory low water 3472, delta 392
Object: 30 entries, memory low water 3064, delta 408
Object: 31 entries, memory low water 2672, delta 392
Object: 32 entries, memory low water 2264, delta 408
Object: 33 entries, memory low water 1872, delta 392
Object: 34 entries, memory low water 1464, delta 408
Object: 35 entries, memory low water 1072, delta 392
Object: 36 entries, memory low water 664, delta 408
Object: 37 entries, memory low water 272, delta 392

Wow. I am completely blown away.

And it sounds like a memory leak to me too. I could maybe buy that the memory wasn't deallocated when it reached the end of the loop, or in the case of the link message version, even when it reaches the end of the event handler's scope. But that would mean that the extra hit was taken on the 1st iteration, and the rest should have been clean. Since it's taking the hit every time, that points to lost memory.

There's no background garbage collection, is there? So this wouldn't be different if I made the list additions with long time delays in between?

And... thanks a lot. That goes without saying

P.S. The fact that I built up the string using the + operator - could the same thing have happened there? Those are constants, not variables, but I have no idea how the memory allocation is handled. I'll try your hand-built string with my original list manipulation code and see what happens.

And also... I'm assuming your code works because of the right-to-left order of evaluation of parameters?

That is correct. Except for lists and function paramaters are evaluated left-to-right (but the expression in each paramater is evaluated in the regular order, right-to-left).

I just removed the addition to clear up some memory (though it would have been a better idea to use a loop to build).

list entries;

default
{
    state_entry()
    {
        string message = "Test String Name 1##Test String Name 2##";
        
        integer i = 0;
        message = (message =  "") + message + message + message + message + (message = (message + message));
        llOwnerSay("String length is " + (string)llStringLength(message));
        
        integer oldLWM = 0;
        integer newLWM = 0;
        
        @loop;
        {
            llOwnerSay((string)i++ + " entries, memory low water " + (string)(newLWM) + ", delta " + (string)(-(oldLWM = newLWM = llGetFreeMemory()) + oldLWM));
            entries = message + (entries = []) + entries;
        }
        jump loop;
    }
}

Thanks.