Lists and Memory Management

Folks,

As part of a legitimate educational project I need to save the chat of consenting avatars, sometimes when no avatar is around to do it the easy way (i.e. using the viewer tools). There are plenty of simple scripts out there, but they all fall over after a while due to memory constraints.

I'm assuming that one logical way to do this is to save a certain amount in a script, checking for available memory until it's almost 'full' and then sending it by email and flushing the list to start again - though presumably I'd then end up losing at least 20 seconds of chat with the delay after mailing...

I've also seen the use of various scripts - and believe that might be a better way of doing it. What I don't know is the way that works - how does script one know when to pass over to script two, etc., and how is it done? And when I need the chat in the X scripts to be emailed, how would I go about that?

Any pointers much appreciated.

That sounds like a very laborious way of doing it!

As an alternative, you can have a look at the SLOODLE WebIntercom if you like... it is used to send chat from Second Life to a Moodle site. It does that by sending each chat message individually to Moodle via HTTP requests. It all gets stored in a database for a Moodle chatroom, and/or as a permanent log.

http://code.google.com/p/sloodle/source/browse/trunk/sloodle/mod/chat-1.0/sloodle_mod_chat-1.0.lsl

Before to start using external ressources, you may want to try a full in-world solution. Here is just an idea about how I would do it.

I'd split the script in 2. One listener, one mailer.

The listener accumulates chat upto around 1/2 KB (512 bytes) and then sends it to the mailer through linked message. It clears the accumulator and restarts storing.

The mailer accumulates the linked messages upto around 3 and 1/2 KB (3584 bytes or 7 linked messages). Then it sends an email, clears its accumulator and restarts waiting for the next linked message.

This way, no chat will be lost. While the mailer is sitting on its hands during 20 seconds, the listener continues its work and, eventually, data will be stored in the linked message queue.

I chose 0.5 KB because it's a fraction of 3.5 KB and a mail cannot be longer than 4 KB, so there's room to go a little over the limits and still keep a lot of room for whatever headers SL wants to add.

Important detail: You do not have to worry about lists and memory. You just store everything in 1 string in each script and your heap will never collide with your stack.

You've got the problem of knowing when to clear the accumulator then though. If all you get are ASCII characters, then you can obviously just determine the size from the string length. However, Unicode characters can be several bytes each, I believe.

The 'listen' events will continue to accumulate during the 20 seconds of script delay imposed by llEmail(). It's just that the script won't start processing those events until the delay is over and you return from the current event handler. If chat is pouring in quickly enough it might cause events to get lost (apparently there is a limit of 64 on ALL event queues, not just link messages; at least that's what is implied in http://wiki.secondlife.com/wiki/LSL_http_server) or keep your script from being able to keep up, but otherwise the script delay shouldn't cause loss of information.

You might want to add a timeout so the e-mail is sent when EITHER the amount of data is nearing the limit OR a certain amount of time has passed (since the last chat or the last e-mail).

Keep in mind that logging chat without the knowledge of the people being recorded is against the SecondLife TOS. Be careful and ethical about how you use this system please.

Perhaps the easiest way of doing something like this would be via a bot program, the bot could sit and listen to all conversation and log it to a local file.

Hmm... There is a simple way to totally ignore the llEmail() delay: sub-scripts. I use them when I must send email at undetermined intervals.

So you have one listener and several mailers. When one mailer is busy, you switch to the next, and so on, until you switch back to the first. Just 4 mailers imply only a "real" delay of 5 seconds in between each email.

In 5 seconds, you can (almost) be sure that you won't hear megabytes of chat.

Now, there still is that problem of calculating the real size of a string in bytes...

integer uuRealStringLength(string zz)
{
integer total = 0;
integer i = llStringLength(zz) - 1;
for (; i >= 0; --i)
{
string char = llEscapeURL(llGetSubString(zz, i, i));
integer c = llStringLength(char);
if (c == 1) { ++total; }
else { total += (c / 3); }
}
return total;
}

Tested in world... It just works. My idea (See my previous post.) isn't so foolish after all.

Have some listening scripts IM a bot, and the bot can save chat or email it. Then you don't need to worry about it being in a certain place, or much about delay, either.