Extending Script Memory via llEmail

Ok people throughout the grid with scripting experience eventually come up with a project where its implimentation would benifit by some form of storage space. For non changing information some people have settled on using notecards and reading them a line at a time.

To me this method has proved slow, aggrevating, inconvienent and a total burden. My solution that I've suggested to people in world and have been using awhile is to take advantage of the fact that objects have email capabilities.

The llGetNextEmail() function allows someone to specify a specific subject line to grab. Also up to 12 emails can remain in the queue without automatically triggering the email() event thus can be left for some time without expiring.

The method I use is to store up to 4kb of specially formatted data in a string as the body of an email with a subjct line labeled in a manner that I can grab JUST the information I am needing to load.

Upon loading the information the email is removed from the queue so the only thing you need to remember to do is to perform any needed edits on the data (maintaining a valid size) and having the script email the data back to itself again so its again ready for later retrieval.

Normally a script is limited to 16kb of total data as most know. With emails being able to hold roughly 4kb of data, and with 12 emails being able to be queued up at a time, that adds ~48kb of additional storage space per script. The only thing to keep in mind is that the main script itself MUST keep 4kb of ITS permanant storage free to have memory to load the email. If you dont keep that in mind you'll grab an email message only to cause a stack overflow.

The good news is that emails are not lost when a script crashes. So once reset, a script has only lost the data in the ONE email it had de-queued prior to the crash. The rest is still preserved and can be accessed.

Convince me this is better than notecards...?

I'm not saying it's not... but I'm not convinced, either.

Adman

Well, for one, the script can change the data that's 'saved' in the emails, you can't do that with notecards.

why use email to store data only? Email can be unreliable, go down on a whim and get lost over the internet. Instead just use another script in the prim, or if you don't want deal with seeing a bunch of scripts in one prim, and the object is multiple prims, use llMessageLinked's to pass data back and forth. This is better because the cap on email is 1000 characters including the header, for a linked message there is no cap. I think this would be easier for you, unless your planning on having two objects using the same data, then you'll have to use emails or off world communications. Another advantage over emails is that the email address gets changed every time you rez the reciever (it gets a new key, and email addresses are just <key>@lsl.secondlife.com).

I find it easier to just use dedicated scripts for storage. If the script is properly constructed memory leakage can be eliminated (but at the cost of a bit more bytecode). On the flip side, using email does give the advantage of better memory cleanup. One disadvantage of using email is if you duplicate the item or detach to inventory and reattach, you loose the data stored in the email queue.

Sorry should have been more explicite. This is intended for objects that have a permanant residence in the world and that are not meant to be shift copied or picked up / rezzed routinely. Considering the post was geared towards people with experience enough to desire taking on large projects requiring a LOT of memory in few scripts the concept of UUIDs changing should be a given (should actually be something understood by anyone above a Building 101 level).

The 1000 character limit is flat out wrong. Thats the limit of the SUBJECT alone. An email has 4kb of data allowed but that 4kb includes the entire set of headers,subject and message body together. That is why i referred to it as ~4kb per email of additional storage that can maintain across script resets / crashes.

The email is not unreliable. Have had hundreds of emails and never lost any. Also the emails never leave the asset server. Emails sent from SL objects to SL objects are an inhouse operation with the Internet completely unaware and Linden Labs local Intranet / LAN doing all handling needed. Short of some DOS attack going on that locks up the mail daemon nothing is getting lost.

Link message handling works great too. I've used that for various database modules myself though havent played with them in a few months. The advantage of this over script memory modules is retainability. Using lists to store data in memory modules and link messages to transfer means a script reset can wipe your data as would an accidental heap overflow.

As for notecards the whole point of this thread is to show ways around a notecards weakness. Notecards can not be written to as that would give scripts access to editing asset server resources. Bad Bad mojo there. So this allows for saving where NCs dont. Also notecards must be giving delayed queries that respond with a single line of info and you have to continue doing yet additional slow / delayed queries for each additional line.

With email you can use llParseString2List with intelligent data seperators of various types for seperating rows / columns for intrisic DB functionality. Since the ENTIRE EMAIL is contained in the message body you only have to use one event() to grab everything and can immediately perform any actions you deem nessisary on the entire thing.

As an example lets say you are creating a game and need to save player data. You put the players UUID along with some float, string and integer stats in a comma seperated list. You can then about a hundred entries all into the same string by seperating them with some discernable sequence such as !%>, or some other set of s ymbols you consider unlikely to ever appear in the normal data portion of your string.

If you enter your information into the string by doing

UUID + seperator + CSVList_Of_Stats + seperator

and just append as many of those as you want together you can use the Parse function to dump that string into a list where each major ENtry in it would be a element with the players UUID immediately followed by an entry for the players complete statustics in a comma seperated value string. Since your defined seperator was NOT a comma and you used parse this is easy to do.

You can then do a llListFindList() to look up the position of a player in that list and use pos+1 as the element for his data. Edit that data as needed and replace that one entry.. then dump the list back into a string using your prechoosen seperator and email it to yourself again with a subject line informative enough to tell which GROUP of stats you have.

Of course there are the other methods but I prefer this one for the sake that it doesnt have as much of a risk factor. Take it or leave it, just figure id share the concept for those scripters just getting into dealing with persistant storage implimentations.

Strife:

btw I still use link messages for one purpose. If i have a no-mod script that i want a user i sell it to, to have some ability to customize, i will factor out the values i want to leave open into a SECOND script with full perms in the object. I will set the variables that need open access in that script and set that scripts state_entry() function up so on saving, it automatically link_messages the updated variables to the no-mod script. I only leave open the values i want changed so its still secure while giving customers some tweaking ability on options.

One problem - hacking

Nobody can hack a notecard that is not their own.

Anybody can find the key of the object in question and send it a bunch of dummy emails thereby totally destroying your queue. ( even if you filter on the from or subject the 12 email queue will become overburdened and you wil lose some or all of your original emails )

Nice concept - but not secure.

False. Objects are granted a queue of 12 emails. Any emails being sent after there are 12 in the queue are silently discarded not clogging up the queue. If you always resend the email after reading its data you are removing it from the queue right before putting it back thus your slot is nearly gaurenteed. If you are still worried about the issue you can just put the emails you personally need taken care of in the queue, and then add several more 'buffer' emails to fill out the 12 total thus all spam emails will be dropped at any time.

Its quite safe.

Nearly doesnt cut-it - if someone wants to disrupt your object they just sit an object which constantly resends (via multiple email senders if necessary ).

In the time it took you to remove - process and replace a new email has entered the q.

Somone who wants who hack will not take nearly as a deterrant.

Actually email is as reliable as any email on the internet. you WILL lose one eventually, it will happen. As for the 1k limit, depends on what you want to do. If you plan on reading the message, the email event only handles 1023 characters for the message string. Sure you can send more, but you can't read it, unless you know a way around that, and if you do please do share, since I've run into the 1023 character per message that the event handler only picks up.

The other main problem with e-mail is that you can't simply looking an entry, say you want entry 3 of 12, how are you going to get at it? Take the first 2 and resend them? Not a very nice solution for lag etc, while a link message can filter so it only deals with requests to it based on a filter (e.g num = 1024, integer comparisons WAY faster than string ones!) which is fast and non-wasteful, plus when the script you want is called, it can find what you need and sned it instantly back.

The only real disadvantage to link messages is that for every script that is triggered, you need a full copy of the message, unless LL has optimised this to be a variable by reference (unless modified at which case it can become a copy).

it is also possible to loose events, the event queue is only so large; it is important to remember that someone who is dedicated to breaking it could flood the event queue between the time the email is requested and the time the email is retrieved and to be placed into the queue. If the event queue is full, events are silently dropped (to do this properly you need to generate a few hundered events in a few seconds).

One issue with email, any script in the prim can read it, though the same can be said about link messages. Security + LSL = Non existant.

Yes.. and if you're the owner of the object for a personal project all scripts in the object will be yours. So thats not an issue.

Also you can read email 3 out of 12. Because (as said above) you use subject lines that let you differentiate what data is stored in the email. And you request llGetNextEmail() using that subject line as the filter. So thats not an issue.

I've sent emails just over 2kb in data from an object to itself and all data was valid. Was a list of UUID along with various text and other variable data and it was all still there upon reading the email. This was in the end of the 1.7 lifecycle and including the first release of the 1.8 client. I'll write up another test script to check that in 1.9 this limit is still maintained and post back.

Lastly what would allow someone else to KNOW exactly what method you are using to store data? Why would someone target your object? Are you gonna advertise your implimentations? Are you gonna tell everyone 'do XXX in order to break my system' ? Id really hope not. Also you could occasionallydo an unfiltered sweep of all emails in your queue checking their subject lines and resending as needed and if you find one that doesnt belong to you personally and that is being used to disrupt your object, FILE AN ABUSE REPORT. =-)

The email you recieve will have headers containing the object key and location in world. Use that in the report and the abuse team will have an easy time investigating. There is nothing in SL that is 100% hack or abuse proof. Nothing. So using ANY method will have strengths and weaknesses depending on what you are trying to accomplish. And this method works great with high usability and is the only method of saving persistant data that doesnt require you to run an out of world server.

Well, barring specially crafted packets and modified clients, there ARE things which are 100% hack or abuse proof. For example, I'm pretty certain that using link messages to communicate with auxiliary memory scripts won't be broken.

I'm rather leery of using email any more than is necessary. The twenty second delay, along with such problems as abuse make it highly unsuitable for a memory script.

Why not just use MySQL database with llHTTPRequest? Nearly unlimited storage

Believe me it isn't that expensive to get a webhost with MySQL support.