Persistent Data Storage

I've been silent on this issue in anticipation of XML-RPC, but now since it appears that RPC isn't going to do persistent storage for us without kludgy polling, I guess it's time to re-open this topic...

Here's some opening suggestions...

1. llCreateNotecard(string name, string message); // create a new notecard (does not edit existing)
   
2. llList2Notecard(list data); // hi Hank! =)
   
3. llSetDataMap(string name, string value); // dictionary / hash map
   llGetDataMap(string name);
   
4. llSetPersistent(list data); // store a list
   llGetPersistent(); // retrieve the list
   
5. llEmailPreview(integer queueindex); // grabs email without deleting
   llEmailDelete(integer queueindex); // removes an email from the queue
   
6. llSetDescription(string text); // not much would fit, but it's better than nothing...
   

Feel free to add your own ideas...

How about just full-blown SQL access from LSL? Why not go for gold? Allow each "premium account" avatar a certain amount of space (say 10MB) on a SQL server hosted inside of SL, or maybe charge L$ for each MB of space used on a server each week.

If not, the DataMap idea sounds great. As an economy sink, to prevent abuse, and to encourage LL to do it, charge for data storage using DataMap (or SQL) with L$, say L$10 per MB per week, or something.

This was actually one of the plans I had for XML-RPC. I was going to "rent" data storage space through my web site/SQL Server, for those who don't have their own web sites w/ database storage.

:-/

I'm all for data storage, any of these would be ok with me just as long as it works.

Come on Lindens, are we beating a dead horse here? Are there no plans for this?

Kex is damn right. 2-way email and XML-RPC will be nice for monetary transactions, but what about those of us who want to develop things *IN*-world (remember that place?) like wrist computers or NPC's?

Give us llWrite2Notecard() now.

If you can spend an entire version allowing us to create doggie-style custom animation, can't you spare a week figuring this out?

Well, if anyone needs, I have a persistent storage medium in SL.

You can store 96-character strings persistently in an array-like medium.

To store a string, you just do "STORE-=-" + (string)iIndex + "-=-" + (string)value

To retrieve it, you do "RETRIEVEREQUEST-=-" + (string)iIndex and listen for linkmessage "RETRIEVERESPONSE-=-" + (string)iIndex + "-=-" + (string)value

Note that the storage medium uses one prim per string, so if you want to store 5 96-character strings, you'll need 5 extra prims in your object.

Azelda

Average turn around time (request - response)?
Does the index have to be an integer?
Is the information stored server side or off site?

Thanks for the offer Azelda.

But like Xytext, both are amazing feats of LSL Scripting knowledge and talent. However, both are extremely wasteful of server resources.


What we need is efficient, persistent storage INSIDE of SL. Requiring storage outside of SL is...

1. Kludgey
2. Slow
3. Unreliable
4. Expensive (if you don't already have dedicated server and database space on the Internet)

This could take the form of at least llList2Notecard and llNotecard2List (asynchronous with event, just like email), which would cover what most people want, the ability to store lists of information for later retrieval.

It could also take the form of what Kex suggests is some data mapping scheme to a Linden run database. Overuse could be metered by attaching a weekly charge to blocks of data storage, like Places listings are charged for.

We could also have true SQL access to LL-run MySQL databases from SL, with similiar use charges. That would enable so much in SL to be done, a lot of things would then be possible.

Right now, as it stands, you have to have your own SQL database available, have to send emails in and out of SL (I know XML-RPC will be available, but it's eccentially the same thing where you have to send "packets" of data in and out of SL), have to have some "program"/"script"/etc. running on the outside to interpret these packets, then have this "program"/"script"/etc interact with the database. If you want it to be up 100% for anything serious in-world, you have to host this "program"/"script"/etc on a dedicated server on the Internet.

All at great expense, lots of extra programming, delay, time wasted, etc.

The data is stored in texture keys.

Why I'm doing this I dont know since you could actually get higher data density by just using the prim name

It'd be possible to increase the data density to 4 char(96)s per prim, or 9 char(4s, by combining texture keys + prim name, and packing a little tighter.

For a tourguide computer, you could rez a storage farm on your land somewhere, and get the tour-vehicle to send requests to the storage via intraSL email.

For dance-bracelets, you could store a single char(345) containing llList2CSV of your dance moves, ie you'd only need a single extra prim within the bracelet.

Azelda

Edit: for dance bracelets, you could actually rez a single centralized storage prim on your land, and then you could send animation updates automatically to all bracelets

But, but, but...

This would hinder my plans to sell data storage space on my SQL Server.

/me kicks dead horse

/me kicks dead horse

/me chops the head off the horse and lays it in philips bed whilst he is sleeping.....


I agree, we really have no good solution for this inworld and I think keeping it inworld is the best solution but I will agree to any data storage option so long as its reliable and not running a close second to molasses in january.

Let me repeat and expand on the idea I just posed in the other horse-kicked thread.


New type of object in your inventory: Data

o You can move Data objects from inventory to object or to world (where it becomes a box like other invetory items without 3D do).

o You can set permissions on Data objects, including separate read/write permissions with lists of individuals, groups, and object keys.

o You can refer to them by key, which does not change if the information in the Data object changes.

o You can set and get the data by index via script commands:


integer llGetDataAsInteger( key data, integer index )
// and other basic types
list llGetDataAsList( key data, integer start, integer end )

llSetData( key data, integer start, integer end, list stuff )
llInsertDataBefore( key data, integer start, list stuff )
llAppendData( key data, list stuff )

llDeleteData( key data, integer start, integer end )
llClearData( key data )
key llCopyData( key src, key dest )
key llCreateData( list permissions )
// Copied or created data objects go into the script's object's inventory
// Copied data objects copy the permissions as well

integer llGetDataType( key data, integer index )
integer llGetDataLength( key data )

integer llFindDataIndex( key data, list match_this )


For the create, you need to provide the permissions list. This is complex, so perhaps a llCopyDataAsEmpty function might be better, which copies the permissions of an existing Data object instead of trying to create one from scratch. Then the permissions can just be set via the UI. But for llCreateData's permissions, you use a list of names and/or keys separated by integer flags:

// PERMISSION_ALLOW
// PERMISSION_DENY
// PERMISSION_ALL
// PERMISSION_READ
// PERMISSION_WRITE
// PERMISSION_READ_WRITE

So: [ PERMISSION_DENY, PERMISSION_ALL, PERMISSION_ALLOW. PEMISSION_READ, key, name, key, key, PERMISSION_READ_WRITE, llGetOwner(), llGetObjectKey() ]

...Would allow me and the script's obect full access to the data, while allowing four people or objects read access. Nothing and no one else could touch the data. Actually, I think I'd make the object and the owner have full access no matter what, so those three terms would be redundant in this case.

The permission list is easy to read: deny: all, allow: read key key name key, (allow) read&write: owner me.


Hmmm... What else have I missed here?

EDIT: Added llCreateData, llFindDataIndex, and made llCopyData return a key

The ability to create a new data object from code (CREATE TABLE xxx)

The ability to find a data index (or list of indexes) based on a partial match to the data. (SELECT id FROM table WHERE user_type = 'schmuck')

I'll go back and edit the post...

EDIT: I made the change, but had to go into a whole bucket-o-worms to permit the create function to set the permissions.

On top of that, I thought of a problem while I was doing it: concurrent access. What happens when two different scripts are both trying to read and write from the same Data object?

The only solution I see is to add two functions:

bool llLoadData( key data )
llUnloadData( key data )

Where the load returns true if it can load the data, or false if some other script has it loaded right then. The data would be automaticly unloaded when the block the data was loaded in goes out of scope. Trying to access a data objet that you have not loaded would be a runtime error.

while( llLoadData( "data name" ) == FALSE )
    llSleep( 1 );
integer index = llFindDataIndex( "data name", ["foo"] );
llSetData( "data name", index, index, ["bar"] );
llUnloadData( "data name" );

*bump*

Any LINDEN response on this? This missing feature is holding up a lot of interesting stuff.

I certainly can't speak for the Lindens, but I imagine that there would be a lot of reluctance on their part to the idea of allowing any of the automated data "store" ideas put forth in this thread. The main problem, from their point of view, would be the disk storage requirements such a feature would entail.

I could easily write a script that creates gigabytes worth of storage requirements.

for(i = 0; i < 1000000000; i++)
StoreData("some arbitrary string";

Let that puppy run for a while, and soon the Lindens will be keeping EMC corporation in business..

There'd have to have some sort of quota or storage usage fees, and it would be a major pain for them.

My Iraqi War Memorial has a database of over 900 names, hometowns, and photo information, all cut from a text file exported from a spreadsheet, and pasted into notecards. I certainly understand the appeal of having a true database API, but I don't see it happening anytime soon.

So long as the Lindens limit us to permantly storing data only via manual cut/paste, they'll not have to worry about ballooning disk requirements.

Once XML-RPC is fully operational, these sorts of storage functions will be possible, but all data will be stored on remote servers, and not the Lindens' machines.

My $0.02...

Ace

Give us a MySQL server. Let us configure our tables with phpMyAdmin. My web service provider gives us phpMyAdmin, automatically provisioned whenever we create a new database hostname, and it works really well.

In LSL:
integer llSQLConnect(string dbName, string dbUserName, string passwd); // Returns hDB
string llSQLGetString(integer hDB, string query); // Similar functions for ints, vectors, etc.
list llSQLGetList(integer hDB, string query);
integer resultCode = llSQLQuery(integer hDB, string query);
llSay(0, "Result: "+llSQLResultCode2String(resultCode));
integer llSQLClose(integer hDB);

If SL is meant to be the foundation of a metaverse, it's going to need more coders, and a lot of them will appreciate the ability to do data storage with SQL rather than playing around with notecards and wierd storage hacks. I also support llWriteNotecardLine() et. al., since there will be situations where it is more appropriate.

No it wouldn't. Plenty of RDBMSes have quota management features, either bundled or available separately. I just found a free one that works with MySQL on google. When someone goes over their limit, you revoke their CREATE and INSERT privilleges. Turn them back on when they go below the limit. It's completely automated, you run it out of a crontab.

He's got the idea! AWSOME! I support!

*kick*

/me keeps beating this decaying, dead horse's carcass until he gets a Linden answer. I notice that you give Island owners what they want, when they want it. What about the rest of us?