Welcome to the Second Life Forums Archive

These forums are CLOSED. Please visit the new forums HERE

Announcing: Persistent In-World Data Storage

Caoimhe Armitage
Script Witch
Join date: 7 Sep 2004
Posts: 117
03-06-2005 04:40
Hello all and sundry -

Just announcing the imminent posting of some code to the Script Library. It's going to be under the thread title: Script File System. I imagine that there's some of y'all out there who could figure out the trick just from what I've said so far, so I'll say no more and let the code speak for itself :)

See y'all in the library

Edit: code is posted and waiting approval (20050306 1314GMT)
Edit: found a small bug in post-posting testing, serves me right
for doing it while jet-lagged. Will fix tomorrow (20050306 1404GMT)

-- C
Jeffrey Gomez
Cubed™
Join date: 11 Jun 2004
Posts: 3,522
03-06-2005 20:12
/15/78/37866/1.html

It's scripted SQL! Well done! :eek:

However, I don't think the major want for scripted storage is strictly to store data in SQL format (though this does that, and thanks!) - rather, the want is to arbitrarily pull data from a storage device on demand.

Say for example that you want to create a catalog for users to access remotely from anywhere in SL provided they have the client. The want, then, is to call the "persistant storage" and grab out of it the products, then beam that back to the client. Perhaps I'm wrong here, but persistant data storage a la ease-of-transfer would be the real utility for me. Storage that piggybacks in a script buffer, by contrast, would not.

My advice, then, is if you're truly interested in this "feature" (and I'm sure you are), add a set of simple-to-use llEmail calls back and forth from the storage device. That would have some real utility in my opinion, especially where databases inside of SL are concerned. :)
_____________________
---
Caoimhe Armitage
Script Witch
Join date: 7 Sep 2004
Posts: 117
03-07-2005 01:29
From: Jeffrey Gomez

It's scripted SQL!


Nothing that ambitious at all. just simple storage of named strings. really.

From: Jeffrey Gomez

However, I don't think the major want for scripted storage is strictly to store data in ... rather, the want is to arbitrarily pull data from a storage device on demand.


That's easy enough to do just by writing a different front end. The console client was really written just to demonstrate the functionality. I'm expecting to have a lot of different applications. currently use it to store animation data.

From: Jeffrey Gomez

but persistant data storage a la ease-of-transfer would be the real utility for me. Storage that piggybacks in a script buffer, by contrast, would not.


It's a building block. You're talking in terms of applications. I haven't done myuch with the llEmail() call yet, but I may give it a try just to show you :)

And very nice to meet you in-world, btw. I'm not arguing, just want to share with the rest of the world too.

-- C
Tiger Crossing
The Prim Maker
Join date: 18 Aug 2003
Posts: 1,560
03-07-2005 17:22
There's still the probelm that sripts can be reset when a sim crashes, or an upgrade goes through, or any number of other random occurances. Data held in script variables is unsafe for long-term or misison critical data storage.
_____________________
~ Tiger Crossing
~ (Nonsanity)
Caoimhe Armitage
Script Witch
Join date: 7 Sep 2004
Posts: 117
03-08-2005 01:22
From: Tiger Crossing
There's still the probelm that sripts can be reset when a sim crashes, or an upgrade goes through, or any number of other random occurances.


Scripts lose state during an *upgrade*? Really? That is very bad. It's a good thing that it's easy to write a save-to-chat function (which can then be pasted into a notecard) with my interface.

From: Tiger Crossing

Data held in script variables is unsafe for long-term or misison critical data storage.


Nolo Contendre. But it's better than what we have now, which is nothing. I do not contend that this solves all the storage problems for code. It does solve an important class of problems until the Lindens give us something better. If you need better storage methods there are a number of ways to do it if you're willing to push your data outside of SL.

-- C
Eggy Lippmann
Wiktator
Join date: 1 May 2003
Posts: 7,939
03-08-2005 01:59
I wonder if everybody who complains about persistent storage realizes that, short of manually recompiling or resetting your script, it's pretty much impossible to lose its state.
Script buffers ARE persistent. At least mine have persisted across a few dozen grid reboots. The servers save everything's state before going down, reloading it when they come up.
I ran a casino for about a year or so and it worked pretty well. But what do I know.
If you are THAT concerned about storage, separate code from data, save your data store's state to email backups, and allow your program to load its entire state from a notecard, just in case.
Torley Linden
Enlightenment!
Join date: 15 Sep 2004
Posts: 16,530
03-08-2005 03:25
From: Eggy Lippmann

I ran a casino for about a year or so and it worked pretty well.


Wow, I learn something new every day. :)
_____________________
Caoimhe Armitage
Script Witch
Join date: 7 Sep 2004
Posts: 117
03-08-2005 03:37
From: Eggy Lippmann

I wonder if everybody who complains about persistent storage realizes that, short of manually recompiling or resetting your script, it's pretty much impossible to lose its state.


That was my thought, but it is those recompiles/resets that make me nuts. I do custom work for certain builders, and when new features come in it's annoying to lose saved data about object positions and networks.

From: Eggy Lippman

Script buffers ARE persistent. At least mine have persisted across a few dozen grid reboots. The servers save everything's state before going down, reloading it when they come up.


yay! that's just what I wanted to know.

From: Eggy Lippman

If you are THAT concerned about storage, separate code from data, save your data store's state to email backups, and allow your program to load its entire state from a notecard, just in case.


Well I have a generic notecard->data piece which I use all over the place to avoid having users need to set params by editting scripts. This gives me some of that separation on the data write side of the game. Of course what we *really* need is llWriteNotecard(), but I'm not holding my breath waiting for it.

-- C
Zeno Concord
To infinity, and beyond!
Join date: 28 Mar 2005
Posts: 51
Writing to script?
04-13-2005 17:43
Is it not possible to write some specified text to a script contained in an object? I thought I saw some mention of doing that, but perhaps they were only copying whole unmodified scripts into other objects. If it were possible, then the scripts could be the persistent storage, assuming they were saved, recompiled, started up, some time after they are written into.

Some way of preserving modified data is central to an idea I am working on, so I am very interested in the state of persistent statefulness. I will use the method Caoimhe has provided us if nothing better is available (Thanks!), but the potential loss of data will ruin the long term value I seek.

Zeno Concord
Eggy Lippmann
Wiktator
Join date: 1 May 2003
Posts: 7,939
04-14-2005 02:01
Sorta. You can't write text to a script file in a way that you can also open it up and edit by hand.
You can send link messages to it, though, and store things in the script's runtime memory space.
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
04-14-2005 06:16
That WOULD work!


Have the script setup this way:

When it first rezzes, it sends an e-mail to the owner and requests a data upload. It will not respond to outsite stimulation until the data is uploaded or a command is sent by the owner.

Have it to where the data can be uploaded by either e-mail, or copy-paste to a notecard, and then activate it with the command. If e-mail, then have the data-upload also activate it.

Every day (or six hours), have it send an e-mail with all the data to a predesignated e-mail address.

If for some reason the script restarts, it will send an e-mail again requested data, and wait quietly for it. If you have a good e-mail system running, you can have it scripted to resend the last backup to the system and restart automatically...

Backups and persistant data anyone?

Update:

Ok, you can send a max of 4000 characters using e-mail. Just have the first line say "Block X of Y" and and have it send a series... Course, each e-mail has a 20 second delay, but you can always make a series of scripts and have each one send part...

Even better, It can do incremental backups. Send the main data once per week, and send daily updates the other 6 days...

Thoughts?
JayD Torgeson
Registered User
Join date: 24 May 2004
Posts: 5
04-14-2005 08:42
Go Vote for "Prop: 47 - Database object" and all our in world storage problems will be solved (one day) :o
Caoimhe Armitage
Script Witch
Join date: 7 Sep 2004
Posts: 117
04-16-2005 21:11
From: Eggy Lippmann
Sorta. You can't write text to a script file in a way that you can also open it up and edit by hand.
You can send link messages to it, though, and store things in the script's runtime memory space.


I have to point out that this is *exactly* what my persistence script does. All packaged nice and neatly with an API and everything.

- C
Caoimhe Armitage
Script Witch
Join date: 7 Sep 2004
Posts: 117
04-16-2005 21:14
From: Foolish Frost

Have the script setup this way:

When it first rezzes, it sends an e-mail to the owner and requests a data upload. It will not respond to outsite stimulation until the data is uploaded or a command is sent by the owner.
...
Every day (or six hours), have it send an e-mail with all the data to a predesignated e-mail address.


If you're willing to use out-world resources, there's way better ways to solve it than by using script storage. The whole point was to find the best you can do in-world.

- C
nonnux white
NN Dez!gns
Join date: 8 Oct 2004
Posts: 90
14.5Kb is nothing!
07-17-2005 15:23
do u remember the Spectrum 48K?

i started building a game, long time ago, on this "PC"... i was only 14 years old, and i run out of memory in the midle of production on this spectrum. there was 48K only!!! later i acquired an 128K +2, so i could finish my game :)

back to SL now... i am already at memory limit. 16Kb is not enought. this persistent storage is just great, but it uses around 1.5Kb of memory, so in the end u can save up to 14969 bytes.

let me just remember you that we are on the Giga age... everything is giga... our hard drives have gigs, we got gigs of ram, our connection is not gigs, but megabytes - with gigs on local network... even in a cellphone u can store up to 1gig!!!

so, i am trying to get more memory. i am saving the minimum on this file system, and is not enought. the solution is to use several scripts (each listening on diferent data and meta channels) but i am not getting a way to store the data sequencially.

i tryed to store the data randomly, but it crashes often, cause random is... err... random :)
it could happen to fill up 1 script memory, while several others are still free.


so, can u help me with this? i need to store - not gigs - but maybe the 128Kb :)

i need one ideia to do it sequencially, so after a bank is filled, it will just jump into the next bank.

any ideias?
_____________________
Caoimhe Armitage
Script Witch
Join date: 7 Sep 2004
Posts: 117
07-18-2005 01:00
From: nonnux white

i need one ideia to do it sequencially, so after a bank is filled, it will just jump into the next bank.


well you could always bother a Linden :)

but seriously, there is no way I'm aware of fixing the problem you mention. The llGetFreeMemory() call isn't worth the bits necessary to encode it. it reports some arbitrary number that *may* be related to the amount of memory a script is actually using at the time you call it. Hashing is the only solution that I've been able to puzzle out for this, and was a planned, productized upgrade (before I got pulled into making lively builds :)

Since we appear to have TZ troubles, I'm not sure what the best way to get together on this will be, but a clearer statement of your requirements might help.

- C
nonnux white
NN Dez!gns
Join date: 8 Oct 2004
Posts: 90
07-18-2005 09:30
From: Caoimhe Armitage
Since we appear to have TZ troubles, I'm not sure what the best way to get together on this will be, but a clearer statement of your requirements might help.


what i need is very simple: store 2 pairs of strings. 1 is the key, the other is the data itself.
the ScriptFileSystem do it perfect, but it goes out of memory too soon. and as u say, llGetFreeMemory() is not accurate. i tryed to place a limiter inside each script, but it crashes anyway. i have to change the stored data sometimes, and is adding more data to the string, 1 char or 2 is enought to stack heap colision :)


i think i am gonna abort this ideia, and sit down and relax till we got fully implemented XML-RPC, since memory will stay at 16Kb for long time :]

i could use email + php + mysql, but is not fast enought for my plans...

the future is XML - SL is stoped in the past :)
_____________________
Trep Cosmo
Registered User
Join date: 3 Mar 2005
Posts: 101
07-18-2005 12:22
Alright, here's my two cents on the subject (please convert to your currency as needed).

I'm currently working on a project that I'd like to have store thousands of data sets. (someday) I explored this script, and the possibility of creating my own pseudo-ram system as described. The only way I figured it could be done is if you emulate the PC architecture. Create a memory controller script that handles the individual pseudo-ram bank scripts. It processes all requests, takes the data returned from the pseudo-ram bank scripts and puts it together to send back to the requesting script.

I deemed this to be a huge effort that's better solved using XML-RPC.