Lack of Persistent Data Storage

Come on Lindens, when are we going to get persistent data storage? The lack of this feature in LSL is keeping a lot of neat things from happening. I've pretty much given up developing anything new because we can't simply store bits of information (don't add to this thread kludgy prim name storage hacks).

When will the feature come? (Note: not really expecting an answer as the Lindens don't monitor or care about this stuff anyways).

they do care hank and they will answer. they'll likely say that a write to notecard feature is impossible to implement because of unique key foo. then they'll claim to be working on several ideas none of which can possible be implemented successfully until at least version 6.

this will of course follow many posts asking why you don't just use xml/rpc or email the data out then stream it back in using the new audio to parcel feature. surely you can write a dehasher that can translate sound into your search result ha ha ha ha ha

If you know any scripting or programming language at all that can parse strings, persistent storage is yours. You can do it with PHP, .Net, Java, Perl, Python, whatever. Just throw out an Access or mySQL DB and you're set. You can even just write to XML files. I personally am much more satisfied with these mechanisms than I would be with write to notecard. Managing XML files or a DB is way more productive and robust than yanking text out of notecards via the ugh data server.

Email works fine too unless what you are talking about is real time storage and retrieval (which even XML RPC isn't ideal for, and again ugh the data server isn't idea for at all). For "save game" or tourney tracking functionality a la Unreal Stats, it works perfectly. I know because it was part of the game I was building for the competition; it was fairly easy to do and worked perfectly...with a hotmail account!

Just dig into email, XML-RPC, and your programming language of choice, and there ya go.

External programs, dedicated external databases are not the solution. XML-RPC is not the solution. We need the ability to store data in-world, without needing to rely on the outside world. Communiation with the outside world is great and all, but it shouldn't be the "catch all" for deficiencies with LSL.

OMG, dialup internet access from SL! in the spirit of America Online, I'll startup a Lifers Online.. 'LOL'?

We'll have to agree to disagree Hank. I really think you're not looking at the whole picture regarding these amazing capabilities, and would hardly call implementing XML-RPC and Email to enable external messaging for persistent storage and incredibly powerful (essentially unlimited) data processing to fall under "making up for deficiencies in the game".

Think about it...for what you suggest, how would it work? Say you had a game, and wanted to do "save games", or track tourney scores. Would you have one notecard for every player? So you could potentially have, say ONE-TWO THOUSAND notecards after a few months? Or would you have one notecard and just keep writing to it, and start a new one when that one was full...how would you know which notecard to query for a given player? Scan ALL of them? How would you issue reports (say you wanted to scan for the top 100 players), how would you do rolling totals (i.e., how many players are at this stage of the quest), what would happen if you needed to alter the structure of the data (would you manually retool every existing notecard /shudder), how would you query data (would you suggest they implement full XML parsing/XPATH or tSQL in game) and so on. Effortless in a relational DB or XML file. Horribly unwieldy with notecards. I've thought about this a great deal for a long time and concluded that write to notecard is not the way to go. You'd crash into the limitations in no time, and I'm certain the devs knew this, and saw the endless complaining that would ensue regarding the clear shortcomings of it (in no time we'd all say, "this sucks!". So instead, they gave scripters the capability to do quick 'n dirty AND sophisticated processing/storage all at once with mechanism of your choice...now we ALL have a solution.

Not having a solution, and not having a solution that you like, are two different things. Sending an email from in game to out with structured data (you can just make the body of the email an xml document) is one line of code. Yanking it from an Outlook data file, parsing it, stuffing it into a DB, and sending an email response back, is simple. Anybody with an email account and MS Access or mySQL can do it. You can now offer a visual tool that shows land ownership patterns in sims...track your own dwell...the possibilities are amazing. None of this is possible with just notecards.

Sure for some things it may seem like a sledgehammer to swat a fly...but in the event I encounter a bear, please don't make a fly swatter my only option, and given the choice of one or the other, I'll take the hammer and just practice getting good enough at it to hit flies (actually, turning it around and just flicking the handle would probably work best...hmm. There's wisdom in that somewhere I think).

Anyway, what exactly are you trying to do? I'm sure if you posted it in scripting all the LSL jockeys would suggest a slew of ways to get it done.

My Turn to come in swinging for Hank.

Having written the perl and stuff for mysql backends for SL, I'll say its not the be all and end all for persistant data storage.

The main problem is you are reliant on external servers to maintain information, and network connectivity for the tranfer of information.

It takes 3 seconds for email to be recieved by lindenlab for an object. It takes 10-15 secs for email to be sent from an object. Why? Because of mail queues and the sheer volume of email comms traffic generated by the objects in SL.

This is completely useless for something that has to happen in realtime. People get frustrated if things dont happen in a timely manner - or not work at all.

I tend to feel the few thousand notecards argument is flawed. People who need to use a databse will do so, and people who will need to use a notecard will use a notecard. Both have their own specific functions and features, and therefore have their own different purposes.

hell - you could use a notecard to store data in the immediate future, and have a timer to say email all the notecard data to a parser every ten mins, and blank the notecard. You get the realtime speed, whilst still putting the information where it needs to go.

What is needed is something to handle real time data, and until that happens, there are many things that cannot be done in SL

Have you actually used the data server (notecards) extensively? It is slow. Your suggestion would not work. Expand that to multiple calls over a number of notecards, having to sweep through lines and lines of data...omg even email would be way faster.

What benefit would you obtain from writing to one persistent mechanism, then at intervals, writing to another? That seems terribly inefficient. Just write the data at intervals to the primary in bulk.

You don't use persistent data storage for real time data, i.e., every time somebody scores a hit, you don't write the new score to a DB. Any program architecture you look at will bear this out...for instance, character data is saved incrementally at key points (like at transactions/trades, or at timed intervals) from in-memory data. This data is never real time in the way you suggest (ever see a MMORG crash and get a rollack?). And what kind of callbacks are you suggesting, something more than "save confirmed"? The player doesn't have to wait for that (i.e., iyou don't kill a monster, then wait for "Character Saved" before you kill another).

And if email isn't fast enough...use XML RPC.

I've written these systems too, as well as persistent mechanisms for a variety of other kinds of programs. The numbers you indicate do not bear out with my experience. I would also say that if you are attempting to write and read in real time, your architecture is unrealistic. That's why things like session variables and so on exist, because you need that sort of speed. I'd be very curious if you could show me any program that uses persistent storage for blink-of-the-eye speed read and write retrieval. SL itself relies on external storage for its persistent data! Your persistent data would actually exist externally! And just think about the features you'd want...no way would it stop at a simple text write. They'd have to extend the scripting API dramatically. And speed up the dataserver responsiveness. And...etc.

We could go back and forth forever until what is trying to be done is known. Write to notecard would be convenient for some quick 'n dirty storage, nothing more, and if you think through how you are pushing and pulling your data, you can get what you want with the current system. I completely disagree saying that writetonotecard would enable many things that can not already be done. It may make them easier to implement since you'd just write all the script in game, but that's not the same thing as "we have to have it or can't do something".

So, propose your idea. What are you trying to do that can not be done? I'd be very surprised if your idea for persistent storage was realistic and was not doable.

I'm just really pissed that most of the "persistant storage" capability of XML-RPC was completely uprooted by the non-inclusion of outbound connections. I, for one, do *NOT* want to have my server poll every single one of my scripts asking "Hey, do you want to access my data now? ... How about now? ... Now?" Multiply a, say... 1 second polling interval by 1000, or even 100 scripts? Eek.

We cant even kludge it and make a client wait more then 10 seconds for a script's reply with the current implementation.

We need TRUE XML-RPC bi-directional communication for it to fuffil what LL promised and advertised it would.

Sure, I can use email to send a message out, but that in itself is an arduous task. Wait 20 seconds just to ask for a 50 character string? NO WAY!

==Chris

I'd also like to start a debate on allowing scripters to access remote functions synchronously, without using the remote_data event, but I'll save it for when we actually *get* outbound.

I'm with Chris on this one, pretty much 100%.

Chris, we need to talk. I need your in-world help with something, and we need to have a brainstorming session!

The main problem I have with *any* external solution to in-world scripting and storage is connectivity and speed/lag. What happens if I create something that relys on a computer program on my computer. What if I quit SL? Boom, there goes the object i created. What about my Internet connection? It's a consumer Internet connection, they fail all the time with no guarantee of uptime.

I agree that all these external "hooks" are a wonderful thing. They will really help people make grand and wonderful solutions that fit their particular situation.

However, they don't fit everybody's situation. You may work in a datacenter with profeesional, 99.9999999999% uptime T3 connections and databases/servers. You may have a 100% secure job, and may never leave SL. Some of us aren't rich or don't have that kind of access.

We pay a lot of money for server and database space inside SL already. I don't think it's too much to ask for some database space for LSL scripts, for those of us who want to keep our creations in-world.

Hank, why don't you contact me, too. We'll all brainstorm together!

I have no argument at all with the fact that SL ain't cheap (25+ a month is WAY premium). I've lately been balancing whether or not the price of SL, and what I actually do with it, is worth the money anymore. The only way to keep it going is to destroy what you've built and start over, and that's just not a useful model. I'm not interested in making money/GOM, but that seems to be a requirement nowadays for anybody that wants to do more than basic work and doesn't want to pay all out of pocket.

Chris, there are ways around what you are talking about. Get into your programming model's pub-sub event structure (I think you're a java guy, and it has one), which can send notifications of when changes to your data structures occur, or when particular methods are accessed, without script polling; your scripts don't have to ask, they can be told when they need to. One of the reasons I like .Net so much (and actually teach it) is event pub sub is way, way easier than Java, so those kinds of systems are easier to write (I eagerly await the fully implemented "new" Java)...but it's certainly possible in both. I've already done this and it works, though I had to get a little inventive with architecture (basically you need to write a callback to a callback). I have a lot of experience with this sort of thing because I've been working with XML and related techs for years, and am used to working in distributed loosely coupled environments.

I'm still curious to hear the proposed idea. I don't work (anymore) in some high end data warehouse/uber tech environment, I have a DSL conn at home, have plain old space on a public web hosting site (which is lovely since it hosts .Net, which makes web services/XML work easy), and do a lot of remote work for all kinds of environments. I might be able to give you some direction, or may look at the problem and say, "yep, this ain't possible", though that's darn unlikely; a lot of what I do for a living now involves, "I know they said it ain't possible. I'm telling you it is."

Unless, of course, you're just not interested in all that. Which is fine.

The problem is that there is no way to actually *update* the data structures in the first place. If all I'm using my server for is just a big memory bank (which is the point of "perminent storage", there is no way to:

A. Tell the server "Hey! I want to store some data."

B. Tell the server "Hey! I want the data in <variableName>"

Unless I'm missing something. Please explain your position a bit more, I'm not familiar with anything that will allow you to predict when an in-world script needs or wants to store data.

I have got to be missing something.

Are you saying that there is no way for your in-world script to send a call to your server to say, "take this data, process it/route it to <function object whatever> if need be, add it to <table row column xmlfile whatever>, and let me know the processed result (or just confirm that it happened)"?

As that's the whole point of XML RPC (to enable disparate, loosely coupled systems to exchange messages to trigger operations and return processed info/confirmations/error reports), and that's indeed what it does, and you seem to be saying it doesn't, I'm confused. Is this real, regular, plain old XML RPC?

Not sure where you got "predict". All I said was "notify". What I mean is, say you have three objects in world that would want to know if some other object updated an external datastore, and possibly do something in reaction to it (or just say "ok". You could do that without having those three objects poll your external source. They could just sit there quietly until they got a notification. I know it's possible, as I've done it. Of course, none of this is "real time", but as I mentioned above, that's not what these mechanisms (or any persistent storage really) are for. That's why we have arrays lists collections datasets recordsets blah blah...so you can handle structures of data in memory, and write them at optimal times. I think what could help Hank would be a notion as to how to manage data effectively in memory, and some routines with matching external technology to handle data updates and callbacks. It might also be nice if the devs gave us another "type" that wasn't just a simple list...some kind of in-memory dataset that serializes naturally as XML would be great. Sure we could write a wrapper for a list, but ugh.

Maybe you could just iterate (procedurally in plain language, without code) what you are trying to do?

Yes, Tcoz, that's what he's saying.

XML-RPC is only half-implemented. We can create LSL scripts that are servers, but there is no LSL XML-RPC client at this point.

So, the in-world object cannot initiate a connection via XML-RPC, only by Email-RPC. But with the recent instability/speed issues with the SL mail servers, Email-RPC does not present a viable solution for data storage. (I've sent test messages that took 2 hours to be delivered. I understand others have seen even longer delays.)

So, the only XML-RPC workaround at the moment is to write an out-of-world XML-RPC client and have it continuously poll all of your XML-RPC-enabled in-world "servers." The problem with this being that the request/response are reversed, and it becomes difficult to correlate one response (which is really a request) with the subsequent request (which is actually the response.)

Ah ok, that's informative Grim. That's what I needed to know.

The model I've been working with is sending calls out, and getting responses, or using event pub sub to initiate emails to the inside. It works fine...I don't see two hours and such, and have sent hundreds of mails (literally) though I suppose that's certainly possible, as I did read that LL was wrestling with their mail servers a little while ago.

Anyway, I defaulted to the XMLRPC out - email in - model naturally as it never occurred to me people wouldn't have seen this (since it's obvious you can't write a client in SL yet...if ever. Tell you the truth I never expected to be able to).

So it seems LL has some serious issues to get around here. A model does exist, and it works, but it's just spotty because the underlying systems are not stable (a la a great many things in SL...new and serious problems seem to still crop up all the time...ghosting seems to be at an all time bad, I logged in yesterday, tried to build and delete some detailed models, and oh god I had a whole roomful of ghosts, and my connection is ROCK solid). I'd also make sure that your email server is responding correctly.

Looks like THAT's what we should be complaining about I suppose, not asking them to write in another way to store data internally (which would use the dataserver, which would crack open a whole host of other problems). Please get this stable first!

So...we have a model, and it can prevent polling. It works...now and then. Ugh. Here we go again.

This is where I'm lost. I dont know how to make an object update an external datastore without using email or polling. I *loathe* email. Back when inbound email was implemented, I attempted to make a network based around it, which failed miserably (too many issues to deal with: high latency, bad outbound times from inworld objects (20 secs), horribly complicated inworld object setup (to beat the 20sec delay), ISP email server issues, etc, etc, etc). Currently LL's XML-RPC doesnt allow for it. Do you have a solution to this? Am *I* missing something?

Christopher, check the way off topic forum, please!!!


(Sorry to hijack, but the guy apparently doesn't check his e-mail OR the off-topic forum. :-/)