backing up inventory

As a newbie, it seems to me that if Linden want more users to test on the beta grid, the most important new feature should be inventory backup. It would then be possible to build in beta and transfer items to the live grid. So forget voice and sculpted prims, the priority should be to give us backup and restore!

Inventory backup should be in SL anyway. Even if the backup file would be encrypted to prevent tampering or multiplication. But losing items again and again one paid money for is just not a permanent solution. I think one should be able to have a LM folder with fav locations as well. Only that this folders content would be displayed on the login screens drop down menu when you launch SL. Now that be something I'd love to see, besides the Inventory backup

1) Backup unique purchased item.
2) Sell item.
3) Restore unique purchased item.
4) go to step 2.


Find a way to prevent the above and it'll improve our chances of getting inventory backups.

The Unique item in step (1) will have a unique key.

So in step (3) check if an item with that key exists in the database either as an in world object or belonging to a new owner, and only restore if neither of those is the case.

What I'm not sure is what happens to objects that fail to rez inworld - do they vanish from the database or are they still there but at a random location or just not existing anywhere. If the latter a garbage collection should really put this into the lost and found of the owner.

Another useful feature would be to get a report of the locations of all in world objects owned by you (this could be scheduled via the website, scheduled to run when the load on the grid is low, and e-mailed say within a day or two of the request, possibly with a small L$ fee to avoid overloading the databases).

backup asset
sell asset
tamper with backup to insert random new keys
restore asset
repeat

Backing up your inventory should be something that has been built into the SL experience. Each and every resident should have the ability to back up their inventory to their own HD as well as the main database backup. It makes perfect sense at the end of the day. Goodness knows there are enough techies out there that are capable of getting this whole 'Open Source' project going in the direction it should be going, and with a lot less hassle. Perhaps the problem is more 'Insular' as opposed to 'Open' when it comes down to the 'Source' of the matter!

Rusty,

Oh, thats easy to prevent - use PKI to encrypt/sign the backup data

i.e. the backup file is created on the servers, encrypted with Lindens private key and then transferred to the client.

The client can read the contents of te backup file by decrypting with the public key, but can't create a new valid backup file to send back during a restore without the private key.

If that imposes some problems due to silly US export limits on encryption technology, then an alternative is:

create backup file on server as before, create and encrypt a hash of the backup file with the private key. Send both the unencrypted file and the encrypted hash.

The client can read the file, and the end user could tamper with it, but they couldn't create a valid encrypted hash.

To restore you send the backup file to the server and the encrypted hash. The server descrypts the hash using the public key, calculates the hash of the backup file it has received and only restores if the two hashes match.

In this scenario the PKI technology is only required on the servers, so never leaves the US.

Both these approaches are based on the standard encryption techniques used through secure internet communications.

Of course the above both assume that thr backup file is stored on the local client, since most (if not all) of the issues are due to software failures rather than hardware failures even having backup system where the files are kept by LL on a dedicated RAID 5 server with tape backup and never leaves the LL networks would be a major step forward!

I don't see why local back-ups are needed, it all should be backed up on the Lindens' end.
But from what I've heard, local back-ups are likely only going to be for items you have copy-rights for, which kind of defeats the point as it's the non-copyable items whose loss is the biggest.

Keep in mind that it's not necessarily the object being lost,
it's the "proof of ownership" of that object.

Transfering a unique item from inventory to in-world must
remove the proof of ownership from your inventory and
place it into a sim. If load/lag/packetloss/errors cause that
operation to fail partway through then the proof of ownership
can be lost... other problems could result in duplicate proofs
of ownership, (not that anyone has reported that).

It's not so much the object that needs backup protection
but the transaction history for that object, and reconciling
inconsistencies, especially with untrusted data is risky.

...

I thought it was 'mod' items not 'copy' items that would be
backupable to local disk... though likely it should require
both.

...

edit: It would be interesting to have 'insurance' for valuable
objects. A claim that guarantees a proof of ownership
usable once to replace the lost item.

Even if it is, good luck getting it recovered.

I've asked about the backups in the past and got nothing resembling an intelligent answer.

Rusty,

The ownership does change if you are giving the object to somebody else. If a problem occurs in this case then I can see how it may result in an object in the databases existing without an owner.

However, if you are rezzing an object in world (or into another object), the ownership doesn't change just its location - given that every object has an owner property, rezzing in world doesn't and shouldn't change that information.

As far as i can see three possible things could happen when a failure to rezz occurs

i) the object ceased to exist in the database (my impression from other posts is that this doesn't happen)

ii) the object exists but is at a random location in the world (there is a post suggesting that it may end up on the sim at <0,0,0> but I own the parcel on that corner of the sim, and haven't found a pile of junk building up there; i did go searching for a sim at global co-ordinates <0,0,0> but couldn't find one. I do wonder if the Lindens created a Linden owned sim at that location and set it to autoreturn we would see a whole load of lost stuff appearing in lost and found)

iii) the object exists in the database but with no location (running a daily or weekly job to detect those and return to their owners in lost and found would help here).

Very good points...

I'm pulling this out of my ear, but I don't believe objects actually know where they are.

An asset uuid can be in a prim's contents,
linked in an object,
an linked object,
a texture applied to an object,
a coalesced bundle of returned objects or unlinked-picked-up-at-once objects,
or it can be in someone's inventory, or in-world.

For each of trilliions of objects, you'd have to search:

every object and all it's prims and all their contents ..
in every sim and in the owner's inventory,

just to see if it didn't already exist somewhere.

I might be wrong, and objects might know where they are, but assets (uploadables like sounds, textures, animations) exist many places with the same uuid, and certainly don't know where they.

I'm guessing here, obviously, but if assets knew their locations keeping those updated would add a lot of work to the asset servers.

I think probably the bulk of people who may be working on back-up facilities would be better spent making the rezzing system more robust to these problems. Currently the main issues we seem to have are:
- Items removed from inventory which don't appear in-world (e.g lagging server)
- Items which are rezzed in-world but are then lost (due to reserver restart)
- Inventory/Object inventory items that just go missing ("TYPE is missing from database!"

The latter should definitely have automatic back-ups as it can be easily confirmed (SL just has to try and access the key in the database) if it's missing. If the back-ups are sensibly done (only recording updates/changes and not actually removing items for say a month) then it can be found again and automatically recovered.
This would also allow for item recovery if you purged your trash by accident or forgetting there was a no-copy item in it that you wanted, as while it would be gone from the live database, it wouldn't disappear from the back-up database until a while later.
This case has happened to me recently with a few unimportant items, but from the sounds of it could happen again now that the database is distributed to scale better. Cory's blog post mentioned it only being likely to happen when the database is re-balanced, so the back-ups could be taken as a part of that process and be used to recover items that are later requested and can't be found anymore.

The other two cases are very hard to confirm; the second one could possibly be improved by having sims keep a note of when they were restarted and (most importantly) to what date and time they were restored. Then we have a note of no-copy items being rezzed and where. Then, if an item goes missing from a simulator, a recovery tool can check that yes your item was rezzed in that simulator, and the simulator was restarted to a state BEFORE your object was rezzed. Thus it can safely return the item without risk of breaking permissions.
Example:
- I rez my object "No-copy Chair" into Koss on Sunday 6th May 2007 at 18:37pm.
- At 19:23pm the simulator crashes and is restored to its last back-up at 17:13pm.
- I go into an object recovery tool and select "No-copy Chair" from a list of possible items to try and restore.
- The restore tool will see that the object was rezzed in Koss and when.
- It will then go into Koss' restart/restore records for immediately after that time and see that it in fact was restored to a back-up made BEFORE my object was rezzed.
- It will give me back my beloved Chair that I couldn't possibly live without and paid L$900,000,000 for.
I realise this is limited in its scope, but it would protect against quite a few examples of missing items that I know I've experienced myself (but fortunately weren't anything expensive or important to me).

The first case however I can't think of anything that doesn't have some obvious flaw. The two I suggested may even have faults I didn't pick up one so please point them out!

Actually, for those that are familiar with it... something like 'syslog' on a remote server that was notified whenever unique objects were being collected, rezzed, purchased, or deleted would probably sufficient.

If the server crashes, there's still a transaction log of where that unique asset was somewhere else. Sims are supposed to checkpoint their state, ?hourly? ? more often? So if they crash anything that's been around longer than that isn't lost. If the sim comes back up after a crash... have it scan the unique item transaction log and return the stuff it should have but doesn't.

If the attempt to rez fails, then the transaction log records "removed from inventory" and there's a missing "rezzed in world at location X" follow-up that's easy to verify.

If someone purchases something using a scripted vendor it might be more difficult to catch the transfer of a unique asset if it failed, I suppose it would have to log: "Attempting to copy and grant item X" followed by a "gift of X verified received."

If the user detects something is lost they could use a "tools -> report lost unique item" or some such, it would comb the transaction history for objects owned by the resident and possible incomplete transactions... and auto-return anything that didn't succeed.

Of course, I'd rather have more robust transactions...

All transactions need to make them more secure is to have it require a response. ie:
- Server sends item to user along with a unique key for the transaction
- User's inventory receives item
- User's inventory server tells simulator that it received the item and sends the key back
- Simulator can now mark the transaction as complete

If there is no response for a while, then the simulator can simply re-send it with the exact same unique key, in this way the inventory server can identify a duplicate transaction (in case it was lagging a bit) and remove any further requests with that key that are within its queue (if the keys are part time-based then this can actually be done really easily).
This is of course assuming these things occur purely because some transactions are being lost somewhere, so the repeat request should ideally allow it to get through. Sims can then limit the number of repeats they perform and attempt to undo a transaction if it cannot be completed.

Something which would help lessen this problem would simply be a transaction_complete() invent or something better named, which will be returned if llGiveInventory() was successful (ie the user got the dialog box or the item was otherwise sent), along with the key of the avatar or object the item was being sent to. This way scripters could build in checks themselves, if it isn't sent within some reasonable period then the script can offer a refund.