How safe are our inventories?

Oh. Well, I don't think the landlords can do that.

coco

Depends, really, on what perms they have. If you've shared them with the group so it doesnt get autoreturned or whatever, they can 'take' them, therefore they can steal them. If you've built on a private island they can block you from it and effectively steal your work by denying you access to it. I've had people remove my stuff from their land loads of times without them ever being returned to me - I almost NEVER get my vendors back when a mall closes, changes hands or whatever because people just delete the stuff, not return it. Or maybe they 'take' it. Who knows?

The absolute worst thing is when you actually have given someone edit perms so you can work with them. I once had someone accidently take an entire building of mine.. the only copy. Of course, when they gave it back, it was no mod, no copy. Even to me *cry*

In short, perms suck.

Oh my! I don't understand all of what you said, but I think I ought to try.

coco

No, Eggy's solution would only let you make backups of prim items that you have full permissions on. It does nothing for clothing, animations, textures, or objects you don't have full permissions on - so I am not sure what the emphatic "this is the local backup, for crying out loud" is for. It certainly doesn't provide an effective solution to this problem.

Blaze,

The very fact that it does go to the fundamental heart of the stability of SL is exactly why I raised the issue. Not raising the topic and continuing to let it go on in the background is far more dangerous than raising it.

Raising it is not an effort to bash, Linden Lab, though admittedly, their performance in this area has really shocked me. When I signed into Yahoo this morning, I had an IM from my friend thanking me for starting this thread and saying that he still has heard nothing. His next step up the ladder is dealing with Philip directly, since nothing below that has produced results that were promised.

Restoring a backup is always an incredibly time sensitive thing, so for this to drag on for over a week with no response doesn't leave a lot of hope for him. Even after contacting someone directly and going around support, the "if you don't hear from someone in 24 hours please contact me" assistance was followed with nothing. No follow up from support, and no follow up from the person who said to contact them again. It's not a great cycle.

It's great for players to theorize on solutions to this problem, and certainly backing up the stuff you are able to back up on your own is one of them. However, only Linden Lab is in a position to provide a comprehensive solution to this problem. As the frequency of this problem seems to be increasing (and as Schwanson pointed out, many people may not even be aware until they try to rez some items), it needs attention and pressure to get it resolved on our behalf. This is our data that is being lost, and LL is comfortably protected by their TOS. What protection do we have?

This is a very pointed problem and has been stated concisely by the OP.

Over the last day, I've gone out of my way to talk to folks about their inventories. What I have heard underscores in a way nothing else could just how serious this is and has been for some time.

Simply put -- backups (with verify) are a fundamental process in any data-based business. SL can be considered nothing else BUT a data-based business, as everything that nets them revenue IS data-based.

With this in mind, there is simply no excuse whatsoever for the current state of affairs. None.

No, sorry, not even 'we only have 50 people, give us a break'. Reliable backups are THE safety net and backbone of your revenue stream. It is completely unacceptable that you cannot restore a customer's data to some safepoint.

I think everyone can accept that losses will happen (though admittedly, to the extent they are here, I would say something is very wrong in your schema or your procs), what no one should have to hear from you, LL, is SILENCE or 'Sorry, I'm afraid you're just SoL.'

Do something, LL, do something, do it quickly, and for pity's sake -- talk to us about it.

I have had a lot of problems with being unable to retrieve items from the database. Some were early things I made myself, but a number were things I bought or was given.

Since 1.6 I have had some very weird things happening. I have had objects which I made, for which I of course have full perms, moving from my inventory for all the world as though they were made by someone else. In one case, I gave a copy of a necklace I had made to a friend, and instead of copying, the original disappeared. I had fortunately given it to her with full perms.

I have recently had several items simply pouf when I take them off...clothes and attachments. Initially I assumed they had vanished into a folder and I wasn't finding them, but I have searched to no avail. I haven't even bothered reporting these things because of the lack of response I got when I reported major losses after a previous update. The reply said that no verification could be made that I had the items in question, although I had at that time a folder bulging with items for which no object could be retrieved.

I agree that it seems inexplicable that LL can't do something about this. Even more inexplicable is the way that they don't seem to have understood the strength of feeling that losing inventory items can engender in the player.
Cali

This is the part that gets me the most. Why do we have to jump through so many hoops to get a response out of technical support? I have never had a single good experience with LL's tech support. I have rarely used it, and the times that I have needed their assistance, I was talked down to, asked what kind of firewall software I was using (as if this has anything to do with all of my items disappearing in a sim), and given no assistance. Many times I would not even receive a response, even a promised one. My friend has received zero response to this issue, other than being told to contact someone if they got no response, after having no response for days.

I recognize that there are limited support resources - but the statement "I never heard anything back from support" is way too common for that to be an acceptable excuse. It also often gets blamed on spam filters and such things. If that is the case, move to a web based ticketing system - it would make it a lot easier to track the resolution of issues. It is one thing not to respond to a bug report, but to not respond at all to "Help, I have missing items in my inventory", or even worse in my friend's case, whose entire inventory is gone, is inexcuseable.

Reading through all these posts is really starting to concern me. I haven't 'noticed' any issues with my inventory. I have been assuming that since the folder is there...then the objects should be in them. I have seen statements that the folders may remain, and they may be empty. So I will be going through my inventory and checking everything...that should be fun and quite time consuming.

I have heard suggestions that people should just make alts and keep copies. Well I personally don't have an alt, but if I did...am I just supposed to buy 2 of everything just in case and put the extras on an alt? That seems...ridiculous to say the least. Not only counting the cost involved...90% of the stuff I have purchased are no transfer and most are no-copy.

I know it is better to be safe than sorry and that technically LL is not to be held responsible for such losses as stated in the TOS. There should be a way that every time you log off...your inventory is backed up. What if everyone's inventories just disappeared one day...without LL being able to do a thing...would SL survive? How many people would leave because of that? I haven't put as much into things as others...but what I have bought and made I want to keep. I have spent quite a bit for what I have and I don't want to have to worry about anything vanishing into the great abyss.

This is not a new issue, around the v1.2-v.1.3 era I knew a player whose inventory was also in the same state of corruption and it was a crapshoot to know whether a created item would actually be stored or not.

After losing a big chunk of inventory, including many novel creations in addition to a well crafted avatar, this player spent countless hours working with the developers (back when you could talk with the developers) to try to help them isolate the problem in order to get a better handle on fixing it. After much work trying to help LL debug their system (with all possibly cordiality and a good measure of technical savvy) the player stopped playing despite having a "lifer" account and therefore not even paying monthly; not to put words in his mouth, but I think he just got fed up and being in SL to make stuff moreso than socialize, it was a big exercise in frustration.

Cristiano is dead right, LL is marketing this in one way - including the much publicized 1.2 Intellectual Property waiver - and operating in another. I've more or less given up on SL due to similar frustrations. Not that the grid much noticed nor cared, and certainly not LL's revenue stream as I bought a lifer account as a vote of confidence in the emerging technology about 2 years ago. I'm not bitter, just tired of having my work disappear from underneath me. I had the same feelings about WordPerfect in the dinosaur era of personal computing... well, my document *was* there.

I noticed this morning that the US Supreme Court decided that file-sharing entities such as Grokster can be held liable for illegally-traded media files (story here). Could this herald a general trend toward holding service providers more liable for content that their users post? Could this mean that LL will have to be more heedful of copyright/trademark violations within SL?

More germane to this thread, will a result of the increase in responsibility for service providers to police the content manipulated/distributed by their users also mean that service providers will have to shoulder more responsibility for data loss or corruption of users' materials?

I agree with Cienna.

I'm getting the sneaking suspiciion, for some reason, that I bet you this happens more the older you are. My inventory is just fine, but then, it is relatively new. I only lost things when the landlord moved me.

Do you suppose there might be something to that? That older inventories get hit harder?

coco

Absolutely. The entire online gaming industry has been fighting tooth and claw to avoid it, as it means they will have to spend time and money to insure availability and be held accountable for the value of items their bugs or negligence has caused to be destroyed.

Frankly, it might be the death knell for the industry when it does occur, as there isn't a game on the market that budgets for or could bear the cost of reparations for lost items.

On the other hand, it may clear the way for some small development house without a lot of market equity to show up with an elegant product that take into account these things and they will be the next Studio 989/Verant.

For now, the only known liability is that of deliberate negligence of basic and well accepted practices... you know... like backups with verify to be able to restore upon data loss.

I would agree Cocoanut but......

I have been working with a new user. She spent several hours last week working on a prim dress. She was very excited and wanted to show it to me, but it was no longer in her inventory. She is very computer savvy and was quite sure as to what she had named it, she says she is 100% certain that it was in her inventory when she logged. Her account is 2 weeks old.

Welcome to SL.

I would defintely like to see some kind of backup system myself.

I just started noticing issues with my inventory as well. For example, I had a posing stand in my inventory. I dragged it from my inventory to rez it on the ground (specifically Anshe's Mature Sandbox) and it would tell me something like item does not exist in database. I tried to make a copy of it and rez it and it didn't rez it AND it didn't notice me that time. Ok, weird. So I decided to just trash the posing stand completely and then grab the copy at my place. Went back to the same sandbox and it rezzed just fine. Hrmmmmmm.

I try to keep my inventory organized as possible and trash things I do not need so I don't think I need to worry too much about a bloated inventory.

I fear much more corruption happening with our inventories as the clock ticks away. Help Linden Labs.

In a way, I suspect you are not far off. My guess is the new inventory database is lacking some much-needed tuning and that, as players either:

- Cross a certain threshold (most likely to happen to long-time players, ironically) for size,
- Log off or change areas before a savepoint,
- Lag out and never trigger a call for save of an item newly added to or modified in inventory

The seive opens and things are just being lost. Unfortunately, we don't know how LL has implemented calls to write or update their database... so we don't know when or where we are most vulnerable.

For example:

- If you knew that every time you put something into your inventory, the system begins writing that item to the database, but, depending upon process queuing, it takes up to 2 minutes to actually do it, you might be less likely to do a lot of things, wouldn't you?

- If you knew that the actual update process doesn't happen upon creation, but only every X seconds or minutes, that would also affect how you do things, wouldn't it?

(Note -- I am not saying this is how they do things. I'm saying these are possible scenarios and posing the question of how knowing it would change how we do things.)

Personally, I think things are written to temp tables before being inserted and something is terribly wrong with how that migration from temp to permanent is happening.

Anyway -- A gen +1 backup rotation isn't that hard to do. For that matter, mirroring temp tables isn't exactly rocket science. I mean, we are living in an age where things like 'high availabilty' are status quo. There is no reason LL cannot restore from backup except that they either:

1) Haven't written the code to do so.
2) Haven't been keeping up with backups as they should.
3) Haven't been verifying backups, and they're all unusable.

Any of these reasons are more frightening for what they say about happenings inside LL than anything. I would hate to hear any of them are true.

After all, anyone of a technical bent knows how excrutiatingly painful it is to perform a backup with verify, or remember to do the backup every night, or code a "select where key =" and "insert where key =" statement.... let alone actually execute it.

(Warning: That last paragraph was laced with a lethal dose of sarcasm.)

Dear Second Lifers,

We have sold our platform to Microsoft. Thanks to all of you for helping us develop this ground breaking technology.


Sincerely, Linden Labs




/sing "We're in the money !" /singoff

Gee Cloudy thanks a lot. Jarod just wet himself and now I have to clean it up.

Cienna's right. The hardest part of automated system backup is keeping quantities of removable storage media around, and deciding for how long.

Executing backup can be automagically scheduled.

The data itself can be put on any number of large-volume media, even in a multi-server system like LL's that probably takes up most of one floor of a building.

The question, however, may not be LL's Disaster Recovery Plan, but rather the issue itself of quirky random, unreproducible inventory bugs that tend to drop bits between your avatar and their server. Given the flukiness of my connection at times (even on cable), I suspect the issue is something in the connection pipeline and handling of packet loss rather than the manner in which backups are performed. The backups are the safety net for *after* a problem surfaces. Meanwhile you have to have a redundant way to cross-check the data as it flows in and make sure you haven't dropped anything on the floor of the server room.

I want to start by thanking Cristiano for bringing this issue up in the forums. I generally don't like exposing LL's problems in "public" like this, but I have to say after the lack of response I am receiving through the normal support channels, I am thoroughly glad I have a friend like Cristiano to expose these sorts of issues more publicly.

I have been missing the majority of my inventory for a few weeks now. I first noticed that many of my top-level folders were missing all of the items that were directly underneath them. Then I found some items still existed in subfolders underneath those, but not always. I have been able to piece together a bunch of my inventory from objects that I had in-world. However, there is no way for me to restore:

Calling Cards - all items under the root folder are gone
Clothing - same story
Landmarks - same story
Sounds - same story
Textures - same story

I don't know when I first started collecting things in my inventory, but it is at least 2 years ago now. I have happily developed game after game for SL and have up until now felt reasonably confident that LL was doing everything they could to protect the time we invest in developing for their platform.

The last couple of weeks have been disheartening. Support has a *policy* of not responding to support requests. If you ever read their auto-response to bug reports, you will notice it says "If any further information is required, Linden Lab will contact you regarding this incident. Otherwise, please consider the matter resolved as no further communications will be sent." This is the most inane support policy I have ever experienced, from any company, ever. To expect people to report problems like this and just "go away" is simply ridiculous. It invites frustration and disappointment, which is exactly the way I feel right about now.

If I can't garner at least some perception that LL is concerned about this issue and working on a solution for me, then why on Earth would I continue to spend my time developing anything in their world? I can't think of a reason.

I'm not giving up hope. I have seen LL pull off many a last-minute save before and am hoping they are able to do so here. I have asked (pleaded) again today for a response to my issue, including Philip, Robin and Concierge support on the TO: line. If my requests for support go unanswered again this week, I have promised myself to tier down to my $0/month state and never develop another game again until such a time as my inventory is restored to an acceptable level.

I'll keep you all updated on my case as it progresses. I thank you all for bringing up and discussing this issue, as it is very important to me, and I think very important to us all.

This was sent to me on a notecard in world:

CURRENT INVENTORY ISSUES

Kona recently asked Live Help to spread the word with the current inventory issues. Linden Lab is moving to new asset servers.

ISSUE ONE: Residents are reporting inventory items missing from database. If residents try to rez or wear items, the Second Life viewer reports an error: "missing from database". Ian Linden is restoring this as speak in batches, so these items will be back soon.

ISSUE TWO: Returned items while offline are not being found when the user logs in.

ISSUE THREE: Double inventory folders.

ISSUE FOUR: Duplicate inventory folders under Lost & Found.

BE SURE TO HAVE THE USER FILE A BUG REPORT, AND FORWARD THEIR NAME ALONG TO KONA LINDEN!

Wow. Finally some encouraging news in one of the most disturbing threads I have ever read here. I lost a few items myself - nothing too traumatic. But reading this thread has forced me to question how much time and effort I put into SL. Let's hope that LL recognizes how fundementally important this issue is to the future success of SecondLife.

A little feedback from LL here in the forums would be nice too.

Persephone Milk

Great work Cris!

The last time they upgraded to new asset servers...never mind. I began to wonder when I read this reply if we'd reached the end of the rope and would simply begin the perpetual tightening, or in more techincal terms, race to upgrade to new servers faster than the ever growing community could generate new inventory items.

They'll never ever ever in a million years manage to keep up with us as long as we have unrestricted access to asset system storage. Our only hope is that our desire to retain our inventory eventually outweighs our selfish notions of unlimited storage.

Client/server synchronization is a noticeable issue in SL and has been so since I arrived. You may well be correct in that much of the issue arises as a result of this. But it is difficult to imagine this is wholly the issue. Not impossible, but difficult.

Judging from the notecard Cristano received (and why are we not all being advised of this??), the asset server is the issue. I suspect however, it is not the only one -- just the most obvious.

Aside -- I would not be adverse to offline backup on the client side, but I think LL would be concerned about security issues. I would also not be adverse to inventory limits or cost for extreme usage (a yet to be determined amount). Frankly, I was amazed to learn that LL charges you for land, but doesn't seem to realise that content/inventory storage likely outstrips the facilities/related costs by a factor of at least three (and that's being conservative).