New asset server problems?

Has anyone else seen a sudden increase in objects that are suddenly 'missing from the database' or just fail silently to rez inworld? I have been receiving reports from customers about products that I sell doing this. In every case when I check the objects, they appear normal, I can move them in inventory, give them to other people, etc. But when I attempt to rez them, nothing happens.

From the information I got from tech support, it seems to be a problem with the asset server (surprise, surprise), and they have been getting 'some reports' through the week about it. In my case it appears that every instance of the object is missing, even if they are currently in use. My master copies, the backups, the ones in vendors, even the ones in the SLX boxes will fail to rez. Even backups on my alts are missing.

It appears that I have potentially thousands of products sold or in my inventory that may have gone silently missing. I am in the process of retrieveing, unpacking and test rezzing every product I make and it seems to be very rare, and random. Just a single part in a box with multiple parts sometimes. Or the container is invalid, while all the contents are still ok.

If you have anything important you may want to check it out and file a ticket if it is missing. In the end all they could tell me was 'file a ticket with as much information as you can'.

Wow that sounds pretty bad. I've seen my scripts coming up blank when I try to edit them but that's been going on for a while. I think I'll check my inventory. Bad news always starts on Friday.

You can sometimes tell when a region is having problems getting at the asset server. Turn on the Statistics Bar (in the View menu or ctrl-alt-shift-1) and look at Pending Downloads. This is the number of things that the sim is waiting to get from the asset server.

If somebody reports something not rezzing, you might want to ask them to look at this number in the stats bar. Smallish numbers aren't a big deal but when you see it going up and up, something wrong and they shouldn't try rezzing anything until it works it's way back down to 0.

This one is not a region problem. It is happening in regions all over to the same items. I am familier with the number you are refering to, and usually when that happens all rezzing starts failing. In my home sim the pending downloads are at zero, and the bad objects still fail, while everything else is fine.

It appears to me that the asset server purged or lost active objects. Some of these are being sold at least once a day, but the ones in the vendors will not rez anymore. I am trying to track down everything that has been affected, but I have literally thousands of different products.

I just ran into a similar situation: I picked up a piece of furniture two days ago and tried it again the next day (it's actually 6 different pieces) and everything but one rezzed.

The one that failed, did so silently, but trying to attach it returned a "Object missing from database" (although only that once, after that even trying to wear it would silently fail).

When it didn't come back on its own for a day I IM'ed the creator and he said he had someone else IM him about the exact same furniture piece tonight. His originals wouldn't rez either, although luckily he had one rezzed so he was able to "Take copy" and the new copy rezzed just fine.

(As far as I know, prims that are actively being simulated - which are rezzed prims on land and actively worn attachments , items in a prim's inventory are assets again though - exist on the sim only, they don't exist in the asset server until taken to inventory. Which would explain the disconnect, if a linkset vanishes completely from the asset server but there's a rezzed copy somewhere, the rezzed copy won't be affected)

---

One item going missing is trouble enough, *all* asset copies going missing is quite creepy . And it doesn't bode well for a few jewelry items I have that went missing (one pair of earrings came in two colours, of course I only have a red left and a blue right left, they wouldn't leave me one matching set, no ) but where I haven't bugged the creator yet .

Yep, that creator was me. Very disturbing. Just before Kitty im'd me I had someone else asking me the same question. I didn't think much of it, until I tried to rez my original item from inventory. It didn't work, 'missing from database'.

That was very odd. I've never encountered that in my years of sl. Luckily I could retrieve the product because I had it displayed in my shop. But what if? Ooooh the horoor...

one of the avatars I own has the same issue.. the left ear is missing from database

but since I had modded it some , my modded copies are still rezzing

But with the number of items out there affected in various places LL is going to have to devote a lot of time to fixing this real quick.

I discovered yesterday that every single instance of a particular product that I have - from the earliest test to the recent update that I did for Havok 4 - had vanished from my inventory. Which was a tad disturbing.

Luckily I always write my scripts offline, and can get hold of the actual object itself to put them back into. But had that not been the case, well.

I must say that I have had asset failures and "script missing from database" before, but this is the first time that every version of something has gone - I wonder how it could _tell_?

Indeed. This one almost sounds like a manual intervention gone horribly awry. Might want to check with Support for this particular case, just to make sure this wasn't an attempt to rid the Grid of your steam-Choppaman cubes.

More generally, I started noticing more of these "missing from database" things happening about two weeks ago, after one particularly nasty bout of asset service sickness. From reports, this round doesn't seem to be associated with the asset creation date, nor how recently it was accessed, so it doesn't seem to be the old problem of garbage collection losing track and harvesting stuff that's still referenced.

Almost motivates me to make off-world copies of my scripts, en route to testing them on Mono. I've always avoided that because, once I have them offline, I'll be tempted to put them into subversion, and before long this will be just like a job (shudders). Hmmm... on the other hand... wonder if anybody has hacked an Emacs lsl-mode...

Oh, it is definitely worth doing; quite apart from asset loss, there are also issues of simple client crashes, and the inworld editor just not being that good compared to a professional product. It also allows one to work on things in peace and quiet without being inworld (or on a machine capable of being inworld).

This _is_ my job, of course, but I started doing it before becoming a full-time Virtual Technologist, and use it for all my current personal stuff now as well.

It would be insane/ill-advised not to have off-world copies of scripts.

I started copy/pasting copies in my early weeks in SL.
I'd put a few hours into my first non-trivial script - logged out and in - went to resume work and -
Loading .............. ing ........ ing .........ing.........ing...........

A lesson that should not really have to be learnt (again) by a techie was learnt.

Quite apart from the backup aspects, I like to make extensive notes.



If I put some heavily parametrised prims together, I make offline notes of those parameters.
I also note any tricky texturing parameters.
This is as much for documenting techniques as it is for backup.

Depending on SL to preserve work is like depending on the hard disk of your PC to preserve work - particularly if the disk is making weird scraping noises, sometimes becomes inaccessible, is red-hot and the source of a burning smell.

I've been wondering about that...

I always got the impression that certain asset types always refer to the same asset for all copies (animations, notecards, textures, etc) but that prims were different and two copies of the same linkset would exist as two distinct assets rather than two references to one asset (unless things get very complex under the hood I can't really see how they can be references since the old "missing from database" happened per instance, not per copy).

One thought I had was that the script asset went missing (the sim would still have its copy as long as it's rezzed I think)) and that might cause the whole object to fail to rez (and it would persist across any copies that use the same script without modification/recompilation since they're references rather than copies). It doesn't account for everything though.

A more scary possibility would be an automated proggie that's roaming through the asset server looking for certain - likely malicious - objects that's flagging entirely innocent objects for deletion by mistake.

I hope they look into it rather than dismiss it as a "clear cache and relog" issue . Having all the OP''s copies vanish in addition to my and another of Paulo's customers chair, including his own copies and Ordinal's work, all in the span of a few days is just too unlikely to be mere coincidence .

You mean that's an indication of a problem??? Hell, I'd better back stuff up - quick!

This problem has been going on for more than a month. About one quarter of my inventory objects have become unrezzable. There is a JIRA issue about it:

https://jira.secondlife.com/browse/SVC-2129

Numerous tickets have been submitted. So far there has been no meaningful response from LL about this.

PLEASE go to the JIRA and vote for #SVC-2129. Anyone having this problem should also submit a support ticket. Then go to live chat and get a Linden to work on it with you. If your ticket is closed without a resolution, reopen it. It is frustrating because they will just tell you to try the usual things which don't work, but the more tickets in the system and the more live chat sessions about this issue, the more likely that it will eventually find its way onto LL's radar screen.

This is a serious issue and LL is either ignoring it or is keeping secret whatever they know about it. We need more people to speak up on the forums, the blogs, and the JIRA to make other users aware. I think many people have this problem but they don't know it yet because it seems to affect only objects that were created a while ago.

I read and voted for the JIRA. That does sound like the same problem I am having. In my case the items missing are doors and windows. Some are part of the contents of a package, in some cases it is the package itself, and the contents may be ok. I had to rebuild some doors, and in the case of the windows I had a 10 pack I was able to replace the missing 5 pack with.

Tech support told me that this had only been reported be a few people over the last week. Their exact words were...

"I know that there have been a couple other reports like yours this week so it is a known issue which means its top priority right now"

Looking at the JIRA it has been ongoing since the first part of April at least.

During the chat I was instructed to get the UUID of the object, and it was all zeros, but when I checked later all of my prim items return that, so I think they were confused, or were trying to confuse me there.

I suspect that they are purged from the asset server because I got the response... 'that's what I was afraid of' at one poiint. Their final instructions to me were as follows.

"ok our techs are going to have to dig into the problem further for you - what I recommend doing is filling out a support ticket with as much information as possible - full name of the item, what happens when you try to rez it, how its also not working for other people, the lack of UUID - and the tech escallation team will be able to go into the asset server file for your inventory to investigate further and hopefully bring it back from a previously saved state"

Almost 24 hours later the support ticket is untouched.

I'm still having problems this morning. I had a customer complain he could not rezz a product he bought from me on SLX yesterday. Curious, I took a copy from my SLX box into inventory then tried to rezz it in several locations and it would not rezz. So I took a copy from one that was on display at my store, and it rezzed fine. Strange considering they are all supposed to be copies of the same inventory item. ??

I would love to see the stats on LL support tickets.

How many
Problem type
Time elapsed before 1st response
Time elapsed before resolution
Was problem actually resolved

I know there are many many more tickets filed about this than are listed on JIRA.

It's just more LL 'tinkering/fondling' with our assets...

To be honest, just another day in the Lab.. the Rats complain, but are not listened to. Who really cares? Not LL, otherwise they would have stabilized this platform years ago.

What happens today, has been happening for the longest time. How can anyone be expected to do any form of business in this platform? Be warned LL, this has GOT to change, focus your efforts on gainning stability at all other costs, before the cost to you, becomes something that YOU cannot live with!

I was gonna work on my script summore.. but I can't open any scripts.. so I guess I'm going to bed.

Yep this ones hit me hard today. Almost all my structures I sell are packed into jack in the box rezzers. Nearly half of my collection now either rezz a few parts or nothing at all. With a pop up saying Unable to create requested object. Object is missing from database.

This could kill my business if not fixed soon as I've sold thousands of these structures.

Even my backups are faulty, I could be looking at losing months of work here

According to the JIRA entry the problem appeares to be resolving?

My items are now appearing, an a few other people are reporting the same.

Maybe they actually found the problem?

Indeed, I just received a resolution of my support ticket telling me to clear my cache and try the objects again. I did that, and every one I have tried so far rezzes! So at least some of the dead objects have been fixed -- after a month of waiting. This is hard to believe. I had pretty much given up hope -- was even seriously thinking of canceling my account. Glad I hung on, but now I'm feeling really insecure about this whole inventory thing. What if something like this happens again? Oh well ...

First thing I did was to clear my cache... it only took 6 hours to reload it... we are still missing some scripts and some do not either open, or work as they should. Just goes to show how frail this platform really is.