Second Life Forums Archive - Asset server maintenance

Jacques Groshomme

Registered User

Join date: 16 Mar 2005

Posts: 355

05-01-2007 14:09

I'd like to know some specifics about the asset server system.

* How many total assets are there? Spread over how many physical machines?

* How many of these must be scanned whenever attempting to rez an object or load a resident's inventory?

* Is the sheer size of the asset database a mitigating factor in recent "failure to rez" errors? Or can that error be more closely attributed to the networking subsystem or individual sim performance?

* What automated processes are in place to archive (or otherwise set aside as long as needed) legit objects that haven't been queried in any way shape or form within the last 10, 30, 90, 120 days, so that they aren't negatively contributing to query response time or having an effect on server stability and availability?

* What automated processes are in place to permanently delete "dead" orphaned objects that no longer exist in-world, in an object's contents, or in anybody's inventory, such as:
- the plethora of plywood cubes that have been created and promptly discarded
- iterations of temporary objects, such as individual rezzed bullets of a weapon
- various (since overwritten) saved states of a single notecard
- items existing only in the inventory of deleted accounts

Ayu Sura

Registered User

Join date: 1 Oct 2006

Posts: 67

05-01-2007 15:34

Also, plans for updating the way the asset server is handled were mentioned in an article covering VW this year. Could you please clarify?

Iridium Linden

Wikkid Linden

Join date: 12 Mar 2007

Posts: 262

05-01-2007 19:02

Great questions and comments. Thanks.

Kitty Barnett

Registered User

Join date: 10 May 2006

Posts: 5,586

05-01-2007 21:22

Sorry to hi-jack your thread

.

From: Jacques Groshomme

* Is the sheer size of the asset database a mitigating factor in recent "failure to rez" errors? Or can that error be more closely attributed to the networking subsystem or individual sim performance?

"Failure to rez" seems to be a sim-specific problem. If it's happening on a sim, fly to the next one over and you'll find that everything works perfectly there.

From: someone

* What automated processes are in place to permanently delete "dead" orphaned objects that no longer exist in-world, in an object's contents, or in anybody's inventory, such as

When we still had Linden Answers this got answered across several posts:

Timeline: http://blog.secondlife.com/2006/11/10/inventory-issues-its-not-just-you/

General GC - /139/62/137077/1.html#post1284267

From: Kelly Linden

Here is some insight into how our asset server works. First, the message is misleading. This isn't a database issue, we have an asset system that is entirely separate from the database.

On this asset system we store all assets in Second Life. An asset is any uploaded image, sound or animation. Assets are also any created notecard or script or landmark. Objects that are not currently being simulated (they are in your inventory or the inventory of an object or in a notecard) are also assets. There are quite a few assets, and new ones get created at a fairly good rate (I don't have exact numbers on me).

What we do is we run what is called Garbage Collection on the asset server as a whole. What this does is looks for any asset that is no longer referenced anywhere in our system. This means it isn't in any ones inventory or any objects inventory, it isn't referenced in a script or in a notecard, it isn't part of any object (textures). These assets that aren't referenced anywhere are orphaned - there is no way to ever find them again and no one is using them now. We move these assets aside and mark them for deletion - but we don't actually delete them.

Instead we wait for a while to make sure that they really and truly aren't needed, and that GC didn't screw up.

When GC screws up we get what we are seeing yesterday and today. Assets that are obviously still needed are getting moved aside. We are working on both restoring all the moved aside assets from this GC run and fixing the GC. This is going slower than we had hoped. While this is running the failsafe mentioned above is still running - any asset that can't be found will be caught by the failsafe and restored.

Again, I am sorry for the trouble this is causing everyone. Your data is not lost and will be restored, although you are right to expect this not to happen in the first place.

Restore process - /139/c0/137147/1.html#post1284484

From: someone

Under normal circumstances, after the wait period has passed and no errors were found (no missing asset hits to the fail safe process or reports from residents) then the assets are in fact deleted.

Untameable Wildcat

Cute furry fluffball

Join date: 30 Nov 2006

Posts: 15

Orphaned assets

05-02-2007 10:42

An interesting post from Kelly but it does raise an even more interesting issue.

I had an account which got deleted, but on a sim I visit regularly were a number of things that account had created, which included a teleport system and a fully furnished hut.

When I created the Untameable Wildcat account and went back to the same sim, the items were still there. Some of them are still there today. I'd love to take posession of them again if there was any way to do so, but my point is, those items are still in-world over six months after the account was deleted. They aren't owned by me any more, since I no longer have an account with that name - but they're still there. I have the option, as I did before, to return them from the sim, but I don't know if they'd be caught by garbage collection or not, as the in-world ones clearly haven't.

It seems the garbage collection system is far from perfect.

Meade Paravane

Hedgehog

Join date: 21 Nov 2006

Posts: 4,845

05-02-2007 10:58

From: Untameable Wildcat

It seems the garbage collection system is far from perfect.

That or it's working as-expected..

If a user got deleted, the only assets that would get GC'ed would be ones that existed only in that users inventory. That's the whole point of GC - it only tosses things once every single instance of it is gone.

If one copy is sitting around in somebody's trash, it won't get GC'ed.

_____________________

Tired of shouting clubs and lucky chairs? Vote for llParcelSay!!!
- Go here: http://jira.secondlife.com/browse/SVC-1224
- If you see "if you were logged in.." on the left, click it and log in
- Click the "Vote for it" link on the left

Untameable Wildcat

Cute furry fluffball

Join date: 30 Nov 2006

Posts: 15

05-02-2007 11:04

In world. These things are object/items in world, in this case decorations to a hut - furniture, lights etc.

Since they've been orphaned by the user that owned them being deleted, that leaves two possibilities. Delete them by garbage collection, or offer the land owner the option of taking them over or deleting them. Personally I would prefer the latter, because in the case of the orphaned teleport system it still worked, and worked fine - but the notecards within it couldn't be altered because the user that owned them no longer existed. The owner of the land was faced with having to deal with the items on an 'as-is' basis, or deleting them altogether.

I would genuinely like to see a new option for "take ownership" that land owners can use should an account be deleted.

Meade Paravane

Hedgehog

Join date: 21 Nov 2006

Posts: 4,845

05-02-2007 11:10

I think you may be using the term 'garbage collection' to mean something different than what Kelly was talking about in the quote Kitty made..

Asset server garbage collection doesn't (I think) really have anything at all to do with land management and/or ownership. What Kelly was talking about is just figuring out when things can be totally tossed out of SL.

_____________________

Tired of shouting clubs and lucky chairs? Vote for llParcelSay!!!
- Go here: http://jira.secondlife.com/browse/SVC-1224
- If you see "if you were logged in.." on the left, click it and log in
- Click the "Vote for it" link on the left

Kitty Barnett

Registered User

Join date: 10 May 2006

Posts: 5,586

05-02-2007 13:25

From: Untameable Wildcat

Since they've been orphaned by the user that owned them being deleted, that leaves two possibilities.

I think you're either misunderstanding what orphaned means in this context, or thinking of assets as "belongings" while in this context it's just an internal name.

It might be easier to look at it like this: imagine an immense warehouse where the actual assets are stored, and then a whole stack of cabinets that contain cards with a reference to the location of an asset in the warehouse.

If you upload a texture, the actual texture is stored in the warehouse, and what shows in your inventory is merely a reference. You can give that texture to someone else, and instead of creating a second copy of it, you merely end up with two references to one texture. If that other person creates a prim and applies that texture, it creates a third reference and so on.

If the original uploader's account gets closed and deleted, the texture they uploaded doesn't get deleted, merely the reference to it. In order for the texture to become orphaned it can not be present in a user's inventory, in a prim's inventory, applied to a prim, stored in a notecard, referenced in a script and so on.

What garbage collection does is take stock of everything that's in the warehouse one by one, and see if there's any reference to it anywhere else. If no reference turns up, it gets deleted (or in SL's case, moved out of storage to a purgatory where it's temporarily held in case it wasn't orphaned at all).

As far as the asset system is concerned, whatever your old account has rezzed, is still very much in use and not orphaned at all because the sim it's on still holds a reference to those assets so until those objects are actually deleted, they don't qualify for GC'ing.

Jacques Groshomme

Registered User

Join date: 16 Mar 2005

Posts: 355

05-02-2007 15:53

From: Kitty Barnett

Sorry to hi-jack your thread

.

No problem. This is all in the hopes of establishing clarity.

From: Kitty Barnett

"Failure to rez" seems to be a sim-specific problem. If it's happening on a sim, fly to the next one over and you'll find that everything works perfectly there.

OK. Ammended question for LL: Why is this? Is it a problem with the sim or the network subsystem feeding that sim?

From: Kitty Barnett

All your other quotes

Good info. I'd still like to have a simple breakdown of the current garbage collection process though, if it has changed at all.

And of course have my other questions answered as well

******
=====
******

Ooh, three more related questions:

* What is the average/median size of a resident's inventory?

* Has a tipping point, in regards to the number of inventory items, been found where performance becomes noticeably degraded when trying to return inventory information?

* And more of a policy issue... has it been seriously discussed, and if so is it still on the table, to enforce inventory size restrictions? Perhaps allowing nonverified users X number of items, but allowing premium members X + Y number of items?

Asset server maintenance
Jacques Groshomme Registered User Join date: 16 Mar 2005 Posts: 355	05-01-2007 14:09 I'd like to know some specifics about the asset server system. * How many total assets are there? Spread over how many physical machines? * How many of these must be scanned whenever attempting to rez an object or load a resident's inventory? * Is the sheer size of the asset database a mitigating factor in recent "failure to rez" errors? Or can that error be more closely attributed to the networking subsystem or individual sim performance? * What automated processes are in place to archive (or otherwise set aside as long as needed) legit objects that haven't been queried in any way shape or form within the last 10, 30, 90, 120 days, so that they aren't negatively contributing to query response time or having an effect on server stability and availability? * What automated processes are in place to permanently delete "dead" orphaned objects that no longer exist in-world, in an object's contents, or in anybody's inventory, such as: - the plethora of plywood cubes that have been created and promptly discarded - iterations of temporary objects, such as individual rezzed bullets of a weapon - various (since overwritten) saved states of a single notecard - items existing only in the inventory of deleted accounts
Ayu Sura Registered User Join date: 1 Oct 2006 Posts: 67	05-01-2007 15:34 Also, plans for updating the way the asset server is handled were mentioned in an article covering VW this year. Could you please clarify?
Iridium Linden Wikkid Linden Join date: 12 Mar 2007 Posts: 262	05-01-2007 19:02 Great questions and comments. Thanks.
Kitty Barnett Registered User Join date: 10 May 2006 Posts: 5,586	05-01-2007 21:22 Sorry to hi-jack your thread . From: Jacques Groshomme * Is the sheer size of the asset database a mitigating factor in recent "failure to rez" errors? Or can that error be more closely attributed to the networking subsystem or individual sim performance? "Failure to rez" seems to be a sim-specific problem. If it's happening on a sim, fly to the next one over and you'll find that everything works perfectly there. From: someone * What automated processes are in place to permanently delete "dead" orphaned objects that no longer exist in-world, in an object's contents, or in anybody's inventory, such as When we still had Linden Answers this got answered across several posts: Timeline: http://blog.secondlife.com/2006/11/10/inventory-issues-its-not-just-you/ General GC - /139/62/137077/1.html#post1284267 From: Kelly Linden Here is some insight into how our asset server works. First, the message is misleading. This isn't a database issue, we have an asset system that is entirely separate from the database. On this asset system we store all assets in Second Life. An asset is any uploaded image, sound or animation. Assets are also any created notecard or script or landmark. Objects that are not currently being simulated (they are in your inventory or the inventory of an object or in a notecard) are also assets. There are quite a few assets, and new ones get created at a fairly good rate (I don't have exact numbers on me). What we do is we run what is called Garbage Collection on the asset server as a whole. What this does is looks for any asset that is no longer referenced anywhere in our system. This means it isn't in any ones inventory or any objects inventory, it isn't referenced in a script or in a notecard, it isn't part of any object (textures). These assets that aren't referenced anywhere are orphaned - there is no way to ever find them again and no one is using them now. We move these assets aside and mark them for deletion - but we don't actually delete them. Instead we wait for a while to make sure that they really and truly aren't needed, and that GC didn't screw up. When GC screws up we get what we are seeing yesterday and today. Assets that are obviously still needed are getting moved aside. We are working on both restoring all the moved aside assets from this GC run and fixing the GC. This is going slower than we had hoped. While this is running the failsafe mentioned above is still running - any asset that can't be found will be caught by the failsafe and restored. Again, I am sorry for the trouble this is causing everyone. Your data is not lost and will be restored, although you are right to expect this not to happen in the first place. Restore process - /139/c0/137147/1.html#post1284484 From: someone Under normal circumstances, after the wait period has passed and no errors were found (no missing asset hits to the fail safe process or reports from residents) then the assets are in fact deleted.
Untameable Wildcat Cute furry fluffball Join date: 30 Nov 2006 Posts: 15	Orphaned assets 05-02-2007 10:42 An interesting post from Kelly but it does raise an even more interesting issue. I had an account which got deleted, but on a sim I visit regularly were a number of things that account had created, which included a teleport system and a fully furnished hut. When I created the Untameable Wildcat account and went back to the same sim, the items were still there. Some of them are still there today. I'd love to take posession of them again if there was any way to do so, but my point is, those items are still in-world over six months after the account was deleted. They aren't owned by me any more, since I no longer have an account with that name - but they're still there. I have the option, as I did before, to return them from the sim, but I don't know if they'd be caught by garbage collection or not, as the in-world ones clearly haven't. It seems the garbage collection system is far from perfect.
Meade Paravane Hedgehog Join date: 21 Nov 2006 Posts: 4,845	05-02-2007 10:58 From: Untameable Wildcat It seems the garbage collection system is far from perfect. That or it's working as-expected.. If a user got deleted, the only assets that would get GC'ed would be ones that existed only in that users inventory. That's the whole point of GC - it only tosses things once every single instance of it is gone. If one copy is sitting around in somebody's trash, it won't get GC'ed. _____________________ Tired of shouting clubs and lucky chairs? Vote for llParcelSay!!! - Go here: http://jira.secondlife.com/browse/SVC-1224 - If you see "if you were logged in.." on the left, click it and log in - Click the "Vote for it" link on the left
Untameable Wildcat Cute furry fluffball Join date: 30 Nov 2006 Posts: 15	05-02-2007 11:04 In world. These things are object/items in world, in this case decorations to a hut - furniture, lights etc. Since they've been orphaned by the user that owned them being deleted, that leaves two possibilities. Delete them by garbage collection, or offer the land owner the option of taking them over or deleting them. Personally I would prefer the latter, because in the case of the orphaned teleport system it still worked, and worked fine - but the notecards within it couldn't be altered because the user that owned them no longer existed. The owner of the land was faced with having to deal with the items on an 'as-is' basis, or deleting them altogether. I would genuinely like to see a new option for "take ownership" that land owners can use should an account be deleted.
Meade Paravane Hedgehog Join date: 21 Nov 2006 Posts: 4,845	05-02-2007 11:10 I think you may be using the term 'garbage collection' to mean something different than what Kelly was talking about in the quote Kitty made.. Asset server garbage collection doesn't (I think) really have anything at all to do with land management and/or ownership. What Kelly was talking about is just figuring out when things can be totally tossed out of SL. _____________________ Tired of shouting clubs and lucky chairs? Vote for llParcelSay!!! - Go here: http://jira.secondlife.com/browse/SVC-1224 - If you see "if you were logged in.." on the left, click it and log in - Click the "Vote for it" link on the left
Kitty Barnett Registered User Join date: 10 May 2006 Posts: 5,586	05-02-2007 13:25 From: Untameable Wildcat Since they've been orphaned by the user that owned them being deleted, that leaves two possibilities. I think you're either misunderstanding what orphaned means in this context, or thinking of assets as "belongings" while in this context it's just an internal name. It might be easier to look at it like this: imagine an immense warehouse where the actual assets are stored, and then a whole stack of cabinets that contain cards with a reference to the location of an asset in the warehouse. If you upload a texture, the actual texture is stored in the warehouse, and what shows in your inventory is merely a reference. You can give that texture to someone else, and instead of creating a second copy of it, you merely end up with two references to one texture. If that other person creates a prim and applies that texture, it creates a third reference and so on. If the original uploader's account gets closed and deleted, the texture they uploaded doesn't get deleted, merely the reference to it. In order for the texture to become orphaned it can not be present in a user's inventory, in a prim's inventory, applied to a prim, stored in a notecard, referenced in a script and so on. What garbage collection does is take stock of everything that's in the warehouse one by one, and see if there's any reference to it anywhere else. If no reference turns up, it gets deleted (or in SL's case, moved out of storage to a purgatory where it's temporarily held in case it wasn't orphaned at all). As far as the asset system is concerned, whatever your old account has rezzed, is still very much in use and not orphaned at all because the sim it's on still holds a reference to those assets so until those objects are actually deleted, they don't qualify for GC'ing.
Jacques Groshomme Registered User Join date: 16 Mar 2005 Posts: 355	05-02-2007 15:53 From: Kitty Barnett Sorry to hi-jack your thread . No problem. This is all in the hopes of establishing clarity. From: Kitty Barnett "Failure to rez" seems to be a sim-specific problem. If it's happening on a sim, fly to the next one over and you'll find that everything works perfectly there. OK. Ammended question for LL: Why is this? Is it a problem with the sim or the network subsystem feeding that sim? From: Kitty Barnett All your other quotes Good info. I'd still like to have a simple breakdown of the current garbage collection process though, if it has changed at all. And of course have my other questions answered as well **** ===== **** Ooh, three more related questions: * What is the average/median size of a resident's inventory? * Has a tipping point, in regards to the number of inventory items, been found where performance becomes noticeably degraded when trying to return inventory information? * And more of a policy issue... has it been seriously discussed, and if so is it still on the table, to enforce inventory size restrictions? Perhaps allowing nonverified users X number of items, but allowing premium members X + Y number of items?

Welcome to the Second Life Forums Archive

Asset server maintenance