Second Life Forums Archive - Robin, something to help the database situation

Jennifer Christensen

Registered User

Join date: 28 Dec 2005

Posts: 112

11-26-2006 12:24

The Linden Blog was not updated, but last night we had yet another case of database going haywire. Perhaps it is overloaded?

Why not implement a tiered system of Inventory? As things stand, a free account can put just as much load on the database as a paying one. Change this!

For instance, a free account would be limited to 250-500 items. A premium would get maybe 10,000 items. A sim owner would get unlimited. A premium who wanted more storage space would pay an extra dollar per month for maybe 5000 more items allowed.

As things stand right now, Linden Lab offers hardly any incentive for a free account to upgrade other than owning land - and a free account could rent if they wished, bypassing this.

Argent Stonecutter

Emergency Mustelid

Join date: 20 Sep 2005

Posts: 20,263

11-26-2006 14:53

How about just giving us a better inventory tool so we can clean up our stuff more effectively?

I used to be ruthless about my inventory, but I've given up... it's too much work.

Psyra Extraordinaire

Corra Nacunda Chieftain

Join date: 24 Jul 2004

Posts: 1,533

11-26-2006 15:03

Here's a Q...

Backing up whole dirs of things into the inventory of single boxes..... a space saver? it seems that the inventory of boxes are loaded when the box is loaded, not every time you see the box.

Just a Q, since I've started backing up dirs into boxes and it'd be nice to know if it's any help to do so or not, cuz it's a long process.

_____________________

E-Mail Psyra at psyralbakor_at_yahoo_dot_com, Visit my Webpage at www.psyra.ca

Visit me in-world at the Avaria sims, in Grendel's Children! ^^

MenuBar Memorial

WaterMoon Artist

Join date: 20 Nov 2005

Posts: 214

Psyra's suggestion

11-27-2006 08:49

It has been established in Linden Answers that Psyra's solution is indeed workable and would decrease the load on the asset server.

And if this does decrease the load by a sufficient amount, it needs to go one step further - would it be so hard for LL to implement a solution like this in the actual interface, instead of us physically making a prim and shoving things into it - perhaps the folders could become boxes of stuff that can be opened from the inventory list.

I've been hoping for a file system in SL where we could double-click a folder and it opens in a separate window - and this might just be the solution, while conserving the load on the asset server at the same time!

If everybody's stuff was in boxes in their inventory, it would remarkably relieve the asset server load - with each person loading perhaps 100 items as opposed to 10,000+ items!

If this is at all possible, it should be something for the developers at LL to be really excited over!

WTG Psyra!

Haravikk Mistral

Registered User

Join date: 8 Oct 2005

Posts: 2,482

11-27-2006 13:21

It seems as though searching of inventory is done client-side, which is nice and all, but really it's backwards as it means a user has to download their WHOLE inventory before it is properly searchable. And how often is search used? I know I use it now and then, but most stuff I want is recent. So mb 70% of the time I use SL I don't need to search to find something I want.

Here's how I'd envision a better system:

Better interface!
The whole reason people search is because the interface as it stands is god-awful, a list? This is supposed to a modern program not a decades old operating system! OS X Column view is a very nice alternative. Or at the least let us open our own tabs and drag over them to 'pop' them open!

Group "the same" items
If an object is rezzed in-world then it should store a note of it's original UUID from inventory (I'd be surprised if they don't already, it seems obvious). When you take it back into your inventory, it'll get a new UUID but keep the old, 'parent', UUID noted in its data.
From here, all items with the same name AND same 'parent' UUID are grouped together, and if ordered by date are ordered by the date of the most recent item. This way if I have fifty historical variations of a particularly nice chair, they are grouped as a single item, if I drag it in-world then I got the most recent variation, otherwise I can expand it (just like a folder) to access past versions.
A tab that displays only 'grouped' items would be nice, as it allows you to then see what you have duplicates of and quickly tidy these up.

Index downloaded items!
When I first log-in, I'll download all my inventory as normal, it is then indexed into a local file. Then when I log-in all I have to do is download inventory items newer than this index file. Every item I find in a search of my index file has the contents of its enclosing folder hashed (or a hash stored in the index is pulled out), if the hash does not match that of the inventory server, then that folder is re-downloaded, so if I search for an item that isn't in the index (e.g I was logged into another machine when I deleted an item so my home index was not updated) then the maximum penalty incurred is downloading that folder again. But for the majority of users the index file will always be up-to-date, thus no downloading of inventory lists is ever required.
So basically:
- All inventory downloaded into a local file which generates 'hash' codes for each folder
- When you log-in these hashes are compared with your inventory server-side, if any don't match the folder is downloaded again
- You now have up-to-date inventory without waiting a year to download it all!

_____________________

Computer (Mac Pro):
2 x Quad Core 3.2ghz Xeon
10gb DDR2 800mhz FB-DIMMS
4 x 750gb, 32mb cache hard-drives (RAID-0/striped)
NVidia GeForce 8800GT (512mb)

Mark Gjellerup

Too Much Gjellerup!

Join date: 20 Mar 2006

Posts: 35

Version Control

11-27-2006 19:15

I have an idea about inventory, and I might as well post it here...
-----------------------------------------------------------------------------------------------

My idea is to create a version control system (or a Workspace) for new objects. Instead of making a new version of an object each time you save or take something back into inventory, why not manage version control on the client?

For example I have hundreds of versions of a HUD I'm working on, because every time I "Take" the object into inventory it creates a new Asset ID. Isn't that unnecessary and wasteful?

Basically, I'd like to see old versions of my HUD stored locally, and only the current version stored on the server, which in the end will mean less objects to index and a faster Asset Server.

Ok, I'm a programmer, so here is how I would implement it... (even though I don't know the database architecture, so I hope this isn't naive)...

From: Implementation (No One Has to Read This Part)

I would create a "Workspace Window" completely different than objects in Inventory. In the Workspace, there would be a Directory Tree Structure for the objects, where you can expand an object to see its contents. An object in the Workspace gets an Asset Server ID (with a flag that says it is a workspace object and a version number). These objects CANNOT be given to someone else. Only the original creator can rez these special objects.

The directory structure of the Workspace is copied on the Client Machine. Each object in the workspace has a serialized version on the client with a folder for its contents, which can contain more serialized objects and LSL scripts (their contents).

So why is this better for the Asset server? Well, each time a person "Takes" an object into their Workspace/Project folder, only the most recent version of each named object is stored on the Asset Server. So something that has a name "Door" in the Contents of the "House" object... each time they take the Door into the Workspace, the SAME Asset ID is used to store the object. No new Asset IDs are created when they take the object into a Workspace. No two objects in a Workspace Folder can have the same name.

What about old versions? They are stored on the client. Each time an object, or a folder is changed (you add, replace, or delete an object), a new version of the whole object or folder is written locally to the hard-drive of the client. I don't think this would be a major security issue because only the owner of the object has the data stored locally on their system (and scripts sent over the wire).

In this system, the client would be managing the version control. So on my local disk, in a workspace folder, you would see folders that contain versions of the serialized objects and their contents:
House v1.000.00.001
House v1.000.00.002
etc.

As for the server, objects on the grid only reflect the latest version (remember only the creator is rezzing these objects so it shouldn't be too much of a problem).

Then I would implement a "Revert to Old Version" option in the Workspace Window. Then no one would care about having multiple versions of the same object. At any time the client could upload an old version from the local hard-drive, and all scripts/objects would be recompiled etc. and then the content-creator would have the old version on the server.

Once a content-creator is done with a project, they just drag the objects (or project folder) out of their Workspace Window, and into their inventory. This makes new Asset IDs. Now the objects can be transferred and sold. Better yet, when a person has a final version done, they can even delete a folder in their workspace... and since you know no one else is rezzing objects that are within this person's Workspace, you can safely permanently delete Asset Server Workspace objects that are deleted by the user without worries.

The Lindens probably already thought of this, right? It's common sense to any programmer who uses an IDE every day. An IDE doesn't store every new version of the software on the Production Server. Only the latest version needs to be on the Production Server. This is the same idea.

I think it's doable, it wouldn't be too much of a pain for content-creators (who might get their hard-drives filled with old versions of their objects) and it would make for a leaner Asset Server with only the objects people need. Am I right, or am I missing something?

_____________________

Buckysoft

Argent Stonecutter

Emergency Mustelid

Join date: 20 Sep 2005

Posts: 20,263

11-27-2006 20:16

From: MenuBar Memorial

And if this does decrease the load by a sufficient amount, it needs to go one step further - would it be so hard for LL to implement a solution like this in the actual interface, instead of us physically making a prim and shoving things into it - perhaps the folders could become boxes of stuff that can be opened from the inventory list.

One of the enhancements I'd like would be "take this folder and turn it into a 'coalesced object'" and "take this coalesced object and turn it into a folder". Where a coalesced object is the thing you get when you pick up a whole build in-world.

Having it do this with boxes would be fine, too.

Being able to look at a box's contents in inventory (open up the box contents like you did an "open" from the pie menu) and move things in and out without rezzing it explicitly in world... that would pretty much solve a LOT of organizational problems I have.

(there are hacks like "open it in your HUD" but you're limited in what you can do to the contents of attachments)

Robin, something to help the database situation
Jennifer Christensen Registered User Join date: 28 Dec 2005 Posts: 112	11-26-2006 12:24 The Linden Blog was not updated, but last night we had yet another case of database going haywire. Perhaps it is overloaded? Why not implement a tiered system of Inventory? As things stand, a free account can put just as much load on the database as a paying one. Change this! For instance, a free account would be limited to 250-500 items. A premium would get maybe 10,000 items. A sim owner would get unlimited. A premium who wanted more storage space would pay an extra dollar per month for maybe 5000 more items allowed. As things stand right now, Linden Lab offers hardly any incentive for a free account to upgrade other than owning land - and a free account could rent if they wished, bypassing this.
Argent Stonecutter Emergency Mustelid Join date: 20 Sep 2005 Posts: 20,263	11-26-2006 14:53 How about just giving us a better inventory tool so we can clean up our stuff more effectively? I used to be ruthless about my inventory, but I've given up... it's too much work.
Psyra Extraordinaire Corra Nacunda Chieftain Join date: 24 Jul 2004 Posts: 1,533	11-26-2006 15:03 Here's a Q... Backing up whole dirs of things into the inventory of single boxes..... a space saver? it seems that the inventory of boxes are loaded when the box is loaded, not every time you see the box. Just a Q, since I've started backing up dirs into boxes and it'd be nice to know if it's any help to do so or not, cuz it's a long process. _____________________ E-Mail Psyra at psyralbakor_at_yahoo_dot_com, Visit my Webpage at www.psyra.ca Visit me in-world at the Avaria sims, in Grendel's Children! ^^
MenuBar Memorial WaterMoon Artist Join date: 20 Nov 2005 Posts: 214	Psyra's suggestion 11-27-2006 08:49 It has been established in Linden Answers that Psyra's solution is indeed workable and would decrease the load on the asset server. And if this does decrease the load by a sufficient amount, it needs to go one step further - would it be so hard for LL to implement a solution like this in the actual interface, instead of us physically making a prim and shoving things into it - perhaps the folders could become boxes of stuff that can be opened from the inventory list. I've been hoping for a file system in SL where we could double-click a folder and it opens in a separate window - and this might just be the solution, while conserving the load on the asset server at the same time! If everybody's stuff was in boxes in their inventory, it would remarkably relieve the asset server load - with each person loading perhaps 100 items as opposed to 10,000+ items! If this is at all possible, it should be something for the developers at LL to be really excited over! WTG Psyra!
Haravikk Mistral Registered User Join date: 8 Oct 2005 Posts: 2,482	11-27-2006 13:21 It seems as though searching of inventory is done client-side, which is nice and all, but really it's backwards as it means a user has to download their WHOLE inventory before it is properly searchable. And how often is search used? I know I use it now and then, but most stuff I want is recent. So mb 70% of the time I use SL I don't need to search to find something I want. Here's how I'd envision a better system: Better interface! The whole reason people search is because the interface as it stands is god-awful, a list? This is supposed to a modern program not a decades old operating system! OS X Column view is a very nice alternative. Or at the least let us open our own tabs and drag over them to 'pop' them open! Group "the same" items If an object is rezzed in-world then it should store a note of it's original UUID from inventory (I'd be surprised if they don't already, it seems obvious). When you take it back into your inventory, it'll get a new UUID but keep the old, 'parent', UUID noted in its data. From here, all items with the same name AND same 'parent' UUID are grouped together, and if ordered by date are ordered by the date of the most recent item. This way if I have fifty historical variations of a particularly nice chair, they are grouped as a single item, if I drag it in-world then I got the most recent variation, otherwise I can expand it (just like a folder) to access past versions. A tab that displays only 'grouped' items would be nice, as it allows you to then see what you have duplicates of and quickly tidy these up. Index downloaded items! When I first log-in, I'll download all my inventory as normal, it is then indexed into a local file. Then when I log-in all I have to do is download inventory items newer than this index file. Every item I find in a search of my index file has the contents of its enclosing folder hashed (or a hash stored in the index is pulled out), if the hash does not match that of the inventory server, then that folder is re-downloaded, so if I search for an item that isn't in the index (e.g I was logged into another machine when I deleted an item so my home index was not updated) then the maximum penalty incurred is downloading that folder again. But for the majority of users the index file will always be up-to-date, thus no downloading of inventory lists is ever required. So basically: - All inventory downloaded into a local file which generates 'hash' codes for each folder - When you log-in these hashes are compared with your inventory server-side, if any don't match the folder is downloaded again - You now have up-to-date inventory without waiting a year to download it all! _____________________ Computer (Mac Pro): 2 x Quad Core 3.2ghz Xeon 10gb DDR2 800mhz FB-DIMMS 4 x 750gb, 32mb cache hard-drives (RAID-0/striped) NVidia GeForce 8800GT (512mb)
Mark Gjellerup Too Much Gjellerup! Join date: 20 Mar 2006 Posts: 35	Version Control 11-27-2006 19:15 I have an idea about inventory, and I might as well post it here... ----------------------------------------------------------------------------------------------- My idea is to create a version control system (or a Workspace) for new objects. Instead of making a new version of an object each time you save or take something back into inventory, why not manage version control on the client? For example I have hundreds of versions of a HUD I'm working on, because every time I "Take" the object into inventory it creates a new Asset ID. Isn't that unnecessary and wasteful? Basically, I'd like to see old versions of my HUD stored locally, and only the current version stored on the server, which in the end will mean less objects to index and a faster Asset Server. Ok, I'm a programmer, so here is how I would implement it... (even though I don't know the database architecture, so I hope this isn't naive)... From: Implementation (No One Has to Read This Part) I would create a "Workspace Window" completely different than objects in Inventory. In the Workspace, there would be a Directory Tree Structure for the objects, where you can expand an object to see its contents. An object in the Workspace gets an Asset Server ID (with a flag that says it is a workspace object and a version number). These objects CANNOT be given to someone else. Only the original creator can rez these special objects. The directory structure of the Workspace is copied on the Client Machine. Each object in the workspace has a serialized version on the client with a folder for its contents, which can contain more serialized objects and LSL scripts (their contents). So why is this better for the Asset server? Well, each time a person "Takes" an object into their Workspace/Project folder, only the most recent version of each named object is stored on the Asset Server. So something that has a name "Door" in the Contents of the "House" object... each time they take the Door into the Workspace, the SAME Asset ID is used to store the object. No new Asset IDs are created when they take the object into a Workspace. No two objects in a Workspace Folder can have the same name. What about old versions? They are stored on the client. Each time an object, or a folder is changed (you add, replace, or delete an object), a new version of the whole object or folder is written locally to the hard-drive of the client. I don't think this would be a major security issue because only the owner of the object has the data stored locally on their system (and scripts sent over the wire). In this system, the client would be managing the version control. So on my local disk, in a workspace folder, you would see folders that contain versions of the serialized objects and their contents: House v1.000.00.001 House v1.000.00.002 etc. As for the server, objects on the grid only reflect the latest version (remember only the creator is rezzing these objects so it shouldn't be too much of a problem). Then I would implement a "Revert to Old Version" option in the Workspace Window. Then no one would care about having multiple versions of the same object. At any time the client could upload an old version from the local hard-drive, and all scripts/objects would be recompiled etc. and then the content-creator would have the old version on the server. Once a content-creator is done with a project, they just drag the objects (or project folder) out of their Workspace Window, and into their inventory. This makes new Asset IDs. Now the objects can be transferred and sold. Better yet, when a person has a final version done, they can even delete a folder in their workspace... and since you know no one else is rezzing objects that are within this person's Workspace, you can safely permanently delete Asset Server Workspace objects that are deleted by the user without worries. The Lindens probably already thought of this, right? It's common sense to any programmer who uses an IDE every day. An IDE doesn't store every new version of the software on the Production Server. Only the latest version needs to be on the Production Server. This is the same idea. I think it's doable, it wouldn't be too much of a pain for content-creators (who might get their hard-drives filled with old versions of their objects) and it would make for a leaner Asset Server with only the objects people need. Am I right, or am I missing something? _____________________ Buckysoft
Argent Stonecutter Emergency Mustelid Join date: 20 Sep 2005 Posts: 20,263	11-27-2006 20:16 From: MenuBar Memorial And if this does decrease the load by a sufficient amount, it needs to go one step further - would it be so hard for LL to implement a solution like this in the actual interface, instead of us physically making a prim and shoving things into it - perhaps the folders could become boxes of stuff that can be opened from the inventory list. One of the enhancements I'd like would be "take this folder and turn it into a 'coalesced object'" and "take this coalesced object and turn it into a folder". Where a coalesced object is the thing you get when you pick up a whole build in-world. Having it do this with boxes would be fine, too. Being able to look at a box's contents in inventory (open up the box contents like you did an "open" from the pie menu) and move things in and out without rezzing it explicitly in world... that would pretty much solve a LOT of organizational problems I have. (there are hacks like "open it in your HUD" but you're limited in what you can do to the contents of attachments)

Welcome to the Second Life Forums Archive

Robin, something to help the database situation