A few words about databases...

I don't read/post here too much, but I thought you all might be interested in the following. A friend and I had been talking, and he asked that I send a note in to Phillip... I gather he and Philip are at least acquaintances.

I did so, and It's been several days and I haven't gotten any kind of acknowledgement. I don't want it to drop into a black hole, so I thought I'd post it here. I removed the name of the SLer involved, since these words are all mine anyway.

***************************************
[An acquaintance] gave me your email address...


[removed] asked that I try to write up some of what he and I had been talking about earlier. I'm a sysadmin from way back, and rightly or wrongly, it appears I've been nominated to try to verbalize these issues.

As I was saying to him, I think perhaps you don't realize what business you're in. You've probably read that book about disruptive technologies and the sad fate of the buggy whip makers: they didn't realize their real business. They thought they were buggy whip makers, which put them out of business when carriages didn't need horses anymore. They were actually in the business of transportation acceleration, and any of them that figured this out at least had a chance to survive.

I suspect that you believe you're in the game business. You are, but only in one sense. In a broader sense, you're in the market of selling access to a shared database. In your particular case, that database happens to be one of primarily 3D objects, and is mostly used for entertainment. Ultimately, that's almost irrelevant; the fundamental fact is that SL is all data. The entire "world", in fact, is data. And unless you realize that that data is THE CORNERSTONE of your business, you will, I believe, eventually fail.

I'm not disparaging the amazing work your programmers have done. It is, by any measure, brilliant stuff. But ultimately that's just the access layer to what really matters, the data. That user-created content is why people keep coming back.

You are IN THE DATABASE BUSINESS. Not in the sense of selling database software, but in the sense of selling access to a database. And your crown jewels, the reason you exist, appear to be stored on systems that are slapped together with strings, bailing wire, and duct tape.

Now, that was probably critical to getting SL off the ground.... you had to prove it was a viable concept without spending the kind of money that serious databases require. But I think it's been proved at this point, and now that choice is no longer a good one. Many (most?) of the bugs in the Second Life system appear to be database related. The hardware/software/administration setup you have is entirely inadequate, by at least an order of magnitude. Consider: after a simple reboot of the system last week, the databases were so overtaxed that it took HOURS and HOURS for the systems that serve textures to even come CLOSE to catching up. It almost didn't happen at all.... your system was only JUST BARELY fast enough. In another couple of months, you may not survive a simple reboot without doing something drastic like bringing up the world in pieces, or rationing logins for several hours.

Over and over and over, the databases are failing. What prompted my specific conversation with [removed] was that my hat was lost in a DB crash. (my avatar is a little guy in a top hat). Now, admittedly, it wasn't that complex an object and it didn't take me that long to recreate, but it had a great deal of sentimental value to me, and it should NEVER HAVE HAPPENED. I spoke with Kona at some length, and I was told that there is simply no way to get back data deleted in this way... something bad happened in the sim software when I changed outfits, and everything I had been wearing was gone forever. That is NOT ACCEPTABLE. This stuff COSTS users MONEY. This data has VALUE, and it needs a great deal more integrity than it has.

You are dealing with data on a scale that few companies ever approach, and even without knowing the details of how you do it, it's quite obvious that your admin staff is underskilled, your hardware is underfunded, or both. I'm not trying to blame anyone or shake my finger at you for getting into this state, but it IS true. You need to find at least one BRILLIANT database guy/gal, like the senior staff working for Amazon or EBay or a big bank. And you need SERIOUS hardware.

I am of the very strong opinion that many, possibly most, of the bugs that we see as users are database related. I completely applaud your emphasis on fixing bugs in 1.6, that's exactly where you need to be going. But fixing the databases is, I believe, a central part of that. You CANNOT build a big structure on quicksand, and that is what you are trying to do. You need a rock-solid, SERIOUS foundation to build on; you need to be able to absolutely predict transaction times and throughput. You need to be able to have 'contracts' with your databases, where you KNOW what the limits are precisely, and can design around them. You're nowhere near that state yet.

Part of what got you into this problem is the prim limits. I believe that this may have fundamentally been the wrong decision. Prims are constantly scarce, there are never enough. So a very great deal of ingenuity goes into using textures instead of prims; one-prim tables are a pretty common example.

The reason this was wrongheaded is simply that prims are incredibly efficient to transfer, maybe 100 bytes each. But textures are huge and slow. A five-prim table in a basic wood texture is going to be 5 or 6k, and only about 500 bytes of that will be the actual prims. A one-prim table is going to be on the order of 50k, possibly more. You have only one prim, saving 400ish bytes, but then you have either three or five textures... and high quality textures are often 30k and up.

In other words, you shift away load from the very fast graphic cards which have insane triangle and fill rates, and load down your bandwidth, your databases, and the texture memory on the cards instead, all of which are both much more limited. It sure looks to me like you shifted load from a strong point to a weak point.

So would undoing this limit fix things? I don't know. It may be too late. Look at the load that the city sims create versus the regular ones. One idea, which depends on whether your internal architecture is set up how I think it is, would be to let people have 5x as many prims, but only a certain number of SCRIPTED ones.

It may be too late, you may have already focused people too strongly on textures instead of prims. If that's the case, then serious money for the database and BRILLIANT admin staff would appear to be the only way to dig out of the hole.

The databases simply aren't trustworthy anymore, and that is, in my opinion, your business. Your core competency, more than any other thing, needs to be managing that data. It is your heart, your soul, your ENTIRE PRODUCT, and I don't think you're treating it that way.

Thanks for listening. I'll forward a copy to [removed] too. He might have something to add.

Although the 15k limit on prims is a client side issue. I always wondered why they dont have there own database system.

Hey Trimming,

I did also receive your message and I forwarded it on to the people who should read it.

My apologies for not acknowledging receipt of your message.

I'm sorry if it went to the wrong place, I was told explicitly to send it to you. My apologies.

I used to work for a large national ISP and our sysadmins estimated it would take several hours to restart all of our services and get them back up to capacity in the event a catastrophic power failure caused them to have to shut everything down. In fact, that theory was very nearly confirmed one day, when a truck took out a power pole outside the building. The backup generator fired up alright, but the automatic transfer switch failed! We came so close to running out of battery power that the SAs were starting to shut everything down. One of the engineers managed to bypass the ATS at the last minute, averting catastrophe.

The point of this little story: It should not be taken as a point of weakness that the grid can take several hours to restart. Any product this large and complex is going to have a ton of background stuff that has to be up and running, and you just don't blitz the foundation stuff by flipping the switch on several hundred sims all at once. Certain services have to be started and allowed to come up all the way before certain other services.

Huns, the system has to be good enough to take a restart. All the other MMORPGs I'm aware of can successfully start up without user glitches.

If the database can't be restarted and then provide normal services to users logging in, then it's not powerful enough.

Any initial startup requirement should be scheduled into the initial downtime. Once the system is 'up' and all processes are running, then let users log in. If, at that point, the system drowns because it can't handle all the logins, then it needs an upgrade, pure and simple.

You can weasel all you like, but it doesn't change the fact that the system isn't powerful enough to handle a routine function.

Trimming: I totaly agree with you on about needing a solid foundation.
I do not believe it is textures that are the only bottle neck. Before i get into explaining the bottle necks i would like to try and show that if anything prim allocation is a good thing but not the best solution.

Ultimately it's the quantity of assets any sim has to deal with that generates lag. Eliminating assets should speed things up.

The best solution is complex mess with a single texture.

Take a prim table. A single prim table has up to 9 texturable faces; resulting in up to 10 assets needing to be downloaded. A 20 prim table has up to 180 texturable faces with as many as 200 assets needing to be downloaded. Using more prims encourages the use of more textures by allowing for more different textures.
If you could express the same level of detail of the table in the same space as two or three textures ultimately i believe the reduced texture & object load would result in a reduced number of assets required for download.

--------------

Responsabilities:

1. At Sim Boot
   1. Assets download
      1. Objects
         
      2. Textures
         
      3. Scripts
         
      4. Sounds
         
      5. Others?
         
      
      
   2. Physics
      
   3. Scripts
      
   
   
2. Regular Operation
   1. Scripts
      
   2. Chat
      
   3. Physics
      
   4. Asset download
      1. From adjasent sims
         
      2. From the asset server
         
      
      
   5. Asset Creation & Updates
      
   6. Client Serving
      1. Asset Handling (avatars, textures, animations, objects, etc)
         1. Serving Assets
            
         2. Determining which assets to serve
            
         
         
      2. Asset Updates
         
      3. Agent Updates
         
      4. Chat
         
      
      
   
   

Bottlenecks:

1. Too many agents - requires too many agent updates
   
2. Too many listens - requires extra cpu time to handle chat
   
3. Too many object - requires complex optimized asset serving.
   
4. Too many physical objects - Physics engine can't handle the number.
   

The net effect of lag is that it effects client serving.

--------------

base assets - objects, avatars, sounds & animations.

Shopping malls and the city sims are the worst when it comes to object generated lag. If you look at these places you will see that the textures in use are all very different. Not because they are trying to get around the prim limit but because they are trying to display many unique items. The fact is these areas are full of unique items that require unique textures.

I do not believe the bottleneck is across the board for Asset Handling. I believe the real bottle neck is with determining which assets to serve. In high density places this can saturate the asset serving.

One possible solution to the high density problem is to have free floating asset servers, taking the bulk of the responsability of serving assets away from the sim. The sim could tell the the server which base assets need to be served to which avatars and have this server actualy deal with the serving. This would allow for the sim to only need to every download objects and scripts.

Ultimately the solution that will yeild the best result will be the one that reduces the number of assets.

-------------------

As to other MMORPG's they lack the shear amount of dynamic data that SL has.

I was answering your assertion that it is unreasonable for the grid to take several hours to come up, which is a separate issue from the problems people have once the grid is open. So let's discuss the issues one at a time, shall we? The discussion will move much more smoothly if the points being discussed do not become convoluted from one post to the next.

How many of those MMORPGs are streaming the entire world to their players, rather than having them download every possible asset beforehand? The point I addressed is that you seem to think it shouldn't take several hours to bring the grid up. Several hours seems perfectly reasonable to me. Not necessarily desirable, but I can understand it. When a sim comes up, it downloads every asset it contains, and caches it locally. (This was revealed earlier in the year by a Linden.) Now, imagine what the load would be like on the DB server if they brought 300 sims online, all at once. It would get blitzed with millions of requests, resulting in tens of millions of rows returned, over the space of a few minutes. Early this year, a Linden indicated that the world had a little over 100GB of assets. What is it now? A terabyte? More? The majority of that would have to be sent as well.

How reasonable is it to expect them to build a database server that can do all that, in less than an hour, when it only has to be powerful enough to sustain a full grid restart a few times a year? Seems like overkill to me.

I will agree that the current state of the asset server is not optimal. We already knew LL was going to upgrade that before you posted this thread. They are also going to make some changes to key data structures in order to attack the border crossing issues. More info is in this transcript of Cory Linden's recent Town Hall.

just FYI the grid is around 530 sims now...

very well stated. some of us have long argued that linden lab's primary business is tantamount to 3d web hosting. the dream is that one day they'll provide open client code, free guest accounts and a service guarantee. as websites and web stores migrate to the grid, the money will come rolling in like they've never imagined.

but then they'll have to maintain a steady service. that'll require a consistant database system across the board. but that means starting at the beginning. things such as group affiliations have to be normalized into their own table. storage units such as notecards must retain their key when their content field is updated. these types of changes require rewriting a lot of code. are ll gonna do that? i dunno.

i agree wholly that most of our problems are asset related. they must be. i also agree that we should have limited scripting cycles, texture's/sm, and inventory entries. prim limits worked to sell more land and force tier increases. it promoted rapid growth. but from a hosting perspective, it's totally bogus to limit the number of prims i can put in a sim and not limit the amount of scripting memory & cycles i can command or the number of textures i can display.

oh the creative geniuses have all threatened to leave if these things come to pass you understand. then we'll only be left with the people, businesses, schools and government offices that want to use the technology as a viable commerical and information delivery platform. what a dismal world that'll be.

Just a random idea, but what if they used some sort of p2p network to distribute assets? There would still be one mane asset server, who would have athority, and the final word on what key gose to what object, and what that object contains.

Trimming,

I wanted to comment on your letter, coming from a slightly different perspective than the others who have responded so far. Unlike many people here, my background and professional expertise lies not in technology (although I am fairly knowledgeable on the subject) but in interpersonal communication. In this respect, I feel that perhaps I can be of some benefit to you here. Just as you wanted Linden Labs to take your suggestions into consideration, I hope you will consider mine.

Let me start out by saying that while your letter was well written and it brought up some very good (if sometimes speculative) points, I must tell you that you made a crucial mistake right from the start in how you chose to structure your message. This may possibly be one reason why you didn't get any initial response. Allow me to explain.

You began the letter by telling Phillip Rosedale that he doesn't understand his own business. While I’m sure that your intent was not to show any disrespect, I must tell you that that kind of statement is INCREDIBLY insulting. This is a man with an immense track record of success who has been pioneering technologies for over a decade, and you more or less accused him of being inept. Again, I doubt that insult was the message you had intended to send, but I’d be willing to be that that was how it was received. Having spent 10 years in the corporate world, and a business owner myself, I can assure you that no CEO in the world would be able to read a statement like that and not be offended, especially one who has as many revolutionary accomplishments under his belt as someone like Phillip.

Your suggestions were good, but no matter how good or bad they might have been, no CEO is likely to give them the time of day unless you present them in a positive and, above all, respectful manner. Had you begun your letter with something to the effect of, "Hey, you guys are great, but let me tell you where I think you could do even better..." I can just about guarantee you that your reader would at least have continued to read far enough to get the gist of your suggestions, something he likely would not do after feeling insulted.

Understand that the average CEO works upwards of about 100 or more hours per week, which means when and if you are lucky enough to get one's attention, you absolutely must demonstrate a compelling reason to hold that attention right off the bat. Otherwise, they will invariably move on to other more pressing matters. Obviously, accusing the head of an organization of not understanding his own business is no way to keep his attention.

So, how do you demonstrate a compelling reason to hold attention? Well, there is something in communications called "the sandwich approach". This very simple formula is pretty much always the best way to present criticism. It goes, quite simply, compliment - criticize - compliment. In other words, your start and your ending are both designed to make the prospect feel good, while anything that could possibly make him feel bad gets "sandwiched" into the middle. This is how our brains are wired to constructively process criticism, so this is how you must present it if you want your reader to absorb what you have to say.

The reasoning here is simple. Starting out with a compliment puts the reader in a good mood and demonstrates that you are someone who should be listened to. After all, you as the writer must know what you are talking about if you had the good sense to know that this particular reader is deserving of praise. Now your reader is encouraged to hear you out, so you move on to the "meat" of your letter, the criticism. This is the part where you run the risk of insult, so it is therefore crucial that as soon as the criticism is done, you end on positive note with another compliment, one which reaffirms your belief in the greatness of the reader, and in his furthered greatness should he listen to your suggestions. That's how it's done.

You on the other hand took the exact opposite approach. You started with an insult by accusing your reader, a successful CEO who has accomplished a great many things, of not understanding his own business. Then you made matters even worse by suggesting he is going to fail. By the time you got to your proposals (assuming he even read that far), he was probably already so convinced that you don't know what your talking about it's doubtful he was in any shape to take them in. Finally, you nailed the coffin shut with the ultimate pessimistic ending, "It may already be too late."

Trimming, you are an intelligent, articulate, and knowledgeable person, and your suggestions were good. However, I can't help but feel they are going to end up falling on deaf ears due to the manner in which they were presented. I hope you will take what I’ve had to say to heart. Your ideas have a lot of potential, and if properly presented, I think they could make a lot of difference.


-Chosen

That was an excellent post, I learned something from it. Your post was structured the same way you suggested in your post a post should be structured. A+ for tact.

> Well, there is something in communications called "the sandwich approach". This very simple formula is pretty much always the best way to present criticism. It goes, quite simply, compliment - criticize - compliment.

Hmmm, they should teach this stuff in school! Whilst this tip is something I'd sortof felt to some extent with time, it's not something I'd ever verbalized explicitly and clearly. Do you have any good references with more similar tips like this?

Azelda

Nah.

Philip is not going to feel threatened by a techy trying to write an email. He'll laugh off the silly parts and focus in on the meat.

Unfortunately

a) there can't be any meat, because nobody knows what LL is doing
b) LL has already stated several times that the database is their core focus, and back end data fixes are what they are working on right now

Didn't you read release notes 1.5.11?

Thanks for the positive feedback, Azelda and Strafe. You're absolutely right that they should teach this stuff in school, Azelda. I've been saying that for years. The world would be a much better place if everyone were educated in how to communicate with other human beings.

To answer your question about reference material, someday maybe I'll write a book or two on the subject, but in the mean time here are a couple of my favorites:

The Seven Habits of Highly Effective People by Stephen Covey
Secrets of Closing a Sale by Zig Zigglar

Don't let the titles fool you. These may sound like business books, but they are applicable to far more than that. Everything in life revolves around how we communicate with eachother and ourselves. Family, work, education, friendships, concepts of self, everything.

I totally agree with your take on the importance of communication. I've taken a few courses in communication as well, and found it well worth the time and money.

One thing about the sandwich approach... It may seem to the uninitiated or the cynical that the point is to trick the person into reading the meat of the message by buttering him up with false praise. I think the point is actually to create a rapport and a foundation for agreement. By spending time considering what you can acknowledge the reader/listener for, you can gain insight into their perspective and so can be more effective speaking with them. This is based on the idea that the most important part of communication is listening.

Effective communication is based on honesty, not deceit or manipulation.

By the way, it's quite refreshing to see a forum comment that is so thoughtful and positive. Thanks for taking the time to write that.

uh you guys! thanks a lot.

you successfully derailed the point, this time, in under ten posts; now i owe jarod fifty bucks! serious talk of developing this software as anything other than a toy is never going to go anywhere is it?

Well, I thought the original post covered the point quite nicely. What more is there to say that hasn't been said already?

Sorry, I didn't realize there were all the replies here... for whatever reason, the forums didn't email me any notifications after the first.

I just thought I'd point out that today's problems completely prove my point. A SINGLE DRIVE FAILURE took down all of Second Life. For NINE HOURS. At least! It may be even longer.

I was commenting to some friends last night that LL appears to have no bloody idea what the they're doing with databases. I said something to the effect that "christ, they're probably trying to run all the millions and millions of dollars worth of data through MySQL." And, sure enough, from Cory this afternoon:



No bloody WONDER SL is constantly borked. They're so incompetent at their database management that they don't even REALIZE they're incompetent. And they definitely don't appear to realize that the databases are the ABSOLUTE CORE of their business.

Wake up, Lindens! Either get serious about your databases, or someone will come along who IS. You've proved that this concept works, and can make money.. now someone can come along and do it better. You've done an awful lot right here, but if you keep doing this one thing wrong, it will eventually kill you.

I, personally, am very irked that they're wasting all that time and effort and money on Teen Life when they can't get Adult Life working. I'm thinking very seriously about tiering down to 'free' level. Since they haven't bothered to answer my letter, I don't know how else to express my dissatisfaction.

Well, if they do that, I hope they keep the toy. I like the toy. That's why I'm here.

There are plenty of pieces of software that can be used for serious business. And they're popping up all the time. It won't stop, 'cause there's so much money in it.

If the idea is to use the other uses of SL-tech to subsidize SL itself, well, that's cool.

But I didn't join so I could have a super-immersive Net Meeting tool. I joined so I could have a Second Life. The business side of things, for me, is a dependency of that, a way of integrating myself deeper into it. And I think the same can be said for the vast majority of users here.

And, if I am not reading him wrong, I think the same can be said for Philip. It seems to me that he really enjoys "the toy" as well.

Not to disparage the idea of taking this technology deeper into the "real" world, I think you have made a lot of good points on these forums, Khamon, I'm just saying that the toy IS the reason for everything else, not the other way around.

Trimming -

Your criticism and view of the problem is overly simplistic. Not only was there a critical drive failure, but it happened shortly before a big upgrade / update was planned to the system that was affected.

I was ON when the original crash happened; and the system was only down for a very short time. It was many hours later, when they decided to try to move up the time-tables on the new / updated system installation, that things went to hell-in-a-handbasket. If you want to criticize them for over-reaching when under duress, and being optimistic that a new system would fix their problem, so be it... But insulting and calling them incompetent when you don't even know all of the details (hell, NONE of us do) is just irresponsible. They're taking a ton of time from their vacation to go BACK to Work, so that the rest of us can have fun on our vacation.

It is in LL's best interest to have the system up as much as possible, to attract the most customers. They WANT and NEED business from folks. So all of this crap about them being incompetent, or sitting around not trying, is just impatient B.S. They have THOUSANDS of customers, and only 30 or so employees to deal with us all - they can't pay individual attention to each user. They also have to prioritize problems and issues - sometimes making tough calls and settling for trade-offs.

As with any new system (and yes, at 1 year old SL is *still new*), there are going to be some mistakes and some learning-curves. Everquest wasn't nearly as complicated; and it had a bunch of outtages and problems during its first year! And how about Anarchy Online? I was in on the Beta-test and the first couple of months of commercial ops; & there were HUGE bugs and exploits that weren't fixed for over a year - as well as major system outtages.

So please, think before you complain / insult - and remember that its only entertainment! If you want to, there are plenty of other ways to spend your Real Life time having fun; instead of raising your blood-pressure & stress-levels here.

Take care,

--Noel "HB" Wade
(Tread Whiplash)

oh

well

okay. i'll just focus on playing the game.

Be creative, people. Direct your anger where it makes a difference.

Grief about Microsoft's Windows

What's wrong with MySQL? It can take a hell of a load, with the proper machines and resources. I have heard of a European supermarket that stores all it's customer loyalty card data on MySQL, millions of records. I have also worked for some pretty big companies using MySQL for their database, and they are fine.

If there are database *issues*, I don't think it stems from the fact the database is MySQL.