A transcript of a chat with Philip Linden....

This is a transcript of a chat I had with Philip Linden. Kate Hanks, Jonquille Noir, and I were there. Siggy Romulus was indirectly there, looking over Jonquille's shoulder. Both Kate and Jonq thought they didn't contribute anything to the conversation and asked to be removed. I have pulled many early lines from them, but they both said some intelligent things that took the conversation in interesting directions, so I didn't feel I could remove them completely. Anyplace I took something out, I put a [snip] in. Other than that, this is unedited.... no typo fixing, exactly as it transpired.


Philip Linden: hi there!
Kate Hanks: welcome Phil, thank you for this
Philip Linden: OK now I see the top hat
Kate Hanks: yay!
Philip Linden:
Philip Linden: OK let's talk databases.
You: ahem, the SECOND top hat, the original was lost in a database crash, which is what set off my original rant
Philip Linden: I know.
Philip Linden: I agree that content loss is unacceptable.
Philip Linden: I can try and justify how we designed SL to now allow it,
Philip Linden: where we are today, etc.
You: all I really ultimately have to say is: you are in the business of selling access to a shared database, and that database is your reason for being, and I don't think you're treating it that way.
Philip Linden: Well let me a add a couple things...
Philip Linden: I 100% agree we need to protect every single piece of content.
Philip Linden: but let me give some more data on how things work...
Philip Linden: what sort of predicament we have here.,
Philip Linden: First,
You: and I don't think the new server is reliable, either, the reboot the other night and now people complaining about a lot of corruption
Philip Linden: we are not in the game business.
Philip Linden: we are in the digital world business.
Philip Linden: and I agree that is a database problem in many ways.
Philip Linden: but a kind of database the world has never seen before.
You: ultimately all a digital world IS, is data
Philip Linden: The objects in SL, like your hat...
Philip Linden: are actually not stored in a database.
Philip Linden: your INVENTORY is stored in a dabase,
Philip Linden: but the objects in-world are not.
Philip Linden: let me explain why.
You: files?
Philip Linden: well not really...
Philip Linden: something far more sophisticated.
You: I'm a sysadmin, though I'm not a database specialist by any means
Philip Linden: the objects that are resident in a sim,
Philip Linden: they are constantly being interacted with.
Philip Linden: so for this reason,
Philip Linden: we had to design something new.
Philip Linden: the 'transaction rate' that SL would be generating
Philip Linden: if it were a central DB,
You: I've often thought about that, actually, that the traffic would be unbelievable
You: that you'd have a database that would look like the federal government's.
Philip Linden: would already be FAR greater than any system in the world could handle.
Philip Linden: so instead what we have is effectively
Philip Linden: 500 parallel 'databases' that hold their objects in RAM
Philip Linden: so that they can be interacted with very fast.
Philip Linden: now...
Philip Linden: there are some databases in SL...
Philip Linden: I can explain where,
Philip Linden: and what problems we have had with them.
Philip Linden: Your inventory...
Philip Linden: everything that is not resident in the world,
Philip Linden: that has to be stored in a database.
Philip Linden: that database IS a central machine...
Trimming Hedges nods
Philip Linden: a NetApp with MySQL
Philip Linden: when you log in,
You: is the netapp itself running mysql?
Philip Linden: we get your invetory from that database.
You: when I last worked iwth them, they didn't do that
Philip Linden: I don't think so... the netapp is just the storage.
You: ok
Philip Linden: there is fast multiprocessor machine (I think) running mySQL
Philip Linden: bear in mind I am the CEO here... haven't been coding this stuff in a while.
Philip Linden: but I have a very technical background.
Philip Linden: Anyway...
You: *nods*
Philip Linden: in your post you mentioned the textures...
Philip Linden: it turns out that those are not stored in the database either.
Philip Linden: There is about 3TB of texture data so far.
You: someone mentioned that. Ouch.
Philip Linden: that is distibuted onto the sims,
Philip Linden: and a fast machine which we call the 'asset servers'
Philip Linden: is used to keep a central copy,
Philip Linden: so that when you teleport,
Philip Linden: your textures can go with you fast.
Philip Linden: so that machine(s) is the one that we have had some problems with.
You: teleporting with attachment has never been reliable
You: attachments*
Philip Linden: Those are apache machines serving the textures as files,
Philip Linden: behind SQUID proxies.
You: probably much more efficient than pulling them with sql
Philip Linden: the storage machines are being switched to Isilons.
Philip Linden: yes MUCH more.
Philip Linden: I am talking about Gbps class data rates....
Philip Linden: for example when we do a restart,
You: I know it's a hard problem, I've always thought it was
Philip Linden: and have to redistribute the data to the sims...
Philip Linden: can you imagine what that would do to Oracle?
Philip Linden: 500 machines making demands.
You: lol death for sure
You: or you'd need a mainframe
Philip Linden: so anyway that is the design.
Philip Linden: we've grown a bit too fast,
Philip Linden: and we have some bugs and tracking problems.
Philip Linden: so for example the teleport with attachments...
You: the world is hanging together by strings, bailing wire, and duct tape
Philip Linden: those are just bugs we need to keep fixing...
You: it is barely working
Philip Linden: basically race conditions and other problems with getting you to the new sim and requesting those assets simultaneously.
Philip Linden: I agree there are big problems.
Philip Linden: I guess we could shut down the whole grid for a few months or something,
[snip 3 lines]
Philip Linden: but that hardly seems the right idea.
Philip Linden: we are working as fast as we can.
You: well, your solution needs to include staying in business
[snip]
Philip Linden: well I really don't like losing content.
Philip Linden: I'd be F'ing furious if I lost something I built.
Philip Linden: in here.
You: philip, I know I'm not as smart as most of you there -- but I don't think you guys really have a handle on this data thing, and as I was trying to tell you, I think you need to find at least one seriously brilliant database guy
[snip]
Philip Linden: Well we have a couple... really very smart people.
Philip Linden: Also I have some database background,
Philip Linden: but not world class.
You: and then the corporate focus to way massively overfix this problem so you can grow
Philip Linden: But honestly if you saw the schemas... the size of the system.
Philip Linden: you would be blown away.
You: because if the foundations aren't stable -- and they aren't -- gluing on more stuff just breaks more things
Philip Linden: We would love to hire more great DB folks,
Philip Linden: but just bear in mind this is a big challenge.
Philip Linden: thing of the complexity of SL compared to Amazon,
Philip Linden: or Windows,
Philip Linden: or Google.
You: I have said that
Philip Linden: WAY more complex.
You: that you guys are like ebay or amazon
Kate Hanks: Trim has been a huge supporter
Philip Linden: We need to build the best team in the world.
You: but it feels like you're trying to spend money like you're Joe Smith running a webserver -- I realize from what you're saying that this isn't true
You: but that's what it FEELS LIKE
Philip Linden: any ideas on how to do that or people we should recruit....
Philip Linden: I'd love.
[snip]
You: I don't know any truly superb database people, sadly
Philip Linden: kate we haven't launched the teen grid.
[snip]
[snip from Philip responding to Jonquille]
[snip 4 lines]
Philip Linden: I think we will get many great people from within SL.
Philip Linden: We have plenty of money to spend,
Philip Linden: but sadly that isn't an instant fix.
You: you're dealing wiht a problem as big as the ones amazon and ebay have faced -- keep in mind they spent millions on their systems
Kate Hanks: how about the good minds in SL but working or meeting....gasp....in SL
Philip Linden: much bigger problem.
Philip Linden: Actuallly kate we are starting to do that.
Philip Linden: we just hired someone in England.
Philip Linden: so yes we are trying to use SL for work.,
Kate Hanks: good! i know you hired liasons and someone from Australia?
Philip Linden: yes a number of liasons,
Philip Linden: but I meant a programmer.
Kate Hanks: oh nice! so that IS possible! very cool
Jonquille Noir: What programming language?
Philip Linden: yes it is.
Philip Linden: Lots J...
Philip Linden: C++ is the core.
Philip Linden: OpenGL is the graphics.
Philip Linden: also every sort of linux/script/etc.
[4 lines snipped]
You: I wish I knew some brilliant DB people to point you at, philip, but I just don't
[snip 7 lines]
Kate Hanks: Phil, i'm so glad you and Trim talked
Philip Linden: well it doesn't get his hat back.
Philip Linden: which is totally lame.
Kate Hanks: no, but he can recreate
Philip Linden: really I am very frustrated.
You: the other thing I was telling people....
You: is that at least in one sense you're not thinking properly about the databases you do have
Philip Linden: how so?
You: ONE HARD DRIVE took your ENTIRE COMPANY down for like three days
You: that is totally unacceptable from an admin standpoint
Philip Linden: acually it was two failed drives in a 5-drive array.
Philip Linden: there was a backup,
Philip Linden: but the backup didn't have enough capacity.
Philip Linden: and now we are on a netapp
Philip Linden: with many backups.
You: ok, what you're saying there is 'growing too fast to keep up'
You: I'm sure your admins are desperately busy
Philip Linden: well sometimes that is true, yes.
You: I've been in a company growing nearly as fast as yours and I remember how desperate it was
Jonquille Noir: Sig: Has the suggestion been made to back up content off-world as raw data?
Philip Linden: I guess we could restrict the number of new users,
Philip Linden: but again that seems really bad.
Philip Linden: you mean back up locally for protection?
Kate Hanks: yes
Philip Linden: yes we are looking at that.
Jonquille Noir: (In the same way you'd download a texture back to your hard drive)
You: well, ya know, that's what blizzard is doing
Philip Linden: I think we will give something for that.
Philip Linden: yeah I agree it's a good idea.
Kate Hanks: make it another cost sink?
Philip Linden: it would be cool to be able to back up the contents of a parcel or something.
Philip Linden: but technically there is a problem... with
Jonquille Noir: I'd do it for my really important builds. My shop
You: the problem with that is that there would be permission problems
Philip Linden: backing up the stuff that isn't owned + created by you,
Jonquille Noir: True
Philip Linden: because it is a way to dupe things.
You: if I back up my database today, and then restore it tomorrow, and I transferred out stuff, I'd get that stuff back
Philip Linden: But we could do it only for YOUR stuff, which would be OK
Kate Hanks: yes, permission bugs fixed first
Philip Linden: and would cover many cases.
You: that would be cool
Jonquille Noir: I'd be happy if it could be done just with our own creations
Philip Linden: no kate it isn't that...
Philip Linden: if we allow full access to the data,
Philip Linden: it isn't a matter of bugs...
You: but fundamentally it's a bandaid
Philip Linden: it would always be possible to copy things then.
Kate Hanks: oh jeez
You: and then what if people try to restore corrupted data? on purpose?
You: it's a can of worms, a nasty one.
Kate Hanks: wow, how do you get around that?
Jonquille Noir: Sticky
Philip Linden: well we don't let you back up stuff you don't have full perms on.
You: for the same reason, user-controlled servers are going to be very scary and not likely to ever happen, as an admin I'd be terrified of remote servers that I didn't have root control of
Kate Hanks: ok, i suck when it comes to technical
Philip Linden: yes can you imagine that?
You: sure, but what if I download stuff and then monkey with it? .... oh, you oculd sign it with a public key
Kate Hanks: yes
Philip Linden: well we'll get into that problem when we come to it...
Kate Hanks: bad!
Philip Linden: happily not today.
Jonquille Noir: Like Siggy with the Sims. *hack hack hack*
You: sorry, I'm way out there
Kate Hanks: i think we tired Phil out
You: but you could sign the database with a public key - the db file
Kate Hanks: haha Jon
You: and then if it was changed in any way, not allow it to be uploaded back in
Philip Linden: yes that is correct we could do that.
Philip Linden: we are thinking about that too.
Philip Linden: that may be an approach that works.
You: it's still a bandaid
[snip]
You: from an admin perspective, I'm used to thinking of backups as MY problem, not the USER'S problem
You: it's just, in your case, a very large problem
Philip Linden: Yes fundamentally we need to get total protection.
Philip Linden: But simply a very hard problem.
Trimming Hedges nods
Philip Linden: OK you guys.
Kate Hanks: ha, never thought you'd be answering these questions, eh?
Philip Linden: I'm going to go and make sure the parrots are all talking on the welcome islands.
You: Rule #1 of holes: when you find yourself in one, stop digging.
Jonquille Noir: hehe
Philip Linden: oh it was fun to talk to you guys!
You: thanks for taking the time!
Philip Linden: just not a fun subject, clearly.
Kate Hanks: Phil you have been so great
Kate Hanks: thanks for coming
You: I appreciate your effort
Philip Linden: thanks for suffering the bugs.
Philip Linden: but I think we are making progress.
Jonquille Noir: Bugs aside, it's still the best thing going
Kate Hanks: yes
You: is anything you have told me confidential?
You: would it be okay to repost on the forum or should I keep my mouth shut?
Philip Linden: no you can talk about that stuff on the forums... that is fune.
Philip Linden: fine.
Kate Hanks: i think it would be GREAT to post in the forums
Kate Hanks: just filter out ME
You: I'm thinking that maybe just a transcript might be interesting
Philip Linden: the isilons are not online yet (I think).
Philip Linden: but will be very shortly.
Jonquille Noir: You can filter me out too
Kate Hanks: haha
Philip Linden: they are up and running.
Kate Hanks: thanks Phil, again. you never fail to amaze me
You: I think people will be very interested in this.... ok to post a transcript?
Philip Linden: Yes it is OK.
You: ok, got it, will do
You: tomorrow

so sims do have the ability to store objects and textures themselves using an independent mysql server box to keep everything properly indexed. that means there is no inherent centralized support on which small, independent grids must rely. they could license the server software today. well not today obviously; it'd have to wait until monday wouldn't it.

thanks for the explanations philip. very enlightening.

Who the hell is Philip Linden? He sounds very feted to me.

Isn't this a violation of the TOS?

Geez!

Always do your research before criticizing, Merwan.

Not a criticism Blaze - it's a question.



_/_/

I would think not seeing as how he asked all parties concerned a few times and Phillip said that it would be fine...There have been several chat log interviews as of late and the consensus seems to be that so long as all parties are made aware that the chat will be posted to forum it is alright.

I got permission from everyone involved and even offered the transcript to Kate and Jonq for approval before posting. Can't see how that could possibly be a violation of anything.

It is true. Not only did he ask Philip if it was ok to post on the forums. He also asked Jonquille and myself. And we both told him it would do much good for the SL community to post the transcript. There is no TOS violation that I know of, he has full permission.

Back to the chat... rude. That's how you treated him. You interrupted him, told him his database guys are sub-par when he was sitting there trying to tell you it's not a database like you're thinking a database is. You kept hammering him and not even listening to what he was saying.

Kudos to Philip for keeping his cool. For crying out loud, he's a programmer not a war criminal.

That was my thought too, Aaron. That was customer service way way above the call of duty. Very good information came out of it though. Thanks for posting it, Trimming.

Heh heh yeah, Philip is quite the dude.

Wow, perhaps I shoud have prefaced this chat by saying I IM'd Philip in world and pointed him to Trim's original forum post about database losses and asked if he'd talk it out. Not only did he read the post, he sought Trimming out through me. It was very much a mutual and not a hostel chat. A chat I thought might be useful to the rest of the SL community. How silly of me to presume.

Definitely very useful, Kate. Thanks

Excellent!


Thank you for setting me straight.

Thank you Kate and Trimming for having the moxy to ask the tough questions and for posting Phillip's responses. I think it's easy for us to sit back and assume that we have solutions based on what we see day to day in SL. For example, I have long thought that it would be a great idea to be able to save our content client side and wondered why we did not do it.

After seeing Phillips response - I sort of feel like smacking my forehead with my best Homer Simpson "Doh". As it is obvious that they have listened and weighed the issues against the benefits and that there are no real easy solutions.

I think this is called a growing pain and that it is a plateau that all businesses go through. One thing I applaud though is anyone's ability to IM Phillip or email him and get a response. The fact that they are hiring more people also indicates that they are aware of the necessity of communication with their customers.

One of the interesting things about SL (and one of the many things that keep me here) is that there are a lot of experienced, creative, brilliant people who are part of the SL Community. This does however present a dilema for the Lindens in that controlling the expectations of creative, brilliant people is sort of like herding cats. Not easily done.

So Kudos to Trimming who asked the hard questions and kudos to the Lindens who believe that honesty is the best policy.

I agree.

I'm not sure why some people believe it's okay to post to the forums flaming everyone at LL, threatening to withdraw from the game, and generally throwing a tantrum, and yet they believe it's bad form to ask a Linden when you have a question or address an issue with them directly.

For those earlier who claimed Trimming was being rude, and interrupting; Please note that the log is not time-stamped. Many of the posts were coming through at the same time. It wasn't 'interrupting,' and this wasn't a Town Hall setting where Philip was to give a speech and then open up the floor for organized questions. It was just a chat. Also, a lot of what was filtered out for being non-pertinent was friendly banter, which made the mood much lighter than it might seem in the transcript.

I certianly hope Philip didn't feel cornered, berated or ganged up on. I didn't have that impression at all, and it definitely wasn't the intention as far as I'm aware.

Very interesting chat that raise some serious questions and very few suggested solutions.



===
Philip Linden: I 100% agree we need to protect every single piece of content.
Philip Linden: but let me give some more data on how things work...
Philip Linden: what sort of predicament we have here.,
Philip Linden: First,
You: and I don't think the new server is reliable, either, the reboot the other night and now people complaining about a lot of corruption
===


Good to see the problem stated - can LL fix it?
LL needs to protect our inWorld owned content, knowing that new server is not reliable is a start - can LL fix it?



===
Philip Linden: we are not in the game business.
Philip Linden: we are in the digital world business.
Philip Linden: and I agree that is a database problem in many ways.
Philip Linden: but a kind of database the world has never seen before.
You: ultimately all a digital world IS, is data
===


Which world is Phil referring to? the SL world or the RL world?


========
Philip Linden: we've grown a bit too fast,
Philip Linden: and we have some bugs and tracking problems.
Philip Linden: so for example the teleport with attachments...
You: the world is hanging together by strings, bailing wire, and duct tape
Philip Linden: those are just bugs we need to keep fixing...
You: it is barely working
Philip Linden: basically race conditions and other problems with getting you to the new sim and requesting those assets simultaneously.
Philip Linden: I agree there are big problems.
Philip Linden: I guess we could shut down the whole grid for a few months or something,
[snip 3 lines]
Philip Linden: but that hardly seems the right idea.
Philip Linden: we are working as fast as we can.
========



Too much growth too soon, coupled with serious tech problems and LL is working fast!
I'm hearing that they're not getting the problems fixed - and would a 3 month shut down actually be required to fix things?
Can things be fixes while the grid stays up, or is this how SL will be from now on?



==========
You: philip, I know I'm not as smart as most of you there -- but I don't think you guys really have a handle on this data thing, and as I was trying to tell you, I think you need to find at least one seriously brilliant database guy
[snip]
Philip Linden: Well we have a couple... really very smart people.
Philip Linden: Also I have some database background,
Philip Linden: but not world class.
You: and then the corporate focus to way massively overfix this problem so you can grow
Philip Linden: But honestly if you saw the schemas... the size of the system.
Philip Linden: you would be blown away.
You: because if the foundations aren't stable -- and they aren't -- gluing on more stuff just breaks more things
Philip Linden: We would love to hire more great DB folks,
Philip Linden: but just bear in mind this is a big challenge.
============


Yes - is LL up to this challenge?

What's being said here? Maybe what's been created can not be managed with the current user growth? Phil sounds as though he's wondering out loud, loss of confidence?


===========
Kate Hanks: Trim has been a huge supporter
Philip Linden: We need to build the best team in the world.
You: but it feels like you're trying to spend money like you're Joe Smith running a webserver -- I realize from what you're saying that this isn't true
You: but that's what it FEELS LIKE
============



Will the best team in the world want to work for LL?

Focus on the best team for LL to fix the the problems.

I'm also a huge supporter of SL/LL - and I have to agree with Trimming about his feeling. And in light of Phil's next statement -


===============
Philip Linden: any ideas on how to do that or people we should recruit....
Philip Linden: I'd love.
[snip]
...
Philip Linden: I think we will get many great people from within SL.
Philip Linden: We have plenty of money to spend,
Philip Linden: but sadly that isn't an instant fix.
================


In light of the above statement, that is EXACTLY how Joe Smith would hand things.

Phil seems to be saying he does NOT know how to "...build the best team..."

I suggest again, build the best team for LL to deal with the serious problems facing SL. Start by retaining a professional tech recruiting firm and focus your time, attention and considerable skills at solving this problem. And, do not hesitate to consult with those who can compliment your skill set - effective CEO's know how to consult, how to attract skill and resources!


=========
Philip Linden: I think we will get many great people from within SL.
Philip Linden: We have plenty of money to spend,
=========


Ok - spend it wisely - and retain a competent firm that can do an international search - let them handle this by bringing their industry resources to bear. The fix maybe be beyond the SL user base - bring in the recruitment people who do this for a living!



=========
Philip Linden: really I am very frustrated.
...
You: that is totally unacceptable from an admin standpoint
Philip Linden: acually it was two failed drives in a 5-drive array.
Philip Linden: there was a backup,
Philip Linden: but the backup didn't have enough capacity.
Philip Linden: and now we are on a netapp
Philip Linden: with many backups.
=========


Another technical error, system design flaw, faulty foundation, of the most basic sort. There was a huge out-cry when the FORUMS Scripting Thread was lost due to back up problems. Get it fixed, test it, plan for the worst case.


============
Philip Linden: I guess we could restrict the number of new users,
Philip Linden: but again that seems really bad.
=============


Again, no solution offered. Phil has stated SL is growing too fast and there are uniquely serious problems - where are the solutions?


========
Philip Linden: it would be cool to be able to back up the contents of a parcel or something.
Philip Linden: but technically there is a problem... with
---
Philip Linden: But simply a very hard problem.
==========


=========
You: Rule #1 of holes: when you find yourself in one, stop digging.
=========


Agreed!



_/_/_/

Trimming, thanks for sharing this chat after asking Philip and present company about it so that they knew... I hope things iz alright.



Holy moly continued thoughtline, LOL! *blush*

Okay... great to hear the parrots being addressed. It means the Bug Report system works. *grins* I was wondering about that, since newbies were coming up to me wondering if they did something wrong.

I have known alternarules from my timeline when it comes to holes, though, such as:

Rule #1 of holes: when you find yourself in one, keep digging -- you could end up in China, or find buried treasure! (And who knows how deep the rabbit hole goes?)

I have to agree with Aaron, Trimming. You were directly assaulting him AND his brain child in that conversation.

Phillip's a cool dude.

dude, I can't stand "interviews" where the interviewer goes in there with an agenda and can't let their subject finish answering a question without butting in.

So they grew fast and are working on scaling issues -- big friggen deal.

Don't butt in there with accusations and slanted questions right to the head honcho when you haven't the slightest idea what's really going on and what they're dealing with.

"You're in x business and you can't be running it like blah blah" is a rather big insult to a founder who runs the company; 'cause he's the damn CEO and knows precisely what business he's in and what the issues are.

If someone did that to me, I'd call them on it. Tell them to shut their yap and do it themselves then.

This whole chatlog I'm sure is in a TOS violation and only serves to add insult to injury.

LL is doing a fine job.

It's a pain in the ass to lose some work, but it happens. The service was never garaunteed in any of the EULAs or TOSs or anything.

And it's not "practically beta" software either. Since there's nothing in the market to compare SL to yet, I'd say this is the best there is. If there was something better, wouldn't you be flocking to it right about now?

</rantaboutpeoplewhobitchaboutSL>

Yeah, Trimming, you're a real twit for trying to engage in a debate about the technology here. Clearly you're just a greedy guy who is going to get rich from this by .. uhh .. ummm .. oh I don't know! there must be something!

(sarcasm off)

Give it a rest folks. It's probably the most interesting chat log to be posted in these forums ever, and Trimming is halfway responsible for it.

So he's got an ego, that's no crime. I'm sure Philip has one to.

This wasn't an interview, Jade. It was just a chat, and one that was thrown together completely by surprise to me, and apparently to Philip too, on maybe two minutes' notice. I wasn't told he'd read my forum post, so I was trying to repeat the core issues there.

And I remain convinced that, no matter how hard the problems involved are, that SL is fundamentally a shared database, and that database needs to be ROCK SOLID.

However much data there is, no matter how complex the access methods are, the fundamental goals of a database are:

1) Data integrity;
2) Reliable,fast retrieval of data;
3) Reliable,fast acceptance of new data written;
4) Data integrity;
and did I mention:

5) Data integrity.

As far as I can see, SL fails on all three points; it does go fast, but it is very poor for reliability. It handles a lot of data very quickly, but it doesn't do it WELL. The foundations are broken. First you do it right, THEN you do it fast.

I have been a sysadmin for many years, and hearing that the loss of two drives pretty much stopped this entire company dead in the water sent my eyebrows nearly through the ceiling. By any measure of performance I would use, this was a horrific failure on the part of the admins. I realize they're probably running like crazy, I have been in a company growing at this speed, but a machine that central to the whole system should have been better protected than that.

Now, I originally thought, from the forum post, that it was a ONE drive failure: had that been the case, it would be GROSS incompetence. It was a two-drive failure, but losing two drives isn't that unusual. In fact, if all your drives come from the same batch, when you lose one of them, it's generally best to assume you're going to lose more in short order. They have a nasty tendency to go in clumps. A critical machine like that should have had another mirrored array running, preferably with drives from a different manufacturer. And there should have been hardware components to replace EVERY SINGLE PIECE of that machine on hand, sitting in the parts closet, ready to go. When you put all your eggs in one basket, you WATCH THAT BASKET, and I don't think that was done. For a machine like that, you *keep parts on hand*.

Now, I'm not holding them to a ridiculous standard here... I think that two hours or so of downtime are acceptable, and four hours might be JUST BARELY acceptable, in the case of really drastic emergency. I'm not expecting them to be up 24x7x365, which is enormously more expensive and difficult to do.

The simple fact is, though, that a relatively simple failure took them down for DAYS. That is incompetence, pure and simple. Whatever the reasons are, whatever the complexity is, if a failure that simple takes your whole company down, then the organization was incompetent, and needs to be fixed. I'm not pointing at any individual person, but the organization as a whole completely and spectacularly failed in one of its major objectives. If it's rude to say so, then consider me rude. I'd rather have someone be rude and tell me the truth than smile and flatter me and tell me it's okay when it really isn't.

As a related issue, I also consider the fact that my original hat was lost irretrievably due to a sim crash to be completely unacceptable. Admittedly, it wasn't that much work to replace, but what if it had been something I'd put real time into? I have some things that would take me two weeks to rewrite, and that scares me. Data Just Going Away is the absolute worst thing that can happen in a database, and that's COMMON here. Corrupted textures and lost inventory are LEGION. If you haven't been bitten yet, count yourself lucky! They have backups, but the backups aren't actually usable for anything but a complete system restore, which does no good whatsoever for the reasonably common case of corrupted data on the part of an individual.

I haven't really sat down yet to write up a response to Philip based on what we talked about, but I suspect this fundamental architecture is probably never going to stabilize. There's just too many hands in the cookie jar to manage all of them. 500 separate, simultaneous databases, none of which are on terribly reliable machines, is just a management nightmare. The fact that it works AT ALL is nothing short of amazing.

In other words, it's my belief that the SL system, as presently designed, will never work reliably. It is too complex. There are too many points of potential failure.

And, again, if I'm rude to say so, then so be it.

Thanks for the insight, Trimming.

I am curious though... "Jade"... who are you referring to in this thread? My Jadey, for one, hasn't posted.

Is he implying Icon is Jade's Alt?!!!