Absolutely Amazing : SL Infrastructure

I am absolutely astounded, the current blog entry regarding the logins outage and the cause.

Ten odd hours outage for something that, if a reasonable infrastructure or sensible configuration and problem recovery processes were in place, should only have been a maximum of 2 - 3 hours outage in any normal enterprise environment, if not, with a highly available network design, should only have been a few minutes outage. It would appear that the infrastructure services and design leave "a little to be desired" to say the least. If this sort of outage was observed on a financial firm, high profile website, or government service, all hell would break loose in the press, with public enquiries. In SL? Will be bypassed, ignored and blame the 3rd Party provider, and become just another forgotten blog entry when the next piece of infrastructure fails.

If this is the best infrastructure that Linden has in place, or can achieve from it's providers for the Second Life grid, then either LL is running the systems like a cheapskate $1 hosting firm, using shared resources or the network and infrastructure teams should seriously be reconsidering their design and career choice.

If I was to put in place such poor infrastructure, containing obvious single point-of-failures, inherent design flaws, shared services, poorly delivering 3rd party suppliers, limited scalability and recoverability, including unreliable support and communications tools, my business and reputation would be shot within hours.

Considering the costs involved in sims, inventory items, lost time, "member fee's" etc, any normal business would be dragged through a hedge backwards if it performed and operated with such disregard for it end-users.

If LL are serious about SL, do you not think that it is time....? Time to adequately invest in your own future, infrastructure and end-user goodwill.

Actually, the time for a class-action lawsuit against those con artists known as "Lindens" may well be approaching. Right now, I cannot think of another firm which more deserves to be slapped than LL (a.k.a. Lagging Lab) for literally stealing too many people's time and money.

And exactly how is LL stealing your time and money? Do they have a gun to your head forcing you to use their service? I think not. If you don't like their service, feel free to leave, and don't let the virtual door hit you in the virtual ass on the way out.

Do you threaten to sue your car maker every time you car breaks down? Do you threaten to sue the cable company every time the cable goes out? I mean seriously, you need to get a grip on reality.

This is not a perfect world we live in, and SL is a imperfecet simulation of an imperfect world, *in* an imperfect world. Ponder that if you can wrap your brain around the concept.

Umm... yeah. If the cable went out three times a week and the communication company continued to collect payment with out refund for lost time, there would most certainly be a class action law suit. We agree and chose to pay for a service, they agree and chose to provide that service.... so provide the fucking service already, or a class action could be a good thing. It's one of the ways to keep companies in the marketplace providing par service, and weed out those that do not. I'm not one of the irate, but i most certainly see the potential value to the bussiness community in all of the sabre rattling. Was a little taken back by the "grip on reality" thing though. Reality is, law suits are tools to check and balance bussinesses. So please, grip reality rather than try to express your fantastic creation of it. People and groups sue sub standard bussinesses, end of story.

SL hasn't even been down 24 hours yet. And, it's not even their doing, it's their ISP. ANd people are talking class action lawsuits? Again, i think it's you that needs to get a grip.

There is a Lemon Law...

However, class action lawsuits are on their way out. Better hurry and get one going before they are no more!

isp outage? With the recent crap that's been put upon us all, pics or it didn't happen!

An ISP that hasn't heard of "failover" and "redundancy"? I wonder whose garage the ISP is running out of?

Precisely my first thoughts about this outrage...oops, "outage."

This was more my point, we all like the service/game, we all wish to continue, but it is such a shame that LL appear to have been unable to keep up with demand and subsequently have not chosen to improve their resources and infrastructure to even allow them to continue to grow in a measured manner without so many of these types of issues.

There was no mention of "legal" activity, with no intention of driving the post this way. Merely an observation and statement. For those that wish to talk such high-brow, and unnecessary actions, please open another thread.

I can understand that demand has out-striped the original infrastructure, but LL appear not to have moved on and put the infrastructure and resources in place to support their new found fame, which is not only disappointing, but also determental to their own business and growth. Without change and a more professional approach and measured growth planning, maybe they will join the ranks of Compuserve and other such VRML worlds previously, or even just those online service providers who lost market share previously by not keeping up with trends and market forces. People get bored with outages, continuing issues, and seriously bugged application software, changed is always needed, changed always need to be monitored, measured and managed, not simply thrown out there on a wing and prayer, this includes the actual service under offer also.

As for the statements of "no-one forced you to be there" and "what money" These are in effect basically true, but once a service has been offered, accepted and in use, providers should be bound to deliver on the service tendered. The lost of items in inventories, not accepting item transfers, whether from in-world or out-of-world and items getting lost in the process or actual costs to the end-users. How impressed would you be if your bank lost $1 on every bank transaction you did? So why would folks not be annoyed when resources are lost from inventory or not successfully transfered from SLX or On-Rez or get corrupted in inventory and now cannot be used.

These are not minor issues, and this is only from the "personal user" perspective, not mentioning the businesses that have been whoo'd and now part with larger amounts of cash and peoples time with SL, to only have the same issues, but to a greater and more costly degree.

Good luck with getting people who refuse to financially contribute to sue for loss of income.
Yep I can see the mass campbot farmers now suing for lost RL income..............
Oh no, just got in and I didn't get a chance to RP feeding my pond fish, and now they are RPing, being dead, I'll sue!

Linden Labs has been having MASSIVE issues for the past 2 - 3 weeks. These issues have been a DAILY occurrence... several times a day, and for hours at a time. All this can be verified by reading the blog.

You want people who are sim/island owners, who pay Linden Labs REAL LIFE money, and a lot at that... on a monthly basis via tier alone, to get a grip? These are the people who are losing REAL LIFE money on a daily basis. This outage affected businesses grid wide. Considering the length of ALL issues, the very least Linden Labs can do is offer some "credits/refunds", to appease their PAYING customers.

According to your inworld profile, you are a business owner yourself. Surely you have felt the hit of these issues yourself.

How about showing some support for your fellow business owners, sim owners, island owners out there? Perhaps if the other issues had not been a continuous, day after day after day happening... the rest of us might believe this 12 hour outage was NOT Linden Labs fault.

You're trying to use logic on somebody playing the class action lawsuit card in a web forum???

In the land of the blind, the one-eyed man is king. In SL, the Lindens are gods. Do not mention all the lameness lest thine crops wither. Bow before them, trembling in fear of their mighty thunder.

SL has more downtime and database issues than any other company I've ever seen on the net. Like has already been stated, how would you feel if for every transaction you made, your bank would keep $1? With all the problems and downtime that SL constantly seems to have, it wouldn't surprise me if the loss of $ runs in the several hundred thousands for people, if not more.

I may not be a premium member, but I have lost plenty of real life money due to SL's constant issues. Have I ever gotten anything to make up for this? Yes, more and more frequent problems.

I don't think it's us that need to get a grip, I think it's LL that needs to get a grip by finally setting up an infastructure that's made to handle the load and moving over to a ISP that actually has a backup system in case of failure.

You can't haul 30 tons of cargo with a Lada, and you can't provide a decent internet service to 30,000+ people a day with inadequate material.

yes you're right, sometimes I like tilting at windmills I guess ^.^

I do run a business, and I have 15 sims currently. But I sure as hell don't see any justification for a lawsuit because an ISP had an issue. My final say on the matter.

First, your sims must be both worth nothing and generating zero income. Why else would you not care about "business?" You must also not be paying any tiers and thus it doesn't matter b/c you're getting exactly that for which you are not paying.

Second, not one network engineer to whom I've spoken about this sees anything but abject failure by LL to plan and prepare for *inevitable* contingencies. Three out of four of those high-level engineers stated there should definitely be a backup ISP in place. Clearly, there is not. This has gone beyond mere incompetence; it's now sheer stupidity.

Third, you are taking LL's word for that being the problem. You believe them after all the many times less than accurate information has been given us--this melt-down being a perfect example? You believe their clearly manipulated "public" statistics? (Didn't take a college stats course, eh?) LL doesn't even know what the word "resolved" means. I refuse to believe anyone that dumb. But should it be the truth, refer back to my second item.

A backup ISP? Hm.. Interesting idea. What if, as has happened before, it's some big regional router that goes critical and melts down?

Also, since you've got high-level engineers on tap, how much do you think LL's monthly ISP bill is now for the 15-20,000 servers they have wired to the net and how much more are you willing to pay every month in tier for the land you have now?

Precisely what is being discussed here.....!

There are some absolutely brilliant technologies, methodologies and techniques available today for disaster recovery, business continuity, continuous operation, clustering, storage networking, mirroring, aggregation, alternate DC's, load balancing, parallel operations and computing, proper grid computing (not connectivity being termed as a "grid"

And..... all of these are not just "interesting idea's" but are actually implemented and in use at small to medium business and Enterprise level. Planning, not only for the actual unexpected event, but also for what will happen and how things will be managed and recovered if those events happen without full DR protection, (which in many cases is still a valid way of operating) is essential for any business in todays high-speed, high-tech world if they are to continue to grow and stay at the forefront of any technology based business.

Having the appropriate infrastructure in place withn your own environment is relatively simple (although costly, yes) but ensuring that your third-party suppliers (ISP's, Telco's etc) are not your businesses weak-link is still part of the task for Infrastructure Architects, otherwise it is partially pointless putting good internal systems in place, if one hiccup outside can stuff your systems up regularly or consistently.

I have been noticing the number of members that SL has, and the number of members that have been logged on for the past 60 days......

That's a big gap. Sounds like there are a LOT of people getting fed up and just not logging in. I am kind of new in here, and I like being able to do stuff, but it's even getting old for me. And I haven't even been here long.

I sure hope the Lindens can get things straightened out. I'd hate to see this game go away.

And, again, how much of a tier increase are you willing to pay to get any of these?

I know this stuff exists. I've worked at a 5-9's company for years. I also have some idea how much it costs even at a small site and have some idea of how absolutely ginormous SL is.

If LL is to continue to "court" business and education use of the service, then they will need to absorb some of these costs and price their fee's appropriately, those, as you say that are aware of 5x9's environment do understand these costs, better than most, but a "Service Level Agreement" stating what the desired and offered Service Levels are, that is appropriate for the implemented technologies is the minimum requirement. Five 9's is not necessarily the required Service Level, there is a lot of ground between how things appear to run now and five 9's availability.

SL does not "need" to be five 9's in most, if not all cases, but at the moment there is no definition of service, apart from the end-users being in the dark, not setting some form of performance or service goals is actually also deteremental to the LL business, as their own staff have no goals to attain or maintain, leaving scope for massive frustration for those staff that try to achieve high service levels, but the infrastructure does not allow it, and for such staff as the support teams that are left high and dry handling the fall-out without any management support or backing or more importantly a point of reference for what is good or bad delivery. It also follows, that with no set service levels, any third-party provider is open to delivery what-ever level of service they wish to LL with no repercussions if the LL / SL services are disrupted due to their inadequacies.

Staggered/Tiered SLA's would provide LL with the opportunity to deliver different levels of Service, with those that wish higher levels paying for them, those that do not, not paying for them. There is a vast array of potential models available that are in use by many other business that deploy differing technologies with alternative price structures.

It's not hard, it's well documented, already in place good business practices, not black magic and smoke.

So are you avoiding the question of cost or do you really think they can get a higher level of availability for free? Or do you expet them to eat the cost?

Sindy, I am not spoiling for a fight or argument, merely discussing services, availability and options, I also thought my answer was actually quite explicit concerning the costs of the proposals and suggestions... and in several different terms.....

Like you, I am also very well aware of the costs of these type of technologies, bandwidth, hardware, software and administration, working with them everyday in five 9's, mission and business critical, clustered and grid Financial, Scientific and Geo. environments, which by far exceed LL/SL size, including small to medium Enterprise continuous computing environments much smaller than LL/SL environments where up time is essential.


Previous Responses:





Does that answer your response?

I'm also not looking for an argument but every time SL explodes, we get people here who want to redesign the world but never _really_ step up and say what they'd be willing to pay for it.

You're getting closer but you still haven't said what _you_ would pay to have a more reliable SL. Is it 50% more in tier? 5%? 500%?