Welcome to the Second Life Forums Archive

These forums are CLOSED. Please visit the new forums HERE

Weekend Grid Outages

Ann Otoole
Registered User
Join date: 22 May 2007
Posts: 867
01-19-2009 17:25
They just brought in some guys that probably studied Joe Celko's books. (Although I seriously doubt they ever studied Vadim Tropashko's book)

Do you think they can wave a wand and fix the issue in a minute? No they can't. This pain shall continue until they knock out some of that corporate speak "low hanging fruit".

But none of whatever a so-called mysql crew of "experts" can produce the effect of a properly engineered (under a solid architecture) SL V2 that is only now possible to figure out.

Best bet is to write off 2009 as the year SL V2 was developed and reap the benefits in 2010.

In the meantime people that are premium (only) should have precedence logging in. Island owners that are not premium can afford 10 bucks to buy a pass. In fact it should be mandatory to even buy an island that you are premium. Duh. LL needs help in so many areas of business logic.

Switch to Oracle and grow up for pete's sake. Go see if Larry will trade some Oracle licensage for face time and prestige.
Kay Douglas
Registered User
Join date: 20 Feb 2007
Posts: 41
01-19-2009 17:26
Network latency is a big problem for more europeans users and another country. For exemple if ping -t agni.lindenlab.com server the time is 192ms. and i ping in another server for example london the time is only 44ms. This difference create more lag in second life for other country users.

From: Ann Otoole

Switch to Oracle and grow up for pete's sake. Go see if Larry will trade some Oracle licensage for face time and prestige.


Lol, Oracle is expensive more website used MySQL (Yahoo and more ....) or PostgreSQL. But optimized and tuning BDD gourou is necessary for the future of the Second Life Grid and a new network architecture.
Lyla Tunwarm
Registered User
Join date: 10 Jul 2008
Posts: 179
01-19-2009 17:29
From: FJ Linden


Our longer term strategy could, in fact, be a replacement of mySQL. This would, obviously, be a major undertaking, but if it is the best way to achieve scale and stability we will pursue it. However, much more analysis needs to be done before making that commitment and it is clear that we have not yet done everything possible to properly manage queries, distribute load, and tune mySQL to operate at its highest performance level.

I understand your position as I deal with management daily that resist the inevitable as long as possible but they always kick themselves for not upgrading sooner as the longer you wait the harder it is to switch something over.

You know that optimizing MySQL to its most efficient state is nothing more than a temporary bandage. SL will grow to a point where the amount of queries will be caused by concurrency instead of whatever is causing it now, then what? There is a reason MySQL is not uses for things of this nature.

I am curious after all this time why MySQL is not already optimized? Why were all these other fancy add-ons over the years added before MySQL was fully optimized? Should that have not been the #1 priority as it is one of the top parts of stability and scalability?

I know the answer to that and it isn't what I want to hear. lol
Tara Lorakeet
Registered User
Join date: 6 Dec 2008
Posts: 3
Appreciate the update...
01-19-2009 17:48
I am new to SL, and as most people here can identify, I have spent a lot of time on SL, learning the environment, buying land, creating and scripting. I really do enjoy it. However, I feel the same as some of these other people in that I don't know that I'm getting my money's worth. I'm not on a free account. I just wish that my friends with the free accounts were reporting the same frustration that I have with lag and getting booted. Sure, they have lag, get booted, but it seems to happen to me more.
Ryker Jacobus
Registered User
Join date: 7 May 2007
Posts: 4
Unable To Log In
01-19-2009 18:03
I am a long time user of SL from a place far far away... gee actually it's just New Zealand. I have not been able to log my account in during peak times AT ALL for the past 6 DAYS!!!. The only times I have been able to log in are between the hours of 3am and 6am SLT... equating to 12am and 3am local time, when the user numbers online drop below about 45K. I have filed two support tickets and added at least two comments every day. At first it was suggested to me that the problem was at my end and I received an email tagged "Problem Solved". Even though I know very little about computing, let alone how a structure like SL functions, from what was happening to me and my account, I knew my problems were data and asset server issues - this weekends grid fiasco supports my theory. I have a paid account, am an in-world business operator and content creator, I own Mainland and Estate parcels equating to a whole SIM. I sympathize with other users who are having a tough time and with LL, who true to the Faustian myth seemed to have created a monster they cannot control, and are now busily scratching their heads wondering what to do next.
We all understand this is new territory but in my little backwater we have laws that protect consumers' rights and if the service/product doesn't work.... you don't pay. We are not willing supporters of some grand experiment, we are in the correct analysis, consumers paying for a product/service that currently, at least for me, does not work MOST of the time. Millions of users have 1000s of USD tied up in inventory and land and only God knows the true value of our/their IPR, yet in the TOS LL offer no responsibility or recourse for compensation should (God forbid) Second Life fail. As a previous blogger stated "I love Second Life but right now I don't like it" echoing my sentiments exactly, I don't want to see SL fail and all credit to LL for doing what they can but surely the time has come, to cease the unbridled expansion of user numbers until the infrastructure can support and deliver a proper service to the current users. Bite the bullet... get rid of all the bots if possible... and yes, when restrictions are required give priority to the paid up users. I have listened to the non-paid account holders' arguments and they simply don't stack up, the choice not to pay LL is yours and sure you may contribute to the economy in other ways but no one is saying (yet) to terminate those accounts as their arguments imply, simply restrict entry to the paid up account holders as a privilege for being a direct Second Life supporter when restrictions are necessary . I too, shop like a crazyman, own Estate Land, create content and run several businesses but I also choose to pay LL tiers and own Mainland parcels, that is my choice but currently I am excluded from the grid at the expense of bots and the many many many non-payed account holders who make little or no contribution to Second Life.
Vittorio Beerbaum
Sexy.Builder Hot.Scripter
Join date: 16 May 2007
Posts: 516
01-19-2009 18:15
From: FJ Linden
Our longer term strategy could, in fact, be a replacement of mySQL.


Prior to do that, release an *official* contents exporter to give us the possibility to backup our work... then do what you want.
Kimo Junot
Registered User
Join date: 12 Feb 2006
Posts: 29
01-19-2009 18:31
Hardware is not the current issue, as was the case last April. We are not CPU or IO bound, so throwing more hardware at the problem will not relieve it (this was one of the first things we looked into).

OK this gets me tottaly confussed....... I Thought THIS WAS The problem with the OS Sims...now hardware is not the issue anymore?
Lyla Tunwarm
Registered User
Join date: 10 Jul 2008
Posts: 179
01-19-2009 18:38
From: Kimo Junot
Hardware is not the current issue, as was the case last April. We are not CPU or IO bound, so throwing more hardware at the problem will not relieve it (this was one of the first things we looked into).

OK this gets me tottaly confussed....... I Thought THIS WAS The problem with the OS Sims...now hardware is not the issue anymore?

Different issues... He is talking about the MySQL database hardware not individual sim hardware. Two totally different subjects/problems.
Yosemite Aero
Registered User
Join date: 28 Nov 2006
Posts: 4
01-19-2009 18:41
From: Catherine Linden
. These 5-10 minute "blocking periods" are substantially less Resident impacting than a full database crash, and a 50-60 minute restart cycle. However, neither is acceptable and I wanted to continue updating our efforts to stabilize the infrastructure.
5-10 minutes? Huh?
But now to my actual point:
I'm just a casual user of SL (although I have a premium account) but my shopping habit does manage to drop a few real dollars into the virtual economy every week and (although your downtime is probably saving me money) I'm getting a little frustrated. That being said, I can only imagine the level of frustration those who are more deeply involved might have (DJs, sim owners, merchants, designers and builders, those involved in research or spinoffs of RL support groups, to name a few). The other day, I made a less-than-flattering comment about your customer support to a lady who informed me that she, as the owner of multiple sims, receives excellent support ("they bend over backwards.";) That's great. For the amount she pays in land tier, she deserves support.
But please remember that it's the casual users (those with "payment info used" whether they be premium acounts or even free accounts who have used a credit card to buy a few Linden $) who keep this virtual economy afloat. If sheer frustration drives us to leave, your economy collapses. (look around the Real World for a minute, folks). Are the DJs going to tip each other, the sim owners rent to each other, and the fashion designers sell each other gowns?
Here are three steps you can take to reduce MY frustration:
1. First, be realistic, truthful and honest with posting that word "RESOLVED" on the grid status report.
2. Make an attempt to give us an "honest" evaluation of the severity of the problem and an estimate of the time it may take to resolve it (I realize that is not easy but at least try). Incidentally, both of these suggestions might reduce the load on your live chat and telephone support staff.
3. When load becomes an issue block the non-paying alts and bots first, before you block the paying customers. Notice here I did NOT say block the "FREE" accounts, just those who have no payment info.
Dil Spitz
Registered User
Join date: 6 Dec 2008
Posts: 9
01-19-2009 18:43
Hey,

yes, yes, its horible sad if you cant log in.

Many seam to have "the idea" what LL shall do or dont.
But please keep the noice a lil bit low.
Please be fair.
LL has managed to set a vital metaverse with a functionally micro economy, thats cool and the first time this really worked!
Also LL's breakeven seam to be reached, so they gets out of the this cashburners-corner. If they wouldnt the world economy crisis would blow them and our SL away over the next year.

I am really sure the Lindens work hard to fix&analyze this problems.
FJ Linden is doing a fancy good job and seams to love that too! Even if on this days he may not love his job as on other days, but he will have his satisfaction if found the bug and fixed it with the team. Thats his benefit and he know it. I am sure on this, even i do not know him.

We all will see a better SL maybe not this month but the way is the right. Forsure we get emotionaly if we think back to the OpenSpace/homeland-pricing and other things of the past, but hey, thats the deal, we life in a material world (real or virtual, what ever). LL has to fix the bugs, not because You or me. This has to be fixed so LL can survie and grow.
Certainly LL's business model will change over the month/years and maybe "3d-webspace" wouldnt be part of the core-businesses...

This Login situation is real painful.
I personaly wanted to go sailing and couldnt do now. Hey, thats really hard! Ok other may have lost money, You have my compassion.
But no one can help us now, there have things to be fixed and they will be, cause its a matter of survive of the company LindenLab.

So maybe You may find this charts i found on the web helpful for Your planed offline time.




Maybe it may not harm the database if anyone clears out her/his trashcan or delete some from lost&found.

sorry if You feel my words to harsh, but im from old europe ;)
Jack Abraham
Lantern By Day
Join date: 11 Apr 2008
Posts: 113
01-19-2009 18:51
Frank, thank you for your (and pardon the pun) frankness about the problems Second Life is facing. I'm a sysadmin by day and scripter by night, so I feel for you and your team as you struggle to keep the system up for us during trying times.

I'm not going to propose solutions. Few of us out here know enough about the system to offer intelligent suggestions, and I'm not one of them. You've already acknowledged the need for better resident communications, which seems to be chronic throughout all Linden Lab/resident interactions. You, personally, are doing your part; if we can bring the status page up to the level of your blog posts, we'll be doing well.

Best of luck.
Oblivious Destiny
Registered User
Join date: 23 May 2008
Posts: 3
01-19-2009 19:00
i am just wondering how many more years of failing database issues do we have to endure before someone at linden labs wakes up and finally realizes the database you are using does NOT work ,,,this is ridiculous making us suffer and our businesses suffer for years on end waiting for you to all wake up this is NOT working its broken and has been broken for so so so long, please FIX THIS. Again no one can ever seem to answer this question as it is deleted 99% of the time or ignore when ever i have seen it asked

edited to add ,yes i realize you are all working on it but that has been the case for well over 2 years and it is still broken every week
Leonardo Zimring
Registered User
Join date: 15 May 2008
Posts: 70
01-19-2009 19:01
From: Ryker Jacobus
I am a long time user of SL from a place far far away... gee actually it's just New Zealand. I have not been able to log my account in during peak times AT ALL for the past 6 DAYS!!!. The only times I have been able to log in are between the hours of 3am and 6am SLT... equating to 12am and 3am local time, when the user numbers online drop below about 45K. I have filed two support tickets and added at least two comments every day. At first it was suggested to me that the problem was at my end and I received an email tagged "Problem Solved". Even though I know very little about computing, let alone how a structure like SL functions, from what was happening to me and my account, I knew my problems were data and asset server issues - this weekends grid fiasco supports my theory. I have a paid account, am an in-world business operator and content creator, I own Mainland and Estate parcels equating to a whole SIM. I sympathize with other users who are having a tough time and with LL, who true to the Faustian myth seemed to have created a monster they cannot control, and are now busily scratching their heads wondering what to do next.


That was the Frankenstein story, not the Faustian myth -- or rather legend.

From: Ryker Jacobus
We all understand this is new territory but in my little backwater we have laws that protect consumers' rights and if the service/product doesn't work.... you don't pay. We are not willing supporters of some grand experiment, we are in the correct analysis, consumers paying for a product/service that currently, at least for me, does not work MOST of the time. Millions of users have 1000s of USD tied up in inventory and land and only God knows the true value of our/their IPR, yet in the TOS LL offer no responsibility or recourse for compensation should (God forbid) Second Life fail. As a previous blogger stated "I love Second Life but right now I don't like it" echoing my sentiments exactly, I don't want to see SL fail and all credit to LL for doing what they can but surely the time has come, to cease the unbridled expansion of user numbers until the infrastructure can support and deliver a proper service to the current users. Bite the bullet... get rid of all the bots if possible... and yes, when restrictions are required give priority to the paid up users. I have listened to the non-paid account holders' arguments and they simply don't stack up, the choice not to pay LL is yours and sure you may contribute to the economy in other ways but no one is saying (yet) to terminate those accounts as their arguments imply, simply restrict entry to the paid up account holders as a privilege for being a direct Second Life supporter when restrictions are necessary . I too, shop like a crazyman, own Estate Land, create content and run several businesses but I also choose to pay LL tiers and own Mainland parcels, that is my choice but currently I am excluded from the grid at the expense of bots and the many many many non-payed account holders who make little or no contribution to Second Life.



I sympathise with your problems but I'm sure that LL is correct. Your problems were local, not to do with LL. I live in nearby Australia and I have had NO problem logging in at any time OTHER than the weekend when the grid was down. And when I needed to be in at 2.00pm slt I simply persevered for half an hour and eventually got in. It wasn't great but it was not a catastrophe either --- it was, after all, just a few hours.

But I would really like F.J. Linden to say something about the long, long term strategy. What IS linden labs doing to prepare for the eventual 200,000 users? I'm sure this is a staple of water cooler conversation over there, so let us in on it, guys. What is the strategy?
Desmond Shang
Guvnah of Caledon
Join date: 14 Mar 2005
Posts: 5,250
01-19-2009 19:04
FJ, great openness and clarity with regard to the issue - would it be possible to tell residents roughly what the bottlenecks are?

I've noticed that problems tend to correlate with times that *actual users* are on the grid, so even while I'm all for banning bot-herds to save the 10% or 2% overhead or whatever it is, it seems pretty clear that normal human actions are driving a lot of load.

So what sort of thing is it - wearing 1/2 a region's worth of prims on an avatar, or high numbers of avatars in one place, or draw distance nice and high downloading eeeeeverything, or... ?

I'd love to put out a message on an estate channel that said: "those of you who don't mind, please drop draw distance down to X" or something. Residents are *amazingly* helpful when given a chance.

Best of luck with all this.
_____________________

Steampunk Victorian, Well-Mannered Caledon!
Nika Talaj
now you see her ...
Join date: 2 Jan 2007
Posts: 5,449
01-19-2009 19:09
From: FJ Linden
Our longer term strategy could, in fact, be a replacement of mySQL. This would, obviously, be a major undertaking, but if it is the best way to achieve scale and stability we will pursue it.
Bravo! I'm amazed that mySQL is handling the transaction volume you now have, I'm sure you have it tweaked pretty thoroughly.

I imagine that all sorts of custom kruft has grown up around your mySQL implementation that will make the conversion painful, but the sooner begun, the sooner ended. Realistically speaking, there aren't THAT many alternatives to look into, so hopefully you can move into a design phase fairly quickly.

Best of luck and thanks for the update.

:cool:

p.s. It looked like you were gathering lots of statistics this weekend. Is there no way to instrument database performance and limit logins more proactively? Of course you will hear complaints whenever logins are limited, but if you do it a little earlier at least those inworld will be able to actually use SL.
Micheil Merlin
Registered User
Join date: 3 Mar 2007
Posts: 32
01-19-2009 19:10
From: Lyla Tunwarm
Different issues... He is talking about the MySQL database hardware not individual sim hardware. Two totally different subjects/problems.


Actually, backend resources, including the asset server, was one of the issues that appeared in LL's postings about openspaces.
Kraelen Redgrave
01010101
Join date: 27 Apr 2007
Posts: 63
01-19-2009 19:22
Frank,

would it be possible to have a queue system for logins, like many other online games have? It is rather annoying trying to log in, when I have to sit around clicking login over and over. It would also be a lot fairer for normal users, trying to be faster than the 'bot army.

You could have a little window that pops up in the client, telling us how long we can expect to wait, or how many people are in front of us in the queue. You could even have it tell us how long until logins are expected to stay down, and when they come back up, it automagically puts us in the login queue.

It would ease the tension I think, because then at least we know we are not trying in vain. And you could even write some code to allow logins at a certain rate (that would be awesome), and then you wouldn't have to turn logins off completely (maybe).
Coventina Dalgleish
Registered User
Join date: 30 Apr 2006
Posts: 78
Want to kill bots kill traffic
01-19-2009 19:32
Well, you do not have to kill traffic just change the way it is registered. Your formula of 1 point of traffic for every minute an AV is on a sim is the main problem. This will most likely take a bit more work on your part but change it to entry's to a given plot for a minimum time of 2 minutes equals one count of traffic and no extra for length of the stay. Yes I am sure someone will come up with a traffic bot that leaves and enters every 2 minutes but most of the 100 bot in a 10 x 10 space will cease to exist. Of course this will also do something that you do not want to have happen by reducing your concurrence numbers making it appear that you are not growing. Traffic numbers are a bogus way to draw attention anyway.
Reacher Rau
Reach Isles
Join date: 9 Mar 2007
Posts: 40
01-19-2009 19:40
at what point do the revenues lost from these outages become greater than the cost of a real enterprise database + support contract? why is second life a mySQL guinea pig? it's not right.
India Mathieson
Registered User
Join date: 27 Dec 2006
Posts: 1
Hard Crashes
01-19-2009 19:43
I am on for about 5 mins at a time--then wham! CRASH!
Eva Ryan
That's Eva Ryan™
Join date: 26 Aug 2006
Posts: 197
01-19-2009 19:49
From: FJ Linden
Our longer term strategy could, in fact, be a replacement of mySQL. This would, obviously, be a major undertaking, but if it is the best way to achieve scale and stability we will pursue it. However, much more analysis needs to be done before making that commitment and it is clear that we have not yet done everything possible to properly manage queries, distribute load, and tune mySQL to operate at its highest performance level.


Ya might wanna take a look at how Google manages their database. I'm sure they have far more load to deal with, and yet do it (mostly) flawlessly. If queries could be handled in such a way, life in SL would be good, except for intermittent drama.
Varahi Lusch
Registered User
Join date: 8 Sep 2007
Posts: 0
Bot solution
01-19-2009 19:51
The best way to eliminate bots would be to completely stop reporting traffic.
Madison Brenham
Registered User
Join date: 24 Sep 2006
Posts: 30
take care of the people that pay the bills
01-19-2009 19:58
From: Nerolus Mosienko
I pay 300.00 a month for a private estate (and not easily...), and I know many people who pay more. Why can't the people that pay this insane amount of money to be a part of Second Life be allowed in before bots and non-paying members of SL? That seems only fair.

Usually the people that pay monthly voice themselves about Second Life on the internet much more than the non-paying members. Good for publicity, and good for just all around fairness.



I agree. Please, when this happens, can you let the people back on that actually pay the bills at LL? I spent hours trying to get back on yesterday and yet it showed 70K + online. When I finally got back online, I couldn't move or teleport and then would crash. I spend way too much money on this "game" to be kept out while bots and non paying newbies look for free sex.
Lyla Tunwarm
Registered User
Join date: 10 Jul 2008
Posts: 179
01-19-2009 20:39
From: Micheil Merlin
Actually, backend resources, including the asset server, was one of the issues that appeared in LL's postings about openspaces.

MySQL database doesn't need a ton of computer power, just needs the code properly setup which it isn't or MySQL is just to light of a database to handle it. More computer power will not fix either of those problems.

I understand your argument however but I don't belive it pertains to the MySQL issue discussed here.
Bob Bunderfeld
Builder Extraordinaire
Join date: 10 Apr 2003
Posts: 423
01-19-2009 20:50
I'm only going to say this once, and you can all crucify me for it.

But Frank, Philip, Robin, and the rest of you down at Linden Lab, isn't pulling in the Big Boys of mySQL exactly what I suggested you do LAST YEAR when the Asset Server was throwing up all over the place?

Gee, nice to be right about it, even if I was a year early; then again, if you would have done this a year ago, perhaps we'd be a year down the road from this problem, instead of a year later and the problem still exists?

Good Work there guys, really fine work!
_____________________
Bob "The Builder" Bunderfeld

"There could be a 13 year old Genius out there smarter than I am." - Blake Rockwell
1 2 3 4 5 6 7 8