Second Life Grid Update from FJ Linden

May I suggest you relocate to some communist paradise, such as North Korea? Linden Lab is, in the end, in business to make money. Without that incentive, there would be no Second Life for any of us. A premium account (and mine is not) should be worth something, and the stipend isn't it.

If logins are disabled, then could the information on http://status.secondlifegrid.net/ please be automatic, reliable, informative and timely.

It's 1:30pm on a Sunday with 78K people online. Logins are disabled, and attempts to log in are met with an instruction to check http://status.secondlifegrid.net/.

Here we find the following 2 hour old information:

[RESOLVED 11:26 AM PST] Logins have been re-enabled.

[11:18 AM PST] Unfortunately we are having continued problems with the database. Logins have been partially restricted again whilst we clear a momentary database bottleneck. We expect to re-enable them very shortly, and In-World will not be affected.

[9:47 AM PST] Logins have been restored and all inworld services appear to be functioning normally. Please feel free to resume your regular activities, including purchases and rezzing.

[9:39 AM PST] Technicians are currently investigating an issue which may interfere with inworld functionality. Logins are blocked. Please refrain from making inworld transactions and rezzing no copy objects for the time being. Please monitor this post for updates and further information.

So logins have been yoyoing up and down all day, but they're disabled now, nobody has given status or outlook, and the login page still has a green ONLINE status. If the system knows that logins are disabled then at least it should be able to report that before people try to log in.

From 12.27.08 to Now every single day there has been at least ONE Log Ins Disabled Message.

To be honest it is nice to hear what improvements you made but.......

you know since months that you cannot handle more than 75k traffic at once and the only solution you have is to disable logins for people!! I am really upset about it by now, can you tell me when your network will finaly be able to handle that user base??

You allow prolly 10k bots to be online and have normal users not to be able to log in because of it!!

PLEASE take some actions to solve that problem!

When they stop being noobies and implement the correct distribution of their software across their whole server network rather than leaving 95% of the servers idle while 5% work overtime.

"Its been a bumpy few week" a bumpy few weeks? you got to be kidding

A long time ago LL explained they were working on a process that would restrict logins based on user status, i.e. no payment people would be the first to be restricted. I'm sitting here with an event scheduled, can't log on and I PAY to use this service. At some point are your paying members going to get any sort of perks for paying for this service over people who use it for free? At least the NPIOF people who never have supplied any money to LL for using this service? This would block bots from trying to log in at the same time.

While you say it's not caused by concurancy, aren't the "database query spikes" directly related to the amount of users online? I fail to see the real difference in that lot's 'o people logged in = problems. Everyday this week at some point money transactions have been broken and my account balance reflects this. What really bothers me is higher users online -should- equal more sales yet when this happens, no one can buy anything. This has been going on for a long time, this asset server problem/dead assets etc should have been a top priority a LONG time ago!

How about really bumpy?........The last three days have been really bumpy with logins closed down more then I can recall. Perhaps the asset server needs an antacid.

This is what Sunday looks like so far....can you respond to this please?


[REOPENED 3:09 PM PST] Logins have been restricted again. Please watch this post for continuing updates.

[RESOLVED 2:26 PM PST] Logins have been re-enabled.

[REOPENED 2:01 PM PST] Logins have been partially restricted again. Please watch this space for continuing updates.

[RESOLVED 1:46 PM PST] Logins have been re-enabled. LindeX should also now be available.

[1:27 PST] The Linden Exchange is temporarily disabled. You will not be able to purchase or sell Linden Dollars from in-world or the website.

[REOPENED 1:08 PST] We are continuing to have problems with the database. Logins have been partially restricted again.

[RESOLVED 11:26 AM PST] Logins have been re-enabled.

[11:18 AM PST] Unfortunately we are having continued problems with the database. Logins have been partially restricted again whilst we clear a momentary database bottleneck. We expect to re-enable them very shortly, and In-World will not be affected.

[9:47 AM PST] Logins have been restored and all inworld services appear to be functioning normally. Please feel free to resume your regular activities, including purchases and rezzing.

[9:39 AM PST] Technicians are currently investigating an issue which may interfere with inworld functionality. Logins are blocked. Please refrain from making inworld transactions and rezzing no copy objects for the time being. Please monitor this post for updates and further information.

Even more amazing, defending your use of bots

BTW if they add no load to the system, why would they be working on the "tiered" database system? Every account made is more data being stored by the asset server. When one user creates and loads 50 bots onto the grid, that is not a "miniscule load". As someone else stated, this causes other people to not be able to access their land etc. And the land bots should have been banned a LONG time ago!

This is getting ridiculous. 18th January continued database issues from early morning. When are you going to stop being ridiculous enough to keep allowing more than 70K online at any one time when your database cannot handle it.

Get rid of the bots and you might be able to cope, with realistic numbers on line, instead of giving the bots free reign.....cos it keeps your online figures up....!!!!!!

I wish Phil wouldn't defend bots.

I disagree with almost everything he says and find it a little disturbing that somebody would actually be so ok and so vocal about the way they cheat the system. On this point he's right, though - things like traffic bots really add very little load to the system. Land bots are a different story but traffic bots are pretty harmless. Honest? No. Harmless? Yes.


Why would a traffic bot have anything in it's inventory?

Time 16.55pm Sunday PST..... I crashed out. logins disabled... OOO.... just 38K and a bit on line, and the database is still screwed

Time 17:05pm PST.... numbers still dropping now 36.25K and dropping......Is the ship sinking ?????

Not just today, but reviewing the blog entries for the past couple of weeks reveals that this has become a daily occurance (there was only one day last week where the system wasn't brought to its knees).

I don't know much about how how SL's infrastructure is set up, but I do know enough about databases to know that MySQL can easily handle hundreds of thousands of concurrent hits IF the database is set up correctly. If SL's database is being hosted on only one server, then that would be the problem right there. Distributing the database over a large number of servers in multiple geographical areas (to eliminate the possibility of backbone bottlenecks) seems to me to be the best way to cure this problem once and for all.

Someone mentioned freebie accounts as being the blame, and - while I cannot necessarily disagree with that, I still take exception to that. I am not a premium member, and really cannot be until the RL economy improves. However, that does not mean I am not a contributing member of the SL society. I work in-world to earn my Lindens, I rent land and pay my "tiers" (the money from which does eventually make its way back to LL's coffers). I buy things and I build things to sell, thus helping to keep SL's economy going. While I acknowledge the point made that a lot of "freebies" do little more than use up resources without contributing to the game, we're not ALL like that.

LMAO SL hits over 80k in log in's today and just about totally fails...of course this is NO suprise... I have been watching this go on for almost 3 years now...EVERY sunday

Well of course its related to the number of people online. This sort of thing only ever happens at peak times during the week and the weekend (specifically sundays) when people are off work.
LL will never say 'we don't know'. All they will ever say is something rather long and random in the hope that it will 1) sound intelligent and 2) you won't understand it.
These problems are caused by concurrency.

I agree that free users should be restricted. I pay my fee annually. When you go to the shop and buy a new mobile phone. You don't get sold a broken one, do you?
Second Life is essentially a product and, while its been around 5 years, its clearly NOT out of beta at all (because not being in beta implies all the bugs are worked out and that everything is going hunky dory). I don't know how LL expects to make Second Life and OpenSim (Which is something whose days are numbered from the outset) to be attractive to corporations like IBM when they can't get the thing to work properly. If I worked for IBM and I was on a deadline to complete a project and missed it because I couldn't log in, or the database was down, I would not be very happy.

so 80000 is all the more people can log in what about the over 1 millions people that have join do they get screwed.

No. They just have to sit around waiting to be able to log in when the cluster goes kaboowie.

THe BIG problem with restricting logins to premium users is that ALOT of what "used" to be premium members are now free accounts...like myself! I Got sick and tired of paying for crappy service and not gettting anything in return.. so I down graded my account. MAYBE one day when LL gets there acts together I will upgrade my account again and I am sure many others will also...but till then...FORGET IT

Please kill the bots and re-think or do away with the traffic system.

The Bulk Storage idea is a great step forward IMHO. This will relieve a lot of the load from the database and probably should have been done a long time ago. Who needs 20,000 objects in their inventory?

I would like to say, as a old Database Administrator, that whenever there are issues with the system the first thing to say is well "it's a database issue". Actually it hardly ever is. More times than not its the new code that has been written inefficiently and/or not tested properly or its a network problem.

Sounds like that the current problems are capacity related and the steps being talked about in this thread should help alleviate some of the congestion.

I think FJ's ideas all sound good. However I hope this weekend is coincidence, and not because bits are being changed over.

I would *love* to see change, which I am sure wouldn't be to hard, is to show logins are down on the login screen on the Second life grid status. I'm sure this used to happen. And also until about August (I'm guessing) the status/blogs were all indexed on the login page, so you could easily spot problems, new blogs, or even watermelon tutorials. It would mean we don't have to keep looking in three places.

Rosay

As an antipodean payed account holder, I have been unable to log my account in during peak times for the last 5 days (prior to the current fiasco this weekend 17 & 18th of Jan). The 'Support Team' initially tried to fob the problem off as my settings/computer /ISP but I knew the issue was to do with the asset servers, I suggested this to them and even though I have posted several comments regarding what I have/or have not been able to do, they have not responded in the last three days.
I don't really care why anyone chooses to use a non=payed account, that is completely their business, but as some one paying for a service and because that service is now so poor as to be described as dysfunctional, I think it is time that instead of restricting logins across the board, the time has come to restrict non-payed accounts and in particular 'bots'. I am a simple user and business provider in SL... I don't pretend to know anything about the intricacies involved in keeping the grid afloat but it is a question of ethics... why should the people who pay so that the grid EXISTS be unable to log in when there are unpaid accounts and bots inworld? I don't know how true it is but have been told at anyone time there can be up to 10% bots inworld - that is an unacceptable situation even if the grid was functioning in a decent manner . I would be more than happy to have the grid shutdown for as long as it takes, repaired and put back online so that it works for everyone to enjoy, or rolled -back to a point where it does work and then a steady more methodical approach taken to its update and improvement. The blog before this one is a LL gloat about the increased numbers and cash-flow but for me and I assume many others Second Life simply doesn't work. The current 'Blog Grid Status' reads as a roll of shame... the last one about how at this time LL are unable to announce inworld that there are extreme technical functions, is an example of how bad the problems really are. It is time people in blogs like this stopped making excuses for LL and forever blowing hard about the technical issues ad nauseum that most of us don't understand. It is less than 6 months ago that an update roll-out occurred with little inworld instability... What went wrong? And why can't you fix it? I know the response to this entry will include those who will bleat on about how everything is fine for them, and sure that's great for them, it still doesn't help me. LL can't tell me why and you can't tell me why but the fact is Second Life, for all intents and purpose, has ceased to function as a quality service/product. The first time user logging in this weekend (if he/she was even able to do so) will probably not be back and will certainly not be telling their friends what a great product Second Life is.

THis is for all of you folks that think you should get special treatment because you are still a premimum meber.....just look at all the "special" treatment we got for investing in all of this OS sims! hahahahahaha (point made)