Weekend Grid Outages

hello,

and can we expect a return of the lost inventory items ???

I was working on my teeth when it went down on Sunday. There I was in limbo for three hours with all my teeth unlinked!

im laughing at that one !

As a business owner and over 4 year resident it has become horrible every single weekend not just this weekend and frankly it is killing business. I understand more people are logging in which you encourage with free accounts so perhaps until you can handle the load we should stop allowing new sign ups for free? You don't seem to be able to handle the ones that are here and it is seriously getting tiresome to not only not be able to conduct business each and every weekend for those of us that would like to work on product production on the weekends you are making that impossible as well.

Again with the bots taking up people slots...

Frank, LL have said a few times that you all know about bots and keep an eye on how much of the population is actually alive.

Any chance you could pencil in a blog/post about the various types of bot that are out there, what LL thinks about their impact to the grid is and what, if anything might be done about harmful ones? Feel free to use the words "arms race" as often as needed.

TY!

Hi, I wanted to get a quick update before afternoon meetings. We have a daily grid status call kicking off shortly. Just to respond to a few questions, peak concurrency is not necessarily the main contributor to our current issues. We've actually crossed well over 80K already. Right now its the number of queries hitting the database, and the structure of those queries. We are in the process of identifying and offloading those highest volume queries, while also looking at configuration settings (e.g. database cache size) that will tune performance.

We are certainly operating in a tactical mode this week, with the objective to get through our next peak on Sunday. I'll check in again later this afternoon when I have more information about existing efforts. Also, this weekend's crash was very unusual. We have had regular problems, but recovery has been a fairly straightforward effort (albeit with resident pain). Something else was a miss this weekend, and is likely a configuration change that caused the actual database crash and long recovery time.

This will likely never happen as too many components are hard-coded, for example, notecards are recreated with new keys every time they're written to.

Thanks for the updates Frank, interesting reading. The proper terminology, for future reference, is "The Central Database All Hail The Central Database" kthx

Only the enterprise data model needs to reflect normalization. Anyone that has spent any time on massive enterprise systems knows there is a jump between the academic model and the high performance data structures.

But with Second Life we are talking about the need for an enterprise model that is evolved into an object model and the persistent data stores support the object model properly.

But it all starts at the metadata layer. If there is no enterprise metadata repository then it is an epic fail to begin with.

Enterprise, of course, means "add an extra zero to your contractor's hourly rate". Or in the case of Oracle, it means "Add an extra two zeros to the size of the software package".

I don't see why with all the issues and lost of business for IN world folks. perhaps set a date for all to know and shut down for 6 -12 hours.. what ever needed and FIX...then we all would regain our time and fun ... and business opportunities... because its more than a pain to try to log in then be told its down. then up then down.. I get tired of trying and don't bother.
Its why I canceled my paid account. its only gotten worst... not better.
I spend more time just trying to TP or sign in...

I think it will help business as well as others if you fix it by shutting down for few hours .. No one in SL .. then no lost items or stale transactions. Just my quarter...

(hides from the yelling)

FJ Linden: Thank you for saying all this in public. It's nice to see a 'yeah, there are problems and here's what we're doing to fix them.'

CAN HAZ SCALE?

They used to do this, and we all bitched just the same as we do now.

The problem there is that they need the load testing to find what the problems are, so closing down would not help at all.
I would guess that things will get worse before they get better as far as the database is concerned, but as they move parts off to different areas, things will improve, hopefully quite dramatically.

That sounds to me as if you have a problem with your video card driver software, check the manufacturers site for the latest drivers, that should cure the problem.
If ypou already have the latest ones, reinstal them.

Hi FJ,

Thank you for all the great information. I have a question for you regarding object inventories.

One of the ongoing annoyances I have in SL (even when things are "working well" is trying to update vendors or similar inventory boxes that already have many items (100, 200, or even more) in their inventory contents.

Adding, deleting, renaming, etc. often takes an incredibly long time (sometimes as much as 10 or more minutes) to update a single change before the next change can be made.

Will the database restructuring improve this object inventory issue or is it primarily caused by something else (one Linden told me, if I recall correctly, that basically the reason was due to communications that weren't very efficient)?

Thanks!

sure we will...just like the ones in 2005 that said 2006 was all about stability and the ones in 2006 that 2007 was all about stability. We even read the ones in 2007 that said 2008 was going to be all about stability. This new message that they are going to focus on stability is very refreshing.

For a while, back in the spring or summer (I forget just when), specific services were shutdown when things got very sick. Presumably those mapped to specific problematic queries, or some other concurrency bottleneck at the database. We haven't seen those in a while. The current SOP seems to be to throttle logins when the DB is feeling fussy. (I guess I'm assuming that the login service itself isn't what's causing problems.)

Unless it's already been explained, why the change in approach?

Not a fix, but a sort-of-workaround I use for that:

Find three or four other objects in the area, each with a tiny bit of inventory (just some random thing with a script in it will work, even your hair etc.) With the contents tab still open, if you select a few other items in turn, then go back to the one with the giant inventory, that seems to flush things out so that the contents of actual interest reload a lot faster.

Seeing that logins have been disabled again and there is nothing about it on the Grid Status page... I'm wondering if there is some way to speed up the posting of information on that page. I have now been trying to login for about 10 mins, but still nothing about it on Grid status - raises all kinds of questions...

Also perhaps to try and give some indication of the severity of the problem.

For example when you found out the DB crashed on the weekend - it would have saved me a lot of time to know that it was a "severe" issue and I could stop sitting there trying to login again and again as opposed to what ever the smaller issues are that take 10 minutes to solve. Just a flag for it minor, unknown, major - something like that would help.

I think we understand the problems you face - and appreciate your efforts to fix them. The more information you can provide the happier we will all be (or at least that's what I would expect)

anyway just some suggestions - hope that helps.

These outages always occur when there are high volumes of users. Over the weekend it seemed to avarage about 75,000 online. Maybe it would help to stop creating new FREE accounts until the system has been upgraded to cope with the traffic. As a PAYING Customer I feel I am justifiably annoyed when I am unable to gain acceess due to non paying accounts overloading the system. We do not get a discount or refund when the system is not available yet we are the people providing the content that is bringing you customers.

You have stated the outages are not dut to peak usage but in the next sentence you say it has something to do with Queries? surely if there are less online there will be less queries?

PS I just tried to log in again and surprise surprise logins are blocked as there are over 80,ooo online.

PERHAPS ALL PAYING CUSTOMERS SHOULD DOWNGRADE TO FREE ACCOUNTS SO WE ARE GETTING VALUE FOR MONEY!!!!!!!!!

I pay 300.00 a month for a private estate (and not easily...), and I know many people who pay more. Why can't the people that pay this insane amount of money to be a part of Second Life be allowed in before bots and non-paying members of SL? That seems only fair.

Usually the people that pay monthly voice themselves about Second Life on the internet much more than the non-paying members. Good for publicity, and good for just all around fairness.

Shut off the bots, for starters.

As a paying member I am starting to wonder what it is that I am getting. I see 90% of my friends online that dont pay a monthly fee and they seem to have no problem getting on line when SL crashes.

I as a paying member am off line more lately than on due to the crashing. It takes hours to log on while I sit here and see all my non paying friends continue to have access with out interuption.

So I ask what am I getting for my monthly fee? Why continue to pay it if I can get better service as a non paying customer?

As Hurly stated" Maybe it would help to stop creating new FREE accounts until the system has been upgraded to cope with the traffic".

You think that payment status has something to do with whether or not you hit bugs?

It surely doesn't matter what people bitch about. What matters is what works. So I'd be disappointed with LL if the easy route of bitching had any impact at all on the actual work they should do for best results.