Weekend Grid Support

Repost from FJ Linden's blog post found here: http://blog.secondlife.com/2009/01/23/weekend-grid-support/

Before we hit the weekend, I wanted to update you on our progress on Grid stability. We have taken a number of major steps this past week to improve database performance, which included offloading high impact queries to slave databases, and completing a master database conversion on Thursday. These actions, along with many other performance tweaks, have significantly improved query response times and reduced load on the central database.

While I am optimistic that these measures are helping to stabilize the Grid, I also want to be ready for any event contingency. We have mapped out a detailed response, escalation, and triage plan across the entire company. One of the major changes for this weekend will be the corrective steps we take if the central database becomes overloaded and unstable. Past procedures were to almost immediately block logins, to allow for load to subside and protect the database from a complete crash (which is what happened last Sunday). This is a rather drastic approach to protect the database, and we’ve now put some “throttling” mechanisms in place that would be our first step to reduce load vs. blocking logins. Throttling literally means turning features and functions off, and potentially degrading in world experience. One of these throttles would disable group queries to the central database, and would cause group features to be unavailable during that period of time. While degrading the resident experience is not a preferred step, we believe that it is far less punitive than the drastic measure of blocking logins. So if some in world features are temporarily disabled, you would still be able to log in, and enjoy other in world features that do not tax the central database.

We have an entire support team that is standing by to react to any potential problem, and will aggressively communicate with the residents, through multiple means (in world, web, blogs, etc) to keep you informed of any problems and progress to correct. While I am optimistic that none of these recovery steps will have to be taken, I think it is very important to continue proactively reaching out to the resident community on our plans.

Ok, so will group queries be limited because they are seen as the less valuable query or because group queries are such a major drain on resources? I'm hoping it's the former but fear it's the latter.

It never should have come this far. LL should have anticipated much much sooner on growth and database strain. Nonetheless, FJ is being honest about the problems and at least tries to take measures. I appreciate the openness.

But really LL, you should be ashamed that you weren't prepared.

thanks for all your efforts.

any chance though, major upgrades could be done early in the week (monday/tuesday) so that by the time the weekend hits, it's likely that the kinks will be ironed out? weekends are the busiest time for merchants and also land sales, and when the service is down on weekends, we really feel the hit, much more so than from weekday downtime.

the major database upgrade that started the latest downtime snowball rolling down the hill was done on the 15th (thursday), which is why i bring this up. the timing was not good.

Will group queries being disabled affect building on group land?

Will you please send grid notices before you start "throttling" and tell us what is being throttled with potential impact?

Thank you.

P.S. I have heard many people say things have improved with the conversion you just performed. Many thanks for that.

Congrats on proactive measures

Thanks for the update, it's appreciated!

Could you please send a global in-world notice to the grid when you disable / throttle features?

I personally like the part about agressive communication.

*Finally* there is a will to keep a decent communication (and hopefully in a rather fast one too?) with the userbase (aka residents). That should have been done more last week than yesterday...

So thank you for being more honest and for actually telling what is going on. Anyone that knows something about networks, databases and similar things will appreciate good communication deeply.

Now if you only could be a bit more specific and less techtalk-y on what the group queries are and how they impact everyday life in SL... I'm having a blonde moment and I'm redhaired!

Please keep the communication going!

First of all I want to say THANK YOU!!! I'm glad to see I'll actually be able to log in this weekend during peak hours, I've got many things to upload - hopefully that will be one of the last things to be turned off (I doubt it though )

One question though: What is the problem with limiting logins to premium accounts during the stressed times? I don't understand why LL hasn't done this yet. Many people not only own land, but manage an entire estate, or numerous estates. I really don't like the fact that my weekend plans and work are hindered or completely cancelled while I'm paying $300.00+ a month... I think it would be a great way to keep customers happy.

Haven't we beat this dead horse enough? It's time for the burial.

You don't get it do you? YOU have yourself chosen to pay that much for a virtual platform.. everybody is a customer in a way on the grid.. incl. non premium users.

Infact, most users are NOT premium. And those peeps, make the economy flow, by spending, etc etc.

Most premium users are land owners (mainland and regions)
and don't spend themselves as much as basic accounts do. (ecept directly to LL, but that's out of scope for the inworld economy)

If you block non premium users.. well.. what's the use of bein able to login, if nobody buys your goodies.. or is able to pay the rent tiers on land? (if your a landbaron)

think about that for a minute.. you actually don't want income from your venues as you state it yourself.


ANYWAY...

keep up the good work, FJ and others at LL..
but think also byond the current solutions.. think big scale, spread accross the world. (open your server code, or make it possible to host regions on the customers machines (if requirements are met) either through licensing (cheap) or as opensource (free), like the viewer codebase..

I have a question for you please, I was told by SL technitions that the problem for not being able to Log In to sl lyied in my PC. How ever i am running the highest Graphics Card and resolution on my PC. When i checked the recomendation on SL it turns out that I'm within the limit. Now SL techs say that I need to upgrade my Graphics Card, but some reason I was then able to Log In for a couple of minutes.

If my Graphics Card needed updating then why did I Log In for these few minutes, it shouldn't have let me in ever. Right? And furthur more why aren't the SL Techs getting back to me with all these answers to my questions? Something doesn't add up here what is SL getting to? I have a business on SL and loosing money everyday im not there, now how is this problem going to be solved?

You see these are questions that no one of the SL Techs can even answer. I even went to extreems and un-installed the SL program from my PC and re-installed it back again to see if however the problem lied with me, but obviously not.

Please if you have any answers for would really appriciate to hearing them.

Thank you,

Camron L.

For another informative posting.

Whether or not matters should have been allowed to reach the current situation is moot. They have - and that you are ever-willing to come out and both post the reality of matters and face what can be the howling masses is appreciated.

Clear, honest communication may not solve the issue - but it does give people faith that actions are being undertaken, and thus hopefully encourage them to be a little more understanding when things do bork.

Certainly, I'd rather read postings that are direct, honest and to the point - such as yours - than the more spin-loaded postings made by some of your colleagues.

Could you possibly have a word with them about the benefits of honest communication?

ok everything has now gone kaput. I cant see anything but myself as a cloud cant interact with anyof my objects on our land. keep getting booted too. our land has lost its name is not showing on maps either. this is crazy your fix made it all much worse at least for me and my husband.

Eva the problem lyes with SL entirly, you are not alone I pay a Tier of $20400 a month and own my business, I am loosing an average of $3000 a day. So u see you are not the only of with Log In problems and communication with SL...

There are alot of people out there the same as us, but SL Technitions dont want to do anything about it. I say shut it down completly and upgrade and update anything that needs to be done and have consideration for the people that have slower PC and the rest that have businesses to run...

All I know is that we as users have reall money involed on this game and would like someone from SL to consider that...

So what was even fixed really cause I am still havin the odd problem when I go to rezz etc.

Interesting theory. Have anything to back it up?

Personally, I find this article interesting:

http://gwynethllewelyn.net/2008/10/13/the-hard-facts-about-the-second-life®-economy/

Here's the money quote:




obOnTopic: FJ, you and your people (who we don't get to see) and Prospero and Vektor and Andrew and Dan and Bao and Ramzi and bunches of of other Linden folk who we don't interact with (and don't get to see either) have pulled together a metric buttload of critical technical improvements to Second Life and delivered them in rapid fire in the last week; including the database rescue, server 1.25 and viewer 1.22.6

It may seem amongst all the traffic from the residents that there's little appreciation of how much work it takes to perform at that level, under the big top and working without a net. There certainly isn't enough appreciation expressed, and perhaps not felt.

But some of us out here do appreciate it. And if this is for some reason not a completely successful weekend, it's surely not because you haven't been trying.

Yes i can back every word so far, I have been trying to recitfy this problem for the past 26-30hours now and still I'm not getting anywhere with SL. They have contacted me with some answers but when it comes to sirious questions no is there to answer me. Which to me only prooves one thing so far. No one from the Technical Support knows anything.
You have no idea how frustrated I am or how Angree I am, that they take no consideration of even getting back to me and informing me of the current issue. I have and so dose everyone out there REAL MONEY involed in this game but you would think that they have been here for as long as they have been, have this problem escilate to this extend is byond words. LL have no idea of the amout of people out there that have the exact problem I'm facing right now.

And just let you know I have to pay a Tier of $20400 for my business a month, and furthur $2100 a week for my home. Now what happens to these two places if I'm not able to go back into SL? I tell you I WILL LOOSE THEM!!!!~ But does LL give a stuff no, they dont otherwise they would have contacted me and inform me of the current situation.

Now If you or anyone out there can defend them (LL) bring it on.....

Yes, this issue is bogus. I added more than 2.6 million L$s to my account in 2008 alone, and I am not a premium customer nor do I plan to become one, ever. And you can't measure by land ownership either, I almost bought two full sims last year, but held off due to the platform instability issues, persistent bugs, and watching as Linden Labs arbitrarily and unilaterally jacked the prices of open sims and focused their resources on things like Dazzle, XLStreet acquisition, new website design, and so many other things, while ignoring crucial issues like the inventory interface, camera view that flys in and out of the screen, persistent avatar clouding, and a whole long list of other things that don't work right, or the service provision is bare bones.

Instead I rented from someone else who owns a sim. So I pay them tiers and not Linden directly. So you can't kick people off arbitrarily just because they don't have a premium account or own a sim and pay Linden direct, this would kill LL's business model.

As far as login restrictions go, I hardly care any more if I can log in or not. The in-world experience is so bad, so often, that it just isn't fun anymore. So I am scaling back. and if things continue to go as they have been, I'll be out of here and find something else that doesn't cost anywhere as near much money and find a supplier that can deliver a consistently reliable platform.

Laska

Camron can you log in and stay logged in from a different computer?

If not then when you manage to log in you need to edit-detach all immediately. If that works then something you had attached is messed up. I have seen people with defective animation overrides or dance chimeras have such maddening issues.

Also can you log into the beta grid and stay logged in?

Um, I wasn't actually talking to you.

Are you claiming you are a typical basic user? I doubt very much that you are.

I'm a basic user too, although my family pays for an island, it's not registered to my account. But I'm not a typical basic user, and it sounds like you aren't either.

I'd go premium if I got anything for it...like access when the grid is throttled. You should consider that option too.

I would like to know what "to disable group queries to the central database" means in real terms...does this mean that residents will not be able to access group chat...or that they wont be able to invite residents to a group during these "throttled" periods? Will group invitations be queued and performed when the database is active again, or will they be ignored alltogether? Will residents be able to change from one group tag to another...or will all of this be disabled during these periods...so that more people can log on that ALSO won't have access to group resources? I would like a definitive list of resources and abilities that won't be available to use when group queries are disabled, please. What you have said is far too general in terms to know what the true impact of this announcement means to residents and businesses.

Temporal Mitra

I have a question too.
Could we know exactly what about group functions would stop working?
I have the whole SIM set to no scripts except for group
No parcel entry or building except for group
and my private building area is group access only.

If how any of the land settings work will be effected by limited group features, it would be good to know now.
Some preparation would obviously be needed, just in case.

Thanks

Good Luck Every Sunday has pretty much sucked as far back as I can remember. I usually avoid Sunday like the plague. Perhaps this will change... I hope so.

Much luck to the team

Mar

Finally proactive communication! This weekend will be a fun ride, we will see how far over 80K the grid can go until something else breaks
Keep the information coming to us and we will keep the feedback coming to you.

Good luck to us all!