Grid Emergency Maintenance 1/22/09 5:30-6:30 AM SLT

Reposted from FJ Linden's blog post

http://blog.secondlife.com/2009/01/21/grid-emergency-maintenance/

I blogged earlier this week in response to our grid outages over this past weekend. We have put enormous efforts into fine tuning the data layer, specifically in optimizing queries and cleaning up the data structure.

However, there is a major maintenance step that needs to be completed tomorrow. We are scheduling a 60 minute maintenance window beginning at 5:30am PST. During this window, we will be migrating our central database to and optimized slave database, and making that slave the master. In this way, we expect a significant bump in performance and wanted to take this action as quickly as possible, most especially before our highest load times over the weekend.

During the maintenance window, we will be blocking logins, but those residents who are already in world will not be bumped offline. However, there will be a degradation in performance, as access to the database will be blocked, so transactions, teleporting, asset management and actions requiring a database call will not be available. Movement within a region, chat, and voice will all remain available during the database migration.

While we are sorry about this late notification, and the need to complete this maintenance work, I believed it was important to move quickly and aggressively to address our current data stability challenges, so I advocated completing this work tomorrow. We have made some real progress this week, and this maintenance activity will begin to take full advantage of this work. Thanks for your patience.

I say go for it!

yay... Glad this is being done, hopefully it helps

Hope it goes smoothly!

any more technical details about what you are doing?

Just curious

Why not just block logins for an hour instead? This way no one gets a surprise.

so this one is planned as opposed to the random nonsense we put up with lately. you people amaze me. not only do we lose money (and we pay for this nonsense remember) but once again no thought of even a little 'thankyou for your patience' gesture. what happened - slex and onrez proved too difficult for you?
get it fixed

Jeeze, they give us warning and people still complain.

Hey Frank!

Thank you for the heads up and addressing this issue on the blog, and thanks for the serious approach on the subject, this sounds a lot of work.

Another great point is this being done during week days, with specific time etc. I believe this kind of announcement is very important; its like a rl tornado, where people know it might happens, but still they blame the govt for not alerting about it. We know its really a hard effort to make SL all work together in real time, so database/performance downsides are at least expected once in a while!

Thanks again for the work LL is determined to do regarding performance/database, and for the sharing those efforts on the blog! This is the kind of thing that builds trust.

Satir DeCuir

This is the way to go FJ. Thanks for your heads up and good luck!

The improvements in the central database performance over the last two days has not gone un-noticed. Well done! If this switch-over adds even more optimizations, then I say yay!

Been Threading the use of Bots in SL over a few Threads now.

Result: closed, because i was out of order according to the moderator.

Now i am totally confused. I really am. I truly am.

Half the Posts are about those Bots, right ?

First i get the advice to Thread it... closed, because not allowed to publish region names.
Everyone laughing at me, because of those posts. Linda, Linda, Linda.
Got an advice to take it up to Jira. Advice Linden employee... Jira is for Techies.
Have posted Abuse reports... nothing.
Have filed tickets... nothing.

So... Linda is not to be taken seriously as she is not a techie huh.

Now where can i post regions that use those traffic bots on a major scale.
I am even risking an IP ban. So LL you can check IP's right? And thus Alts & Bots

Get rid of them PLEASE !

This is not a personal vandetta, geeeez, it is sticking my head out.
We know those bots are no good for the performance.

do what you need to do..

The instability of the grid over the weekend was annoying, to be sure, but I have studied LL grid infrastructure long enough to know just simply running a grid with any kind of stability is an amazing feat. my hat is off to LL.

Most businesses including where I work in my RL life job we shut down systems completely and even then 100 percent up time is never achieved all of the time, computers will crash thats the nature of teh beast, you want 100 percent up time I couldnt afford to play here and help make SL a better palce myself.


The downtime will be beneficial for all residents in the long term and LL is still growing at an amazing rate, with new technology ever pushing the barrier of what CAN be accomplished over the internet.

530 am to 620 am scheduled downtime is a must do, otherwise we may never get past these bumpy rides. Hang in there everybody.

I think Ill sit in my newly built house and my newly bought land and watch tv while that is going on tomorrow.

Shouldn't there be some mention of this on http://status.secondlifegrid.net/ ?

Hey Linda, I hear you,

There was a type I was really annoyed with the regions/stores that use bulk loads of alts in the sky, like 80+, to boost traffic... that usually comes with a group of noobs on the ground IMng every person that tps there to put their place on picks, and get 25l / week... I think these traffic bots jam the grid, not to mention its cheating the search system...

I was like you, crazy to do something about it, specially when really lazy bad quality stuff is like on 1rst page on search cause of sleazy techniques, but my worry about this soon got due cause I got a bigger worry, content theft issues... so its like worrying about a cut in the toe when you are shot in the head.

I don't think posting region names is the best solution, maybe LL needs to see the general picture... a high "users online" and a high user numbers are great, but LL needs to see at which price does it "pays" for those high numbers... does it worth, a 10, 20, 30% "users growth", if the grid performance and search engines are compromised? not to mention content theft with alt accounts being made every minute. Maybe a bit more quality then quantity would be a nice try ... well that's just my very humble opinion.

Thanks,

I am pretty sure LL does see the big picture, however, they don't seem to want to listen.
Sure Bots/Alts boost Concurrent users too.
My fingers are anxicous to name regions....

Ok, i'll open a whitty Thread about it, not call names but refer to my blog where they can be named to file to LL, how's that !

I hear you too.
Content theft is bad, i understand, however it is not about a peformance issue which is the root annoyance of our social thingy performance ( SL ).

Good luck to you and Prospero too.

/me wanders off to set his alarm clock a bit earlier.

Ooooh, I'd love to help with that, hehe. Can't wait to see said thread. =3

Second the motion. One big failure of communicating with LL customers is that it is inconsistant.

Bravo for the upgrade though!

I applaud the upgrade as well but I keep wondering:

WHY did they take notices like this off the login page? When I first started SL, they had one blog for info like this and you could easily see the topics on the login page in the viewer. Then they removed all that info, separated the info into multiple blogs and added these "rss feeds" and "twitter" stuff that, while I'm fairly techy, I have no idea how to access those nor do I care to learn. What was simple for everyone, turned into a complex confusing inconsistant mess that IMHO looks to be intentional? Now you can easily log into SL, have no idea they are having issues until you get burned by it. I have no idea where to look to find any consistant grid reports.

On the bot issue, what I don't get is if they banned bots, more REAL users with REAL USD to spend would be logging in. The concurancy numbers would probably be even higher and imagine this, instead of 50 avies from one IP, that is making one tier payment, they might have 50 paying users? Or 25 more anyway. And that one "bot user" probably wouldn't leave because bots are banned either. I would think they would like to have less strain on their system which in turn would reduce their overhead costs. It just doesn't make sense to me to WANT these leaches on their system..

i am all in favour of attempted (attmpted emphsis) upgrades but it does not excuse the fact that 05:30 is 08:30 here on the east coast and even later in the rest of the world. good luck and i hope you win brownie points with the suits - oh look we came in early. if there are any suits

Thanks again for the feedback. Just wanted to close this forum with an assurance that we have a full plan (including multiple contingencies for problems) and have gone through a great deal of testing in a short period. We're trying to balance speed (to implement improvements) with the correct level of caution (testing is good!).

i hope it will help and it will be better, we are paying members and may get something good back for our money, but i keep hope, greetings from holland