Load Problems with SL

Hi,

We are still having problems with heavy login load on Second Life, as of 2:30PM PST on 12/31. The grid is currently open, but logins are very slow. We are working on some quick fixes that we are testing right now, and additionally are getting new hardware ready to replace the equipment that is still running slow. I don't have an ETA for repairs. We may get some speedups in the next couple hours, otherwise there will likely be grid downtime tommorow (1/1) as we change hardware.

We're here at the office - doing all we can. Apologies for the problems.

Still working.... update as of 5:15PM PST 12/31

Logins still very slow. We've experimented with shutting down some of the database processes that cause load (there were cases where we could do that without kicking everyone off, so we tried it). No signifigant relief - login servers are still overwhelmed.

We are continuing to try to debug and look at changes that we can implement now. We will also likely have a downtime tommorow morning to update the database servers to new hardware.

Update, 7:45PM, 12/31:

We found a bug in the viewer that was causing most of the trouble, and have released a patch. Your viewer should auto-update and you should now be able to login. This is a patch to the Windows viewer only - a Mac update should be available sometime tommorow. If you have a Mac client, you may still be able to get in without the update, as the load level drops on the system.

The problem was not in the database, but rather a race condition in the viewer that caused logins to totally fail when the database was slow. The database disk failure slowed login performance enough to trigger the bug, which is what has blocked many logins since yesterday.

We will not need to bring the grid down tommorow morning, but will still schedule downtime for hardware updates to the database sometime early next week. These upgrades will further speed login time.

Very sorry for the problems, and Happy New Year to those enjoying it in SL tonight. Great work from the LL team who came in today and got this fixed.