Second Life Forums Archive - CEO's Comments Thank You Phil!

Merwan Marker

Booring...

Join date: 28 Jan 2004

Posts: 4,706

12-31-2004 14:35

We are still having problems with heavy login load on Second Life, as of 2:30PM PST on 12/31. The grid is currently open, but logins are very slow. We are working on some quick fixes that we are testing right now, and additionally are getting new hardware ready to replace the equipment that is still running slow. I don't have an ETA for repairs. We may get some speedups in the next couple hours, otherwise there will likely be grid downtime tommorow (1/1) as we change hardware.

We're here at the office - doing all we can. Apologies for the problems.

__________________
Philip Linden
CEO & Founder, Linden Lab
blog: http://secondlife.blogs.com/philip

_____________________

Don't Worry, Be Happy - Meher Baba

Danny DeGroot

Sub-legendary

Join date: 7 Jul 2004

Posts: 191

12-31-2004 14:47

Many thanks to the Linden Honcho and his grid monkeys for battling this in the middle of the holidays. That has to suck.

== danny d.

Aces Spade

Raise you One♠

Join date: 22 Sep 2003

Posts: 2,774

12-31-2004 14:50

Yes Thank you Linden Labs for the super job on getting this all fixed.. we know you are working hard on restoring us

Happy New Year to all

_____________________

From: someone

Posted by ZsuZsanna Raven
So where is the "i don't give a shit'' option?

Dazzo Street

Registered User

Join date: 1 Aug 2004

Posts: 71

12-31-2004 14:53

make a sign in the office that reads

"a stitch in time saves nine"

dont blame luck, blame yourselves for working those servers to DEATH instead of getting additional ones months ago.

Merwan Marker

Booring...

Join date: 28 Jan 2004

Posts: 4,706

System Update From Uncle L.

12-31-2004 15:07

Never mind.

Thank you!

_____________________

Don't Worry, Be Happy - Meher Baba

Aaron Levy

Medicated Lately?

Join date: 3 Jun 2004

Posts: 2,147

12-31-2004 15:17

Remember, Lindens... no "extra stuff" in the punchbowl at work tonight... tomorrow you'll wake up with more than server problems!

_____________________

Rezzations.com

Home of the Trivia Stadium

Leran Charlton

Registered User

Join date: 21 Jun 2004

Posts: 5

12-31-2004 15:19

Thank you Phil, I appreciate the report!

Antagonistic Protagonist

Zeta

Join date: 29 Jun 2003

Posts: 467

12-31-2004 15:27

Many of us realize that upgrading and troubleshooting real time, mission critical hardware & systems under heavy load is not trivial... and when something goes wrong, it can be catastrophic. As customers, we do expect our inconveniences to be acknowledged when we recive our bills... but as artisans of the craft we feel your pain and share your lament.

We are rooting for you.

Hopefully it won't take writing backwards opcodes in blood to get it working

With Regards,
Antagonistic Protagonist

Mojo Bliss

Registered User

Join date: 29 Dec 2004

Posts: 213

12-31-2004 17:11

Very nice that the ceo of LL would bother to explain for the company.

Helps me feel less concerned for the events of past few days when i began.

Merwan Marker

Booring...

Join date: 28 Jan 2004

Posts: 4,706

Update at 5:15 pm Game Time

12-31-2004 17:29

till working.... update as of 5:15PM PST 12/31

Logins still very slow. We've experimented with shutting down some of the database processes that cause load (there were cases where we could do that without kicking everyone off, so we tried it). No signifigant relief - login servers are still overwhelmed.

We are continuing to try to debug and look at changes that we can implement now. We will also likely have a downtime tommorow morning to update the database servers to new hardware.

__________________
Philip Linden
CEO & Founder, Linden Lab
blog: http://secondlife.blogs.com/philip

Click

Thanks Philip.

_____________________

Don't Worry, Be Happy - Meher Baba

Hayden Hedges

Registered User

Join date: 11 Mar 2004

Posts: 138

12-31-2004 17:29

It's sods law that anything that can go wrong will go wrong at the worst possible time. I wish everyone would relax a little, enjoy the break from what appear to be extremely full second lives and thank god you're all not stood looking at a misbehaving server with the 'Oh shit what's it doing now' mantra running through your head

Peace be with you all guys and I hope 2005 brings you all the hapiness you deserve.

Now Im going back to my conversation with Mr J Daniels.

Night

Jasmine Divine

Registered User

Join date: 5 Dec 2004

Posts: 33

Thank you

12-31-2004 17:33

Thank you so much for letting us know, I'm fairly new around (and easily confused lol

)

I think the majority of people appreicate your taking the time to fix the problem and you should get some extra time off

Happy new year from the UK

Hugsss

Jas x

Neehai Zapata

Unofficial Parent

Join date: 8 Apr 2004

Posts: 1,970

12-31-2004 17:34

Poor Philip working so hard.

He deserves a special night at my place to relax after all this stress.

_____________________

Unofficial moderator and proud dysfunctional parent to over 1000 bastard children.

Pendari Lorentz

Senior Member

Join date: 5 Sep 2003

Posts: 4,372

12-31-2004 17:55

Another reply from Philip in another thread.

From: Philip Linden

Sorry, we're doing all we can. We feel terrible about this outage.

Though our planning can be improved and certainly will be, I might provide the small defense that the aspects of SL that will fail under load and expansion are much harder to predict that, say, a retail e-commerce business. For example, one of the things that is currently creating the load on the login system is very very heavy IM traffic, where much of the load is coming from objects in-world. This is caused by vendors, games, and other objects that use IM to communicate in a particular way. We didn't initially anticipate the need to decentralize the IM services on this aggressive a schedule, because we didn't know how many and in what different ways people might use object->user IM. Now we know, and we are scrambling to speed that system up. Ditto for heavy inventory use (the root problem that is slowing login).
So in many cases we are forced to be reactive where of course we would like to be proactive. It is very frustrating, but I for one am willing to suffer the frustration for all the amazing things that such an open-ended world has brought us.

From this thread: http://forums.secondlife.com/newreply.php?do=newreply&p=325430

_____________________

*hugs everyone*

Khamon Fate

fategardens.net

Join date: 21 Nov 2003

Posts: 4,177

12-31-2004 22:23

thank you philip

_____________________

Visit the Fate Gardens Website @ fategardens.net

Merwan Marker

Booring...

Join date: 28 Jan 2004

Posts: 4,706

01-01-2005 06:15

Update, 7:45PM, 12/31:

We found a bug in the viewer that was causing most of the trouble, and have released a patch. Your viewer should auto-update and you should now be able to login. This is a patch to the Windows viewer only - a Mac update should be available sometime tommorow. If you have a Mac client, you may still be able to get in without the update, as the load level drops on the system.

The problem was not in the database, but rather a race condition in the viewer that caused logins to totally fail when the database was slow. The database disk failure slowed login performance enough to trigger the bug, which is what has blocked many logins since yesterday.

We will not need to bring the grid down tommorow morning, but will still schedule downtime for hardware updates to the database sometime early next week. These upgrades will further speed login time.

Very sorry for the problems, and Happy New Year to those enjoying it in SL tonight. Great work from the LL team who came in today and got this fixed.
__________________
Philip Linden
CEO & Founder, Linden Lab
blog: http://secondlife.blogs.com/philip
---
Last edited by Philip Linden : Yesterday at 08:14 PM.

Click Here

==

(Thanks Aces)

_____________________

Don't Worry, Be Happy - Meher Baba

CEO's Comments Thank You Phil!
Merwan Marker Booring... Join date: 28 Jan 2004 Posts: 4,706	12-31-2004 14:35 We are still having problems with heavy login load on Second Life, as of 2:30PM PST on 12/31. The grid is currently open, but logins are very slow. We are working on some quick fixes that we are testing right now, and additionally are getting new hardware ready to replace the equipment that is still running slow. I don't have an ETA for repairs. We may get some speedups in the next couple hours, otherwise there will likely be grid downtime tommorow (1/1) as we change hardware. We're here at the office - doing all we can. Apologies for the problems. __________________ Philip Linden CEO & Founder, Linden Lab blog: http://secondlife.blogs.com/philip _____________________ Don't Worry, Be Happy - Meher Baba
Danny DeGroot Sub-legendary Join date: 7 Jul 2004 Posts: 191	12-31-2004 14:47 Many thanks to the Linden Honcho and his grid monkeys for battling this in the middle of the holidays. That has to suck. == danny d.
Aces Spade Raise you One♠ Join date: 22 Sep 2003 Posts: 2,774	12-31-2004 14:50 Yes Thank you Linden Labs for the super job on getting this all fixed.. we know you are working hard on restoring us Happy New Year to all _____________________ From: someone Posted by ZsuZsanna Raven So where is the "i don't give a shit'' option?
Dazzo Street Registered User Join date: 1 Aug 2004 Posts: 71	12-31-2004 14:53 make a sign in the office that reads "a stitch in time saves nine" dont blame luck, blame yourselves for working those servers to DEATH instead of getting additional ones months ago.
Merwan Marker Booring... Join date: 28 Jan 2004 Posts: 4,706	System Update From Uncle L. 12-31-2004 15:07 Never mind. Thank you! _____________________ Don't Worry, Be Happy - Meher Baba
Aaron Levy Medicated Lately? Join date: 3 Jun 2004 Posts: 2,147	12-31-2004 15:17 Remember, Lindens... no "extra stuff" in the punchbowl at work tonight... tomorrow you'll wake up with more than server problems! _____________________ Rezzations.com Home of the Trivia Stadium
Leran Charlton Registered User Join date: 21 Jun 2004 Posts: 5	12-31-2004 15:19 Thank you Phil, I appreciate the report!
Antagonistic Protagonist Zeta Join date: 29 Jun 2003 Posts: 467	12-31-2004 15:27 Many of us realize that upgrading and troubleshooting real time, mission critical hardware & systems under heavy load is not trivial... and when something goes wrong, it can be catastrophic. As customers, we do expect our inconveniences to be acknowledged when we recive our bills... but as artisans of the craft we feel your pain and share your lament. We are rooting for you. Hopefully it won't take writing backwards opcodes in blood to get it working With Regards, Antagonistic Protagonist
Mojo Bliss Registered User Join date: 29 Dec 2004 Posts: 213	12-31-2004 17:11 Very nice that the ceo of LL would bother to explain for the company. Helps me feel less concerned for the events of past few days when i began.
Merwan Marker Booring... Join date: 28 Jan 2004 Posts: 4,706	Update at 5:15 pm Game Time 12-31-2004 17:29 till working.... update as of 5:15PM PST 12/31 Logins still very slow. We've experimented with shutting down some of the database processes that cause load (there were cases where we could do that without kicking everyone off, so we tried it). No signifigant relief - login servers are still overwhelmed. We are continuing to try to debug and look at changes that we can implement now. We will also likely have a downtime tommorow morning to update the database servers to new hardware. __________________ Philip Linden CEO & Founder, Linden Lab blog: http://secondlife.blogs.com/philip Click Thanks Philip. _____________________ Don't Worry, Be Happy - Meher Baba
Hayden Hedges Registered User Join date: 11 Mar 2004 Posts: 138	12-31-2004 17:29 It's sods law that anything that can go wrong will go wrong at the worst possible time. I wish everyone would relax a little, enjoy the break from what appear to be extremely full second lives and thank god you're all not stood looking at a misbehaving server with the 'Oh shit what's it doing now' mantra running through your head Peace be with you all guys and I hope 2005 brings you all the hapiness you deserve. Now Im going back to my conversation with Mr J Daniels. Night
Jasmine Divine Registered User Join date: 5 Dec 2004 Posts: 33	Thank you 12-31-2004 17:33 Thank you so much for letting us know, I'm fairly new around (and easily confused lol ) I think the majority of people appreicate your taking the time to fix the problem and you should get some extra time off Happy new year from the UK Hugsss Jas x
Neehai Zapata Unofficial Parent Join date: 8 Apr 2004 Posts: 1,970	12-31-2004 17:34 Poor Philip working so hard. He deserves a special night at my place to relax after all this stress. _____________________ Unofficial moderator and proud dysfunctional parent to over 1000 bastard children.
Pendari Lorentz Senior Member Join date: 5 Sep 2003 Posts: 4,372	12-31-2004 17:55 Another reply from Philip in another thread. From: Philip Linden Sorry, we're doing all we can. We feel terrible about this outage. Though our planning can be improved and certainly will be, I might provide the small defense that the aspects of SL that will fail under load and expansion are much harder to predict that, say, a retail e-commerce business. For example, one of the things that is currently creating the load on the login system is very very heavy IM traffic, where much of the load is coming from objects in-world. This is caused by vendors, games, and other objects that use IM to communicate in a particular way. We didn't initially anticipate the need to decentralize the IM services on this aggressive a schedule, because we didn't know how many and in what different ways people might use object->user IM. Now we know, and we are scrambling to speed that system up. Ditto for heavy inventory use (the root problem that is slowing login). So in many cases we are forced to be reactive where of course we would like to be proactive. It is very frustrating, but I for one am willing to suffer the frustration for all the amazing things that such an open-ended world has brought us. From this thread: http://forums.secondlife.com/newreply.php?do=newreply&p=325430 _____________________ hugs everyone
Khamon Fate fategardens.net Join date: 21 Nov 2003 Posts: 4,177	12-31-2004 22:23 thank you philip _____________________ Visit the Fate Gardens Website @ fategardens.net
Merwan Marker Booring... Join date: 28 Jan 2004 Posts: 4,706	01-01-2005 06:15 Update, 7:45PM, 12/31: We found a bug in the viewer that was causing most of the trouble, and have released a patch. Your viewer should auto-update and you should now be able to login. This is a patch to the Windows viewer only - a Mac update should be available sometime tommorow. If you have a Mac client, you may still be able to get in without the update, as the load level drops on the system. The problem was not in the database, but rather a race condition in the viewer that caused logins to totally fail when the database was slow. The database disk failure slowed login performance enough to trigger the bug, which is what has blocked many logins since yesterday. We will not need to bring the grid down tommorow morning, but will still schedule downtime for hardware updates to the database sometime early next week. These upgrades will further speed login time. Very sorry for the problems, and Happy New Year to those enjoying it in SL tonight. Great work from the LL team who came in today and got this fixed. __________________ Philip Linden CEO & Founder, Linden Lab blog: http://secondlife.blogs.com/philip --- Last edited by Philip Linden : Yesterday at 08:14 PM. Click Here == (Thanks Aces) _____________________ Don't Worry, Be Happy - Meher Baba

Welcome to the Second Life Forums Archive

CEO's Comments Thank You Phil!