Havoc 4, Mono, Stability

Linden Lab’s Effort to Improve Grid Stability
Friday, August 24th, 2007 at 4:42 PM PDT by: Ian Linden

I want to explain our ongoing efforts to make Second Life as robust and stable as possible, while at the same time continuing to improve the quality and depth of the in-world experience. Innovative new features like 3D voice, WindLight, and Sculpties are essential for the continued evolution of Second Life; feature stasis just isn’t a good option. However, we are now devoting the majority of our development resources to considerably less visible behind-the-scenes improvements. These back-end changes have long-term benefits for the entire community, and will also help the grid withstand the sort of infrastructure failure we’ve encountered recently (described in my post here). I’ll use the rest of this post to highlight this work.


Bug Fixes
Fixing individual bugs and system failures remains a high priority for us, and our largest (and growing) internal development group is dedicated to these tasks. The open source community is also a great help in this effort, and our team integrates a large number of fixes submitted by external developers. Some of these improvements were recently highlighted in Sardonyx Linden’s post on this blog.

Modern Infrastructure
Much of the server-to-server communication happening within the Second Life grid is based on old technology which has served well beyond its initial design. When a failed database causes grid-wide problems, a de-rez doesn’t complete, or a teleport explodes on lift-off, this is usually why. So, there are currently multiple teams working on the wholesale elimination of this technology, replacing it with a modern web-service based approach. This is a big effort and will take a long time, but the benefits are myriad: when finished, this will improve reliability, eliminate data bottlenecks on crowded sims, allow multiple versions to co-exist, enable global scalability, and allow us to dispense with VPNs altogether. We’ve started to see first fruit in the form of fewer planned outages; most new releases no longer require us to shut down the whole grid.

Fewer Failure Points
In addition, Linden Lab engineers are working to improve our database failover capabilities and to reduce single points-of-failure in the network infrastructure. We’re also refactoring all of our recurring data maintenance processes to reduce the impact of these on the main grid. Ideally, all of this work will be completely invisible to residents, but it will mean less grid downtime.

Viewer Stability
Generally Second Life client crashes are handled by the ongoing bug-fixing project mentioned above, but a group of our graphics specialists have recently formed a team to improve the rendering code which is at the root of most crashes. This should lead to a more stable client on a wider variety of computer hardware.

Simulator Stability
We’ve been talking about upgrading the Havok physics engine included in the sim software for years, and haven’t managed to deliver. However, this project is making real progress now and will be ready for a beta test soon. The Havok 4 version in development can withstand the majority of physics glitches and attacks that cause most sim crashes. A little farther out, we’ll be replacing the guts of LSL execution with Mono, which was on hold for a long time but is moving forward again. This will result in massive performance gains for scripts, but will also make sims (especially very busy sims) more stable.

A New Response Team
As Second Life has grown, so too has Linden Lab. We’ve recently created the Production Operations team (a virtual fire department, if you will), which will add a new level of grid monitoring and trouble response. This group will provide superior 24/7 monitoring, and will diagnose, solve and escalate technical issues that arise on the Second Life grid, network and website. These engineers will also develop tools and automation technologies for monitoring and repairing Second Life more efficiently. We’re currently hiring new members for Production Ops all over the world – the job listing is here.

Communication
We’ll periodically be posting updates on our continued efforts, and their relative success or failure. In the meantime, we depend on your feedback – please continue to report the bugs you experience.

As Second Life matures - more residents have more time, effort and money invested in-world than ever before - Linden Lab must be accountable for making the virtual world robust and stable, without losing sight of all the new capabilities we hope to add. There’s a great deal of work to be done, but we have far more resources than we did a year ago, and there are more engineers dedicated to stability and scalability than are working on 3D voice, WindLight, or sculpties combined. I believe that we’re headed in the right direction.

In fact, contrary to what was said, feature stasis IS a VERY good option when you don't have that as a foundation, because building new features on a platform that you don't have a good handle on usually causes more problems and customer angst than feature stasis alone ever will. That's the simple (and not-so-subtle) wisdom that the Lindens don't grok.

Well, features are definitely "essential" for the continue evolution of any system. HOWEVER, unlike what was said (well, implied, really) there, they are FAR LESS "essential" than a stable, useful platform.

In fact, contrary to what was said, feature stasis IS a VERY good option when you don't have that as a foundation, because building new features on a platform that you don't have a good handle on usually causes more problems and customer angst than feature stasis alone ever will. That's the simple (and not-so-subtle) wisdom that the Lindens don't grok.

Hard to keep from laughing at a post stating that sculpties are essential to the growth SL. Just how is an out side of SL program created texture that replaces inworld building skills 'essential'??

Could not agree more!
Its reallt wierd watching the add stuff to SL when SL is so badly broken... and they are the ones who LOVE pr ya would think they would get the basic done first... but gones to show ya LL really has crappy coders... look at the produts they put out as "fixed" and "resloved" Any other real life company they would be "Fired" lol

For startes SL doesnt have any competors anywhere There and Keneva are just a joke.
Lag,40k online limit,PayPal being screwed, Inventory loss, NO CUSTOMER service, crashes both client side and server side...well the list can go on a VERY long time lol
Ya SL is a broken toy.
Saying SL isnt broken ya gotta have your head in the sand or be REALLY new to the net lol


O and stacking the fix code on top of the broken..ya like thats a good job LOL

I can't say I agree with your reasoning there, Talon.

New features stave off user boredom. Sculpties are the first thing in a very long time that got me excited about SL again. There are many things that could improve the SL experience beyond just having a stable grid. Stability alone doesn't cut it. A stagnant platform won't keep people interested.

..and, Zaphod, 7.5 MILLION accounts (~80%) already have gone "somewhere else".

Where have they gone? Where do you get your numbers from? Where, exactly, is that "somewhere else"? At this point, in terms of what Second Life is, there isn't "Somewhere Else" just yet. That's kinda the point.

However, SL will never be perfectly stable. Never. I do wonder about people who insist that the existing system be perfected before any new features are introduced. Is it a fear of bugs or a fear of obsolescence?

For startes SL doesnt have any competors anywhere There and Keneva are just a joke.
Lag,40k online limit,PayPal being screwed, Inventory loss, NO CUSTOMER service, crashes both client side and server side...well the list can go on a VERY long time lol
Ya SL is a broken toy.
Saying SL isnt broken ya gotta have your head in the sand or be REALLY new to the net lol


O and stacking the fix code on top of the broken..ya like thats a good job LOL

Hard to keep from laughing at a post stating that sculpties are essential to the growth SL. Just how is an out side of SL program created texture that replaces inworld building skills 'essential'??

It's essential because it means we don't have to be logged into a broken Second Life to build content.

Hard to keep from laughing at a post stating that sculpties are essential to the growth SL. Just how is an out side of SL program created texture that replaces inworld building skills 'essential'??

Maybe as Hipihi progresses we will be able to create SL prim creations there for use in SL.

Where have they gone? Where do you get your numbers from? Where, exactly, is that "somewhere else"? At this point, in terms of what Second Life is, there isn't "Somewhere Else" just yet. That's kinda the point.