FJ Linden (Frank Ambrose) November Grid Update

[ Note: You can read the original post at http://blog.secondlife.com/2008/12/04/fj-linden-frank-ambrose-november-grid-update/ ]

FJ Linden here, with my monthly grid update.

It's been a good stretch of grid stability over the last month, with one very poor day in the mix. Some central database issues and then a Level 3 outage in the middle of the month cascaded into a series of problems, although we were able to isolate and fix them in just over 3 hours. However, that event only served to reinforce just how important it is to bring LLnet online, and quickly. On that topic, I’m pleased to start this month's updates with the status of LLnet.

LLnet 30 Days Ahead of Schedule
LLnet, our private fiber optic ring, is a good 30 days ahead of schedule. This network, which will privately interconnect our datacenters, will allow us to move away from VPN reliance. “LLnet” fiber facilities have been delivered into our 3 data centers, and are currently in the configuration and testing phase with the routing infrastructure. This work should be concluded by the end of this week, and we will then start full testing in a production environment. We want to move as quickly as possible, but also do not want to destabilize the grid for the sake of speed, so we will take most of December to finish production testing, and begin cutover of live traffic in late December or early January. We have thousands of machines across the data centers, so the cutover process is expected to take about 60 days, but we have been very good (so far) at beating our projected dates.

HTTP Dataserver
On the infrastructure project front, we've completed most of the HTTP Dataserver project to migrate all C++ mysql traffic from mysql protocol to http(s). This project will allow us to move farther away from VPN dependency as well as off of MySQL wire protocol over the WAN, to better enable tracking and monitoring of queries. We expect to be through testing in the next week.

Agent Inventory Services
Agent Inventory Services is scheduled to be deployed with the server code update in January. This is one of the ongoing projects to address inventory issues for Residents.

These projects are both designed to provide more reliability, especially as it relates to inventory delivery and database queries, by better handling messaging across the databases and simulators, as well as back to the viewer. I intend to use my December/January post to talk about our strategy for inventory services, our storage strategy and our thoughts on our data architecture.

My primary goal has always been to improve grid stability and reliability and we are making great strides on that front. We’re not through the woods yet, but I want to re-emphasize how important I believe it is to address "foundational" issues that have the potential to cause huge impairment (like network problems), and then decide how we scale other components of the infrastructure.

Finally, I have made some internal organizational changes over the past month, that I hope will begin to drive more specialization in some key areas. This included adding a new network director, and more focused team leads managing databases, asset management, and data services. My belief is that, in addition to sound technical strategy, we need the right organizational alignment and specialized technical skills to achieve long term stability and scalability on the grid.

Links:

Second Life Grid Status Reports
https://status.secondlifegrid.net/

RSS Feed for SL Grid Status Reports Page
http://status.secondlifegrid.net/feed/

Second Life Grid Status via Twitter
https://twitter.com/SLGridStatus

Service Disruptions Wiki page
http://wiki.secondlife.com/wiki/Service_Disruptions

FJ's Grid Status Updates are one of our efforts to help you understand how we are moving toward the greater stability and reliability of Second Life.

FJ said, "I intend to use my December/January post to talk about our strategy for inventory services, our storage strategy and our thoughts on our data architecture."

What related topics would you be interested in Frank highlighting in upcoming posts?

Yay!

I like reading about what you are doing, and why, especially concerning data retrieval.
The planned post about "strategy for inventory services, our storage strategy and our thoughts on our data architecture" sounds like it will be a good one.

Did you see here on the forum where someone suggested (as is possible in some other worlds) having a kind of storage plan for little-used inventory? That would be an interesting idea to explore - giving reasons why it would or would not be appropriate.

FJ Linden will review all your ideas and comments, though he won't have time to comment here today.

What would you be interested in having FJ highlight for future blog posts?

I for one would love to see a discussion about the enterprise data model including metadata, object models, and data architecture and how all these things are tied together to form reliable data services.

How about an explanation geared toward non-tech-geeks as to just how exactly your systems talk to each other, esp w/regards to Dallas vs. San Fran vs. London, etc, and how our stuff is stored, esp w/regards to avatars, Inventory, etc. Not absolutely necessary but it would be interesting to learn, and seeing as how I've had the basics of SQL server but nothing beyond that, would be nice to learn some ways to tweak our own accounts/inventories for best performance.

Upgrading from a jalopy to a hotrod won't win the race.

How about a new driver?

I do hope they fix the major fall of rendering speed of objects.
I've done about everything i could, and even variation of the graphic settings in preferences doesn't make any differences at all.
The major fall of rendering makes SL almost unplayable now.
Have sent a report to SL... Answer was very Lez Affair...
"Thank you Ma'am, we'll keep on monitoring SL as always"...
In other words... ty, and back the usual business.

I am VERY sure that many suffer from this problem as i have asked many inworld.
Many objects stay grey over even more than 20 minutes after arrival.
And nothing helps, not even totally re-installing SL.

So... I am totally sure that SL has a major issue to solve, but they won't admit...
as the usual business in SL.

Well, after weeks not being online i'm off to RL again... c u when issues are better.
This sucks now big time.

Hahahahahah!

Getting your own fiber ring is very exciting. So many problems eliminated!

I think the next posting on data architecture is going to be very helpful - the more detail, the better.

Perhaps you could also say a few words about whether the web services will just be CRUD, or will you be providing an extensible layer above that? I ask because I'm interested in whether LL intends to provide a way for third parties to plug into SL's server network, and it seems like a secure yet rich set of, for example, fairly standard SOAP services could be helpful to LL partners who may have a need to plug in.

I am also wondering if a well-defined interface at this level could help with secure interoperability between grids.

Another topic: Is SL planning any upgrade to its presence service? I think that voice and other media presence could be better integrated, and also would be helpful to have more intelligent management of multiple identities (avatars). Perhaps, also, emerging commonality among presence servers might allow links between other communities (e.g Facebook or LinkedIn) and SL avatars.

Who, for what role?

Also 1 question to make something clear, when the post say to have a cutover for about 60 days this is gradually right?....And what exactly it means....?

I would immediately give FJ Linden a great big raise!

The only time I will most likely be saying this, but someone at Linden Lab has FINALLY spoke the truth, not the spin, but the truth.

I would suggest EVERYONE that BLOGS at Linden Lab take a close look at what Frank has posted and FOLLOW HIS EXAMPLE.

Notice that Frank doesn't run away from the issues suffered at Linden Lab, but instead, meets them head on and doesn't try to hide behind some propaganda BS.

While I'm not sure that Frank is on-target with his "rosy outlook" on how things should work once this work is finished, at least Frank is willing to step-up and speak the truth.

So, my suggestion to Frank is that he gives BLOG POSTING CLASSES to the rest of the Linden Lab employee's, including management, and they should follow his lead for speaking the truth, and NOT propaganda.

Perhaps KATT will be the first to follow suit?

I'd like to hear about what we, scripters and builders and such, will be seeing in the future.

Sure, bug fixes are goodness and greater stability is something SL is in serious need of. No disagreements there but I'd also like to see more growth in what resident developers can talk SL into doing. Object-object comms, better scripted physical movement, stuff like that.

/me also inserts a complaint about the recent openspace pricing actions - that's far from being forgotten.

The fiber ring migration is needed. Thanks for making this happen.

The Agent Inventory and Storage is something we are all looking for more detail on. The earlier suggestion on a storage area (perhaps as part of being a Premium Account holder) for storing little used objects/textures would be great. We could offload some of our non-active inventory into this storage bucket and pull it back when we intend to use it over time. It might also be useful for content creators to archive their earlier creations which they have loaded in their vendors and no longer need in their inventory. If you take this service one step further and make this a benefit for becoming a Premium accountholder, you add some real value to that product and might drive more people into Premiums.

Thanks also for noting the organizational changes and alignment of skillsets and personnel to the technology plan. All companies and customers expect this to happen. Unfortunately, in many cases it does not. People and skillsets get fragmented and dispersed across lots of projects and tasks. Good alignment is essential for stability and growth. These are also often difficult to massage through the existing department's skillsets and personnel. Thank you for taking the time to let us know a little how you operate. It helps build confidence.

FJ, I'm worried migration to LLnet means abandoning the level of redundancy that other providers may give you. What assurances can you give this network weenie that you're not becoming your own single point of failure?

May I ask, wether any steps towards Europe-based servers have been made? It is expected that this would speed up the SL-experience for European residents at dramatic rates. The demand for it has been put in claims to the Lindens for years. I'm still waiting to see some action going on.

Yay! Technical blog posts are the best! Thankyou Frank! I hope you carry on with your monthly blog posts.
I have noticed that my ping as dropped from over 250 (some days over 500...) to around 160 over the past couple of weeks. And packet loss has dropped considerably. Is that FJ's doing? If so, give him a huge christmas bonus.


Katt is there any chance of other Lindens making posts much like Franks?

I (and I'm sure many others) would love to hear more about the behind the scenes.
Perhaps a blog post about the upcoming shadows or other projects.

What about the Nimble 3D volumetric clouds that were mentioned long ago?
http://www.windwardmark.net/products.php?page=nimble&sub=technology

Er.. Already done.


Also already done, tho they could do more.

Then kindly leave. There are enough posts regarding OS venting throughout the Forums. I think LL gets your point. The constant OS venting does nothing by clog up forum threads for the rest of us trying to get something done in LL.

Well, there's LL and then there's LL management. I don't think LL upper-management has any idea how large or angry the mob still is.

As long as posts contain something topical, I don't have any objections to them also reminding LL management that this issue hasn't been forgotten.

Really? Having one server per rack dedicated to the inventory of a subset of users has been accomplished? Making it so entering a sim and requesting 20 texures from 20 different owners returns data from 20 different seperate machines instead of one group out in uberAssetClusterLand?

Somehow, I pictured it working better than this.

Well, then we are looking at the same thing and seeing two different results. I believe LL Management and LL Board totally understand the issue. And yes, I also own an OS so I do understand the effect of the policy change.

The continuing consolidation of OSes demonstrates that LL Management understands the effect of their decision. I suspect the consolidation is what they wanted to achieve.

As to the large and angry mob, I spend a good bit of my RL in "politics" and the large and angry mob is a strawperson (as I put on my Kevlar suit). Just because there is a very vocal group in the hundreds (or less) does not mean there is in reality tens of thousands who are upset with the policy change. As we have seen in politics, there is a tipping point. When the "large and angry mob" becomes a distraction, the rest of the population begins to discount the points they are yelling about. That's when the "large and angry mob" become marginalized.

I do agree that if there is something topical it should be posted.

Nuff said, I realize this post has gone totally Off Thread and apologize to those that got distracted by this conversation.

First of all, no, I shall not leave. I am still a user of the game and the forum, and I'm following the guidelines here. You are free to ignore me if you don't like what I say.

Second, I mentioned several other things that need addressing as well as the fiber and the OS debacle. These problems have been around for years and I'm pointing them out (yet again) in the hopes they might finally get fixed, as LL seems to be interested in doing more than just bandaging their tech wounds for the moment. It is possible they will also realize that they have a self-inflicted business wound as well; one can only hope (and wave the money that they don't get in the air over and over again).