SL Asset Server Approaching Critical

I will first say that I have been in SL for around 2 years, I joined in the days of 1.1 where land was still free and "custom animations" were just a distant wish. And never have I been so concerned about the immediate future of Second Life...

From what I understand, SL's Asset server which does the majority of database transactions in the virtual universe, Is one dedicated server serving not only multiple sims, But even multiple grids. In the past, It wasnt uncommon to hear complaints about the asset server not being able to keep up with the load. But most of the inconveniences could be (and were) tolerated.

However the present situation is very different, and to be honest, scares me to death about the capabilies of said system. Ever since I have been here, I have never seen such rampant problems with the asset server. I hear complaints on the forums about objects disappearing from inventories, and other people also seem to experience the same issue. Before problems such as corrupt or missing data were rare and far in between. Now nearly everyone whom I talk to has experienced it. The other day I attempted to rez one of my objects (it was a blueprint for a building I was designing) and instead of recieving said item. A pair of fairy wings were rezzed that I have no recolection of. Upon further inspection, not only could I not modify the wings or copy them... But they were not even under my ownership! Yet, even while not being under my ownership I still had the ability to delete them. In addition, Saving scripts, which used to take under 3 seconds 99% of the time. Are now frequently taking up to 2 or 3 minutes. Objects are taking similar durations to rez. And there is still the matter of rampant teleport failures. All of these indicators seem to suggest a very undeniable fact.

The SL Asset server is not only insufficient for maintaining and operating such a large and heavily used database, It is approaching a critical point where very few actions will even be possible. There is another undeniable fact, The asset databases are growing, Getting larger every day as every identical wooden cube in someones inventory is added to the database. The Asset server does not appear to be growing even proportionally to the systems current growth.. Much less enough to have breathing room and reliable fast performance.

I am interested to hear what others have to say on this topic? Is anyone else as worried as me that I have been unable to teleport off Isle sandbox for an hour now (at the time of writing) and rezzing objects takes nearly a minute?

The lindens usually do not take notice unless they see an overwhelming customer consent to an idea. Please voice your support so that The asset server may recieve a long overdue overhaul that will allow it to scale not only to meet but to exceed requirements for maintaining reasonable operation of the world.

I agree. Something must be done about this issue. Not only is it difficult to teleport, but I have been experiencing multiple crashes, as have others. Framerates in some areas have dropped to as low as one frame per second. ONE FRAME!! It has even become difficult to type and send messages, that's how lagged the server is.

There's also been days of extreme packet loss, IMs and chat messages that either don't go through, or take ten minutes to show for the other person. Objects sometimes refuse to rezz at all, and sometimes don't appear back in my inventory for minutes or don't at all. I've even lost work on some object editing I was doing because of an unexpected crash that occured as I was putting an object back into my inventory. Music streams are also beginning to lag horribly when they play fine in Winamp.

I think the problem has become quite obvious: I just purchased land the other day. In the span of about one or two days, the entire sim I purchased land in has sprung up with very large houses, objects, and buildings that weren't there before. None of these things were an issue before the free account giveaway for this year. I signed up the week before and things worked perfectly. Then people started joining in droves (quite a few because of my own referrals). Said people then started learning how to build and buy land. Now we have a dramatic increase of users which has understandably created a much larger load for the server to handle.

I think it's obvious that people love Second Life. But perhaps they love it just a little too much. The game is quickly becoming unplayable. There's no way that this can continue without people leaving and cancelling their subscriptions. The best idea for Linden Lab would be to either upgrade their servers or increase the amount of them. Otherwise, I don't think any of us will be able to enjoy Second Life any longer.

I agree...

It is a pain when saving a small script can take from 1 second to 30 seconds to save, and random sometimes quick, sometime slow...same with taking things out, sometime it just there, other time it is a wait (and 1 prim objects, not multiple)..

Can't tp in costume anymore, have to be human with no attachments, or I am staying put.

Keep hoping that with each patch it will get better, but with the last 2 patches got worse, and this one stayed about the same...but can't get much worse so has to get better soon!

I was wondering. Would it help if we would remove a lot of the objects and junk we have in our inventories which we never use?

It could, I make sure to empty my trash, delete all duplicates, only keep multiple copies of projects I am working on, once done keep only the finished, and maybe one before.

Threw out most of my drinks, and other freebie stuff I never use, my inventory is around 8000, and it does get bigger each month, but not by alot, in 1 year I got to 8000, so I don't think that is bad, and that includes SL library stuff I think, so I am doing pretty good

Although they wont admit it, LL has known about this issue for at least the last year. The system, like most novel technologies, has long exceeded its design limits and as was noted by Catherine Linden recently (if memory serves) this is an emergent problem that has no clear fix. Indeed, I expect that the fix is coming in the form of SL 2 where most of the development resources seem to be concentrated at present.

What we are left with is a skeleton support crew who is putting instrumentation into place for the 5000 AV crash-a-thon on Sunday where they can see where the worst of the problems are and try to band-aid that.

The only thing I cannot figure out is why they are so hell bent on expanding the userbase and grid at a time where they can least afford it. Recently they started giving memberships away as if having more load which generates no revenue would increase the mindshare of SL. It will, but not in any way that will help; if these forums are any indication, they are broadening a perception of "wow, this SL thing is sheer frustration, what else is there to play?".

SL will likely survive following a population plummet (no one plays SL anymore it's too crowded). As far as I know there is still a hardcore group of ActiveWorlds players. ActiveWorlds was a forebearer of SL that I tried for 5 minutes once before deeming it craptastic; I cannot imagine that the new user experience of SL is any different.

But Linden Lab is privately held, has always been closed lipped about plans, futures, faults, expectations and so on. In the past I've personally implored Philip and Robin to let the customers know what is going on because they will be far more forgiving with credible information in hand than when they hear a fix is coming Real Soon Now and keeps not.

The original poster is correct, the system is "broken by design". As to why Linden Lab insists on shooting themselves in the foot is known only to them.

I bought into the dream and the coolness as a Lifer and am just hoping they will manage to right themselves before all the truffula trees are gone. Sigh.

I've now posted in about 5 threads regarding the TP issue, as well as the asset manager and other core functions not working. Surprisingly, I have taken flack from some of the forum members, saying I should be more patient and accepting of the complexity of the problem. I guess in my 25 years of software development/engineering I have never really worked on a complex system or had to deal with upset customers. *chuckles*

I agree with what you have all posted here. There are serious design issues that need real work. These weekly patches are not getting to the real issues that plague the system. Instead, people continue to pay good money, sometimes in excess of $200 a month to be told "this is a top priority for us and we are working to fix it as soon as possible". Well, how long is that tune going to play? In any other business you don't get away with telling the boss I will deliver as soon as possible. He wants dates and level of effort. I gather at this point, assuming the problem really has been under investigation, that the level of effort is high. The question is how high and are we looking at weeks or maybe even months for a real solution? Or are we actually waiting for SL2 to come out with a complete redesign? I agree, we need more information than "we are looking into it and working to resolve the problem", and unless the wheel squeaks it won't get oiled.

FYI - this is not a hardware issue.

I had the pleasure and privelege of touring the SL colocation facility with the group that invaded LL. There are a handful of machines working as "the asset server", and while I probably am not allowed to detail the specs of these machines, I can say that the CPU power is there.

I will also add that I yelled at them to stop breaking

The issue is not the asset servers themselves, it's more about the architecture. Centralized item databases are simply not scalable, and it's something LL readily admits. Decentralizing is one of the great challenges that LL faces in the medium to long run.

I have a blog entry about decentralization if you're curious to more of my thoughts - check out the Scalability article at secondtense.blogspot.com

Yeah, I certainly never thought it was a hardware issue. If it were hardware the server(s) would probably crash and the problems would be even worse because there would be total failures. It would also be a lot easier to fix, because you could just throw more horsepower at the problem.

This clearly is an architectural issue, and will take significant time to fix properly. Perhaps they can at least create a good interim solution that will work for however long it might take to redesign the asset manager, but this wait and see and hope for the best approach is just not working for the end-users, at least not this end-user.

You know, If we all didn't give a damn about SL...then we wouldn't be complaining. If we all hadn't put our hearts and soul into it...then we wouldn't care. We are allowed to complain...we pay for the service and we want it to work well. I don't expect perfection, but I have put a lot of time and effort and $ into my SL experience and it is so frustrating when basic things can't be done. I get tired of not being able to TP, textures not rezzing in a timely manner like they used to, etc etc. Sometimes people just turn gray all of a sudden or they are gray when they log on and stay that way for awhile. I already have had to put my preference settings down to the bare minimum...which is riduculous that that has to be done. I may not have a pc built by NASA, but it should be able to more than handle SL. I have never had to reduce my settings so drastically before, but up until about 2 months ago I have had to. Even that doesn't really help.

I havent worked on my castle in a month because I'm too afraid that it will get all screwed up. The last time I worked on it...I highlighted it to take it up and only parts of it were taken...leaving a big friggin mess that I am still too upset about to even fix right now. I am also missing several things out of my inventory. I find it interesting to hear about people finding things in their inventory that wasn't theirs...it seems maybe things are being put into the wrong inventories during loading. I have seen some odd things in mine that I don't remember having...makes one wonder where they came from.

I am also in agreement that there was no need to have the free accounts available recently. It makes no sense to keep pumping gas into your car when it is full and overflowing...SL has difficulties as it is without adding more people and more inventories to it. I can understand wanting more people in SL, but how exciting is SL to a new person who comes into a world with a lot of issues? I have seen several new people say that SL was a cool place but there was no way they were going to stay and deal with the issues it has. The only reasons most people stay is because of the time, effort, money they have put into SL...and the friends they have made. We have become too accustomed to dealing with problems here. How long will it be before we stop 'Going with the flow' and shrugging off the issues?

For those few people not having all the problems that most of us are...well wtg...but don't talk badly about us that are, by telling us to quit complaining. We have every right to. We enjoy SL and we want it to work properly. We are all addicted, even if we don't want to admit it. That is the reason we are still here...waiting for the drug called SL to be pure and free from side effects.

ZsuZsanna Raven, I totally agree with every word you said, and it frustrates the hell out of me when people try to sugar coat the issue and say there is nothing we can do besides just sit back and wait, and just accept that Linden Labs is working on the problem. I have been blasted and mocked in at least one thread by multiple people for stating my frustration, saying I should not be "barking" at Linden Labs. Funny, but I pay over $150 of their salary every month, so I think I have a right to be a bit aggitated that the service I am getting is certainly not what I am paying for. I appreciate it takes time to fix some problems, and I am completely confident this is a major architectural issue, which means these weekly patches are not going to address the problem. So, it's time for Linden Labs to say "Yes, we have a significant problem, and we believe it is going to take X number of months to fix it completely".

Like you, I love SL. I think it's an amazing place for creativity, and a fantastic place to meet all kinds of interesting people, but these primary functions that are failing in the system are a serious issue, and I too am getting tired of people telling me I should bite my tongue. That's just not going to happen. So, thank you for a wonderful post.