ass set server

True, but identifying what the problem is does help, Eggy, you know that.

It's the one thing Moopf hasn't done in this thread is identify their problem, merely say "there's a problem" which doesn't help anyone figure out what that problem is.

Without the personal inside knowledge that you seem to have it's actually a little difficult to pin down what isn't working when things don't work. I can tell you that editing linked prims is buggered, I can tell you that moving unlinked prims up and down is buggered (and has been for over a year now as well. bug reported. ignored), I can tell you my client FPS is down a large amount, I can tell you that I've never been able to turn shadows back on since 1.6, I can tell you that textures aren't loading, I can tell you ad. infinitum about a whole host of problems - a list of problems that gets longer and longer as time goes on. I can also tell you that with each release using Second Life feels more like a chore than enjoyment because of all the nice new problems and issues that crop up.

But you want technical reasons for those? Well, forgive me, but that's the Linden Lab job, not mine. My job as a paying customer is to tell them what's not working for me - I'm not here to pin it down for them. However much you enjoy doing that, that's not a pre-requisite of paying Linden Lab a load of money every month. If it was, I'd be off like a shot!

It's all about customer support, often you can say there is a problem but you can't tell those who make the software what's causing the problem, because that's outside your domain. And it's not always possible to give them a reproducable set of circumstances either.

And yes, regards the asset server, I stand by that - it's quite plain to see that when you take something from in-world and it doesn't appear in your inventory, that's an asset problem. It's also plain to see that when you try to rez something and it doesn't for a good 10 minutes, that's an asset server problem. It's also plain to see that when textures won't load, that's also an asset server problem. These problems haven't just started, they've just reached a critical mass (yet again) with the release of 1.7. It gives lots of problems lots of times and has done for well over a year now. Oh and the login server and now the new space server that they've just started mentioning. Oh and squids and what not.

I'd just like to say: NICE EXPLANATIONS ICE.

And more Resis should refer here.

OK, now we're getting somewhere:

Editor bugs: report as Building bugs, no they aren't ignored, it's just there are also a ton of other bugs on the list since there's a lot of new code introduced with 1.7, and thus a lot of new bugs.

FPS being down was observed in a forum thread stating that the average FPS is down by 2 frames per minute, so they are working on fixing that, probobly 1.7.2's release early this upcoming week.

Have you reported that you're unable to turn shadow on?

A detailed description of the problem can be as simple as "Second Life crashes when I check this box" that tells the QA team where to look.

As for bugs, not all bugs can physically be addressed, there are only a limited number of staff members, and usually quantity wins out as far as bugs they're able to deal with.

If an object doesn't appear in inventory, has the inventory streamed in fully, or are you still waiting for items to appear? I personally haven't seen that particular problem except for a day when there was a hardware error since the grid was upgraded to the 1.7 codebase. And I as a rule stop messing with things when the asset server's having a bad day. I honestly recomend anyone who is willing to, to do that as well, as it's easier to repair the server's hardware problems when it's not being slammed by people trying to repeatedly do something while it's having a problem.

Rezzing, oddly enough is handled by the /sim/ not the asset server. Asset Server may be contacted to find out what to rez, but that's done in inventory streaming.

You know, are you trying for a job at LL or something? Seriously, you're more like customer support than they are!

And yes, I report a lot. The shadows bug I reported as soon as 1.6 came out, had Lindens over, lots of other people were getting the same thing and then, finally, we worked it out amongst ourselves that it was shadows. The Lindens even had it happen to them on my island where it always happened for me. Anything get done about it? Nope.

Regarding objects in inventory - have you read the forums recently? It's a popular problem at the moment.

And quantity, not quality, kind of sums it up nicely for the way LL approach these issues.

This "hold off when the asset server's dodgy" just doesn't work in the real world I'm afraid. Their system should be able to cope. it can't. That means something needs to be done. End of story really. They can't expect everybody to suddenly stop whenever the asset server's having a bad time - they should address why the asset server has these bad times and remedy that.

Hmmm.... Ice Brodie?

Someone who sounds more 'in the know' than any Linden Liaison I have spoken to.

A better communicator than any Linden I have yet heard.

Yet, apparently not a Linden...

Discuss.

Quite - well whatever he is, it makes interesting reading.

You have 4 people sorting bugs, 30 people dealing with those bugs... the count of bugs is roughly 10 per reseident, give or take... given that there's 1,000 - 3,000 residents online at any one time...probobly 3,000 - 8,000 residents active total (give or take, either direction, these aren't solid statistics) in that rough range, there are (maybe) up to 80,000 bug reports, the relevence of a bug could also be affected specifically by video drivers, cache status, internet connection and operating system.

Also have you: Updated your video drivers and tried shadows again? Cleared cache and tried shadows again?

I wonder, do people drive /faster/ because their car's making a clanking sound... I've actually seen global announcements of "Don't do (describe cause of problem) until Ops has finished fixing it" and what happens, the people around me actually try doing that... Maybe I just don't understand the need to intentionally drive something towards a limit when there are obvious (described) problems.

When you have a system that runs 100% of the time, that system has /natural/ wear and tear at a physical level, unlike a personal computer which only sees short bursts of full use, the Asset system is being used constantly at a capacity that would make all our desktop systems melt, it's only ever shut down if it /needs/ to be, most of the problems with it are fixed by dealing with something on the operating system level (it is another Linux machine, after all)

Hold on, now you're going back into the normality of Linden Lab support again Yes, my drivers are up to date. Cleared cache? This started when 1.6 came out, not 1.7 I've even changed computers since then and it still happens.

My heart really does bleed for those stats you pulled out of the air. Maybe if they were tighter in theirn QA, addressed issues as they came up, installed a coherent development plan etc. etc. then the number of bugs per user would be dramatically less. Sorry, I'm a heretic I know. These are the dark ages after all.

You've gone into spinning around the problems, which sounds a lot like LL as well. I thought I was getting somewhere with this conversation, you started by talking the right talk for the most part, now you just sound like a puppet of the puppet masters. I guess you're too closely aligned, which is your right and understandable if it's what makes you happy. I don't think it helps the end-user though.

PS. At the end of the day, I don't need to know how difficult all this is to hold together, how they're understaffed, how it's all so hard etc. etc. I pay my money to a company and expect what I recieve to remain at a fairly consistent level of service. I think my bang per buck has gone down a lot over the last year, and that shows no sign of stopping.

ANOTHER Linden alt bites the dust?

(I am not a Linden alt, that's a very common misconception, my resume was turned down due to poor performance on the programming test and the fact that I lack a BA in computer science)

They're the only QA team I've actually talked with when I've reported bugs, they investigate, they work on reproduction, then they add to the developer's huge stack of things to do...

Perhaps, upon reading, you'd notice that I describe that. 1. SL is a huge virtual world simulation unlike anything anyone has yet done, and thus has /unique/ problems on a grand scale that no one prior to Linden Lab would have encountered and 2. They're horridly understaffed for what they're doing. (but hiring!)

The development plan's actually live, development currently on Colo grid (one of 3 total grids, Agni, Siva and Colo) is working on the 1.8 server codebase, early development. Siva, is currently used for 1.7.2 and 1.7.3 client side testing, as well as the 1.7 server codebase. Bugs that are long standing... most likely are either harder to track down or not reported as much. (or if they are, in a fashion that doesn't actually identify a bug)

I point to the fact that I had to ask 4 times for a description to get anything besides "the asset server is a problem." and then got a reply that pointed to other systems, thus someone in QA working with you would have had to check the Asset server event logs, talk with ops, and find out later that they're issues to check other logs with.

People seem inclined to think that tech support can read their minds... makes me wonder what QA sees, honestly. Oh wait, I can answer that... "U R TEH SUXORZ" followed by an incoherant string of explination points, ones, and simular punctuation, maybe a general category if they're lucky. Tends to be what I see in any complaint department environment.

The prior paragraph of this post points to a general phenominon... common in many tech support environments, just ask anyone who's done computer tech support about cup holders...

Detail as much as you can about observed problems... can become the difference between a heated argument and a quick solution.

Ice, with all due respect, I think you're being very presumptuous on (a) my background and experience and (b) the quality of bug reports I have submitted since I've been here. I was not writing a detailed thesis in my first response on this thread as to the reasons why I think the asset server SUXORZ but yes, I do think it SUXORZ quite a lot. As has been discussed so many times since I've been in SL, it is a bottle-neck. It does not scale well. The architecture needs some more thought. You seem to find it acceptable for there to be several days of total instability after a patch? I don't, the system should be built to deal with it.

As for a development road plan, I'm not sure if you read Andrew Linden's recent post that was discussed at length in several threads and gave a keen insight into how Linden Lab operate - there is no coherent development plan, the developers simply work on what they find interesting at the time and on fighting the fire, where the flames seemingly lick higher and higher with time. They work in the here and now with no clear outline of where it will be heading. That, in my opinion, is no way to run a software company and expect good results, especially when staffing is limited.

Maybe, if they're understaffed and the technology is so different to anything else (I'd reserve judgement here, personally), then they should not be charging top wack for something they cannot adequately develop and support at this time. You are giving excuses but no resolutions to the problem. You're just saying how hard it is. Business isn't easy, it never is. But when you consistently fall below normal customer expectations as Linden Lab do, then maybe somebody needs to get a grip of the situation sooner rather than later and stop making the excuses.

Whilst I appreciate "U R TEH SUCOXRS" isn't useful (and is hard to write), and I try to give detailed reports when I can I don't work in a technical setting like LL despite being a moderately able scripter.

I try to give them what information I can, even now when the crash reporter is crashing on my mac (someone who I won't name gets them pasted into an email every time, lucky Linden!). But I don't know if I'm doing well or badly with what I write - I know it's basically grammatical and spelt more or less properly, but useful to them, not sure.

I also have to say that there is a limit to the number of different ways I can write "I spend time in world, scripting seems to reduce that time, when after a critical time has passed I touch a blue key (friends, IM, profile, etc.) I crash."

Back to my scripting, I know from debugging code (much, much smaller code) that it's a painstaking nasty job. It takes hours. I do appreciate what they're doing, all of them, and I'm damn sure they're working hard at it.

But if they'd tell us what is useful and what isn't, give us some feedback rather than an automated bug report and then in the vast majority of cases nothing else I'd be more even more sympathetic. Something that clusters similar bug reports (I'm sure that for many it's not only me) and sends out a bcc'ed email saying "Someone's looked at your bug, it is..." and either fixed, going to wait for a future upgrade, here's a work-around, confusing us and on the back burner whilst we get more data would take a few seconds (possibly less, there must be an automated way to do it from the bug tracking software they use). It might mean one less bug looked at per day. It might also mean WAY better bug reporting over time. Isn't that worth it?

I'm sorry Moopf, I didn't reolize until I posted that last post that the original post wasn't yours. I made a terrible error in that respect. And I do appologize.

Andrew and the other devs are working on more coherant work plans, though the current one has worked very well for them in the past, with the scaling SL has seen in the past year (an increase by roughly 1/3 every 6 months, based on my observations) that they have re-organized, 1.7 is the first version where they've gone into project teams, rather than everyone simply putting all their code into a central project. Yes it's developed without an extremely solid core plan, but so have several open source projects with simular and smaller teams. Linden Lab's work environment encourages productive creativity, not mindless automita (*cough* EA *cough*) who's only goals are to complete subroutines under quota.

Bug repair is as high a priority as humanly possible, but if they stopped new development and fixed all the bugs, it'd be 2 years before 1.8 comes out, and the spirit that drives Linden Lab would be crushed in months.

A repeat of a bug is best described as "Continued bug problem" and mention the prior time you posted the bug, makes less they have to read and more time they can spend hunting it down in that case, honestly.

That's not all that hard to figure out, Ice mentioned the servers are 1 Rack Unit each and there are roughly 1200 of them.

Let's say they are using APC NetShelter cabinets at Linden Labs as APC makes some pretty nice cabinets and they hold a decent 47 Rack Units worth of hardware.

http://www.server-rack-online.com/apc-netshelter-ar2104blk.html

So 47 rack units, Lets allot 1 Rack Unit for a switch so thats 46 servers to a rack if you packed them in without any space between them.

So now we have 46 Servers to a rack, That comes out to about 26 racks worth minimum needed to hold those servers.

90.25" H x 23" W x 42.2" D

Now data center design varries widely so I cant really guess past the last step as to how big the room would be to hold those 1200 servers so I guess I didnt really answer your question either.

26 racks doesnt seem like all that many but spacing them for airflow and access is going to increase the amount of space needed in a room for them, not to mention whatever space would be needed for the UPS and Enviromental systems.

Ice, no need for apologies, I don't mind robust arguments and misunderstandings happen on linear boards all too easily (that's how I have my settings).

Part of my problem with them only introducing a coherent road map for development now is that they should have seen this coming. They courted it with advertising, account offers, press etc. They were caught with their proverbial pants down because they didn't have the foresight to plan for it.

And the open source argument is totally eliminated by the fact that this isn't open source and is charged for and, it has to be said, isn't exactly cheap. Keeping the employees happy is all well and good, but if you're not delivering what the customer expects, that's not a feasible way for a company that relies upon those customers to exist, to carry on with. And plus, one doesn't have to preclude the other.

Andrew's post was made just before 1.7's release (if I remember rightly) so I don't believe that 1.7 was built with the new development plan in place - the developers are still working on what they want to, from what I can gather.

As for fixing bugs. My problem is that they generally appear to tinker around the edges and allow many more important ones to slide. As a content creator, I've found creating content increasingly frustrating as time has gone on because of the bugs. I'm also dissapointed with the quality of new additions such as HUDs - they leave a lot to be desired and have given me the impression of 'just enough to keep them happy' rather than actually having pride in a well designed new feature.

As a business LL has to strike a balance. It can't be all about the employees being happy in their job, that's just not going to work. It's a good way to piss the customers off. That they're only just realising that makes me wonder what planet the management is on, to be frank.

I like your idea of what's meant by a repeatable bug. Kelly Linden's posts, especially, really go to town on stating the need for reproduceable bugs if anything is to be done. That's not always possible, and I do appreciate how difficult they can be to solve, but they can't reply upon their customers to be able to supply it.

You certainly seem to have a great deal more faith in the developers of Second Life than I do. I've just seen things gradually silt up, if you like, since I've been here. What LL really needs, however, is some serious competition. That would have a great effect in focusing their minds I guess. Without competition, it's easy to take your own route, with competition you suddenly have to be a lot sharper and try harder not to drop the ball.

The main Agni colo (yes main, they actually have several facilities) has a section of their datacenter dedicated to Linden Labs for growth. Another colo currently houses Siva and Colo grids. My numbers are not exact, but include all three grids. My exact numbers are not current at this time, this was a few months ago.

The support servers (Squid, Database, Asset, and Space) are all most likely RAID servers with the exception of Asset which is a high volume multi-blade system designed for database management of corporate infrastructures. while Squid, DB, and Space are all likely to be 4U RAID equipped servers.

I do know that the servers have double redundant power supplies, the Asset server has several hot-swap redundancies built in (one prior problem they've had was actually related to a RAID controller that claimed to be able to be hot-swapped but failed when it was tested)

90% of my data is a grouping of multiple observations I've made... so may not be totally accurate... I'm attempting to get a network diagram currently.

As for Andrew's post... 1.7 is the start of the multi-team development, and this was phased in late during 1.7's development (probobly when preview builds where already compiling fully)

Balance, sadly is why deveopment towards bugs, towards new features seems stagnant, HUD is the first phase in a route that will lead to a fully customizable UI, and incidently, they fixed a long standing bug with attachments in the process.

Bugs have priority based on several factors, severity, quantity, age, and reproductivity, that also becomes ballanced with new features, and in some cases, if a bug can be worked with, it may get lower priority (sadly) in favor of bugs that are more problematic and that have fewer or no workarounds.

Sadly some bugs (like agent attachments not being able to move) are cases where a bug is deemed 'not severe' because it was seen as a limitation and not an actual bug, but with HUD... in order to make the HUD work properly, they discovered that it was indeed a bug and fixed it. This happens with a few things, sadly, and is due to the need to balance.

Not everyone percieves that balance as fair, but it's the bane of a project this large and a team this small. It's not an army of coders, it's a small group fighting gourilla warfare on the bugs... (LLs programming staff are about the same size as a platoon of army soldiers. at 30 developers.)

Which maybe should tell them to hire more. I doubt they make little profit enough to do more hiring. A more stable and working SL keeps the old and brings the new.

Given this last month since SLCC they have received a lot of press. If I was brand spanking new, and Neil Protagonist had told me to get on now, I would have said, "WTF, this sucks".

So, hire a few more ppl to sort it out if that is the bottleneck, and do it without needing BA's and whatnot. For twenty five years now I have programmed very successfully, a decade of that self-employed, and have no paper on my wall.

But they don't even need more coders per se, just more ppl answering the bug reports, sorting them, prioritise, and better, faster customer relations.

Currently they don't have a QA manager, and they are hiring for QA team.

http://lindenlab.com/employment.php and the recruiting center in Waterhead. Have had QA positions up for several months.

Nope, a Linden wouldn't have shared these numbers. Number of open issues in 5 figures?!?! Oh, and not 70,000+ residents ... really only 3 to 8 thousand. All the other accounts are, well, "not active." Ten open issues per resident ... well, let's assume five thousand "active" residents is, well you can do the math, I'm sure.

In 20+ years of working in software development, I've never heard the like. I guess I don't get out enough.