The Lag Monster Myths

For the record:

I have experienced the same ***mysterious-out-of-the-blue*** "lag" on several occassions.

Perplexing in nature, because has previously stated NOTHING had changed ... ALL OF S SUDDEN ... performance just went completely down.

This was on an Island sim that friends of mine had owned.

There was no rhyme or reason as to the cause. Sometimes it lasted for minutes, sometimes for hours, sometimes logging off and relogging took care of the problem and sometimes it did not.

this is a complex game and there are many external, internal, client, server and network issues that are all interrelated.

A useful diagnostic tool that SL could create would list server side process, mem usage , cpu usage, etc.

Perhaps a per land usage mechanism that could be configured to show, in some sorted order, what processes, scripts, etc are currently hoging resources. It may identify a rogue process that should not be there.

regards,
wizzie

Since, as private island owners, we've basically *bought and paid for* the hardware that runs our island, how about a shell that we can log into and see real performance data for the server?

I'm not asking for root. Im not asking to be able to run shell scripts. Id just like to be able to run "ps - aux" or "top" and see what the heck is really happening.

Well, I didn't expect to see this thread come back alive. LOL.

To Wizzie, who offered to go to LL with an armful of techs to check out Linden processes... man, I'd love to see that. I think you'd discover some stuff. I also seriously doubt that Lindens would let you do that in a zillion years... because you'd discover some stuff.

I've had a few months to examine this issue further since I last went head to head with Lee in these forums. I've analyzed data. I've spoken with sim owners. I've spoken with LL employees. I'm no longer a newbie by a long shot. And here's the bottom line:

The LL propaganda machine works REAL good. They've got so many people bamboozled it's not even funny. And the LL groupies who think they know a whole lot about computers and are swallowing the whole thing hook line and sinker-- aren't helping matters a bit. (do I sound ticked? Oh yeah...)

We've taken data readings for MONTHS. We've spent untold hours (days? weeks?) correlating statistical analysis and LL claims. Know what we've found?

Fact: Content is NOT responsible for excessive lag. Yes, badly designed excessive content *can* affect sim speed. Standard content absolutely does not affect sim speed to a degree that causes you to walk in molasses. If it did... ElvenGlen would NEVER operate efficiently... because we fer sure have content.

Fact: Active scripting is NOT responsible for excessive lag. We've tested it. We have seen sims running with 1,500 active scripts at 3,600 fps. We have conducted tests where we ADDED 180 active scripts to our sim and measured sim speed...and the speed actually INCREASED over a period of days. The claims that anything over 500 active scripts is excessive is a load of hooey.

Fact: Textures are NOT responsible for excessive lag. Again, if it was, ElvenGlen would NEVER operate optimally. I mean, textures are to blame? chee... must think we're morons.

Fact: Server-side issues, SL programming and internal data layouts and routines ARE responsible for excessive lag. The streaming inventory system is terrible, communications don't work properly and there is significant evidence that the SL system is just chuck full of code that is screaming for optimization.

SL problems are understandable to an extent. All major systems experience growing pains. It's not those facts that irritate users. It's being lied to about those facts that irritates users... especially sim owners who are paying Linden Lab thousands of dollars a year to host a piece of virtual land. We do not like being told that SL lag problems are our fault-- when every chunk of data analysis that comes through the line proves that is not the case.

Like Wizzie, I am tired of the smoke screens, the lies and the user-blame games. I'm tired of being mislead by Linden Lab as to the cause of lag issues and I'm tired of them telling me it's my CONTENT when that content is EXACTLY the same today as it was yesterday-- when we experienced NO lag whatsoever. I'm tired of being told that my RUN TASKS ms jumping to 8.5 today is a result of my content.. when THAT CONTENT HAS NOT CHANGED SINCE THE DAY BEFORE when Run Tasks was running at 1.3.

News flash Linden Lab: we are not idiots (well, at least not when it comes to analyzing data. It might be argued that anyone who pays a company thousands of dollars a year for a piece of imaginary land might be somewhat of questionable mental balance... but, that's what we do). So if anyone out there thinks LL isn't being straight with us, is pulling the wool over our eyes and is pulling underhanded stunts to cover up the way their system really works... the data says YOU'RE RIGHT.

No Wayfinder, tell us what you REALLY think

Don't worry... I'm about to.

Good idea Paulie. Of course, you won't get it, but it's an absolutely reasonable request.

If ACTIVE SCRIPTING is so much to blame for sim lag... why is it, as a sim owner, we can't find out WHO has WHAT active script WHERE? Obviously the system can detect that data. Obviously Linden Lab can. WHY CAN'T WE? How are we supposed to control active scripting when we can't even find out what active scripts are on the sim WE OWN AND PAY FOR?

WHY IS IT LINDEN LAB that private sim owners, in order to properly protect our sims, are forced to purchase expensive third party security devices? Why are those tools not already built into sim owner/manager controls? Linden employees have those tools. WHY DON'T SIM OWNERS?

WHY IS IT that the sim OWNER is expected by Linden Lab to also be sim manager? Has Linden Lab never heard of PARTNERSHIPS? Why is it that there is no MANAGER LIST function with which the sim owner can specify people who are given full sim control authorization? For that matter, why can there be only one sim owner? Two or three people can't go together and buy/pay for a sim?

And while we're on the lack of tools question... WHY IS IT that officers in a group have no way to communciate with all group members at the same time (both online and offline)? How are we supposed to operate a group when we cannot make event announcements, group announcements, or circulate newsletters to ALL our members? Your current communications systems are inadequate and simply do not work. How are we supposed to run a group when we cannot COMMUNICATE with that group (other than by a kluged, irritating and highly inadequate VOTE system)?

And back to the original post... why is it that sim owners do not have the simple tools to tell us what is going on in our sims? One cannot help but wonder if perhaps it's because LL doesn't WANT us to know what's going on in our sims.

It seems that for $1200 in setup fees and $2,340 a year... we should receive a lot better customer service than we're receiving. My group is currently spending more than $7,000 a year for land at Second Life... and considering the above factors, accompanied by the shameful lack of support or even apparent caring from Linden Lab... we find ourselves questioning that investment.

It may have something to do with some chowderhead pointing to this thread in a Hotline to Linden Post:Obviously Comrade Wishbinger, you have not been sufficiently brainwashed. But don't worry, I'm here to help. Just say the following to yourself repeatedly and continually. After a few days, you will suddenly find that it all makes sense and you will wonder why you ever questioned Linden Lab at all:

1. Linden Lab loves me.
   
2. Linden Lab loves all humankind.
   
3. All Lindens are good people.
   
4. All Lindens work hard.
   
5. All Lindens know more than me.
   
6. No Linden is ever defensive.
   
7. No Linden ever attempts to blame bugs on users.
   
8. Second Life® is the most is the most reliable computer ever made.
   
9. Second Life® has never made a mistake nor distorted information.
   
10. Linden Lab is, by any practical definition of the words, foolproof and incapable of error.
    
11. Two plus two equals five.
    
12. What have I done for Linden Lab today?
    
13. Second Life® is too important to allow me to jeopardize it.
    
14. Linden Lab is not making a game. Linden Lab is making a Country.
    *edited*
    
15. There are thought criminals who maintain that lag is real.
    
16. You must love Linden Lab. It is not enough to obey Linden Lab. You must love Linden Lab.
    
17. An empty sim is an efficient sim.
    
18. Linden Lab is very scrupulous about following up and eradicating any error.
    
19. All I have to do is be Linden's pal, say nice things about them, tell the folks at home what a good job they're doing. Either that I will be banished for sedition.
    
20. Ignorance is Strength

Comrade Wishringer, I only give you this advice to make you happy and because I wish for you to be of the fullest possible use, which is all I think that any conscious entity can ever hope to do.

That has to be the funniest thing I've seen appear on these forums. I wanna have that printed on a T-shirt (well, except maybe line 15. Might edit that a bit).

I literally LOL as I read through that list. Now all I have to do is copy it down for posterity (just where DID I put that notecard giver script...). Ah, never mind, a huge monolyth with that list imported as a TEXTURE ought to cover it.

OH NO! I forgot! Mustn't use TEXTURES! They lag!



--ps your message took me right back to 1984. What a great book.

To be totally fair, I understand why LL does things the way they do. I just don't understand how they believe they are going to create thousands and thousands of loyal customers. Instead, eventually their customers are going to throw up their hands in despair and jump ship for the very first competitor that comes along. There are already tens of thousands of people who no longer use Second Life. They tout "50,000 clients"... some 35,000 of which apparently haven't visited SL within the last 6 months. Seems to me their basic business philosophy is absolutely self-destructive. But then to be honest... it does seem like the majority of large businesses today are pursuing that same philosophy: rake in the bucks now, forget about tomorrow.

The one noteable example I'll site is SPRINT cell phones, which was so greatly ticking off their clients that they registered bottom of the list in customer satisfaction-- so finally decided that well duh, you know, ticking off clients isn't the best way to conduct business. From what I can see, the company is working on doing a 180 and although their customer support is purported to still stink, they're at least starting to offer their clients increased services designed for the good of the client.

A close business associate once told me: don't worry about the money. Care about the client, and the money will follow. I think if LL is going to survive in the highly competitive VR field, they need to make that statement their prime directive and paste it up on their wall. I don't think Linden Lab is intentionally a bunch of jerks. From what I hear from those who have had direct personal dealings with them-- they're basically friendly, enthusiastic folk. Too bad that doesn't come across in their business construct. And too bad they're not honest with their clients.

I proposed the shell idea. Go and vote for it if you think it's a good idea:
http://secondlife.com/vote/index.php?get_id=657

I'll also cross-post the idea and the voting link to HOTLINE.

I gave it four votes. I hope that every sim owner in the place votes for this proposition as well. However, do we really believe that Linden Lab is going to give estate owners the tools to prove that lag is LL/server fault? Even if they do provide a user shell, how are we to trust that it will contain the proper tool-- or the untampered tools-- necessary for us to manage sims? We have been told for months that the RUN TASKSms reading is specifically related to sim content-- and have recently discovered that to be a false claim. So even if they did provide such a resource-- are we to believe it will honestly portray actual server activity?


Equally important vote propositions:
(All of these proposals should have been a standard part of Second Life ages ago. Due to the screwy way Linden Lab has set up their voting system, many of them haven't received anywhere near the number of votes they deserve.)

ALTER THE VOTING SYSTEM TO REFLECT TRUE VOTER OPINION
http://secondlife.com/vote/index.php?get_id=639

ALLOW EXPORTING OF INVENTORY CONTENT
http://secondlife.com/vote/index.php?get_id=20

ALLOW PARTNERSHIP SIM CONTROL
http://secondlife.com/vote/index.php?get_id=133

INSTITUTE OFFICER-TO-GROUP NOTECARD FEATURE
http://secondlife.com/vote/index.php?get_id=448

You are right and wrong with these statements. You even say so yourself. It is not the number of scripts, or textures, or content, it is what they do. If you have a lot of llListens, then you will have more lag with fewer scripts. If you have physical objects, you will have more lag per prim. If you have very large textures (greater then 512x512) then that will also increase lag (mostly just in download time). SL is not the best it can be, most Lindens can tell you that. They are doing the best they can with the time they have. Everything that you listed effect a server. Most of the time it is not just one thing that causes the lag, but a combination of them. I have seen sims brought to their knees with just a few active scripts, or a few really large physical objects. It is not the raw number of items, but it is what the items do.

PS, I would love to see your data if you still have it around

To which I'd add proposition #8192, "if you put up a feature voting system or a feature suggestion forum then make some connection between customer's express desires and actual plans" but that would sort of be a self-subverting proposition. That is unless... [sotto voce] they never intended for these to be anything other than a ruse to make customers think they cared.

Woah, my re-education just left me for a moment. Time for another hour or two at the Erewhon Re-Education Camp, Miru (24, 79). Linden Lab loves all humankind. Sorry.

LL has listened to those who have really done solid testing work, and 1.7 is going to see an entirely new sim backend process which runs much smoother and is far better designed to cope with bad content.

and yes there have been sim issues, many of whichhi did discuss in detail here and in other threads...

there have been instsnces of sims booting up 3 to a 2 core machine, and yes this causes all kinds of problems, and can be hard to detect..

but again *most* sl lag is indeed content related

i have never been to a 'laggy' sim that wasn't full of sim-expensive, badli coded content..

people want their poseball chairs to look pretty, not run fast, most wont even know what that means... i've seen single items that caused 500 k/s bandwidth from *everyone* in the sim to be continuously drained, jusy so someone can have their badly made vendor/pictureframe/typing anim keyboard that split secomd more responsive, etc

yes, we do need tools to help people better see the impact of what they create... most people honestly dont know their things lag, much less how to spot it or fix it.

but again 1.7 is going to be far more accomidating to many of these bad objects... and latter versions even moreso still...

but remember LL gave us all the tools we need ro bring sims to their knees... usin them responsably however is not *their* obligation, its ours

Dnate, ya need to go back through all 4 pages of this post and read back posts. You're just repeating things that have already been shot down.

I do know how scripts work. I know what a listener is. I know that one single script can lag a sim to a complete standstill. That doesn't have anything to do with the subject at hand: sims all over the grid suddenly lagging for absolutely no discernable reason.

Dnate... when the pros visit sims and can't find a cause-- the cause isn't content. When content doesn't change but sim lag does... it isn't content. When sim speed drops and stays dropped for weeks and months on end and content hasn't changed... bud, that's server side. Anyone who can't see that really needs to go back to Computer 101.



I think there are a lot of people on SL who would heavily disagree with that statement.

Yeah, I am lazy and didn't read. I know that there are problems with lag for no reason that you can see, but you made it seem the only reason for lag was server side. I have often wonder since more then one sim is on a machine, could the other sim on the machine effect the other? If Lindens thought that SL was perfect, why are there so many bug fix releases?

Many will say that it is the best that they can do, but it is not the best it can be.

While I understand that you are frustrated, it is not appropriate to post inflammatory attacks against anyone in the SL forums, including Linden Lab employees. The offensive attack has been removed.

Jeska, there was no offensive or inflammatory attack posted (at least not one that I read... maybe there was one posted. You didn't refer to the poster/what was posted, so no can tell). The things I've read here are factual statements reflecting quickly growing sentiment of SL landowners-- people who finance LL paychecks.

You can remove whatever posts from these forums that you want to remove. That's not going to lessen the coming storm that's brewing around Linden Lab. LL has been cautioned about their attitude toward sim owners for months. Rumors are a new sim owners group is forming, and I expect it to wield a little more power than individuals. Stuff like that doesn't happen when things are going right and customer service is doing what it's supposed to.

Just some thoughts.

Eltee... does LL pay you to post this stuff? LOL



I think the data presented in this thread has sufficiently proven that is not the case. You want to ignore data Eltee.. your decision.



SURE IT'S LL's OBLIGATION, Eltee. Dude, I've worked as a business consultant for two decades, specializing in computer systems analysis and design. If a company does not have security devices in place, they cannot place blame on their employees and clients when something blows up.

The moment Linden Lab invented a scripting system to work in a public VR world... they took upon themselves the responsibility to try, as much as possible, to insure that scripting system could not be abused.

They did not do that. They allow push scripts so powerful that it can actually knock users offline. They allow particle scripts so laggy that one device can bring a sim to a standstill. They do not govern the number of active scripts that can be used on a 512/1024/4096/sim... and they give clients no way to detect when an active script is being used. They do not control the number of devices that can be used on an avatar, the way AO devices work (talk about lag devices!) or any number of multitudinous things that should be a standard part of any computer security coding evironment.

So yes, it is absolutely LL responsibility when sims lag due to excessive script use. The vast majority of people who build on SL are not professional programmers. You can't give a loaded gun to a child and then blame the child when it goes off and kills someone.

But all this is irrelevant... because the data specs that we and numerous other sim owners have performed prove beyond any doubt that excessive lag is not primarily caused by content... but rather IS SERVER SIDE.

I don't care how many times you make the claim Eltee... the simple fact remains: when content doesn't change and sim speed does.. IT'S SERVER SIDE DUDE!!!!

Good points Dnate. Yeah, the server abuse is about to become a major issue, I think. Linden Lab claims that "our servers are dual processor so they can handle the load".

Well, when 20 avatars come on a sim and lag it to syrup... I don't think that claim really holds up.

And when people pay $1,200 for a sim server and then find out they're actually being stacked on an already existing server... that's not going to make for happy clients.

Stacked sims absolutely do affect one another.

More on this issue is about to strike the rotary oscillator.

I am sorry, I don't see any PROOF that it is all server side. Since all sims run the same code, what is different between 2 sims, one that is lagging and the other that is not lagging? Content. That is what is differant between sims. You can state whatever you want, but a single instance is not proof that all the issues are server side.

One more question, you noticed a drop with sim preformance, are you 100% sure that there was not an object dropped by someone else on the island that could have been causing the lag?

Have you seen 1.7? 107 avies in one sim and it still held up running pretty well. I was getting 8-9 FPS on the client side, and the sim seemed to be running at about 33 FPS. I hope this will really help with lag in areas. Guess we will wait and see.

Dnate, you still haven't read this thread, have you?

You know, it's stuff like this that makes it so easy for LL. People don't pay attention when others who have done the work report their findings. And then... they make the same arguments/claims/statements that have already been discussed and shot down.

I have read it this time, really! It just seems that since all servers run the same code, why do some lag and others don't? It seems that all the sims also have about the same specs, so what is different?

Ok fair question. The reason some lag and some don't is due to a lot of issues.

Some sims are sharing servers (more on that is to come in another thread). Others aren't. Some servers are faster than others. Some servers are functioning at maximum capacity, some are malfunctioning.

And as you mentioned, SOME sims do have awful content. I have never denied that content CAN affect sim speed. I have stated that is not the case with ElvenGlen or most other sims I know of. Shoot, I have one particle script that guarantee will bring any sim on the grid to its knees. So I never use that script. Yes, content CAN affect a sim. That doesn't mean it DOES, on a daily basis, day in day out, at random times for apparently no reason whatsoever.

Dnate, until you own a sim yourself and track stats for months on end... might want to at least consider the claims of those who have.

If the number of sim owners who have experienced flaky, unexplainable sim operation were to post in this thread... you'd see it grow immensely, overnight.

I have to say that I would never consider to do any real work in SL. I am a programmer that has been working on developing games using openGL() along with maya and photoshop. I decided to try SL to see if it would be a suitable platform to get feedback for my games. First off the scripting language is a joke, the lag- be it sever side or client side is ridiculous. I decided to enjoy the game for what its worth - a glimpse at what the future may hold. I think it was chosen who said that maybe they have to start all over. Which will not be the case but i think another development team will set out to create a world much like SL that will be able to keep in mind the problems in SL. Don't get me wrong i still find enjoyment in writting simple scripts and expiermenting with textures. Though i find most of my time is spent in here reading very well thought out ideas presented by all of you. The Lag is not going away its just a fact and with the amount of users getting larger each day the technology that SL is built on will not be able to burden the load. SL is a truely pioneering game/platform and i believe it is just the begining of better things to come.

All sims do share a server with another sim. If you Ctrl-Shift-1 you will see a stat for Sim CPU. You will see that it is at either .5 or .25. If it is at .5 you share the server with one other sim, if it is at .25 you share it with 3 other sims. I am almost positive that all the servers that could only handle 1 sim per have been placed in the void areas and host multiple sims there. Like it has been stated before, scripts do things, and what they do changes. It may seem random to you, but it happens when script x goes to state y. With 100s of scripts all running together, it would be near impossible to find out if it which script and with transition causes a spike in lag.

Jack, thanks for your thoughts. I pretty much agree with all of it. I'm frustrated by SL and LL.. but I enjoy the basics of the concept. I like building and scripting and designing. I don't like it when stuff I build and script and design fall apart and I'm told it's my fault when I know that's not the case.

I do disagree with the idea that the lag isn't going away. Well, not really disagree. If LL keeps up with past performance, it's not going away. But I think that if a really competent watchdog were to get on LL's heels and was allowed to wield some real power-- the majority of lag could be vanquished in 3 months, 6 months tops. I busted LL's chops about the ghosting bug and the teleport bug months ago and was constantly told nothing could be done about it. I guess I and others like me busted them enough... because those problems are largely gone. Same with lag. Just takes us rattling enough cages long and hard enough for them to start listenting to what we're saying and start looking in the right places.

I do 100% agree that it is very likely that before LL gets its act together... someone else is going to come along and eat their lunch. We've seen it before with both IBM and Texas Instruments. Oh, LL will still be around. But they'll have so many people jump ship it'll give 'em a heart attack. And if LL survives at all, they'll almost certainly lose market position they hold right now.

Can't treat your clients with disrespect and expect respect in return.