I triple dog dare you, LL!

Right. So I went a bit insaner today. Emailed this to LL:



Lets see if they're chicken.

I think you're probably right about the facts here, Moleculor, because there is no way in which LL would upgrade all of its servers to the same (or even similar) levels at each upgrade point. Instead, they evolve the grid piecemeal.

Disparity is built in to the upgrade model here. It's an inherent property of statically assigned systems.

I think I understand the reason behind Philip's words too. One must remember that he is a businessman, and a good one. He knows that he must never ever ever ever get himself into the dangerous waters of being seen to upgrade servers at customer request, because once there is a precedent for this, demands would never cease.

Until it all goes dynamic, the unlucky people stuck with the oldest servers will also be stuck with having to fight their corner to try to pressure LL into an upgrade before the planned time for their zone. It's not going to be pleasant.

or mature....

LL has been getting smart about the sims. Just today i came up with a new incantation for crashing sims; and after 3 tests the sim had picked up on the incantation and any time it ran the sim would delete the object before the incantation finished.

On a Sunday? Nice.

I'm saying the server did it automaticaly; without any admin imput. I ran the same script 10 times in less then 10 min. And the first 3 times the sim crashed. The forth and subsequent times resulted in my test object being deleted.

Odd.

I think i know why the "lower numbered" sims would be slower. If LL is using an ID system for their sims starting a 001, then server 001 is the oldest and probably runs on the equivalent of a in-the-garage server (lol) meanwhile sim 900 would be faster since its on a brand new server running better equipment.

As for the crashing.. i have no clue how you suddenly break a scripting engine... I know someone who had a dragon head that would crash the server simply because it existed though, lol.

Well, If the servers are ever replaced maybe we will get lucky and the old servers will be used to fill the rest of the void areas rather than just being decomissioned.

Upgrading your Video Card, Processor, and Ram would help you greatly I think.

Uhh...

1. That's almost identical to the one I posted. Hell, it's WORSE in the Physics FPS and Time Dilation.
   
2. My computer has nothing to do with the simulator's performance.

Waiting on a response from LL...

With my setup I get near zero lag in pretty much any sim with any load. Just saying with a few upgrades you wouldn't even notice sim performance.

You've obviously never scripted or been in a sim with 0.75 Time Dilation.

Pretty much, the only stats your computer can affect are visual FPS and network lag. And even the lag can be out of your control. If your net connection slows, you lag.

I think Moleculor is referring to the lag introduced by scripts/prims/textures/agents that are running and executing on the server-side, not client-side. Yes, your setup can defeat lag introduced by a thousand llTargetOmega rotations, and 30 club-goers with their handheld particle wands... but it won't stand a chance against a few too many physics-enabled objects, or heavy scripts. Especially if the scripts are listening for multiple commands, scanning, calculating and doing whatever else they do.

Even with the best hardware you could purchase for your end, the speed at which the server spits data to you is limited by that server's ability to process the next instance of each running script... the new position of each Avatar and car... etc. So, you will experience lag in these situations.

The main component of network lag under normal conditions (ie. not under fault conditions) is the network latency given by your round-trip times, as seen in ping. This latency is not under user control, except in a very small part. For example, upgrading a 512kbps ADSL link to 2Mbps will probably reduce your RTT by 10 ms or less, which in most cases is a very small proportion.

Typical ping times into the SL grid range from under 70 ms within the US to around 150 ms in Europe to as much as 300 ms in the far east. There is barely any correlation at all with bandwidth: you could be on a 5Mbps dedicated line and still suffer a 200 ms RTT. It's all about backbone path, and that's not under your control, other than by choosing an ISP that employs a fast backbone provider. That's not all that easy to determine though, because it's common for ISPs to be multihomed and to use various alternative paths.

I think that these words which Philip has just written in his weblog are going to be repeated in many a forum thread. They're certainly appropriate here:
Good news.

SL will never feel like an FPS in terms of speed unless and until we are free to setup our own server, like we can do with FPS.
I usually get a 250ms ping to SL. I have no control over it. If people could run their own servers I could just plug into a server in Lisbon, or even at my own ISP, and experience the same laglessness I get when playing an FPS.

The thing about using the phrase "like an FPS" rather than using actual numbers to quantify responsiveness and framerate is that it's nice and subjective and that down the road after the developers stop shaking their heads and muttering, "Philip, what have you promised them now?", it means that they can say, "Oh, yes... well, in this case, it feels like the FPS is Doom 3 and the CPU is a Pentium 2."

I think Philip was more speaking of the rendering speed than network response... they've been talking about doing a major rewrite of the rendering engine for a while, and hopefully this is part of the deal. As to the responsiveness side of "feeling like an FPS", some good client-side prediction could really help with that -- and I don't mean just "continue moving the avatar in roughly the same direction", either. I'm talking about client-side physics and executing scripted object repositioning on the client as well.

Say I have a simple door script. This is a relatively basic behavior, yet the system of repositioning, scaling or rotating prims on the server, then sending those updates to the client is more cumbersome than need be. Imagine if your client could already stream that intended behavior, then when the server sent the "open" signal, the client would perform the appropriate sequence of object animations and sounds.

Now imagine if instead of a simple turning door, it was an elaborate gears-and-hydraulics affair. Rather than update the position and rotation of two or three dozen prims, the actual data sent could be much smaller.

Currently, a phantom prim that both rotates and quickly bobs up and down requires many updates every second. Why? What does it affect? If the timing needs to be synchronized on the server, why couldn't we have whatever it's supposed to be obscuring/revealing/coming out of/etc. on the client as well?

As it stands, I think the biggest barrier to getting these sorts of features implemented -- aside from manpower, that is -- is that the interface for dealing with such a system could be a little complicated. Still, I feel that this would really increase the level of immersion we get in SL. Afterwards, the whole business of touching things in reality and having them move instantly is really jarring!

Cat, in SL, ping time gets in the way of your FPS and vice versa, since comms and rendering are on the same thread. For instance, no matter how fast they make LSL I can't possibly get my game to redraw its screen or even send a simple text message faster than the server can send the changes to me. So there will always be latency unless and until we get some servers on this side of the pond. Heck, I would settle for some servers on the east coast. It might help a little.

well considering i'm one fo the people who sort of spawned this argument to begin with heres the *fact* of what happened two fridays ago:

i'm going to not use the simfps numbers because people get their meanings confused and fail to recognize its an inverse aka 1/n statistic.. instead i will use an actual direct numerical gauge of sim performance, the amount of time it takes processing each 'run' of all the scripts in the sim (the runtasks timeslice)

since the spontaneous 'software problem' that caused SL to be down for several hours one day lusk had been running *fast* much faster than we had ever seen it.. sub 0.5ms runtasks. (it had never been below 1.0 before in the year i've lived here).

Now i noticed that *NOTHING* in lusk had changed content wise, but lusk was now running on a 450 'block' sim (i think it was sim456.agni.lindenlab.com) so those new sims are *fast*.

Now we had a hiccup with a found crash bug by accident that friday evening and the sim went down.

when it came back up... it was exhibiting 4.5ms runtasks, constantly... the overall sim was slowed significantly... very noticably even outside the numbers. a quick glance around the sim showed that *NOTHING* in the sim itself had changed at all.. there were only three of us in sim, and we had all been there since before the crash, when the same content was being run 9x faster.

I looked at the DNS of the sim, and it turns out that it was now no longer running on a 450 block server, it was a ZERO block server (sim16 to be specific). These are the oldest sims in SL and have *NEVER* been replaced, they're still out there and still running on the same hardware they were runing on potentially up to 2 or more years ago now.

Now in the process of verifying the repeatability of the crash bug, the sim went down again and this time it came back at roughly 1ms runtasks. or about half as slow as it had been before that evening, but more than 4x as fast as it had been just before that second crash. Yup you guessed it, no content changes what-so-ever.. and now the dns showed it was a '150 block' server, sim163 to be precise.


So to sum up:

if you rate the ORIGINAL simulators that SL was built upon back in early beta days, which are most definately still running... we have these as base points all with the *EXACT SAME CONTENT* and the same three people in-sim and all within a total of 15 minutes of eachother:

sim16: 4.5ms runtasks
sim163: 1.0ms runtasks
sim450: 0.5ms runtasks

what you get is that the oldest block of sims *CURRENTLY ALIVE* (whether it is the actual starting pre-beta hardwate that has *NEVER* been replaced, or not) is now *NINE TIMES* slower than the latest block of hardware that has been brought up in the last few weeks.

If Philip thinks thats 'not a significant hardware difference' i would be more than happy to trade LL an originalgeforce256 i have laying around for one of their nv40 class developer cards, after all, thats not a significant hardware difference.

thats also about a 9x performance delta there and im sure it would be hardly noticable in SL

(not to lindens readin this: i'm not bein mean i'm jus tryin ta goad ya inta admittin ya kinda got caught there in abit of a mis-statement and prolly need to not just keep adding new machines, but remove the oldest of them from the general pool, to simpler, more dedicated use, say as void sims etc)

Putting a zone server closer to where one lives provides no benefit whatsoever if one is at an event that is not hosted on that server. Using a local server for one's own home zone is particularly pointless unless one wants an advantage over visitors, since the bulk of one's customers will see no benefit.

The only kind of regionalization that could help regardless of where one travels in the SL world is regionalization of cache servers, for caching long-lived, rarely-changing data irrespective of which zone server it comes from. That would benefiit both the grid servers through reducing their loading, and also the cache users who will see lower RTTs for the cached content.

I dont care about visitors or unchanging content, I want to be able to build without my prims snapping back and forth like crazy like they do when it lags. I want to be able to open scripts instantly and save them instantly and have a responsive IDE. Heck, sometimes my scripts dont open at all.

Hmmm, something like public sandboxes hosted around the world would do the trick... you could do your major work there and then bring it back into SL's CaLAGGYfornia-hosted main grid

As to your suggestion of around-the-world caching, Morgaine, that's basically "Akamaizing" SL's main grid Since it's streaming anyway... it would work very well

As discussed, two points:

1. client side physics simulation (beyond the second order prediction we currently do) will help reduce apparent lag, and eventually we will probably do it. But such simulation is quite CPU intensive in a prim-intensive environment like SL, and so initially we've opted to focus on streaming and rendering as 100% of the client CPU. There.com does an OK attempt at client physics simulation, taking advantage of the much simpler environment to allow it.

2. really fast interactions (for example driving two user-driven vehicles into each other and getting a realistic collision) will only be made perfect by having local servers, and even in that case will require that the two drivers be roughly in the same geographic locale. As stated in this thread, the problem is the network delay. Hence our architecting SL to ultimately allow local servers as we grow. Akamai-style caching will help with opening a script or other static objects, but will not enhance interaction - the authoritative server needs to be network-close to you.

As to server performance - yes it is true that we have different machines on the grid, and the higher numbers are I believe generally faster than the lower numbers. We always buy the fastest CPU's available at the time, and we've been doing so for 1.5 years so there has been some change. But the difference is less than 2x (I can get some further data and post it for you) - I think the earlier servers were like 1.6Ghz and now we buy 2.8Ghz or something (but don't quote me on that - I will check). This does not create as large a difference as suggested in this thread - something else must have been different or wrong.

well thats potentially true for raw cpu mhz but that doesn't necessarily correlate to actual server performance.

Memory speed im sure plays a large part, as well as cache size, front side bus speed, northbridge improvements etc. a larger l2 cache specifically could offer the latest hardware significant gains with the total processing time taken by scripts etc...

it is very possible to have two computers side by side with no one component on the faster outspeccing the lower by more than 2x and get a sort of 'worst case' situation where the slower machine is easily 4x or more slower just because all of the weaknesses combine and all take their share of performance as compared to the new hardware. *especially* when comparing server style performance as opposed to a simple grunt cpu benchmark

What i said still stands.. the performance delta within the same exact 'sim with the same exact content was nearly 10x. The three measurements i posted were taken on the same sim, with *zero* content differences, within 10 minutes of eachother.

if you think that theres something wrong by all means please take a sampling of three sims, and an archive of lusk, run it on sim16, sim 160 somethin, and sim 450 somethin... see what happens. I can only say what i saw directly happen after two consecutive sim 'swapouts'

Well, if you'd like to prove me (and the others who have documented the differences on multiple sims using multiple servers over the past few months), post the pictures. I swear, my money's good!

You are the only person who can take three identical copies of a sim and run them on three different servers of your choosing. If you are truely confident that it's "something else" and not the box, then prove it. Show me.

Here. Here's a few more pictures. That new Alt-1 is handy!

I'm not asking a lot. It's a simple thing that can go two ways.

Either you prove that your words are truely accurate, and that performance is equal in all sims regardless of age, and it's the content that creates problems, and I pay you L$1000, or you freely admit that you misspoke when you said that performance was equivilent, and you promise to start looking into ways of solving the problem. It's not a crime to speak incorrectly, and it's not a crime to admit that something is wrong. It is a crime to ignore well documented evidence to the contrary and declare the problem to be associated with some ephemeral goblin that can't be pinned down or proven in any way.

I honestly don't care what the CPU speeds are of the boxes. It's performance that I care about. When I see that older sims are doing worse than newer sims, it's obvious that something is wrong. Just because the numbers on the chips may be "equal" somehow doesn't make the end result equal.

I'm not trying to be hostile here. I'm not trying to say that you're evil, or you should spend all that money replacing sims. Hell, if you can find a way to solve this problem without replacing or doing away with boxes, I'll be all for it. Hell, if you can find a way of even just alleviating the problem a bit, I'll be happy. But don't just ignore it. It's happening in more than one sim, and it's a consistant problem across the board. That means it's a problem for every user.