Are simulators single-threaded?

Reading here: http://secondlife.com/tiki/tiki-index.php?page=How+do+I+read+the+sim+performance+stats%3F and having watched simulator stats for the past hour in Wetheral, I'm starting to get the sense that the simulator process is single-threaded. In the sense that when simulators are sending image data or doing any particular thing like that, the time spent doing this counts aginst the total timeslices allocated for simulation. This seems to cause script delay and time dilation to start falling towards zero. It would appear to me that because of the increased load and therefor time spent on pulling image and texture data from asset database is causing the simulator process to become in effect, I/O blocked.

How close am I here to reality?

Speaking now from my experience as a unix sysadmin and Pc user (including Linux). multi threading is strictly on a cpu usage level.

I/O's have a single stack per device, and are carried out on a fifo system.

In other words, if a process is saturating the network card on a server, all processes on the server will wait in line to get access to the network card.

The same applies to any other device, like physical disks. By striping a sets of disks, a degree of "parallel processing" can be achieved.

By bundeling network interfaces, the same.

But the principle is basically the same; to get to the physical resource (of which there is one), all processes wait in line, even the "bundeling" or "virtual disk" software.

Anyone, please correct me if my view of thigs is too simplistic, or fallacious.

Your view is correct...

In addition I need to comment on an often fallacious misunderstanding about multi-threading.
With multi-threading, the name is somewhat misleading.
There is one CPU per simulator...and that CPU must split its time out among the threads that it has running. Each CPU has only so many clock cycles to split among all of its current processes (Threads). The fact that a single CPU can actually RUN muliple processes is what multi-threading is about...Once upon a time, it was not so; it was, "One Central Processing Unit, One Process." All hail multithreading, where one CPU is capable of running the calculations for more than one process at a time. But at the expense of speed. (There are emergent conditions that make this a more than value appropriate trade-off... do YOU like using a GUI? N-Tiered architecture? The INTERNET as we know it today?)

Multiple stacks for processes to run in...
But they all still have to run through the same toruous little maze of silicon, and there is only one path at a time in there. (co-processors notwithstanding). So even though the Simulators aren't single threaded (They couldn't be), there is only one thread being worked on at any given time...
The rest are waiting their turn in line to have the next step processed.

Yup you're correct That's what DEvice MONitors do, they regulate access to unique device for multiple processes.

The thing is, when a program hits a "block" waiting for a device to be available for use, this program is flagged as waiting, and another non-waiting program then starts running in its place. This is how most modern OSes work. So when you cut your program in multiple parts, one part waiting for the disk won't block other parts from running in the meantime. Each part is a thread of the same program, that's what multithreading is about. It's not to be confused with multi-tasking, which is about running a bit of each program after the other bit of another program.

One thing though: some OSes make a distinction between programs and the threads of a program, and only allow a given program to run on a single CPU at a time, so that none of its threads may run on different CPUs at the same time. LL might be using a similar mechanism to ensure that one simulator running on a double- or quad- stacked server doesn't hobble all the CPUs for the other simulators sharing the server. AFAIK I have never seen a single sim using its "buddy sim"'s CPU (though they certainly exchange CPUs sometimes).

(wew this post sounds academic )

You are correct... Perhaps I should have made the distinction.

Yes, multitasking is when the different stacks are for different programs, and therefor there are also different heaps.
Mutithreading is when multiple stacks are allocated to a single program, and they all share the same heap... (At least in Java and C++ on Unix and Windows... I dunno about ALL languages and OSes, just the ones I work in.)

Exactly

When we switched one database server version for another at work, the new version had up to 250 threads running per "Program", of which there were 2 running (database engines). The Operating System, Sun Solaris 8, allowed us to "bind" each program to a cpu (we had 4), which we did not do; as a result, under peak load the two programs used more that half the CPU power, i.e more than 2 cpu's for 2 programs, although of course each program itself ran only on one cpu at a time. Unless the Sim Programs are bound to a cpu, I would think Linux COULD use more that 1 cpu per sim, theoretically. The resource conflict, if any, is more likely to be elsewhere though, like the network card or the associated TCP/IP stack, or kernel locks, since it is extremely unlikely that ALL CPU's would be at 100% at one time, and all idle time is available for all processes on the computer.

However, in another forum thread Lee Linden has said that resource conflicts between Sims sharing a computer has in test never been found to be the case. So a) No multithreading across CPU's and b) the "single point resources", like network cards and kernel locks, have enough reserves when running several Sims so that there is no issue with blocked, locked or waited processes because another Sim is overusing a "single point resource". These resource problems are extremely easy to see in Linux and the Unix like operating systems, and would not fall through either testing or normal operations.

Sorry, a bit of a ramble off the original topic, but still directly on multithreading and how it works

Addendum: as mentioned above, if a process (consisting of multiple threads) is waiting for a resource, regardless of the number of threads it used up to then, the process, and all the threads that depend on this resource being available, will have to wait. I see this on my client side, when I get to a new area: the textures are loaded, one at a a time, into the graphic card, from the network. And as long as the graphic card is loading textures, it is very sluggish to other changes. It is a single resource, with a very high load. In a separate "thread", so to speak, my IM chat is still fast, except to the extent my CPU is heavily loaded by the graphic card loading textures and rendering them on the screen. This makes SL appear to be single threaded - a key resource is VERY heavily loaded, so the whole program waits. Server side similar things can happen, if for example 20 new avs descend upon a heavily textured and detailed area at once. It settles down after a while, when all the clients (avs) have the stuff they need from the server. Continuous load, makeing for a slwo sim, can come from anything that the clinet continually has to talk to the server for, like scripts on channel 0, or av (also channel 0), and presumably some scripts and graphich effects. Channel 0 is an "all to all" channel - everything coming out on channel 0 (the chat channel) goes to all avs and to all scripts listening on channel 0, that means, client to server and then to all clients within hearing range.

My 10c to the original question

I hate to be a thorn, but the bolded words above (my bolding) show that this assertion of Lee's is, at best, speculative when applied to the production grid.

The production grid shares all kinds of resources that would not be at conflict in a test environment... because, in a test environment, there are not 500+ servers with 1000+ sims on them competing for tight resource clockcycles...

Inside the Multi CPU boxen (This I am most likely to trust tests of... most easily replicable)
Shared buses?
Shared hard drives?
Shared access to shared memory interfaces?
Shared access to anything else??

Among the boxen on the grid (Ttests for these I am least likely to trust, because of the inability to replicate configs and load testing in a test environment)
Shared access to Network facilities?
Shared access to the Asset Server?
How is an asset server request identified for placement in the queue?
By sim or by server?
First come first served, or by Prioritized multiqueue?

sorry, our posts crossed...

Have a look at my post above.

The forums were down, or I would have added it sooner.

But you are right - in test is not like live, the 20 avs at once issue, especially since they ALSO download data from neighbouring sims within "SL Local" range, and upload stuff to avs and scripts in neighbouring sims within range, and then engage in the "upload download" game for all channel 0 stuff and changing texture etc that can not be cached client side, makes live a situation where a server gets hit with completely different loads. I know of a plot on the next door sim to bad girls, and that plot, on a separate and basically VERY quiet Sim, is Dead Laggy cos it is within eye and earshot of Bad Girls. But I bet (I haven't checked) that the bad girls sim and this plot are on different servers. This is for me an example of mainly client side lag, though I presume the quiet sim has SOME load because it is "physically" adjacent to Bad Girls, so they probably communicate with each other.

What Lee was talking about, is that IN TEST one sim can be running at 100% (what is loaded, he does not say, maybe cpu only?) and another sim on the same server does not get any appreciable degradation. Thjis defintely sounds like a sterile "lab test", but it proves some point, I guess...