A Cluster Administrators Two-Cents on Lag

Alright...Don't let that title fool you because I am by no means a professional. I am a network admin and also a freelance developer for all sorts.

I digress.

I have only been playing in SL for a couple of days now and it's really amazing, but the main problem I notice is when people get together in large numbers, and natrually there are all kinds of custom models out there but in general we are running into overloading problems, there is just not enough computing power to keep the FPS floating at a level everyone can enjoy.

I have seen topics on this board about dynamic computing, using grouped clusters you could create a really flexable system that could point processing power where it is needed. That way you could host events like wars, nascar, sports, massive parties, etc. without having to worry about everything slowing to a crawl. I really liked reading page after page on technology like this because it just might be the future of this virtual world.

My idea...well, do any of you remember SETI? For those of you still running it you will know what I am talking about. Would something like this be possible for SL where packets of data are sent to be processed and sent back, this for stability reasons would have to be an official thing needing confirmation from LL labs before activation or something like that but if you could have third party clients processing data, maybe even for their own islands with processing power they own in their own home to help bring up FPS.

I might be getting ahead of myself here, but maybe you can see what I mean? Personally I would not have a problem at all setting a box up to always processing SL data, I might even be able to sneak a couple extra boxes into work too *sneaky* On the bigger side, if you have a organization/group of people that wanted to have seriously like their own cluster working for SL servers you could really help keep lag down.

Your thoughts?

I don't think it'd work: the latency between computers is too high.

I could imagine some interesting things you could do using fractally generated land with computers handelling their own plots though.

Given people's concern with security, indicating you would sneak around behind your employer's back to place machines or divert existing machine's time to run SL is probably not the best way to make people want to share part of the SL processing with you.

Who knows what other kind of sneaking around you might try?

There are really only three things that subdeviding and distribution can help with. Physics, Scripting, Assets. The unit for the devision for scripting and physics by sim (mulitple sims are run on one server). Users experience lag in 3 ways: Slow movement, Slow scripts, Low client framerates. Framerate and slow movement are tied together; when framerates are low movement is difficult. There are three major generators of server Lag, textures, scripts, agents. Sending assets to the agents cause lag, in areas with many high detail textures the cpu time spent serving assets becomes substantial. Scripts slowing a sim is now very unlikely as they have implemented a (name slips my mind atm); it's major effect is that scripts run slower and some function calls get lost (which totaly sucks). Agents cause lag in a number of ways, the avatars have attachments, these can have many scripts, the sim has to import these objects, and associated assets into the sim when the user enters (this is why sim crossings can be ugly in vehicals). The physics engine has to take avatars into account adding extra load. Physical objects add load as well (someone should make a nice table of all this).

Sending assets could be handled by another server seperate from the sim all together.

Scripts that don't depend on physics or sensors could be handled by a different server altogether; this is along the lines of virtual memory.

Physics could be subdevided.

Swarming assets is complicated and unlikely to be effciant unless the entire grid were in a single swarm and cache sizes were around 8+ GB.

I like the idea of having scripts on a separate server (or rather group of servers) as a lot of their time is spent doing things with themselves like changing variables until some output is required.

SuezanneC Baskerville assumes that I am a criminal, but thank you for the input on the subject anyway. Heh. If you don't think people sneak around behind their employer then you must have just been born yesterday...

I figured I would just ask, thank you for those who did reply. It looks like you would have even more problems if you let other people serve content. But what about the idea of decreasing lag in a certain area?

Like if I hang out in one area more so than any other I might want to become a "permanent member" which would cost some money, but this would be a donation/contribution to the area in general. So if me and the hundreds of other people in that area become "residents" for so long that will allow LL Labs to rate areas by how many people call it home. You could maybe give out processing power based on that? I can tell already where this would have problems, because people do roam around, I am just trying to think of something that would not involve changing everything.

Hmm, well I think the main reason a distributed processing model like SETI@Home won't work is that time isn't a big issue as such. SETI models are small chunks of data that require a lot of work to process. However SL is primarily a load of small packets that are needed immediately, so the time taken sending between the server and clients would actually increase lag. It's a nice idea but that wouldn't work.

However, you may have spurred a similar idea. Namely:
Peer to peer content!

This would be strictly things like textures, sounds and animations.

Thinking from the client app's perspective:
- Okay I've entered a new sim, I'll request the data!
- Hey! There are 20 avatars already here and they've already cached everything!
- I know! I'll request bits of it from each of them depending on how much spare bandwidth each one has!

So basically, each sim would keep tabs on how much content each av has cached, and a note of how fast their upload speed appears to be. Now when a new av teleports in, the avs who have already cached stuff will be asked to send chunks of their data to this new person. Thus taking load off the sim and speeding up loading times.
I feel that they still would need a much more improved caching system anyway (and would likely need to for the p2p content idea to work without opening security holes), but it could be a way to drastically improve load times. As my internet connection is pretty good, it's 1mb so not the best, but in a quiet sim I rez very quickly, in a laggy sim however it can take MUCH longer.

I think that while distributed computing has its merits (and as I say, having scripts run on an individual server cluster separate from the sims could work for non-sensor or similar scripts), another big way to reduce lag is to replace scripting with a language that heavily optimises it's code and uses sensible data types like arrays instead of lists, and booleans instead of an entire 32-bit integer for flags.
This would not only vastly improve scripting by allowing more info to be held in the 16k chunk, it would also allow scripts to run faster while maintaining FPS.

It's then things like more efficient physics/collision detection that will also improve speed. So really the improvements need to be more to the sims themselves than outwith. Although the idea is nice I don't think SL would benefit greatly.

I am really liking this idea, this fits along the same lines but is actually just a better idea because it removes the need to send the data again. This would allow for that "dynamic" processing power but more in the sense that if alot of people are in an area (watching a race, a match, fight, performance, etc.) they all pull from each other to keep things running smoothly. You would NOT want to pull from a slow person, you would want to give to them, you could base this system of a AV's ping?

The only thing you would want to look out for it pulling to much from one person, but I don't think that would be a problem especially if we are talking about small pieces (which you get into data consistancy, corruption problems...need some MD5 checksum to make sure there was no modification from the end-user, but these are CPU intensive.) One more thing, at the same time as you pull data from other users around you (assuming there was content control) it would make the finding of malicious users easier, even in a crowd because if he is sending bad data to everyone around him you no longer have to wait until he crashed the grid to know there is a hacker.