Mass communication relay system.

Ok. The discussion of the encryption setup was enlightening. Now I have a new idea to discuss...

Actually... It's a part of hte old idea, but who's counting...


While I already have a relay system that will work on a limited basis, I am trying to build something that scales properly, while still allowing communication to work across SL.

Let me explain the concepts I have and how this would have to work.


-I was wanting, if possible, for all of the system to fit in and be run by SL. No outside servers. That would be cheating.
-It would need to efficiently deal with hundreds of sims, or just one...
- It would have to allow moving devices (probably worn, but vehicles are not impossible) to communicate even when crossing sim boundries.


Now, the method I have is a system of 9 nodes that cover a sim. they are about 80-90 meters apart, and shout dat to each other.

When these nodes cover multiple sims, data travles it in a wave. Feedback is normally prevented by the nodes keeping track of the last 50 or so message headers and ignoring them. If it's heard it once, it does not repeat it again.

Now, the problems are these:

The data being relayed grows exponentialy. Every new sim makes the situation worse. Data will litterally overflow the feedback protection by having too many messages to keep track of. System collapse.

- There has to be a way to limit the spread of a message to limit the bandwidth overload.

If I try to segment the network with bridge nodes, then the bridges eventually collapse under the load if the bottleneck communcation and fail to let all of the data through that needs to get through.

I've tried dozens of ways, but it always seems to come down to too much communication or too little. Any ideas?

This is more of an engieneering problem that a programming. If I can jsut get a method that works, everything else will fall into place. The main limitations of SL are this:

- SL has limited memory. There is a limit to how many past messages it can keep track of.

So, the nodes need to be dumb, but designed so that messages can travel intelligently.

Any thoughts or questions?

Just cheat, trust me and others whom played with this idea in the past - before email and rpc - just cheat and use a backend server and save yourself (and some sims) some grief.

Best tool for the job isn't cheating.

<grins> Your point is made, but I want to see if it's possible to DO.

This is a design question: Can it be done in an efficent way? Try not to say know, think of ways it CAN be done.

Excercise your mind!

Like I said, I can get it to work on a small scale, 9-25 sims. I was just curious if someone had come up with a way to relay data between those groups of sims effectivly. If you move all the processing of SL, you've just chosen to eat up bandwidth and use a CPU with 'unlimited' resources. I want to see if it can be done inside of SL without crashing, or even heaviliy lagging, a sim.

If it can't be done, then fine! But picture the uses of something like this overall in the sims!

What about using email inside SL ? Use one concentrator per sim, receiving emails from objects inside this sim. Each moving object needs to know the key of the concentrator, either by asking it to the previous concentrator or by using a sensor, after moving into another sim.

I think I just figured out a way to scale it for communication growth.

You have the following parts:

Relay nodes: Just dumb relay systems to take data and move it along with shouts. 9 of these poer sim for total coverage.

Master Nodes: Watch the other nodes and make sure they are communicating properly. If not, kill the relay node and rez a new one. One of these per sim.

CPU node: One of these per region. it's design is to make Bridge Nodes like this:

You have 10 regions for a network. The CPU knows of them (let's not worry about how yet) and rezzes one bridge node for each other region. The CPUs also communicate to each other the names of each bridge node and connect it to a single bridge node in another sim. Not that hard a feat, just send the key list of the bridges to all the other CPUS.

These 10 nodes then listen to data-packet traffic and relay information to thier mate in the other regions using e-mail.

Now, here is the scaling part: When traffic load get's to a point that there is worry that the bridges can't handle it, the bridges rez slave bridge. This slave bridge get's relayed all the information that the master bridge can't handle, and IT relays to a slave rezzed on the other sims as well.

When the additional slaves are no longer needed to keep up with the added traffic, then they kill themselves to not hog resources.

All we would need is a way to know if a message was NEEDING to travel across the bridges, or if local traffic only was needed. Perhaps the sender has an escalating system? Send once locally, and if no reply, send by bridge?

Email a primary distribution node in a sim that relays emails to sim subnodes. Subnodes shout the information to mobile devices on a consistent outgoing channel. Mobile devices communicate by shouting back. The nearest subnode receives and sends an email to the sim distribution node which relays it both to other sim sub nodes and to other sim distribution nodes (who distribute to subnodes).

This would eliminate moving objects requiring keys. Email delay would probably be an issue tho.

Interesting exercise. I suspect there's lots of old threads on the topic. The pre-email one's are probably interesting.

[edit: Missed above comment. Sounds similar to my thought but the slave bridge idea is interesting.]

It's the only way to prevent e-mail overflow issues due to 500 people deciding to camp in 10 sims and yell "Boo" at the same time.

<Brrrr>

And I have news for you, I'm not sure if ANY communication system would fail to crash under THAT load... Heck. That could crash SL...

Hey, is their a sim feature to read the number of avatars in a Sim? Like from the Sim Stats screen?

I was thinking about this too. I think your response might work, but I suspect it will collapse under enough demand too.

I'm groping for a half remembered and probably half misunderstood convesation - but you're really creating LANs that then communicate across a WAN in a more limited sense. I'd think it might be worth looking at some of those sorts of documents to see how they make them make semi-smart decisions about handing data on and then seeing if you can do that within the restrictions of LSL.

It might be beneficial to understand some of the IP protocol to implement something like this.

I don't know if this helps but I just created a script to relay information down from a ship to a telepad. The pad is on the ground and rezzes a "scout" which shoots up in the air while shouting on a specific channel. When the ship picks up that shout and the request message sent by the scout it then shouts back it's height. The scout then shoots right back down to the telepad and relays the height of the ship so that it knows how high to go. If it goes past 600m without a response it comes back down saying that there was no message from the ship (the ship left). You could do a relay similar to that. Prims moving along set paths back and forth carrying messages. Just a thought.

It sounds like you're trying to create an IP network. Hence, I would look to the structure of the Internet to solve your problem. Eg: routers, switches and/or hubs, subnets, etc. It also sounds, from the overflow problem you mentioned, that a time-to-live (TTL) flag could be useful in your data packets.

And I think you need an addressing system (eg: ip address). Even a "dns" system to map friendly names. I had been thinking about a "dns" system for another idea... you could map "easy names" such as "mystore.ahern.sl" to a set of coordinates, and a teleportation system could be setup around this.

Come up with an addressing system with subnets. For starters, one sim = subnet. So, 1.255 is some object in sim 1, and 2.127 is an object in sim 2.

Next you need "routers" to relay between subnets.

Anything in the local subnet is "broadcast" and heard by all, similar to the way a hub works. The only difference here is a the limit on shout distance, so your 9 nodes are collectively acting as one hub.

If there is alot of traffic, then yes, your "routers" will get overloaded due to the nature of your communication medium. Your solution of "slave" routers seems reasonable. But even with that, I suspect you will need to develop some sort of buffering system. Email was suggested. Perhaps read/write to a notecard inside the object? Not sure if that's feasible.


However... this setup is ideally suited for relaying communication between two entities. I suspect your original intent was to be able to broadcast to everyone on the network? This is doable... I recommend supporting two datapacket types, one for broadcast and one for regular (analagous to UDP vs. TCP).

Then of course you'll need a standard format for the data packets, so other developers can build network devices for this new network.

Yeah, pretty much re-creating the internet. Very fun! Need help?

I was thinking some more about your "buffer problems". So I've found out you can't write to notecards (I'm new, so sue me). How about this for an idea... Since each script is limited to 16K of memory (hard to believe you would need more, but), what if you had multiple "memory module" scripts, connected in some way. For example, each module script would consist of little more than a list variable to store information. A "master" script would then be created that would handle reading from and writing (via linkmessage likely) to the "attached" module scripts. Altogether, they form a modular memory unit of expandable size.

Your router might be constructed with an inbound script and outbound script. The inbound script receives data from the network, and immediately writes to the master memory script. The outbound script, at it's leisure, reads from the memory script and handles routing to whereever it needs to go.

Just a thought really.

Pretty neat. The Internet inside of SL... may I suggest you call it SLInternet

That's what I was trying to get at. Thanks for expanding upon it, though.

Seriously though, could you manage to get land permissions to have 9 relays in every sim? I think not. What about the different continents?

llEmail is still the way to go for massive communication.

It's not a problem in this specific instance. It's for private sims.

Ahh... I see.

hrmm... why 9? I have seen some effective layouts of sim-wide security systems that use 1 control/hub and 4 slave systems that cover the entire sim with 96 meter sensors. Since the shout range is actually 100m.. this would work perfectly, wouldn't it?

Anyway.. with that said.. why not use the same method.. have 1 control object that sends and receives Emails, and then it shouts out the message to the 4 slave objects. It sounds a lot more efficient than using a 100% llShout() system. And with a bit of tweaking, you could even create your own queuing system for high-demand communicative simulators.

-shrugs-

Relay systems bring back some memories. Before 1.3 there was no long range communication in SL. llShout was the only way to go. Some people, myself included, had to create massive llShout relay systems for something as simple as making a sensor sweep of a whole sim rather than just the 96m sensor range. Christopher Omega and James Miller worked on some crazy telephone network project, like the grandfather of Nexcom or something.
Like others said, I wouldnt touch a relay system with a 100m pole. Just use email.

I'm going to create giant spools of copper wire for y'all. The wire will be tiny but you can use it to connect any objects together at any distance (provided you have enough). It will carry linkmessages from object to object... of course converting them to llwhispers at the endpoints and at the "max distance for a single linked object" limitation points, where the wire will have to be connected to the next segment of itself via a "magnetism" script.

It will be configurable to carry raw data streams, encrypted packets, or your mom. Ok, I may not actually get around to it, but maybe a generic connectivity mechanism like this could help... actual wire may or may not be the winner, but you get the idea.

A single object (or multiple objects acting as a single entity as in the massive span of wire example) with two endpoints (logical endpoints mind you) that carrys any data streams from point a to point b. If the receiving item is too busy at the moment to handle another message, the sending item is sent a busy signal of sorts via the "wire", and is responsible for re-sending the message at configurable intervals until successful. A middle man can be thrown in if needed to marshal messages back and forth and hold a queue (it will now be responsible for re-sending instead of the first object). So the message publisher can send messages asychronous to the actual consumption of them at the other end. Much like MSMQ works...

Make each component completely generic and independent of the rest of the system though. So you've got the "wire", then the "transmitter" and "receiver", possibly an "encryption/decryption box", and a "hub" or "queue". All of which will work with or without any of the other components. All "object a" and "object b" care about is outputting a linkmessage to the transmitter object and/or acting on linkmessages from the receiver object. (linking the transmitter and receiver is probably best so linkmessages can be used at the endpoints, therefore no worry about encryption for the "users".

If you experience trouble due to traffic you'll have to throw in more middlemen. This will be an extremely easy step though since each transmitter (including the queues) automatically re-sends if a busy signal is received, and deletes the message if successful. So "queue 1" is constantly filling up (that node is your bottleneck), meaning the sender objects are always bogged down resending messages to it. Just put another queue object between each sender and the original queue. MAYBE, if you're really good at scripting, each queue object will be able to sense this state and rez another one which will auto-config itself into the network, maybe only temporarily while it's needed. That method will increase the total message limit of each node, and the other option would be to throw in another queue connected directly from object a to object b, so now you've got 2 queues running in parallel on the same node thus increasing total throughput. In that case, the transmitter, upon receiving a busy signal from the first queue, will re-send to the 2nd queue...

So there you have it. The engineering theory is done, now all that's left is implementation.

Using this generic componentized sytem you should always be able to reconfigure the network to handle more load (using sound network analysis, finding the bottlenecks etc..) simply by adding more "hardware". And, just like in real life, you'll have to learn to deal with latency, but the sytem should never truely "collapse" if built (and maintained) correctly.

EDIT: Once this "wired" approach is mastered it should be a simple thing to add "wireless" devices into the mix. If your wireless transmitter is not in range of the network it will report back a network not found message like cellphones do.

Apply Morgan's thoughts for subnet addressing to the above infrastructure and you now have your generic "protocol". Add a dns server and a router, oila, SLInternet!

Ok, not to pop the bubble or anything... this is certainly a fun mental exercise, but if such a network were actually created, what are some possible uses, aside from being a fun experiment (which is enough reason for me)?

One seeming advantage of this system is relay speed. Email incurs a 20 second delay; unacceptable. XML-RPC is 3 seconds; also unacceptable. This system would use a combination of linkmessage and shout/whisper, which should be much faster. However, with greater distance, say, across a continent, I suspect the propogation delay would match or be greater than email or xml-rpc, thereby diminishing any speed advantage. If that turned out to be the case, then this system would only be useful in a relatively local area; say a few to more than a few adjacent sims in size. This limit isn't so bad, if the intended use is for a multi-sim wide game, for example (and it was mentioned this is for a private sim). Of course, we wouldn't know the prop delay until this system were actually built.

To overcome the prop delay upper limit, a hybrid network could be constructed that would use both "fast" messages (link and shout) and "slow" xml-rpc/email. A routing algorithm (if using a subnet system) could determine probable delay, and decide to send either as xmlprc/email or shout/link. This would allow for a system wide network that would only be slow at extreme distances...not unlike the real Internet.

Now again I ask, what types of feasible, practical projects would benefit from this system vs. current methods? Here are some of my thoughts, but I'm wondering what others think...

Perhaps a near real-time network game, where participating avatars aren't in the same spot. A chat relay/meeting broadcast system. Mass information dissemination; eg. quickly updating products in multiple stores, etc.

What else?

Why not just use IM?

Thats why I never understood why Nexcom took off.

1. A massive sensor system (possibly used to track player locations for a game or track locations of objects like GPS)

2. If it were a generic implementation and spread around the world (getting landowners to allow you to build up the infrastructure, again like cellphone networks), you could sell the transmitter/receiver units to anyone wanting to build systems reliant on inter-object communication at large distances, i.e. vendors, banks, etc... so commerce i guess is answer #2. You could even sell access to the network on a subscription basis. LOL, that would be cool to see "SLInterent" service providers pop up and compete.

3. Any remote control / monitoring system

4. SLInternet-enabled objects or display boards around the grid could be updated remotely. Say you have... oh i don't know, 10 or so Tringo boards around the world and you want to host a giant game where everyone sees the same pieces and the scoreboard shows the top scores for ALL boards (essentially one big Tringo game with a massive payout). If they were SLInternet-enabled this would be pretty possible in real-time, whereas it's not really doable with email. I know Tringo is unpopular in this clique but it's a decent example.

There's gotta be a million more uses, and I'm sure more would become apparent if something like that ever actually got built and iteratively improved.

Jamie -- we're talking about inter-object communications via scripts here. Objects can send, but cannot receive IMs.