Second, for secure handshakes, stay with standardized norms, people who don't share their algorithms are typically using silly xor ciphers that are easily broken the minute a spot of zero's is hit, in the industry we tend to call this type of stuff 'encraption'. You should look at how this is done 'in real life', namely look at public key encryption, challenge response or even symmetric encryption.
Typically, maybe, but that's a stereotype. I don't share my algorithms, and I assure you they use as robust a method as possible. I'm in the "industry" as well, and I know a good algorithm from a bad one, too.

As for "real" encryption technologies in SL; not practical. LSL is SO SLOW that it takes seconds to minutes to do any kind of basic "high grade" encryption. I wrote an XTEA implementation and, as fast as I could make it, encrypting/decrypting 100-character strings took from 8-16 seconds each. Not really good for heavy-duty use.
Silly things like negative channels are easily brute forced, (for (i = (2^31-1)*-1; i < (2^31-1); i++) { llSay(i, "abracadabra"
; }, and you shouldn't depend on those types of things for security.
; }, and you shouldn't depend on those types of things for security.Actually, I think you might want to learn a bit about how LSL works before you say "easily" anything with regards to brute-forcing a channel. Random channels are actually "fairly" secure, if used on a temporary, one-time basis for each communication.
Like I said, stick with publicly known standards, and if your security depends on no one knowing how it operates, you're not secure. Bruce Scheiner has a bunch of source code for various crypto implementations on his website, use that as a reference to code a known algorithm.
Security through obscurity actually gets a lot less credit than it deserves. The real axiom to use is "use as much security as you need to make it cost the attacker more to break than what the security mechanism is protecting". In many cases, especially in SL, obscurity is "good enough" for many applications. No, obscurity is not anywhere near as good as a real algorithm, but considering you are probably protecting a few cents' worth of information with it, and it takes dollars' worth of effort to break it, why worry about it? Many times, having "obscurity" heaped on top of industrial-strength cryptography makes it an order of magnitude more difficult to break. Attackers first have to figure out what the heck you are using and how you are using it, THEN they can attack it.
Even using a symmetric cipher would probably be sufficiently strong, I'd probably update the keys on each update and provide a means for you to override and upload a new key if anything goes wrong.
Good advice for a RL product, built on a real platform with a real programming language. It's not practical or feasible to implement strong cryptography in LSL within SL. It's just too limited and too slow. Maybe if the Lindens gave us hooks into OpenSSL or something, that would make it much better, but don't hold your breath on that one.