Second Life Protocol : Technical Chat

http://labs.highenergychemistry.com/slprotocol/

There is already a very nice thread going on for discussion of law and ethics, and things that make you angry right here: /54/af/105371/1.html

This thread is taking a cue from Phoenix Linden's post:



Technical discussion only please. The wiki is great for posting findings and making notes on, but discussion is better saved for IRC and this thread here. To start it off I have some very good news for the project, a mysterious entity donated two important tools to our arsenal. One of them is called snowcrash, and it is the protocol documentation the Lindens have been talking about releasing, it's just built in to SecondLife.exe. Follow the readme to generate your own protocol notes that look like this:

LOW 00001 - TestMessage - Untrusted - Zerocoded
	0074 TestBlock1 (Single / 0)
		1151 Test1 (U32 / 0)
	1401 NeighborBlock (Multiple / 4)
		1149 Test0 (U32 / 0)
		1151 Test1 (U32 / 0)
		1152 Test2 (U32 / 0)
LOW 00002 - AddCircuitCode - Trusted - Unencoded
	0786 CircuitCode (Single / 0)
		0212 AgentID (LLUUID / 0)
		0392 SessionID (LLUUID / 0)
		0549 Code (U32 / 0)
LOW 00003 - UseCircuitCode - Untrusted - Unencoded
	0786 CircuitCode (Single / 0)
		0030 ID (LLUUID / 0)
		0392 SessionID (LLUUID / 0)
		0549 Code (U32 / 0)
LOW 00004 - LogControl - Trusted - Unencoded
	0711 Options (Single / 0)
		0526 Mask (U32 / 0)
		0545 Time (BOOL / 0)
		0590 RemoteInfos (BOOL / 0)
		0925 Location (BOOL / 0)
		1186 Level (U8 / 0)
LOW 00005 - RelayLogControl - Trusted - Unencoded
	0711 Options (Single / 0)
		0526 Mask (U32 / 0)
		0545 Time (BOOL / 0)
		0590 RemoteInfos (BOOL / 0)
		0925 Location (BOOL / 0)
		1186 Level (U8 / 0)
LOW 00006 - LogMessages - Trusted - Unencoded
	0711 Options (Single / 0)
		0902 Enable (BOOL / 0)
LOW 00007 - SimulatorAssign - Trusted - Zerocoded
	0919 RegionInfo (Single / 0)
		0047 IP (IPADDR / 0)
		0242 SecPerDay (U32 / 0)
		0440 MetersPerGrid (F32 / 0)
		0593 UsecSinceStart (U64 / 0)
		0608 Port (IPPORT / 0)
		0739 SecPerYear (U32 / 0)
		0795 GridsPerEdge (S32 / 0)
		0824 Handle (U64 / 0)
		1100 SunAngVelocity (LLVector3 / 0)
		1228 SunDirection (LLVector3 / 0)
	1401 NeighborBlock (Multiple / 4)
		0047 IP (IPADDR / 0)
		0342 PublicPort (IPPORT / 0)
		0495 Name (Variable / 1)
		0608 Port (IPPORT / 0)
		0774 SimAccess (U8 / 0)
		1135 PublicIP (IPADDR / 0)

Things should get a lot more interesting very soon

One of the things we need done is parsing the message template in to wiki pages. Each packet on the wiki will have it's own page, there are 565 different packet types by my count. Instead of writing each one up manually they will follow a generic format with a notes section, and notes can be added by hand after the initial page creation. An example packet layout out of the message template is:

{ NeighborList High Trusted Unencoded { NeighborBlock Multiple 4 { IP IPADDR } { Port IPPORT } { PublicIP IPADDR } { PublicPort IPPORT } { RegionID LLUUID } { Name Variable 1 } { SimAccess U8 } } } 

This file is actually the comm.dat file that comes with Second Life after running a XOR decryption. A decrypted copy of the comm.dat from 1.9.0.21 is available here: http://labs.highenergychemistry.com/secondlife/message_template.msg

So for now, if there are any volunteers, it would be very beneficial to have a simple PHP script that parses this log file. We'll work on what the output is soon, but having the parser is the first step.

EDIT: Edited as new information became available

I see you have handled the packet loss & reordering problem in a reasonable manner; that one can be a real pain. I prefere XnView, using the LuraWave jp2k plugin it can handle incomplete & corrupt streams. Most file systems get grumpy when you dump 20k worth of files into a single directory. I went with converting the cache to a tar file. WinZip does a much nicer job handling that many files and with the 512 block size of tar, you actualy save over the caches 1024 block size.

Another project just came up. The source code to the utility that generated the documentation for us, called snowcrash, has just been uploaded. It's an MSVC.NET 2003 project using the Windows Template Library that's broken up in to an exe that is mostly a wrapper, and a dll doing the grunt work. We need someone to analyze the dll and figure out how it works; how much of the data it generates is pulled from SecondLife.exe and how much is generated on it's own. How resilient is it to new versions of SL and new protocol features? C++ experience necessary, familiarity with reading other processes memory and debugging techniques a plus.

http://labs.highenergychemistry.com/secondlife/snowcrash-source.zip

The official codebase is just short of being ready to take new developers on board. We need to get the code skeleton finished before individual devs start coloring inside the lines, but it's just a matter of Adam merging my classes with his skeleton and importing the whole thing. So here's the big roll call: if you are familiar with C++, the boost library (http://www.boost.org/), sigslot templates (http://sigslot.sf.net/), or any combination of the three we need you!

Repository: http://svn.jhurliman.org/wsvn/libsecondlife/trunk/

I have a question that may reveal just how little knowledge I have, but too bad, I'm asking it anyway.

When I click on the slice link in the protocol wiki, slice , it goes to a page that appears to display a binary file in a form that can be displayed as text encoded in base64/uue form.

I don't know what to do with this. I see references on the web to the use of various forms of encoding used in such places as binary news groups to reproduce binary files in ascii or similar text formats but nothing that gives me confidence as to how to deal with this file presented in this manner on a web page.

Sorry about that Suezanne, that was how the code was donated to us and we haven't got around to changing the page yet. It's a base64/uue encoded file, meaning you'll need to save all that garbled text to a file with a .b64 extension, use Winzip to extract a filename with a .001 extension. Rename that .001 extension to .uue and open that file up in Winzip, and you should be able to get it. Otherwise you can just download the program directly from:

http://labs.highenergychemistry.com/secondlife/slice.exe

Usage involves putting the file in an empty directory, closing Second Life, going to the command line and running slice.exe with no parameters. Please think responsibly when using the debug tools and don't infringe on the copyright of others.

Thanks for the response.

I had already found the directions on the discussion tab on the page in the protocol wiki and decoded the cache by the time I found your post.

The bvh files produced by slice are not the text files that I would have expected. How are those stored? Another XOR? are they compressed?

The ogg files are presumable the same sounds as the files in the cache that have a dsf extension, which are just wav files with the extension changed to dsf.

My assumption (haven't looked to closely at that part of the code yet) is OGG files are stored on the servers, transmitted locally and stored in the cache. The dsf files are decompressed temporary caches of raw audio (wav) that aren't in immediate use, but might be in the next parcel or sim over. So any file you look at directly from the main cache is likely very close to what the server is storing and sending the client. It's very useful for empirical analysis of compression/uploading techniques for the textures, sounds, and animations. The animations stored on the server are in a packed format similar to how ll builds packets. I might even be able to find the spec for it in comm.dat, but it's likely just a bunch of packed floats and bone names, all zerocoded (like RLE compression but only works on zeroes).

Does this mean people are already using this reverse engineering effort to try and re-capture the animations and sounds sent to the client? That didn't take long.

*Adds animations and sounds to list of things that can be stolen in SL*

slice is a tool that was created long before the Reverse Engineering team started peeking at packets, you just weren't aware of it's existence. To answer your question, it would probably be trivial to write a converter from the packed format to something Poser could import, so my guess is someone out there has done it.

I've known about the dsf files for a long time.
I mentioned them in this thread from October 2005. I imagine this is broadly known by people who know such things.

Stealing sounds is much easier actualy. Stealing animations is much more compilicated; the animation format is one that LL dreamed up. It's pretty efficiant for a non compressed format. I have writen code that can generate a BVH though really they would decode better to Poser's format; I decided Posers format was way to complicated to learn. I'll contribute an encoder; when I get around to it.

https://gna.org/projects/libsecondlife/

The project is using OpenSSH, Boost libraries and asio (http://asio.sf.net/). asio was recently accepted as a Boost library, but won't be included until the next release at least. Asio is only a set of header files so it doesn't add an extra requirement to building.

Team members: We should get everyone on the gna/libsecondlife mailing list so communication is more direct than through IRC and the forums. Anyways, I'll be out of town from today until Sunday, so no code from me for the next three to four days. I just committed what could almost be called an alpha release; the connectSim() call in SecondLife needs to call connectSim() in Network inside of a separate thread and return immediately, and some testing needs to be done to see if registerCallback()+tick() actually works as intended, and tick() needs to get a lock on the Network mutex before reading or popping off the list. Also Network needs a login() function that uses libcurl (just for now); I'll probably copy over my code from the old proof of concept for login just to get it working. Once those things are done we'll have an alpha release, so I'm shooting for Monday.

Things that could be done until then:

* Test it it compiles under MSVC6/7/7.1/8, cygwin, Intel compiler, gcc 3.x
* Write instructions on how to install OpenSSL, Boost.Build, the Boost libraries and compile on Linux/Windows/Mac
* Register with gna.org and join the libsecondlife project
* Start writing a new test_app that will register callbacks, login, connect to a sim and run a loop calling tick() that actually does something useful
* Investigate how to use C++ libraries from C# (Mono?)
* Investigate the Python bindings/wrapper

Quick status update, the alpha release isn't ready yet. All of the classes needed some new functions and the Network and Packet classes needed a lot of code added before they were usable, so it's been a long day of coding. Right now the multi-threading appears to be working, which will make debugging from this point on a lot more interesting. Most of the problem areas in the code are threading specific so I will probably wait a day or two until Adam gets a chance to take a look at it, but the alpha release is still right around the corner. Right now the code is logging in, sending the initial sim login packet, and receiving any packets the server sends back. In a narrow sense it's only one packet away from being able to login, but in a broader sense it's got at least a couple more days before being considered useful for writing bots.

Also, my proposal for Google's Summer of Code was accepted today so I will be programming full-time on the MythTV project. I'll still be working on libsecondlife but hopefully some other coders will be devoting spare time as well.