A possible solution to allow for outgoing XML-RPC calls?

Hello,

I don't know if something like this has already been suggested, so I'll go ahead and post it.

From what I can tell, the primary reason Linden Labs doesn't currently allow outgoing XML-RPC calls is to prevent using SL to spam external sites.

One possible way to avoid this would be to have a potential out-calling site register itself with SL, which would then allow scripts to make XML-RPC calls to that site. For example:

My fictitious site, neorebus.com, sends an XML-RPC request to second life, using a specific "registration" channel (e.g. 01234567-baad-c0de-d00d-0123456789ab). The IntValue of the request would be the number of seconds for which the registration is valid, possibly with a maximum (say 86400 seconds, 1 day). The StringValue of the request would be a security code which would be used to allow only authorized objects to talk to the XML-RPC server.

The registration process would record the expiration time of the registration, the security code, and the IP address from which the registration request was received.

After the site is registered, objects within SL could start sending XML-RPC requests to the site. The "channel" of the llSendRemoteData call would have to be the security code listed in the registration, and the hostname portion of the URL for "dest" would have to resolve to the same IP address that was recorded for the registration; if either of these requirements is not met, the XML-RPC request would fail [silently or otherwise].

By restricting access to sites that have already registered themselves with SL, you prevent random DoS attacks, and by including a "security code", you prevent unauthorized access to private XML-RPC servers. (public servers could use a null security code to allow general access).

This was a quick thought this morning, and so there are quite possibly security flaws in this model that I haven't considered; if you think of any, please discuss them here

- Neo

I would want the ability to set multiple "security codes," as well as the ability to rescind a previously approved "security code."

This would allow the existence of a multi-user server, that's still restricted to only specific users.

An additional thought on security: Both inbound and outbound XML-RPC should support SSL encryption.

It would also be nice to have some true cryptography capability beyond the limited MD5 hashing and Base64 encoding that LSL supports now.

Both of these could be implemented given this scheme - there's no reason why you should be limited to a single security code per IP address, nor should you be limited to a single IP address per security code [in the case of "public" services using a null security code]. I could also see that you can rescind a security code by sending in a new request with a 1 second expiration for that security code.

In my original post, I was trying to be careful to leave out "underlying" implementation details, but this could easily be done using a hash table, where the index of the table is hashed from both IP address and security code, and the data stored in the table is the expiration time of that combination.

- Neo

Bumping because the current 4+ hour delay for llEmail() is rendering SL-initiated XMLRPC communications useless.

I have built custom large-scale mail servers from scratch and I can tell you from experience that any one of a thousand tiny little misconfigurations can and will make the server grind to a halt. According to the RFC (I forget which number), email is not a real-time communications protocol, therefore delays like this are not necessarily considered a problem.

In order for this world to really move to the next level, we need a non-queued method of instantiating external communications.

I concur.

I further state that LL should ditch XMLRPC (it's a junk protocol for this - overly complex and no distinct merit.), and move to something simpler, like HTTP (using POSTs bidirectionally), or even simpler than that, some form of plain text ASCII protocol.

-Adam

They should just flip the damn switch to the on position and ban anyone who purposely uses it to spam sites.

XML-RPC already is a plain text ASCII protocol, and it already runs over HTTP, and it already uses POST. Eg. a request goes to xmlrpc.secondlife.com on TCP port 80 and looks like this on the wire:

POST /cgi-bin/xmlrpc.cgi HTTP/1.1
User-Agent: libwww-perl/5.79
Host: xmlrpc.secondlife.com
Content-Type: text/xml
Content-length: 778

<?xml version="1.0"?>
<methodCall>
    ....
</methodCall>

And far from having no merit, it's tailormade for transfer of data, just like we're doing here.

One of the driving forces behind the development of XML was addressing the content+presentation mess that was HTML and the lack of flexible data handling in POSTs, so it's hard to imagine anything more exactly suitable for encapsulating data. Going back to unstructured text in POSTs would provide no advantage, but plenty of disadvantages for the future.

The most important thing about XML-RPC is that it puts us on the first rung of the ladder for transferring large amounts of structured data to and from our SL objects. Passing one key, one string and one integer isn't using much of the power of XML currently, but the potential to do much more is there.

Here's a small example from our Offline uploads thread showing texture uploading into object or avatar inventories, which wouldn't take much effort to layer on top of our XML-RPC system, as one option. And once Don Linden completes his work on arrays for LSL, it would then be quite easy to extend the llRemoteData method to allow us to load the arrays from outside in just a single rapid XML-RPC upload. Or indeed we could start making suggestions right now for how to load up lists in a similar way.

So, calling for a step back in time is a pretty bad idea. XML-RPC is ideal in this role, and gives us huge scope for doing more sophisticated things in the future..

Would an acceptable solution also be to register an external host with LL? One could fill out a web page with user information and the IP address of the server, the web page would generate a confirmation code, and the user would have 5 minutes to send that code to an LL server from the registered IP address to enable a temporary lease on an outgoing connection. The leases could be a month long.

~Ulrika~

I would guess that most XML-RPC users in SL would greet any initiator capability from SL to the outside with open arms, even if very restrictive, on the basis that something is usually better than nothing.

If the potential for generating spam is the sticking point, registering a single destination for outgoing messages per LL account (one change a day max, say?) and not permitting traffic to anywhere else would certainly be much better than nothing. Any spam complaints from sites are then trivially dealt with, using the big ban stick as suggested.

And like many others have said, even zero outbound messaging would be a cross worth bearing if XML-RPC messaging were allowed between objects within the platform. I'd support such an initial step very happily. It can be extended to messaging the outside world later.

I’m not keen on the xml-rpc method ether, for one its not immediately compatible with POST, because xml-rpc is outdated, unless you put a lot of work and effort into POST to get it to be able to do the correct format for the old xml-rpc format.

Another reason, is that it required a web server, http protocol, then xml format, any any other darn special formatting, it requires to much dependences in order to use xml-rpc, if it were standard TCP-IP streams, it could do the exact same thing, only with no dependences, and absolutely no learning curve, and could be run on anything, without the need for a web server, SL could create an dll, which would have to proper TCP sockets, and commands, making it a simple drop in place and go.

whole heartedly agree. xml-rpc would be better than what we have now but it's too complicated, why can't I send a darn post or even get and read back the results as a nice string? that's all the data I need, no more complex than it needs to be

if they are worried about spamming though I do think some form of like domain or ip registration is dooable. have a downloadable script/page that ll has to be able to hit once a day or so at the base domain to keep it registered. ie. I register doda.com, LL has a script that checks to see if the dooda.com/sl.xml is available (IE the domain owner has given permission to use their domain). allow only one domain or ip per user. to keep the db from getting too bad.

Why have the external site send a message into SL?

I saw it suggested once that the outside site place a certain file, similar to a robots.txt file, that is checked for by SL in order to allow communication. Obviously this would have to be cached so you don't ping the site all the time.

I think a better way would be to require the xml-rpc endpoint being called to implement a method, say llAllowData, that simply returns true. The first time a method call is maded to an endpoint, it calls the llAllowData method first. If it returns true, it caches that true value for some period and from then on allows calls to that endpoint. If it returns false, it would disallow (and cache).

In order to not completely hose someone who makes a mistake (such as forgetting to implement the check method) the false-response would be on a sliding scale. The first miss is stored and blocks calls for half an hour. The second one, one hour, the third 2, and so on up to some maximum value. Or, after a certain number of unusable responses it turns off access to that endpoint. Basically this would serve to throttle outbound traffic to invalid destinations, rather than attempt to blacklist things completely.

Personally, I prefer pure XML or SOAP over XML-RPC, and I have no desire to drift backwards to plain strings and then have to write my own XML parser in LSL (OMG what a friggin' nightmare!)

How about a custom XML schema specific to LSL-supported data types, where you can send a single list, but each element of the list can be of any supported LSL data-type (except another list, of course)

<listData>
    <item type='integer'>10</item>
    <item type='vector'>0.5,1.0,0.0</item>
    <item type='string'>Whatever you want.</item>
</listData>

When received in LSL, the event would have a parameter ("data" of type "list" that would be a standard LSL list equivalent to:

list data = [10, <0.5,1.0,0.0>, "Whatever you want"];

Brilliant! This would be a wonderful feature. It reminds me of how Mason/Perl automatically makes URI character conversions completely transparent.

~Ulrika~

You could just as easily map XML-RPC datatypes to LSL ones.

integer = <int>
float = <double>
string = <string>
And for incoming messages or responses, <boolean> would map to an integer (0 or 1), <datetime> and <base64> to a string I would suppose. Although handling for base64 encoded binaries would be of dubious use as things stand in LSL.

For the more complex data types:
list = <array>
vector = <struct> (3 element)
rotation = <struct> (4 element)

XML-RPC arrays are identical to LSL lists (ordered set of mixed types). In order to distinguish between list, vector, and rotation the vector and rotation can be transmitted as structs, with x,y,z,w value names in order to preserve those associations.

Your example would become:

<array>
  <data>
    <value><int>10</int></value>
    <value>
      <struct>
        <member><name>x</name><value><double>0.5</double></value></member>
        <member><name>y</name><value><double>1.0</double></value></member>
        <member><name>z</name><value><double>0.0</double></value></member>
      </struct>
    </value>
    <value><string>Whatever you want.</string></value>
  </data>
</array>

While more verbose, it's a moot point because the already-written XML-RPC libraries will do the grunt work for you translating between LSL and the outside world. The ability for this to be done with XML-RPC already exists, the libraries are there, and it's a helluva lot easier than making everyone parse a custom schema. Facilitating arbitrary RPC calls between multiple platforms in a simple manner is what XML-RPC is designed for, unfortunately only a subset of the full functionality is made available to us via the LSL bindings.

That was a good suggestion, Cienna.

I think we need to apply a little customer pressure to get sublists working in LSL if there's any hope of a future where we can pass more structured data in or out of SL. The situation currently is completely unsatisfactory (confirmed by Lindens): the LSL compiler accepts lists of sublists syntactically without complaining, yet at runtime the wrong thing happens (breadth-first traversal fails, it gets stuck in a sublist) and some essential LSL functions for manipulating sublists are missing anyway.

On top of that, Don Linden said he was working on proper arrays for LSL, so the pressure for more structured data in XML-RPC will become more intense soon, hopefully. It's an area where LSL is pretty poor, but fortunately that's relatively easy to fix compared to other things like architecture.

I wish they'd open-source the LSL compiler and the XML-RPC interface, we could sort out these problems pretty quickly.