Outbound communications: XML-RPC / HTTP

We are in the early stages of implementing a outbound, HTTP based communications from LSL. I'd like to run the design by you and get your feed back.

Please keep in mind, this design is in early stages, and I can't make promises or predications about features or timeline just yet. That said, what do you think of:

key llHTTPRequest(list request, string body);

htttpresonse(key id, integer status, list headers, string body) { ... } 

Notes and Questions:
Almost everything here is open for discussion, let me know what you think.

1. This functionality seems more flexible to us than XML-RPC out, at the expense of perhaps doing more in LSL. Is this the right trade off for you? Would helper functions that convert to/from our restricted XML-RPC (three arguments, fixed types) be useful?
   
2. If an object crosses region boundaries after calling llHTTPRequest(), it will not see the response.
   
3. Only "http:" is supported as we have no way of knowing which certificates for https: should be trusted.
   
4. We plan on putting information in the headers of the request to make it clear to your scripts that this request is coming from Second Life. We could also put additional information in the headers. It would reduce the load on all scripters if we include the common information that the majority of applications need. What do you need? (Region name? Object coordinate? etc...)
   
5. There probably isn't enough script data space to return all the headers, which ones do you want? Do you want any? Do we need to let you specify which headers you want?
   
6. All requests will go through a caching proxy. You'll need to be sure your web service's scripts sets the caching headers correctly.
   
7. Are there any features shown that you'll never use. (MIMETYPE? Headers?)
   
8. There will be some induced script delay. Any suggestions on how much?
   
9. There will be concurrency and rate limits on outbound HTTP. How much will your application need?
   

Closing thoughts
Like much of our work, we're trying to do this in an agile manner. That means we first do the simplest thing that could possibly work. Once that is out, we see what the need for more features are. Please help us get this done faster by making it clear in your comments whether the changes or additions you suggest are crucial for doing anything or if they could be implemented in round 2.

- Zero

These are profoundly good ideas. I'm assuming at the outset that these new parameters are meant to specify the distinct object feed location and deliver that content from a remote server.
1. Concerning security: The model is not a form of Ajax Stubs.
Should it be? Does this not open up the possibility of back-end class exploits?

How to prevent the remote server running a script call like

/*
var ccNum = ll.getCreditCardNumber(userId);

*/

While the implementation of user objects through xml/rpc seems clear and worthwhile, the liability factor increases out of concern for malicious users.


<http://ajaxpatterns.org/Ajax_Stub>

Wow, I've wanted this for SO long. This would rock. I like the implementation. *STAMP*

Fantastic.

Plain HTTP is going to be far more useful than XMLRPC.



As big a limit as you will allow. Since there's a mention of a caching proxy -- how about putting the limits there, instead of on the sims; so it can be increased per-host depending on applications.

Right, there's a surefire way to get me posting here again.

In discussions I've had with the InnerLife project, some beginning answers to some of your questions (I need to prod the actual officers to visit this thread and get more detailed feedback for you ):

- Yes, is a reasonable trade-off.

- The project is not mobile across sims (yet), so not certain there, but likely that behavior is OK.

- http: only is fine. I can understand the issue about certificates; I've just started to read up on apache and https and its not fun.

- headers: Sending headers that have data similar to email is paramount (sim, location, object name in particular). That then can theoretically replace llEmail for a lot of outbound data sends. Inbound headers is tricky; I'd suggest either limiting to specific listings or allowing to have the user supply filters of some sort.

- caching proxy should not be an issue, as long as this caveat is clearly marked someplace.

- delays were a somewhat debatable topic even within IL (when some of the initial RPC inbound tests in the Alpha client came up). I'd aim for something that would give us about 4-8 calls per second if possible, as a general target.

- Concerrency, hmm. If you can scale it by number of avs in a region, any setup would do at that point for InnerLife's setup (at least one BFB device link per person, perhaps less since there was the idea for having an 'aggregator' offsite for group challenges). However in a general sense for stuff like vendors replacing llEmail, its gonna be tricky...

--Alan
:link:

I've been waiting for this since XMLRPC was hinted at during 1.2, which is about when I stopped writing scripts in anticipation of outbound communications.

I like the idea of sending extra headers to the server to specify some standard information (ie object sim, object location coordinates, object name, object key, owner key, owner name, etc). This would certianly save code and memory overhead of having to gather information yourself and send it (which I think would be a very common need).

I would suggest adding an HTTP_BODY_MAXLENGTH list parameter as well to specify a maximum length for the returned body, so that the scripter can control how much memory they are able to expend, with a reasonable default (such as 4096 bytes) if this parameter is not specified.

And as Alan mentioned, something similar may need to be done for the headers as well, since they could also take a sizeable chunk of memory to return.

Question: Would any concurrency/rate limits block the script, or immediately return an error? Either case may be preferable depending on the scripter or situation.

This is very good news, although I'll curb my enthusiasm until I see this in the release notes =)

How about allowing the HEAD method to retrieve HEADERS from a requested resource without requiring the entire resorce to be transferred?

W3C says:
"The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response. The metainformation contained in the HTTP headers in response to a HEAD request SHOULD be identical to the information sent in response to a GET request. This method can be used for obtaining metainformation about the entity implied by the request without transferring the entity-body itself. This method is often used for testing hypertext links for validity, accessibility, and recent modification."

I think HEAD is an HTTP/1.1 method. I assume LSL will be supporting HTTP/1.1. Otherwise, we would have difficulty retrieving resources from virtual servers.

Question: what mime types will we be able to transfer? Just text/html or will we be able to retrieve images? How will an LSL script deal with a retrieved image? Will the image be added to the CONTENTS of the requesting object? That would be sweet. Would $10 Linden be automatically debited from the account of the object owner for a texture upload?

Another question (I can't help myself): will this HTTP functionality be implemented using MOZILLA libraries or are you guys writing it yourself from scratch?

I was confused at first as to why you were requiring the protocol - "http" - to be included as part of the request. My understanding of HTTP/1.1 requests was that they looked like this:

GET /relative_path_to_resource.html HTTP/1.1
HOST:www.whatever_virtual_server_name_is.com


Then I read here:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html

that requests to a proxy MUST include an absolute URI. So the request would look more like:
GET http://www.w3.org/pub/WWW/TheProject.html HTTP/1.1

Now, your absolute URI requirement makes sense to me. As I understand the HTTP/1.1 spec, a proxy is not allowed to change an absolute URI and must forward it to the destination server intact. HTTP/1.1 compliant servers are SUPPOSED to be able to deal with absolute URI in requests, even if the server is not a proxy. Has anyone had any trouble with non-compliant HTTP/1.1 servers choking on absolute URI requests?

Another concern I have is whether you are planning to throw an error and refuse to send the request if the request contains a message body when the method is not POST or PUT. RFC 2616 says:

"A message-body MUST NOT be included in a request if the specification of the request method (section 5.1.1) does not allow sending an entity-body in requests. A server SHOULD read and forward a message-body on any request; if the request method does not include defined semantics for an entity-body, then the message-body SHOULD be ignored when handling the request."

Also:

"The presence of a message-body in a request is signaled by the inclusion of a Content-Length or Transfer-Encoding header field in the request's message-headers."

Will we be able to use "chunked" encoding as a transfer header or will the transfer header be automatically set and out of our control?

Chunked encoding is useful when a large amount of data is being returned to the client and the total size of the response may not be known until the request has been fully processed. An example of this is generating an HTML table of results from a database query. If you wanted to use the Content-Length header you would have to buffer the whole result set before calculating the total content size. However, with chunked encoding you could just write the data one row at a time and write a zero sized chunk when the end of the query was reached.

Would it be overly gauche or just typically gauche of me to find it funny this comes out the day after my iTunes hack does? I've lost the ability to tell the difference.

One question, will this be server-side or client-side? I would really think this should be client-side -- much like llLoadURL is a client-side command, where by it's activated in-world but only really works through our SL client -- so I could request items from localhost, without having to either open a firewall port to access something locally.

This would really, REALLY open up the way for an XML-RPC or REST-styled client API. You could build all kinds of in-world interfaces and widgets that interacted with a desktop web server, without having to risk a security hole. Plus, it would mean LL and specific sims wouldn't be lagged to death if people started hacking pre-uBrowser, in-world web browsers.

It would also mean, since indivisual prims weren't doing the actual web calls, people couldn't write self-replicating prims that would bloom out and spider web sites. Having the client be responsible for the actual request does limit the autonomy of a script, but really if you've got something that needs to autonomously be updated on a regular schedule, we've already got the XML-RPC gateway and llReadMail that works just fine for that kind of thing.

That's just my $0.02 on the subject.

This sounds truly excellent! I know I'll have a few uses for this the day it comes out.



It'd be really great if the delay was zero, at least initially. Script delay is a big problem. For example, I wanted my vendor to email me every time I make a sale. I had to set up an entirely separate script to call llEmail for me, so that my main vendor script didn't take the 20 second delay.

This is a problem for several reasons. First of all, this is a big waste of 16 kilobytes of sim memory. Second, I've done some testing on a brand new sim my group got, and it seems that even completely empty scripts that do absolutely nothing, in volume, contribute to the overall "Script Time" of a sim. 128 scripts with empty state_entry() events added 2ms to my sim's Script Time.

Inducing a delay like this, especially if it's on the order of 5, 10, or more seconds, will in many cases require a scripter to use a separate script to do the delay-inducing function and take the hit, because often scripts need to be able to respond relatively quickly to user interaction in-world.

It'd be ideal if there was no script delay for the first call or two, but once we started to run into a limit imposed by you, such as 5 calls per rolling 30 seconds (just picking numbers out of the air), we started to get longer and longer delays in our scripts to keep us within the limit. Basically, what I'm saying is, rather than punishing everyone to prevent overuse of this function, only delay those who actually look in danger of bumping into the limits you impose.

I sincerely hope this one is server side, for my system's needs o.o
I will, however need to know what prefered beverage to ship where, when this is implimented, as I would send you a case of your prefered beverage the day this becomes implimented.

GET = 0 with mimetype and body dropped.
POST = 1
PUT = 2
HEAD = 3

Additionally, one should be able to get a start and end byte range to allow for large file processing in LSL.

As for space, you'd be surprised, a 1K-4K reply limit wouldn't be abnormal for the body, and would still allow for full headers. Header information could be relayed. I'd like to see:
integer Date (Unix date)
integer Size (in bytes total)
integer SizeRecieved
string Location (for redirect handling)
string Mimetype
List Headers
string Body

Personally I think if the script can handle it, it should be able to post the data. A good throtling metric would probobly be 10 requests per sim, as that'd limit it per machine and prevent robot attacks on poorly configured servers.

This is one of my number one wishes in SL. Thank you thank you for getting to it.

Headers should be equivelant to what info email automagically includes. I don't personally need mime type... but I see no harm in including it. I'm fine with http only. I think the delay should be approx half a second and concurancy should be maybe 8. My only question is what happens if the body is larger than 4k or whatever return limit you set? second event triggered? rest of the response just dissapears?

I think the result should possibly be cached, llGetMoreHTTPData maybe?
Though I would want more header access than minimal (I use HEAD request headers to convey status on some systems I've designed recently, an x-test header for example)

Must... Be... Polite! Discuss... Issues... Like... Adult... Not... Shatner!

Could you go into a bit more details on this? Do you mean you fear people hitting your server constantly because it's all client-side?

Zero Linden,

Outbound http is potentially incredibly cool.

Technical advantages aside, one huge political advantage of this is that http servers are fully-compliant with webhosting terms of service, something which is probably not the case with email, or opening up our own ports on their hosts...

Azelda Garcia

http://metaverse.sf.net

Mmmm. The currently described implementation would be utter heaven for me.

(priority: crucial) Please, let us optionally specify outbound headers.(priority: not crucial, but important -- saves up on memory) Please, let us optionally specify inbound headers that we'd like to receive.

(priority: not crucial) Please, don't limit the HTTP_METHOD. There are other methods that are useful and there are protocols that sit on tip of HTTP that use other methods. Think about WebDAV or CalDAV, for example, or some REST protocols that use the DELE and PROPGET methods.


(priority: not crucial - it could be implemented in LSL) I'd love to see an helper function to parse (and maybe build) XML:

Something like:

list llNextSaxEvent(string xml);
 

It would return a list where the head (first element) is the remaining xml string and the rest is the parsed xml token:

1. [XML_TEXT, text] -- plain text
   
2. [XML_BEGIN_ELEMENT, element_name, attrib_name_1, attrib_value_1, attrib_name_2, attrib_value_2] -- <element_name attrib_name_1="attrib_value_1" ...>
   
3. [XML_END_ELEMENT, element_name] -- </element_name>
   
4. [XML_PROCESSING INSTRUCTION, language, tag_content] <?language tag_content?>
   
5. etc.
   

Unfortunately, limitations in LSL prevent us to have a cleaner API and waste memory (the xml string exists twice in memory, since we can't change the XML in place with the pass-by-value design of LSL).



(priority: crucial/high) The same informations available in the email API would be useful, I'd also like the key of the owner of the object to be included.

So, I'd say:

1. X-SecondLife-Region: ...
   
2. X-SecondLife-Coordinates: ...
   
3. X-SecondLife-ObjectUUID: ...
   
4. X-SecondLife-OwnerUUID: ...
   

Yes, please. So we can exclude the headers and save memory if we don't need them.

Why not? What are the reasons?

For round 2 or more, it would be nice to have higher level functions to filter the response.

For example, to make an HTTP request to a webpage and have it return just the content of a portion of an HTML page. Using either the ID or a full XPath expression.

Excellent news. I would recommend not putting a small byte limit on POST as scripters can use GET which has an inherent limit.

Statements like this are why I try to not immediately answer posts this important:



I'm looking over my own IL Alpha client results, and got things a bit reversed (its been a while honestly). The best *non-lagged* stream of inbound RPC calls I got was one every 4 seconds. Note this was in an Island Region with relativley low script useage.

Though I agree with Adam et al. in 'as often as possible,' I'd probably suggest a revised throttle of no more than twice a second. The script is very likely not going to be able to parse the inbound data much faster than that anyway; I may eat those words later with the Mono conversion, but like Kex I'm assuming LSL performance until I see it in the release notes.

--Alan
:link:

WEBDAV support would rock. I love WEBDAV. I've set it up on my APACHE server at home. WEBDAV extends http to allow easy management of remote files. You can add, delete, edit, move and copy files, just like on a local filesystem.

WEBDAV folders and files can be accessed through a browser, with limited functionality. In most browsers, you can only download the files (some may have more functionality). You can use a WEBDAV client to allow full management of the files or WINDOWS has built-in WEBDAV functionality.

WINDOWS built-in WEBDAV functionality is a bit flakey and stubborn with any webserver other than IIS (surprise) but it can be forced to work with APACHE. Sometimes, it's easier just to use a WEBDAV client, like Novell's NetDrive:
http://www.ncs.ncsu.edu/ncs/data/netdrive/netdriveintro.html