XML-RPC Reply Delays

I'm building an object that communicates via XML-RPC with a website. Sometimes, the amount of data returned from the website can be very large, requiring up to a possible 90 individial RPC events to be triggered.

The website is very fast. What's killing me is the durn three second delay on every llRemoteDataReply command in LSL. Heck, if it was only a one second delay, the transfer would take around 90 seconds. Instead, I'm stuck waiting six friggin minutes!

Does anyone know a way to speed up llRemoteDataReply?

Holy crud! That's a lot of data. I didn't realize there was a limit to the amount of data you could send in an XMLRPC request... what's the limit?

Anyway, I'm pretty sure llRemoteDataReply() is optional. Do you really need the script to reply to every message? I suppose maybe you're acking or something, but maybe you could reply to every fifth message or something. In my (limited) experience, if you don't do a llRemoteDataReply(), the requester hangs for 15 seconds after the request goes through, waiting for a possible response from the object. Maybe you can just cut the connection early or something?

LSL is limited to 255 character strings on each remote_data event.



I'm afraid I don't know what "acking" means.

On my webserver, I'm using the PHP Example 2 from the LSL wiki to be the requester. I don't need to send a reply to every message although it probably does keep the PHP script from sending too many messages to my object and flooding the remote_data queue.

Originally, I didn't know to send a llRemoteDataReply() and without it the PHP script would seem to hang for 20 seconds or so before it continued. I must admit, I'm not entirely sure how that PHP code is written. I'm still teaching myself PHP. If you know any PHP, take a look at the wiki and let me know.



If I could do that and not need to send a response, then that would be awesome... although I do worry that if the PHP script floods the remote_data queue with 90 responses they may not all get processed.

Any thoughts?

Lex, there's a 255 character limit documented on the wiki page.

Harris - depending on how critical and how often, I'd suggest the best ways might be a rotating farm of scripts receiving the xml-rpc just like people use for outgoing email etc. In fact, since you'll be writing it off-world you could write something smarter and bigger than lsl will allow to decide whether to send to one or more scripts since storing their keys is relatively small in terms of off world memory space.

Oops, sorry, that means "acknowledging"... as in, replying with a "yes, I received this" message.

I've never done XMLRPC in PHP, but it looks like the PHP script has to do a lot more of the work than I had to do in my perl scripts, because the PHP implementation won't actually construct and send the HTTP request for you. I'm not sure what exactly would be legit here... where your PHP script hangs for 15 seconds is in the part where it calls "fgets()" a bunch to read the response.

Maybe if you just call "fclose()" on the socket, tossing away any response before the server sends it, you could get on with sending another. The only problem is, LL's servers might see an aborted connection like that as a sign that the request itself was unsuccessful, and it might decide not to deliver the message to the LSL script. It'd take some experimentation. Maybe what Eloise suggests would be better.


I just did some math. You're saying you might have to send up to 90 XMLRPC requests to your script, and each can hold up to 255 bytes. That's 22950, way more than a script can possibly hope to hold. Looks like you're destined to use multiple scripts anyway.

What's wrong with email for this particular scenerio?

Here's what I would do:
Use XMLRPC to start the script's timer event. In the timer event, call llGetNextEmail. Number your emails - if you must send multiple in sequence, let the script know how many there are in the sequence, so it can call llGetNextEmail until it receives all of them. Once all the emails in the sequence are received, then stop the timer event.

You can send a *lot* more data in an email then you can in an XMLRPC request - if you dont need object->external server replies, its the best communication method for your purposes. (At least, to the extent that I'm seeing them, you could be hiding something from us )
==Chris

I considered email but I'm planning on selling this object and there's a real chance that my PHP script would end up sending lots of emails. My account is limited to 200 an hour. While, I might not ever go that high, I'd prefer to avoid it.

I did exactly that and it worked (mostly). Read below.



Yes, I have database scripts where I hold all of the information. I've left myself some safety space by using three DB scripts and each one holding 10 records (each record has an average of 300-400 characters.

I did a bunch of testing last night and here's what I discovered:

1) You do *NOT* have to send a llRemoteDataReply as long as your requestor doesn't need it.

With a quick tweak to the PHP script, I made it not require an "ack" back from my LSL script before it continued processing. This caused the PHP script to rapid fire XML-RPC calls to my object very, very quickly. Woo!

2) If you dont't "ack" back, messages come back out of order

By removing the reliance of hearing the LSL script acknowledgment, my PHP script was free to spam my object with a bunch of XML-RPC events. Without the syncronization, the messages eventually fall out of order. This is a big deal since I've currently written my script to expect a certain order.

Furthermore, my script would expect a "Done" message to be the last message my PHP script would send so the LSL script knew it had received all the messages. But since they get out of order, the done would show up before some of the other messages.

I started to give up then I had an idea. All I really need to do is know exactly how many messages I plan to send and tell my LSL script this first. Then I can spam it with XML-RPC calls and keep a counter. Once the counter equals the expected number of messages my LSL script knows to continue.

I'm going to try this tonight.

You can also prepend each of your strings with a "Packet Number" so you can re-order the information after you receive it.

I think they toned down email to 1024 instead of the old 4096, another one of those silent updates .

I have enough exp on XMLRPC in LSL and server side. It's pretty slow when you start tryign to poll stuff.

Inbound email is 1,000, as I found out the hard way. The wiki says so - and comments contain the bounce message.

Update:

I got it to work. It's a bit sloppy but it works.

I changed my PHP script to count the number of XML-RPC connects it plans to make and first sends that number to the script. (This number can potentially change on each session). Then it rapid-fires XML-RPC connections to the object and the object counts them until it gets the same number it expects.

So, what used to take *eight* minutes to parse using the slow as molasses llRemoteDataReply function to ack back, now only takes *two* minutes to finish. This includes some overhead on my PHP scripts part so there's room for even more improvement I think.

So far it hasn't dropped any messages that I can tell. Even though they come into the queue in the wrong order (which I'm still not entirely certain why) they do all make it there which is good news.

If someone from LL is reading this, this entire ordeal would be unnecessary if only the llRemoteDataReply() function didn't delay so damn long!! Just changing the delay to 1 second rather than three would effectively give me the same performance.