llEmail() Randomly and Silently Failing

Okay, this has been driving me a wee bit batty.

I run a fishing game in SL. The rods (and vendors) use llEmail() to contact a central server with a known key that does not change. The emails basically say "Hey, you, send person X item Y" and lo, the server does so. This is normally a one-way communication, although every six hours or so the vendors will mail to request the latest items for sale, and they'll get a mail back.

It works great! 95%-97% of the time. And the weirdest thing is that in certain sims, ones with heavy server load like Mobius and Avalon Estates, that ratio drops considerably. The emails... just never arrive. I've checked the logs and the vendor will send an email and it never reaches its destination.

I know about email throttling. The rods send out one email every 25 seconds at worst, and the vendors actually temp-rez objects that do the emailing for them to get around this problem. Even the server has 15+ separate email buddy objects to do its heavy lifting for it. So if it's not throttling... why are the emails randomly never being sent out?

what i do with my servers is have a type of fail safe, i have about 1-15 scripts that recrive the emails kinda like a overflow failsafe not sure if that is the problem but that is what it sounds like to many emails coming at once to understand all them, aka silent errors

Well, the thing is, the server in question has a loop of llGetNextEmail() so in theory, it should never MISS an email in its queue. And since the problem gets worse on sims with heavy load that leads me to suspect the client sending the mail has the issue, not the server at my home sim.

I guess I could have multiple scripts in there to parse and process emails, but how do I avoid them stepping on each other's toes? And would that really be the issue?

What you describe has always been this way, for as long as i've been in SL (gulp, heading toward 2 years now). It's probably the #1 gripe among merchants, because 80% of the vendor systems use llEmail for product delivery. It simply just isn't reliable. Many newer vendor systems have started going to HTTP for communcation, as it's more robust, and takes the server portion of the equation out of LL's hands.

So why do emails randomly never arrive? The answer is nobody knows. I could theorize that there is a breakdown in communication between the sims and the mail router, but it's just that, a theory.

Terrific. Yet another broken SL function. I'm getting really tired of the platform fighting me every inch of the way on these projects.

Is there a way to use HTTP calls without having to own your own external server and design backend database software? Can I just call one object to another using HTTP requests instead of llEmail()?

If you notice emails failing, give the sim a kick. a.k.a. reboot.

that should fix it in 105 of 100 cases or so.

I don't own the problematic regions. Anybody can set up a 7Seas fishing area anywhere in SL, including Linden-owned mainland -- it's the emails from those regions that are failing. So a reboot's out of the question.

Researching the wiki a bit, looks like HTTP is outgoing from SL only, and XML is incoming from outside SL only. What I need, since my system is entirely self contained, is a reflector -- something taking the place of the craptastic email system. Takes the string that was HTTP'd out and sends that same string back to our server by XML. I might be able to get space on a friend's PHP-enabled server, if I can find a script that'll do this -- anybody have any ideas?

Of course, this doesn't help my other problem -- scripted fishing rods resetting themselves for no good reason or during a sim crash and thus losing someone's XP -- but whatever...

Heh. If you like e-mail, you're going to LOVE XML-RPC.

Best solution I know of is messages sent to the server on demand via HTTP, queued, and then de-queued by the receiving object via HTTP polling: use a timer to ask for the next message from the web server; if you get one, possibly ask again immediately or shorten the timer until all messages have been dealt with, at which time you return to normal polling.

There IS a way to use XML-RPC to enhance this method without relying on its (non-)dependability. What you do is poll the web server every so often, but if you receive an XML-RPC message, poll immediately. This allows XML-RPC to benefit the system when it works, but not detract significantly when it doesn't. Of course, XML-RPC DOES require some bookeeping, so you'll have to figure out whether the small added benefit is worth the added complexity.

...I have no idea how to even parse those paragraphs. ??

My thinking was to get a friend of mine who knows PHP to make a reflector -- when it gets an HTTP message from InSL(tm), it sends the text it got via XML-RPC message back into SL(tm) to our servers. Basically, it's like llEmail, except it bounces it off the external webserver as a midway step. Is that not possible..?

This would all be vastly simpler if llEmail wasn't made of complete fail. Yay for lousy LSL!

It wouldn't be so bad if objects could IM each other.

really interesting thought there... set it up like the email handler... also throttled to prevent abuse/overuse. however i'd have to note that ims have had their own running set of probs... :\

p.s. hadn't yet truly grokked that xml-rpc stuff much... and seems according to the buglogs to suffer usually as much at the same time as email probs. :\

i have had plenty of that email/comm corruption thing seem to happen in the past... still have little-to-no clue as to what trips it... lately that has settled out in deference to the stale payment transactions bug.

So basically, there is no reliable means of intersim object communication whatsoever in Second Life. Good to know.

Still, I think I'm gonna move forward with the XML-RPC scheme. I've been working it out with a friend of mine who knows PHP, and we're going to make a new means of communication between servers and clients with that system, similar to the llEmail but with a bit of local PHP data variable storage to cut down on traffic. Hopefully it'll be better than llEmail. Hell, if SLX can do it... *shrug*

sounds like a good 'backup' plan really... i'm feeling committed to keeping my systems inworld and as simple as possible. fact is there -must- be some reliable way to do such stuff, else this grid would eventually get superceded by another that can...

Definitely follow the recommendation of using HTTP poling as a failsafe mechanism, or you're not going to see any improvement.

What we're saying is that XML-RPC is even worse than e-mail. The best solution is to take the message from the sender and store it on the web server. Then the "receiver" checks in now and then (also using HTTP) to see if it has any new messages. Just pretend you don't have cell phone reception, so instead you have to call your voice mail every ten minutes and see if there are any messages for you.

I vaguely remember LL saying something to the effect of "We will not maintain XML-RPC going forward", now that they've provided HTTP. I think all XML-RPC traffic for all of SL is handled by a single server, and it often chokes. This is all snippets of memory though, so I could be wrong.

You can do what you're trying to do with HTTP. Email isn't really 'delivered' asynchronously to the receiver, you have to poll the email queue by periodically calling llGetNextEmail (or whatever that call is), right. So you replace that with periodically calling a PHP script on the server. And the sender calls some other script on the server. The sender-side script stores something in a DB, the receiver-side script checks the DB, if there's a message, it sends it back. So you don't really need to 'push' the data from the webhost back to SL (and you're right, only XML-RPC can do that), but instead, you 'pull' it, which you can do using HTTP.

I have a little bit of experience with email, XML-RPC, and HTTP, and so far HTTP has been the most stable by far.

HTTP polling instead of XMLRPC...? Alright, but how often can I poll? Right now I'm polling email every 6 seconds. Would every 10 seconds or so be excessive here? I'm worried if I wait "ten minutes" as suggested before people will assume that their items aren't being delivered and IM me to tell me my vendors are broken.

Ah, good point. Here's a suggestion that may or may not work. The rod sends out an HTTP request, the PHP script at the other end saves the info in the DB, and then sends a response back to the rod. At this point the rod tells the user "Your request has been received, the item will be delivered shortly / in a couple of minutes / later today / whatever". That gives you some leeway in your polling.

You're right, if there's no feedback, people will think it's broken. And I'm pretty sure HTTP requests are throttled to prevent DoS attacks from inside SL, so 6 seconds might be too frequent. I think the specific rules for frequency of HTTP requests are in the Wiki somewhere. AFAIK it was set up to allow short bursts, but not sustained high-frequency calls.

You could also have the server-side communication send back a batch of requests in a single message, so it could deliver multiple items based off a single HTTP response. Or, when you receive a message, switch to a faster timer, then go back to a slower timer once the DB says "no more requests".

I could tone that down to one-a-minute. I'm planning to stick to llEmail for the fishing rods themselves -- it's just too high volume and that way I don't need to issue a thousand new rods. It's the vendor/server refreshes that happen every six hours, and vendor purchases (which people can sit and wait 60 seconds for, if I tell them to) that are gonna be HTTP'd, so that's plausible.

How do systems like SLX and Hippotech work, though, if they have sharp timing for deliveries and such? Are they using HTTP polling?

I don't have the quotes here, but several times I've seen Lindens say that it's okay to HTTP-poll once a second. HippoTech boxes seem to poll every 3 seconds or so; they're very responsive for web updates.

SL Exchange and SL Boutique existed before you could do HTTP from LSL, so I'm sure they started out with email and XML-RPC. I have no idea what they use now.

http://wiki.secondlife.com/wiki/LlHTTPRequest

25 requests per 20 seconds. So I was wrong about the point at which it'll start to throttle requests.

Seven, what you said about not changing the rods makes sense.

Yeah, there's no reason to burden people down with a rod upgrade using a new communications method when it's just to fill out the missing 7%-5% failure rate... fish are so easy to catch, even rare ones, and we're willing to fork 'em over if people IM us. It's the PAID FOR items that are unacceptably undelivered by llEmail().

Okay, I'm gonna discuss with my shiny new database engineer pal to do llHTTPRequest polling once every 30 seconds, to be safe. I think that's a reasonable amount of time to wait for delivery of an item. Thanks for your help; I'll post again if we need assistance, though! What sounds good "in theory" often turns to "crud" when put in practice, I've realized in my SL(tm) tinkerings(tm).

Sorry. The 10 minute thing was more appropriate to the analogy than the actual HTTP scenario. Really it is going to depend a great deal on each application how long the polling period is.

seven!

if you're getting an 85-95% success rate, you are doing WAY BETTER than anybody else i know of! cripes, i get a 98% FAILURE rate on my inter-sim emails. i had to take my mailboxes off the market, as they just wouldn't work at all.

the thought did occur to me.... that objects could actually email a REAL email address -- these, as far as i know, are pretty reliable. then some kind of email bot could parse the email, get the uuid of the intended recipient, and forward it.

of course, doing php databases and http thingies is probably a ton more sophisticated than that.

there was a jira entry to fix this; i begged some people a couple weeks ago to please vote for it. of course, i dont have the link NOW. i really need a 'pimp my jira' link collection....


anyhow, good luck, and im still flabbergasted it works that well for you. :)