llGiveInventory randomly failing?

This is bizarre.

I have a networked vendor system I wrote. The vendor terminals put in item requests using HTTP and an offsite database -- the master server retrieves those requests then uses llGiveInventory to send the items out.

Well, 90-95% of the time, that works fine. But the others, they'll pay, and get no item. "The HTTP must be failing," I thought, so I checked the Handled Requests table... nope! They were all handled. Stored and retrieved by the master server.

So, the point of failure has gotta be llGiveInventory. (This is a summary of my checking, of course, there's a lot of possible point of failures... but I've tracked it down to that.) It's randomly not giving out the items to people. If they'd muted the server I could see it, but that's almost never the case, as I can tell from followup IMs with people calling to me for their missing items.

What's going on? Is there some way to RELIABLY give things to people in SL from across the entire grid? Do other vendor systems have the same problem?

llGiveInventory doesn't return status and the asset server silently drops such requests when overloaded. There is no reliable way to give inventory; This affects all vendor systems and simple in-game "buy" operations. This has been a minor problem for years, a major problem for all of 2008 and it keeps getting worse.

Have a nice day.

Alot of systems will give 2 or 3 copies from different locations to ensure the item is delivered. Obviously this would be for a copyable item. Bear in mind, if the region with the sever is down, it will not receive the call from the http server to give the item. Generally best to have at least two locations, more if possible, and have the item deliver from the server, then have it inform the http server that the item was given, or else after a time out, the http server can ask another server, etc. Or, better yet, the http server can query the in world servers and use the first one that responds to give the item, then report that it was given, if not, assume that the server did not deliver, and have another server serve it up.

Problem is, almost all the items in question aren't copy_ok... so giving multiple copies is a no-no. Also, I've checked the uptime and the problem isn't the server's sim going down, since we're up 99% of the time and the non-deliveries happen randomly throughout the entire day.

I can split the item requests into multiple 'pools' at the HTTP server level, each of which is tied to a different object server in a different sim... but I'm not sure that'd really help. If the problem is not sim uptime and is in fact that SL's asset server in general is made of chicken wire and chewing gum how would having multiple servers in multiple sims really make a difference? Is there some kind of throttling going on at the individual object level?

Check to make sure that certain "other" issues aren't interfering.

The most common cause of inventory delivery failure (in my experience) is that the customer has accidentally muted the shop owner. http://jira.secondlife.com/browse/VWR-4185. If you have a notecard giver, or a landmark giver, or some kind of greeter bot and they're muting that, they're actually muting you. They can pay you all day long, but they'll never receive an item, customer support IMS, or notecards you send them. http://jira.secondlife.com/browse/VWR-1735

The next most common cause (recently) of delivery failure, is that the Transaction was "stale".. the money transacted properly (after a time), but the money event in the vendor was never triggered. Therefore, the script has no idea it's SUPPOSED to deliver an item.

The third most commpn cause of inventory transfer failures, is that the customer is set either "Busy" or "Away".

Occasionally it's a mute -- but we've checked back and the majority of these people don't have me or any of the servers on mute. I doubt it's away mode since they need to actively go and buy an item in order to kick this process off, and with the number of complaints we get I'm doubting THAT many people actively use away mode while wandering about. As for stale transactions, they get the "Please wait five minutes, if it doesn't arrive, IM so-and-so" (so-and-so not being me) message, so the money() event IS firing.

I've studied all the breakdown points, and while each is responsible for a handful of failed deliveries, in the majority of cases everything was A-OK and the item just never arrived.

Good example: They buy the item. It fails. Confused, they pay again. Fail. Angry, they pay AGAIN... and one copy of it arrives, the other two never do. That can't be away mode or muting.

How often does your delivery server check for purchases? Does your data schema account for multiple purchases per resident? It could be that they are impatient in the middle of a polling period, and pay again before the delivery is dequeued. Then it could be the first delivery that arrives, or the schema could be such that multiple purchases only result in one queued delivery. An example of this would be if you have a database schema and the delivery table has a primary key that is composed of resident key and product name, without a sequence number thrown in to take care of duplicates.

Either way you might want to include with each purchase a message to the buyer that says something like "Please be patient. Your package could take up to 5 minutes to arrive." Where that 5 minutes is at least 2-3 times your polling period.

Actually, we DO have a "Please wait 5 minutes" message. Some folks just don't bother reading it. But even if they did buy 3 times, they should get 3 items as a result -- the system is geared to have no problems with multiple purchases from the same person, since each is a distinct request and none overwrite each other. I've tested this by buying a burst of the same item, up to 10 of them, with no problems.

It's polling for requests every 20 seconds for maximum speed, too, so even if the polling mechanism fails once, it'll just take 40 seconds instead of 20 for the request to come around. And we flag every handled request as pulled down and processed by the main server; we've been checking those logs and almost all of the nondeliveries WERE flagged as handled.

That log is critical evidence here. According to the log, everything is A-OK. Everything's going through the gears all the way to the llGiveInventory step... and then, the item doesn't show up.

I have even had the 'foolproof' buy from a box fail on occasion. Nothing in SL is 100% guaranteed unfortunately.