GOM ATMs Offline

Linden Lab's mail server is broken again leaving our ATMs completely disabled. The reply I got from Cory was that they're fixing things. That was about 7 hours ago. I'm giving up for tonight and I'll check again tomorrow morning before work.

Those of you who already have "linked" avatars and who are looking to withdraw can use the "Pay My Avatar" feature on the site. If you haven't linked or if you need to deposit, IM me and we'll work something out. Keep in mind that I'll be in bed for the next few hours and at work during the day tomorrow. If I don't respond immediately, it's not because I'm being rude.

Happy trading, and poke your local Linden for me.

Zep

The email blockage has been fixed, but the backup has yet to clear, so they will continue to be delayed for a few hours.

Thanks to Zeppi and Andrew both for providing notification to customers.

Andrew, is there a cause that is cited for the outage that LL can share?

Just a stab for laffs:

1. Script / code change was pushed to production which broke the LSL - OS - Internet chain of events.

2. DNS lookups weren't working properly (resolv.conf file wrong, pointing to DNS servers that were down, etc etc)

3. Sendmail process wasn't flushing the queue because it was down or server it was on was overloaded


Been there done that

Still delayed.

Zep

Further to that, the main Shinda ATM seems to be back online and actually responded with my balance despite taking 11 minutes to activate itself.

Expect delays, but if you're persistent, it just might work for you. Sorry about this folks!

Zep

*UPDATE*

I have just deleted the two remaining ATMs in Shinda. I will put them back when things are working and I can spend my time doing something other than correcting transactions.

Anyone else using email scripts having insane problems? How about XML-RPC?

Zep

We found a family of objects that were seding out many emails every second. These were the source of the email backup.

It appeared that the objects had been engineered to get around our email throttles, and it also seems that the emails were being sent for a valid purpose (i.e. not a denial of service attack against someone else's emailbox), however I can't imagine why anyone would need to send themselves so many emails.

We had to disable the email scripts to allow the flood to subside.

What have I been saying these past few months? Multithreading is evil, and the scripting system as it's set up right now gives people too much incentive to use it. Top that with oldbie scripters in the forums actually SUGGESTING multithreading as a viable option, and you have a recipe for problems like this.

Yes, the email delay does make sense, of a sort, but not when it's so large that it makes people think "God, this SUCKS. Ok, I just need to multithread."

Rework the damn delays so emails get QUEUED to be sent out, and the script keeps on running!

...And we're back. 2 ATMs in Shinda. The rest will be back once I finish rebuilding them...

Zep

Our ATMs use this same technique. It was the only way to get reasonable performance for them. Perhaps an auditing process would work? ie. I submit my scripts, some knowledgeable Linden has a quick look to see that I'm not DOSing someone and flips a bit to let that instance of the script run free of delays.

Of course LL would have to kill the ability to "multi-thread". And then stuff would really start to suck..

Just my 2 cents.

Zep

Please, please, please don't kill multithreading in Scripts! There is no need.

What is needed is to remove a lot of the artificial delays that are added in, and rethink other avenues for preventing abuse via scripts. Delays do nothing other than actually causing more lag or using more server resources in the end. Also, please don't put in a silly restriction like 100 emails per day. Then you'll force people to autorez objects, and a bigger mess will ensue. Work on fixing the problem as to why people are sending large amounts of email for a valid purpose. Why is that? Is it because XML-RPC is only half implemented? Is it because of the silly length restrictions on emails? Is it poor proramming?

You sir are a misguided fool.

Additional scripts do not cause lag, events cause lag (within those scripts). The Link_message script which is primarily used in multithreading, doesnt cause any lag becuase link_messages are pushed out to prims, rather than the prims poll a master event stack (like sensor).

While the scripts are active (as in sending messages), most of them will be stuck in the sleep(20) after email. While in this state, they are paused and cannot do anything until the simulator resumes the script (which is done on a global push).

I have said this to you before infact, in another thread where you claimed to know it all.

-Adam

Aw, thank you for the flame. I feel all toasty now.



Gee... What was this thread about? Oh yeah. LAGGED email. Problems with the email server being bogged down. Tell me that multithreading didn't cause that!

That said, I'm not, and have not at all in this thread, suggested doing away with multithreading capability as Hank seems to be misguidedly worried about. I'm asking for alternatives. I'm asking that my entire script not be frozen for 20 seconds while it thinks about emailing someone. Just as Hank asked after me.

You assume too much.

That wasnt the content of your wiki post.



Yes, the email delay is annoying, - a product of a time before two-way email infact. However; without it being limited to those who understand multithreading to send frequent messages, these server crashes would be more frequent. Keep also in mind, that it doesnt nessecarily need to be one script that brought the server down (I highly doubt even a 255+ thread script could do it.), but one script in multiple objects all over SL.

What I would like to see, is that LL changes our primary interface for external I/O to be something more flexible and less likely to kill the sims - prehaps some new protocol which is designed to send blob's around, without any excess headers, message re-delivery or error correction, or anything else which will impede it from transmitting messages in an efficient manner.

-Adam

Lessee. Mildly dangerous... yes... mild would be a good word to describe most. At most something could go wrong with LL's code, and combined with your multithread, it could bork something.

Bring a server to it's knees or outright crash it? Gee. What happened here again?



How many people use GOM? Or llEmail? Would you say it's at least two dozen?



This is the only thing that might be construed as incorrect. Would you like me to put a "can cause" rather than "causes"? Would it make you feel better? How is any of the rest of what I said inaccurate?

XML-RPC outbound would be nice.

Multi-threading itself isn't evil (except for perhaps program correctness), but what multi-threading lets you do in SecondLife is very very evil.

Andrew's point about "Why would anyone send so many emails to themselves?" is indicative of a dangerous philosophy to have when you're platform building. If it is possible it will be done, whether on purpose or by accident.

here, here, ice!