Email -> SL Very Slow Currently?

OK, for the last 9-10 hours I've been finding emails coming into objects in SL have been really slow (anywhere from 5 minutes to 10 minutes). Is anybody else finding this as well? I've checked my server and the email's being sent speedily but delivery to the object in SL appears very sluggish.

Yes and it seems to be getting worse as time goes on too. At about 1am SLT the delay was about 10 mins... now at 530am SLT it's at least 40 mins since I sent a command code mail to an object and it still hasnt responded.

I figured at first that it was just a backlog from the grid being down but now I'm wondering if there is something else going on with them.

I spoke to Cyn about it when it first started happening yesterday and was told that QA had been informed. It seemed to speed up again for a while but today it's just been very, very slow.

Seems to be working OK again now by the look of things

Looks like it's on the blink again

It's still being very up and down today. Has anybody heard what's going on?

Hi, is it still being really slow? Things look alright internally right now, but we want to make sure. Please email me (torley@lindenlab.com) a complete copy of an example message that took a long time, with full headers. Anyone else who's experiencing this, please post as well so we can rule out things like specific ISPs.

Hi Torley !

It was going really well earlier but the delay is starting to build up again in the last 30 mins, we are up to about 10 mins again...

I'm going to send you a copy of one of our command mails privately when we hit about 20 mins lag ... instead of posting it here. (Corporate secrets and all that ... lol)

Torley, Ive sent you a notecard in-world with a sample control E-Mail from our system along with log entries etc...

I have some observations though:

At about 21:28 SLT the backlog of command e-mails we had sent started comming in one after another. (About 30 seconds apart... they originally were sent about 3-4 mins apart)

I **THINK** that only e-mails from outside servers were being delayed. E-Mails between objects that were both in SL didnt appear to be delayed (however I wasnt watching until after the backlog started clearing so I'm not sure about it.. I know once the backlog of external server mails started clearing that the in-world to in-world objects could send mails pretty much instantly)

Confirmed I've got your detailed info, Jitar--I've sent it to Operations for further analysis. I'll also have the inworld things you sent me if even more info is needed, but what you provided looks very comprehensive. Thank you.

Yes it was only emails from outside servers that was being affected. This morning everything seems OK again. I'll let you know if it happens again

OK, this is getting pretty tiresome now Email's up and down in service quality all the time and has been for days now. Is this known about and being worked on at all, because it seems very unreliable at present.

I'm seeing it pretty much every morning too. Now at 132am PDT I'm still waiting for our system to show what song I played at 100am .... Our listeners are SEEING the problem and really dont want to know what our DJ's played 45 mins ago... for us, after about 90 seconds the data is totally worthless...


You can use the Orb for personal use too Torley ... well... you COULD if this problem gets solved... **cough**

EDIT: The one transmitted at 01:00:11 arrived at 1:35:55 ... and the others are showing up now about every 45-60 seconds.... just like before... something is obviously clogging the system and once it gets through the clog....

I haven't heard back yet about this specifically; Moopf, could you please send me an email with the full headers too? I'm going to ask again. Thanks.

And, who else is experiencing this?

Torley, with the greatest of respect, your request is a complete red herring. This is happening to my bubble gum machines, some of my vending machines and multiple other of my projects that rely on outside email -> objects in Second Life.

To sum it up: the email service is unstable and has been for about a week now. It's only outside emails -> objects that it is happening to, it is not happening to emails between objects in-world and it's not happening to replies to offline IMs either.

I'm really quite suprised that a week later the issue still hasn't been resolved. Please, if Linden Labs really wish to treat this as a serious development platform, the response to issues such as this must become much quicker and knowledgeable.

I have to agree with Moopf, Torley. Tracking individual E-Mails isnt going to solve the issue. I know why OURS are being delayed, it's because they are being queued at data.agni.secondlife.com while it's working on processing something else. Something is causing that server to get clogged at fairly regular intervals. Once it chews through whatever it is choking on, it starts spewing out its backlog.

And who it's happening to would be anyone who uses any outside server that communicates into SL via E-Mail... which would explain a lot of vendor problems... Networked Vendors (ie SL Exchange and possible JEVIN?) would have use this too since its the only way currently to really communicate to an outside server.

From what I see, the clog is occuring on some cycle and repeats several times a day. Yesterday I saw it at 7pm SLT and again at 1AM SLT...

Apologies for any confusion.

I'll check with our Operations again.

we haven't fixed it yet. the grid needs to be down in order to do it safely. and this week ops has been very short on time in general -- 1/3 of us were sick, 1/3 of us reassigned to an emergency issue, and the other 1/3 (me) scrambling to pick up the slack.

we plan to fix it during the next week's scheduled downtime. apologies for the delay.

This is actually the first time it's been acknowledged, so I guess you were too busy to even do that? That's the part that really irks me. I brought this up with Cyn Linden a couple of days before I posted this thread, she spoke to ops - they didn't confirm anything. And this thread has been here for a week and only now do we get a confirmation.

By the time it's been fixed it will be nearly two weeks since it started happening. How is that conducive to a serious development platform?

ops has no mail from cyn. it's possible she walked over and asked someone, but i can't imagine why we wouldn't have given her the answer. there must be some miscommunication here.

i initially posted to this thread because torley forwarded it to us. (i'm posting right now because i happened to hit reload before closing the tab.) torley occasionally forwards us ops-related questions like this one, and we greatly appreciate the help. we like to respond to these, but ops questions come maybe one in a few hundred posts. we can't realistically hope to read every post and still do our jobs.

as for this particular problem, torley wrote to us on the 29th about it. mark wrote back with the explanation. it looks to me like maybe each thought the other was going to follow up to the thread and it just never happened, but i dunno. this is the one time i know of that we were asked about the problem. i'm sorry that answer didn't make it all the way back to you.

fwiw, data.agni is not getting "clogged". that would be nice -- much easier to observe, replicate, and solve. instead the problem is transient and transport-related. it may seem like data.agni is slow sometimes, but the machine itself is actually purring along with a low-to-normal load.

two weeks isn't so shabby when you consider the fact that **the grid has to be down** to deploy the fix. i believe this is the first planned downtime since the problem became understood, so we are fixing it at the first available opportunity.

Well either Cyn didn't speak to Ops or Ops didn't give Cyn the information. Cyn said there wasn't a known problem. Work it out amongst yourselves, I guess.



I'm not suggesting that you do. In fact, Torley said he'd brought it to your attention on the 30th (see the post on the first page, and according to you he brought it to the attention of ops the day before, the 29th). It took 6 days for a response on this thread, and I've seen no confirmation of the issue elsewhere on the forum before then. You didn't even have to sift through the forum on this one because Torley had let you know.



So are you suggesting that, on the 4th May, when Torley said this:



That Torley was lying? You're saying Mark wrote back with the explanation so Torley did know specifically about this? I'm very confused here. Something doesn't add up, and you know Torley runs round like a headless chicken most of the time plastering the cracks caused by lack of communication, or poor communication. I'm sure if Torley did know, even if he hadn't posted straight away and was only reminded by the thread appearing at the top of the forum again (perfectly understandable), he'd say what he knew, not that he'd heard nothing specifically.



We'll have to disagree here as 2 weeks, with the level of instability we've seen in the email service, is decidedly shabby. Maybe it raises that ugly spectre again of how scalable the SL architecture is if you have to take the whole grid down in order to solve an email problem. That just doesn't seem scalable to me.

I guess at the very least it highlights that internal communication still isn't very good at LL. Unless you want to send somebody some love I guess.

Ok, so why didnt this get fixed while the grid was down?

I sent out re-reg command e-mails to all of our orbs 12 mins ago and none have gotten them yet. (I'm sitting next to one so I know it hasnt even arrived yet because when they arrive the orb's hover text changes.)

Look, lets get simple. I'm paying a premium membership for this. If it dosent work and no one is going to actually fix it, I want my money back. NOW

From the ToS:

1.6 Second Life is subject to scheduled and unscheduled service interruptions. All aspects of the Service are subject to change or elimination at Linden Lab's sole discretion.

Linden Lab reserves the right to interrupt the Service with or without prior notice for any reason or no reason. You agree that Linden Lab will not be liable for any interruption of the Service, delay or failure to perform, and you understand that except as otherwise specifically provided in Linden Lab's billing policies posted at http://secondlife.com/corporate/billing.php, you shall not be entitled to any refunds of fees for interruption of service or failure to perform. Linden Lab has the right at any time for any reason or no reason to change and/or eliminate any aspect(s) of the Service as it sees fit in its sole discretion.

Whilst I can see this issue is a problem to you, there are many other issues that need work.

Lewis

If we are going "Legal" then I would invoke the California State Business and Professions Code (Since Both Linden Labs and I are in California) which supercedes their very one sided "terms".... The law I'm refering to states that a service or product offered, and accepted has an implied contract of sale... which Linden Labs is now in Breach of. They offered a service, and I accepted. They have taken my money and now are not delivering the service. So blow the TOS out your ....

At this point, looking up in this thread, we were told that the fix for this data transit problem would have to wait until the servers were down.... so we all waited patiently for 2 weeks. Sure looks like we were lied to, again.

It does seem to be a bit better, it only clogs to about 18 minuites then clears (instead of up to 40 minuites as before) but thats still not gonna cut it.

Especially when combined with all the other problems that LL are making excuses for.

OK, let's just calm down a bit here.

The logging on registrations for my bubble gum machines etc. tells me that yup, there is still a problem. Round about 2am GMT there must have been just such a time when the email service was crawling again.

But Lewis, just because the TOS says something, doesn't actually mean that the clauses would be upheld in court. That's kind of by the by and why, quoting the TOS doesn't necessarily mean that that's the end of the story.

Personally, this isn't an issue that I'd even think of demanding my money back over, not by a long shot. But, you know, the lack of communication, the promise of a resolution, the off-hand way in which 2 weeks isn't considered shabby, just results in frustration for those who use the tools provided. Personally, I'm getting sick of having to deal with all the IMs when people get my bubblegum machines, or my reseller vendors, and can't get them to register - it frustrates them as well, understandably.

I guess what I'm saying Lewis is that if, in a technical forum, all you wish to do is quote pieces of the TOS, then why bother? What, exactly, do you get out of it? And what does it add to the subject of this thread? Not a great deal. It actually comes over as you just trolling to be frank.

Anyway, please try and keep this thread on-topic. It's hard enough getting the Lindens to listen, confirm and resolve things anyway, as this thread has proved yet again.