Dreamhost bouncing LL mail?

do a

"dig mx lsl.secondlife.com"

from a unix command prompt on dreamhost.. what do you get?

And then try

telnet data.agni.lindenlab.com 25

if you can

and then for fun if that that doesn't work

try

telnet 66.150.244.192 25

which is the ip address for a mx server for lsl.secondlife.com

Have they mentioned if they tried emailing Mark?

This is starting to become very annoying. I would like to thank you, Huns, for spearheading this effort - I just hope LL (or I suppose it still could be Dreamhost - although I am starting to doubt that since my experience has been - and is with this - very positive with them, much better than LL I hate to say) gets this working again.

I will feel absolutely ill if I have to switch hosts due to reason I don't believe is their fault (and I doubt I could find as good of a deal elsewhere).

--------------------------------------------------------------------------------
Re: Blaze -

[lucky]$ dig mx lsl.secondlife.com

; <<>> DiG 9.2.4 <<>> mx lsl.secondlife.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25698
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 3

;; QUESTION SECTION:
;lsl.secondlife.com. IN MX

;; ANSWER SECTION:
lsl.secondlife.com. 3600 IN MX 10 data.agni.lindenlab.com.

;; AUTHORITY SECTION:
secondlife.com. 157459 IN NS ns0.lindenlab.com.
secondlife.com. 157459 IN NS ns1.sfo.pnap.net.
secondlife.com. 157459 IN NS ns1.lindenlab.com.
secondlife.com. 157459 IN NS ns2.sfo.pnap.net.

;; ADDITIONAL SECTION:
data.agni.lindenlab.com. 2724 IN A 66.150.244.192
ns1.sfo.pnap.net. 40613 IN A 63.251.62.1
ns2.sfo.pnap.net. 40613 IN A 63.251.62.33

;; Query time: 29 msec
;; SERVER: 66.33.216.208#53(66.33.216.20
;; WHEN: Thu Nov 3 18:06:11 2005
;; MSG SIZE rcvd: 204

-----------------------------------------------------------------
[lucky]$ telnet data.agni.lindenlab.com 25
Trying 66.150.244.192...
telnet: Unable to connect to remote host: Connection timed out

Well, it's possible that dreamhost is blocking the ICMP packets from SL

My guess is that someone flooded the network with an out of control series of LSL scripts and xmlrpc and/or email and someone at either LL or Dreamhost shut it off at the firewall or has some auto DOS software which has shut it down.

The question is which one.

My bet is on dreamhost, because probably everything coming in/out of SL looks like DOS.

At the moment my guess is either a misconfigured border router (pnap) or internal switch (LLab). It could be a packet shaper or similar device misbehaving at DH. While interesting, this is all academic... the only parties who can solve this BOTH seem to think it's on the OTHER guy's end.

Try throwing UDP packets at a Linden box. If shows up then you know it's at the dreamhost end.

Tell him, "We know you don't *intentionally* block something, but all the data points to a misconfiguration or block on your end. Please fix this or you may lose 40+ dreamhost members who are getting angrier every day at your service."

I'm hesitant to say that to them when I'm really not convinced it's DH's fault. However, I do think it's a good idea for all impacted DH customers to write to DH support about this. Maybe we can light a fire under their asses and get them to make a more determined effort to get in touch with LL. That's the only thing that will fix this, them talking to each other.

This is a very strange problem. Something is very screwy in the network.

Traceroute from DH:
[lucky]$ traceroute data.agni.lindenlab.com
traceroute to data.agni.lindenlab.com (66.150.244.192), 30 hops max, 38 byte pac
kets
1 gw-66-33-192-1 (66.33.192.1) 0.461 ms 0.317 ms 0.239 ms
2 gw-L3 (4.78.192.65) 0.485 ms 0.352 ms 0.362 ms
3 ae-1-54.bbr2.LosAngeles1.Level3.net (4.68.102.97) 0.486 ms 0.345 ms ae-1-56.bbr2.LosAngeles1.Level3.net (4.68.102.161) 0.372 ms
4 so-3-0-0.mp1.SanFrancisco1.Level3.net (209.247.8.89) 12.841 ms 12.699 ms as-0-0.mp2.SanFrancisco1.Level3.net (64.159.0.217) 12.728 ms
5 ge-7-0-0.gar1.SanFrancisco1.Level3.net (4.68.124.210) 12.717 ms ge-6-0-0.gar1.SanFrancisco1.Level3.net (4.68.124.206) 12.818 ms 12.814 ms
6 4.78.242.18 (4.78.242.1 13.976 ms 11.835 ms 11.722 ms
7 border1.ge2-1-bbnet2.sfo002.pnap.net (63.251.63.65) 11.871 ms border1.ge1-1-bbnet1.sfo002.pnap.net (63.251.63.1) 14.082 ms border1.ge2-1-bbnet2.sfo002.pnap.net (63.251.63.65) 13.939 ms
8 * * *

Now, notice that hop3 and hop7 don't always reply with the same host. There's some route flapping going on, and I've never seen route flapping that's this bad.

This might be unrelated to the outage, but it is indicative of a network that's either not operating properly or not properly architected.

Okay, I've been staring at this problem for a few hours.

I'm starting to think the route flapping is not related, although someone at level 3 should probably look into it.

As far as I can tell, the only thing that makes any sense is that either LL or DH has an ingress filter at the IP level.

If anyone figures this one out, let me know, I'm kinda curious at this point.

There's tonnes of bgp software that routes dynamically these days.

Day 17 and we are still down. I haven't heard a word from either support department.

I posted a new thread in the hotline.

/invalid_link.html

I recieved this:

Like Brian said you will need to contact lindenlab.com since it is them who is blocking the connection not us. Sorry, wish there was more we could do.

Thanks!
Javier

Try this:

echo Hi mark! | nc -u ns0.lindenlab.com 9999

And then get mark to go

nc -l -u -p 9999


on ns0.lindenlab.com


(presuming it's linux and has nc installed)

Assuming they're not filtering out 9999 or dreamhost, then they should get the packets. If they don't, then they're the culprits. If they do get the packets, dreamhost should be the culprit.

Let me know if my reasoning is unsound..

At least you got a response. Now what is Mark doing about this? I see Huns posted to the hotline. Any response Huns? I still can't ping as of a few minutes ago.

None.

This makes me wonder if I still want to be in the "external" systems biz.

I got this from Dreamhost support:



I'm not sure if his analysis is correct since I still can't do a straight ping of Linden's IP addresses from Dreamhost. But, there it is for what it is worth.

19 Days of Downtime and Counting... (Are you sure you want to run a business in SL?)

sorry to hear about all your troubles with emails from second life.

i use Gmail and have never and i mean never had any IM's or object messages NOT sent to my in box.
that being said i don't know how it could be on the LL side

but then again i dont know a whole lot about these types of things

Actually, whoeever replied to that, didnt know what they were talking about LL isnt a dreamhost customer, so why they are bothering with LL's domain is beyond me --

However, if I was to bet (given all the evidence, and the ability to access from the thousands of hosts which use SL itself), I'd say this is the fault of a random blockage upstream from Dreamhost, and Dreamhost isnt willing to champion the cause to get them to fix it.

(Having dealt with bandwidth providers in the past, they are very rarely willing to fix a problem, unless you have a basketload of evidence which clearly puts the blame on them -- and even then it takes a while)

-Adam

Considering where the traceroute from Dreamhost to Lindenlab.com fails, my guess is that their is a filter on linden lab's border router device (possible firewall) that is filtering out packets sourced from the Dreamhost subnet.

I will venture a guess to say that somekind of dynamically filtering may have occurred after the 1.7 upgrade when some SL users program on a Dreamhost server started mondo-spamming either email or xml. Or someone at LL manually filtered it out at that time.

This is not an easy decision for me to make. I am going to wait on the hotline thread. Hopefully LL will come up with something.

In fact, if someone from LL would like, I will create a user account and home directory on harpo.dreamhost.com. It won't cost me or LL anything and it won't have visibility into my other stuff, so there should be no question of propriety. The machine has GCC, so anything desired can be compiled right there. I would recommend running something like blaze's suggestion, i.e. using netcat to open a socket on ns0.lindenlab.com and sending packets to it from DH, and then tracing it back from LL to DH to see where the packets are stopping. This might have to involve inter-NOC cooperation on every provider that sits between DH and LL. But I do not see another viable solution at this time.

Francis are you guys running Roam off of Dreamhost? Is it up? Is it using email?

Okay,

I have contact our ISP (Internap), as have some of you directly. I've double checked our firewall configs, just in case.

Internap isn't blocking Dreamhost. I am not blocking Dreamhost. Linden Lab doesn't IP blacklist anybody as policy unless we're under attack.

We are not under attack from Dreamhost.

I have attempted to contact Dreamhost via phone. No one from Dreamhost has called Linden Lab to ask for me, or anyone, as far as I know.

At this point in time, I'm not sure what I can do to help you.

I just had a horrible thought.

We've had some service done on our phone system in the office recently; I'll double check my VM on monday, but maybe something got nuked.

Half of my email from SL goes to my inbox the other half goes into "junk mail" no idea whats going on..