Dreamhost bouncing LL mail?
|
|
Mark Linden
Funky Linden Monkey
Join date: 20 Nov 2002
Posts: 179
|
10-28-2005 11:35
From: Moopf Murray Section 3.6 of RFC 2821: 4.1.1.1. goes on to give more information on that. I know I'm sounding pedantic, but I recently had an issue with a host that wasn't resolving with a reverse DNS lookup, and so mail sent out from it wasn't getting to many different clients - Dreamhost was one of the hosting companies I tested with (as I have accounts there), which is when I realised they were silently dieing in these circumstances. It's becoming very common for mail servers to silently die in situations like this, which doesn't make debugging any easier!  Huh. 3.6 is pretty clear that you don't actually need a PTR record (just that your name resolves to an A or MX record). On the other hand, 4.1.1.1 does indicate that if you don't have a PTR record, you should just send your IP instead of your domain name in the HELO/EHLO step. Neat. In any case, RFC or no, it is how many mail server operators run their servers these days, and as I said, it's how we operate. This is just a bug, like any other. |
|
Moopf Murray
Moopfmerising
Join date: 7 Jan 2004
Posts: 2,448
|
10-28-2005 11:42
From: Mark Linden Huh. 3.6 is pretty clear that you don't actually need a PTR record (just that your name resolves to an A or MX record). On the other hand, 4.1.1.1 does indicate that if you don't have a PTR record, you should just send your IP instead of your domain name in the HELO/EHLO step. Neat.
In any case, RFC or no, it is how many mail server operators run their servers these days, and as I said, it's how we operate. This is just a bug, like any other. I wouldn't of known anything about this myself if I hadn't had it happen very recently. But I got an indication from that just how many mail servers were rejecting because of this now, and I was suprised. And also how few are actually sending a response, and just silently dieing. Hope it helps. |
|
Sapphire Bombay
Avatar
Join date: 8 Oct 2003
Posts: 341
|
10-28-2005 13:16
Just to be clear here. This isn't email. It is IP all together. I can't even ping LL from Dreamhost.
_____________________
Avatar: A temporary manifestation or aspect of a continuing entity.
|
|
Sapphire Bombay
Avatar
Join date: 8 Oct 2003
Posts: 341
|
10-29-2005 04:48
OK we are still down. I know you guys have been busy with the upgrade. But, none of us are going to get SL past the 'game stage' when communications can go out like this for over 10 days. Is there a status on what is going on?
_____________________
Avatar: A temporary manifestation or aspect of a continuing entity.
|
|
Huns Valen
Don't PM me here.
Join date: 3 May 2003
Posts: 2,749
|
10-29-2005 15:34
From: Sapphire Bombay OK we are still down. I know you guys have been busy with the upgrade. But, none of us are going to get SL past the 'game stage' when communications can go out like this for over 10 days. Is there a status on what is going on? Here is some status: harpo.dreamhost.com - 15:20:25  at Oct 29 $ host 66.150.245.67 66.150.245.67 PTR record not found, server failure (That's the IP of Uba, by the way.) IMO, getting out of the game stage is not going to happen until (among other things) we get real XML-RPC. This email business is just a symptom of a larger lack of needed functionality. |
|
Mark Linden
Funky Linden Monkey
Join date: 20 Nov 2002
Posts: 179
|
10-29-2005 15:53
From: Huns Valen Here is some status: harpo.dreamhost.com - 15:20:25 at Oct 29 $ host 66.150.245.67 66.150.245.67 PTR record not found, server failure (That's the IP of Uba, by the way.) IMO, getting out of the game stage is not going to happen until (among other things) we get real XML-RPC. This email business is just a symptom of a larger lack of needed functionality. That IP has had working reverse DNS for about 2 years now, and doesn't appear to be broken at this time. I suspect that dreamhost is blocking access to our IP range at this point; my traceroutes into their network die at their border router, still: [mark@janus ~]$ traceroute harpo.dreamhost.com traceroute to harpo.dreamhost.com (66.33.213.101), 30 hops max, 38 byte packets 1 border5.fe5-23.linden-1.sfo.pnap.net (66.150.245.252) 0.469 ms 0.331 ms 0.273 ms 2 core2.ae1-bbnet2.sfo002.pnap.net (63.251.63.85) 0.390 ms 0.369 ms 0.334 ms 3 4.78.242.17 (4.78.242.17) 0.852 ms 0.882 ms 0.839 ms 4 ge-5-0-0.mp2.SanFrancisco1.Level3.net (4.68.124.209) 0.915 ms 1.028 ms 13.881 ms 5 as-2-0.bbr2.LosAngeles1.Level3.net (4.68.128.15  13.244 ms 13.262 ms 13.236 ms 6 ae-21-54.car1.LosAngeles1.Level3.net (4.68.102.10 13.217 ms ae-21-52.car1.LosAngeles1.Level3.net (4.68.102.44) 13.234 ms ae-21-54.car1.LosAngeles1.Level3.net (4.68.102.10 13.201 ms 7 * ge1-L3.dreamhost.com (4.78.192.66) 13.474 ms !A * 8 * * Have you guys filed a trouble ticket with dreamhost to ask them what's going on? Given that I can traceroute to them from other networks, it looks like they are blocking 66.150.244.0/23 and 69.25.104.0/23 at their border router. M |
|
Alondria LeFay
Registered User
Join date: 2 May 2003
Posts: 725
|
10-29-2005 17:02
From my understanding, tickets have been filled and response was that 1) they have not blocked it 2) they have recieved nothing from lindenlabs.com nor secondlife.com.... Traceroutes from Lucky also cross similar networks as the traceroute from LL.. it appears both ways it dies on the way. [lucky]$ traceroute 66.150.245.67 1 gw-66-33-192-1 (66.33.192.1) 0.530 ms 0.277 ms 0.237 ms 2 gw-L3 (4.78.192.65) 0.483 ms 0.345 ms 0.365 ms 3 ae-1-56.bbr2.LosAngeles1.Level3.net (4.68.102.161) 0.362 ms 0.346 ms ae-1-52.bbr2.LosAngeles1.Level3.net (4.68.102.33) 0.368 ms 4 so-3-0-0.mp1.SanFrancisco1.Level3.net (209.247.8.89) 16.704 ms as-0-0.mp2.SanFrancisco1.Level3.net (64.159.0.217) 14.210 ms 14.458 ms 5 ge-7-0-0.gar1.SanFrancisco1.Level3.net (4.68.124.210) 16.725 ms 12.711 ms ge-6-0-0.gar1.SanFrancisco1.Level3.net (4.68.124.206) 12.852 ms 6 4.78.242.18 (4.78.242.1 13.205 ms 13.323 ms 13.232 ms 7 border1.ge1-1-bbnet1.sfo002.pnap.net (63.251.63.1) 13.103 ms 13.332 ms border1.ge2-1-bbnet 8 * * * If it would help, I will set up a shell account for a Linden to assist with figuring out. Feel free to contact me regarding it. |
|
Eggy Lippmann
Wiktator
Join date: 1 May 2003
Posts: 7,939
|
10-29-2005 17:16
I'm not a sysadmin, so flame away. But since the problem doesn't seem to be on LL's side or on Dreamhost's, and it happens at the IP level rather than being email-specific, could it have anything to do with the recent Tier 1 peering agreement disputes that have been going on?
|
|
Mark Linden
Funky Linden Monkey
Join date: 20 Nov 2002
Posts: 179
|
10-29-2005 17:27
I believe that dreamhost isn't blocking anything on their SMTP servers, but they appear to be blocking IP from two of our subnets at their border router (or, perhaps their ISP is blocking subnets from our border router).
I'm happy to provide evidence to dreamhost and work with their techs to help resolve this, but customers of DreamHost will have to start the process, I think.
Eggy: It could be fallout from the Level3 debacle; I don't know.
M
|
|
Huns Valen
Don't PM me here.
Join date: 3 May 2003
Posts: 2,749
|
10-29-2005 22:48
From: Mark Linden I believe that dreamhost isn't blocking anything on their SMTP servers, but they appear to be blocking IP from two of our subnets at their border router (or, perhaps their ISP is blocking subnets from our border router).
I'm happy to provide evidence to dreamhost and work with their techs to help resolve this, but customers of DreamHost will have to start the process, I think.
Eggy: It could be fallout from the Level3 debacle; I don't know. Mark, I will file a ticket with them and give them LL's telephone number and tell them to ask for you. That is the only way this is likely to be resolved, other than someone figuring out they typed the wrong information into a router somewhere. I will file the ticket with a callback request, so they will (hopefully) call me and let me know. I don't know how many days it will take them. |
|
Huns Valen
Don't PM me here.
Join date: 3 May 2003
Posts: 2,749
|
10-29-2005 23:07
OK, I filed a support request: From: someone Hello, It seems that there is a bit of routing difficulty between various DH machines and Linden Lab's network. This is having an impact on some e-commerce applications that run on Linden Lab's 3-D platform, Second Life, mainly that email is falling into a black hole somewhere between DH and Linden Lab. We, as customers of both companies, are attempting to coordinate some communication between DH and Linden Lab in order to get this resolved - it's been going on for well over a week now. The going theory at the moment is that DH's mail servers are not able to reverse-DNS the IPs of Linden Lab's machines, and because of this, may be silently rejecting them. I don't know if that's the case, but it might help. To give you an example: harpo.dreamhost.com - 15:20:25 at Oct 29 $ host 66.150.245.67 66.150.245.67 PTR record not found, server failure That IP actually does have reverse DNS if you check from outside: DNS server handling your query: ns.kloth.net DNS server's address: 85.10.194.170#53 Non-authoritative answer: 67.245.150.66.in-addr.arpa name = sim131.agni.lindenlab.com. Mark (one of Linden Lab's neteng guys) is driving this issue on LL's side and has reported that traceroutes to DH seem to work from some of LL's subnets but not others. He thinks that 66.150.244.0/23 and 69.25.104.0/23 may be blocked at your border routers. I and several other customers would like to ask that you give Mark a call at (415) 243-9000. I've also checked the "Request Callback" box and would appreciate a call on Monday, or as soon as you've got in touch with Mark. This has also been carbon copied to [email]mark@lindenlab.com[/email], I hope that is the right address. The ticket number is 1148608. |
|
Mark Linden
Funky Linden Monkey
Join date: 20 Nov 2002
Posts: 179
|
10-31-2005 00:15
Should they call the office, I'll talk to them and see if we can figure out what's going on. Thanks for filing the ticket.
M
|
|
Nicola Escher
512 by 512
Join date: 1 May 2003
Posts: 200
|
10-31-2005 07:45
From: Huns Valen OK, I filed a support request:
This has also been carbon copied to [email]mark@lindenlab.com[/email], I hope that is the right address. The ticket number is 1148608. Thanks, Huns, for writing up and submitting that ticket and others for pitching in a trying to ge this resolved. |
|
Huns Valen
Don't PM me here.
Join date: 3 May 2003
Posts: 2,749
|
10-31-2005 14:55
Here's the latest from DreamHost... From: someone Hello, I tried to call you, but got no answer, I left a voicemail detailing the problem. Basically, lindenlab's ISP seems to be blocking us: traceroute NS0.LINDENLAB.COM traceroute to NS0.LINDENLAB.COM (66.150.244.129), 30 hops max, 40 byte packets 1 gw-66-33-192-1.dreamhost.com (66.33.192.1) 4 ms 4 ms 4 ms 2 gw-L3.dreamhost.com (4.78.192.65) 4 ms 4 ms 4 ms 3 ae-1-56.bbr2.LosAngeles1.Level3.net (4.68.102.161) 4 ms ae-1-52.bbr2.LosAngeles1.Level3.net (4.68.102.33) 4 ms ae-1-54.bbr2.LosAngeles1.Level3.net (4.68.102.97) 4 ms 4 as-0-0.mp2.SanFrancisco1.Level3.net (64.159.0.217) 17 ms 26 ms so-3-0-0.mp1.SanFrancisco1.Level3.net (209.247.8.89) 17 ms 5 ge-6-0-0.gar1.SanFrancisco1.Level3.net (4.68.124.206) 17 ms ge-7-0-0.gar1.SanFrancisco1.Level3.net (4.68.124.210) 17 ms 15 ms 6 4.78.242.18 (4.78.242.1 12 ms 14 ms 15 ms 7 border1.ge1-1-bbnet1.sfo002.pnap.net (63.251.63.1) 64 ms 229 ms border1.ge2-1-bbnet2.sfo002.pnap.net (63.251.63.65) 45 ms 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * You'll need to contact them in order to get that taken care of. If you need anything else, please let us know. Thanks! Brian Mark, I suppose at this point the problem should be handed off to your service provider. |
|
blaze Spinnaker
1/2 Serious
Join date: 12 Aug 2004
Posts: 5,898
|
10-31-2005 14:56
You'd probably have more luck convincing dreamhost sending it to their service provider.
After all, Sl isn't blocking traffic to dreamhost.
|
|
Mark Linden
Funky Linden Monkey
Join date: 20 Nov 2002
Posts: 179
|
10-31-2005 18:28
From: Huns Valen Here's the latest from DreamHost...
Mark, I suppose at this point the problem should be handed off to your service provider. Okay, at this point, I'm 100% certain it's not us and 95% certain it's not our ISP. That traceroute he posted could also mean that the dreamhost border router is dropping us. So, here we are, again. Can someone email me (not from a dreamhost account, apparently), a technical contact number for dreamhost and a trouble ticket number that I can use to reference? I'll try to call them tomorrow morning, PST. I'll use the ticket # already posted here if there isn't a fresher one. FYI: the reverse DNS for 72.5.12.0/22 is now active. There may be caching issues for the next week or so, but there's not a lot I can do about that. |
|
blaze Spinnaker
1/2 Serious
Join date: 12 Aug 2004
Posts: 5,898
|
10-31-2005 19:21
From: Mark Linden Okay, at this point, I'm 100% certain it's not us and 95% certain it's not our ISP. That traceroute he posted could also mean that the dreamhost border router is dropping us.
So, here we are, again.
Can someone email me (not from a dreamhost account, apparently), a technical contact number for dreamhost and a trouble ticket number that I can use to reference? I'll try to call them tomorrow morning, PST. I'll use the ticket # already posted here if there isn't a fresher one.
FYI: the reverse DNS for 72.5.12.0/22 is now active. There may be caching issues for the next week or so, but there's not a lot I can do about that. Doesn't that traceroute simply mean that the ISP firewall is droping tracerout packets? Most colos drop traceroute packets because you dont want hackers sniffing out details. |
|
Huns Valen
Don't PM me here.
Join date: 3 May 2003
Posts: 2,749
|
10-31-2005 20:31
From: blaze Spinnaker Doesn't that traceroute simply mean that the ISP firewall is droping tracerout packets?
Most colos drop traceroute packets because you dont want hackers sniffing out details. Oy... That's correct... So anyway, I tried to telnet to lsl.secondlife.com:25 to see if I could send a test message. Dreamhost couldn't find lsl.secondlife.com. I tried querying ns.kloth.net and ns1.earthlink.net and neither of them have heard of lsl.secondlife.com either! (I got that address by checking the from field on the last email I received, which was a couple weeks ago or so.) Mark, I'm thinking of two things that should be verified next. The first would be LL's nameserver configuration. If I can't look up lsl.secondlife.com from two separate nameservers that are not run by either LL or DH, that takes DH out of the equation for the time being. The second would be to check all switch/router configs between your nameserver and the Internet, to see if they are blocking 53/TCP or 53/UDP from anywhere. Once Earthlink and Kloth can look up lsl.secondlife.com, we can look at DH again in the DNS department, see if they can or can't... So, here's my theory as of this moment: - Mail sent from LL to DH: Fails because (presumably) DH's mail server can't reverse-DNS where the traffic is coming from, due to subnets having no reverse DNS, which may clear up as LL's DNS changes propagate. This is usually a matter of a day or two from my experience. As of this minute, 66.150.245.67 (Uba) has reverse DNS according to Earthlink's and Kloth's nameservers, but not Dreamhost's.
- Mail sent from DH to LL: Can't work since at the moment DH will not be able to translate lsl.secondlife.com into an IP.
- XML-RPC from DH to LL: Should work. I CAN resolve xmlrpc.secondlife.com from DreamHost.
Hmm... Mark, can you bump the serial number on lsl.secondlife.com in your bind.conf (or whatever you use)? If there is a stale entry in someone's nameserver cache somewhere, that might fix it. (In a few days at any rate.) It certainly won't hurt anything. It's wierd that I can look up xmlrpc but not lsl, are they on different nameservers? (Is that even possible?) |
|
blaze Spinnaker
1/2 Serious
Join date: 12 Aug 2004
Posts: 5,898
|
10-31-2005 20:42
do a
"dig mx lsl.secondlife.com"
from a unix command prompt on dreamhost.. what do you get?
And then try
telnet data.agni.lindenlab.com 25
if you can
and then for fun if that that doesn't work
try
telnet 66.150.244.192 25
which is the ip address for a mx server for lsl.secondlife.com
|
|
blaze Spinnaker
1/2 Serious
Join date: 12 Aug 2004
Posts: 5,898
|
10-31-2005 20:52
i don't actually think lsl.secondlife.com resolves to anything.
theoretically, this should not be an issue.
|
|
blaze Spinnaker
1/2 Serious
Join date: 12 Aug 2004
Posts: 5,898
|
10-31-2005 21:01
yeah there is no DNS entry for lsl.secondlife.com
However, that shouldn't block emails from dreamhost -> secondlife.com
shouldn't really block emails from sl -> dreamhost either, but I guess I can see it happening
|
|
Sapphire Bombay
Avatar
Join date: 8 Oct 2003
Posts: 341
|
11-01-2005 03:30
From: Huns Valen XML-RPC from DH to LL: Should work. I CAN resolve xmlrpc.secondlife.com from DreamHost.I can verify that this does not work. I had an XML app running between LL & DH for about 6 months with few problems. Now I can not connect. I restarted the channel with no help. Remember this is all IP communications that have failed. So with no communications, no upper level protocols are going to work. And it is not just DNS. I can not ping 66.150.244.149 from DH either. No lookup going on there. Whereas I can ping that from my home PC.
_____________________
Avatar: A temporary manifestation or aspect of a continuing entity.
|
|
Adam Zaius
Deus
Join date: 9 Jan 2004
Posts: 1,483
|
11-01-2005 03:37
From: blaze Spinnaker yeah there is no DNS entry for lsl.secondlife.com  adam@freya:~$ dig lsl.secondlife.com -t MX
; <<>> DiG 9.3.1 <<>> lsl.secondlife.com -t MX
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65014
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 5
;; QUESTION SECTION:
;lsl.secondlife.com. IN MX
;; ANSWER SECTION:
lsl.secondlife.com. 852 IN MX 10 data.agni.lindenlab.com.
;; AUTHORITY SECTION:
secondlife.com. 54269 IN NS ns1.sfo.pnap.net.
secondlife.com. 54269 IN NS ns1.lindenlab.com.
secondlife.com. 54269 IN NS ns2.sfo.pnap.net.
secondlife.com. 54269 IN NS ns0.lindenlab.com.
;; ADDITIONAL SECTION:
data.agni.lindenlab.com. 3252 IN A 66.150.244.192
ns0.lindenlab.com. 59668 IN A 66.150.244.129
ns1.sfo.pnap.net. 9860 IN A 63.251.62.1
ns1.lindenlab.com. 59668 IN A 66.150.244.130
ns2.sfo.pnap.net. 9860 IN A 63.251.62.33
;; Query time: 16 msec
;; SERVER: 202.72.191.199#53(202.72.191.199)
;; WHEN: Tue Nov 1 19:32:22 2005
;; MSG SIZE rcvd: 236
|
|
Huns Valen
Don't PM me here.
Join date: 3 May 2003
Posts: 2,749
|
11-01-2005 05:12
From: someone $ dig lsl.secondlife.com ; <<>> DiG 9.2.4 <<>> lsl.secondlife.com ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16341 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;lsl.secondlife.com. IN A ;; AUTHORITY SECTION: secondlife.com. 1 IN SOA ns0.lindenlab.com. hostmaster.lindenlab.com. 2005102403 1200 300 1209600 1 ;; Query time: 18 msec ;; SERVER: 66.33.216.208#53(66.33.216.20 ;; WHEN: Tue Nov 1 05:02:35 2005 ;; MSG SIZE rcvd: 97 $ host lsl.secondlife.com lsl.secondlife.com A record currently not present Both of these are from harpo.dreamhost.com. OK, now let's try nslookup... From: someone $ nslookup
> server ns.kloth.net
Default server: ns.kloth.net
Address: 85.10.194.170#53
> lsl.secondlife.com
Server: ns.kloth.net
Address: 85.10.194.170#53
** server can't find lsl.secondlife.com: REFUSED
> server ns1.earthlink.net
Default server: ns1.earthlink.net
Address: 207.217.126.41#53
> lsl.secondlife.com
Server: ns1.earthlink.net
Address: 207.217.126.41#53
Non-authoritative answer:
*** Can't find lsl.secondlife.com: No answer
> server ns0.lindenlab.com
Default server: ns0.lindenlab.com
Address: 66.150.244.129#53
> lsl.secondlife.com
;; connection timed out; no servers could be reached
> server ns.kloth.net
Default server: ns.kloth.net
Address: 85.10.194.170#53
> data.agni.lindenlab.com
Server: ns.kloth.net
Address: 85.10.194.170#53
** server can't find data.agni.lindenlab.com: REFUSED
>
> I don't know how dig is finding this info, when nslookup queries of a couple servers outside the LL<->DH corridor can't find it. Does dig query a root server directly or something? It seems unlikely that nslookup would fail but dig would work if they were both talking to the same source of information. Also, let's try pnap, which is listed as one of LL's DNS servers in a dig: From: someone $ nslookup
> server ns1.sfo.pnap.net
Default server: ns1.sfo.pnap.net
Address: 63.251.62.1#53
> lsl.secondlife.com
Server: ns1.sfo.pnap.net
Address: 63.251.62.1#53
*** Can't find lsl.secondlife.com: No answer
> data.agni.lindenlab.com
Server: ns1.sfo.pnap.net
Address: 63.251.62.1#53
Name: data.agni.lindenlab.com
Address: 66.150.244.192
> 66.150.245.67
Server: ns1.sfo.pnap.net
Address: 63.251.62.1#53
** server can't find 67.245.150.66.in-addr.arpa: SERVFAIL
Hmmm... |
|
Adam Zaius
Deus
Join date: 9 Jan 2004
Posts: 1,483
|
11-01-2005 05:52
Your searching 'A' records which are not used for mail. Try MX records instead.
|
