Constant Packetloss since late 07/18

Looks like I started seeing a fairly constant 9-20% packet loss late at night on 07/18. Figured it was a temporary thing, but it looks persistent. From here I see:

Packets Pings
Hostname %Loss Rcv Snt Last Best Avg Worst
1. darkness.nu 0% 233 233 1 1 1 10
2. 209-145-162-1.interface.dsl.accessu 1% 232 233 20 10 12 37
3. stl-gsr-1-vlan-214.accessus.net 0% 233 233 11 10 12 30
4. at-1-2-092.usstls6b-j20c.savvis.net 0% 232 232 12 11 14 38
5. 500.POS3-0.GW4.STL3.ALTER.NET 0% 232 232 13 12 14 26
6. 108.at-5-0-0.XL1.STL3.ALTER.NET 0% 232 232 14 12 15 28
7. 0.so-1-0-0.TL1.STL3.ALTER.NET 0% 232 232 13 12 15 69
8. 0.so-2-0-0.TL1.DFW9.ALTER.NET 0% 232 232 30 27 30 60
9. 0.so-7-0-0.XL1.DFW9.ALTER.NET 0% 232 232 30 28 30 54
10. 0.so-6-0-0.BR6.DFW9.ALTER.NET 0% 232 232 30 29 32 87
11. so-1-0-0.edge1.Dallas1.Level3.net 0% 232 232 45 38 40 77
12. so-1-2-0.bbr2.Dallas1.level3.net 0% 232 232 39 38 41 72
13. so-3-0-0.mp1.SanFrancisco1.Level3.n 7% 216 232 71 65 68 94
14. pos8-0.core2.SanFrancisco1.Level3.n 9% 213 232 67 65 67 96
15. gige7-2.ipcolo1.SanFrancisco1.Level 7% 216 232 67 65 67 85
16. marie.lindenlab.com 7% 217 232 68 65 68 99

From Level3 looking glass in SF on a ping back to me I see:

----207.246.201.3 PING Statistics----
100 packets transmitted, 73 packets received, 27% packet loss
round-trip (ms) min/avg/max = 71/74/85

And a traceroute from Level3 SF back to me:

Tracing the route to challenger.darkness.nu (207.246.201.5)
1 so-4-0-0.mp1.SanFrancisco1.Level3.net (209.247.10.225) 0 msec
so-4-0-0.mp2.SanFrancisco1.Level3.net (209.247.10.233) 0 msec
so-4-0-0.mp1.SanFrancisco1.Level3.net (209.247.10.225) 4 msec
2 so-2-0-0.mp2.LosAngeles1.level3.net (209.247.9.181) 8 msec
so-1-0-0.bbr1.LosAngeles1.level3.net (209.247.9.113) 8 msec
so-2-0-0.mp2.LosAngeles1.level3.net (209.247.9.181) 8 msec
3 pos8-0.core1.LosAngeles1.Level3.net (209.247.10.194) 8 msec
pos9-0.core1.LosAngeles1.Level3.net (209.247.10.202) 8 msec
pos8-0.core1.LosAngeles1.Level3.net (209.247.10.194) 8 msec
4 savvis-level3-oc3.LosAngeles1.Level3.net (209.0.227.46) 8 msec 8 msec 8 msec
5 atm8-0-020.CR-1.ussntc.savvis.net (209.83.222.82) [AS 6347 {DIAMOND}] 56 msec 56 msec 52 msec
6 accessus-1.usstls3.savvis.net (64.240.159.11 [AS 6347 {DIAMOND}] 56 msec 56 msec 60 msec
7 stl-sms500-covad-gw.accessus.net (207.206.148.114) [AS 5656 {}] 56 msec 56 msec 56 msec
8 gw.darkness.nu (207.246.201.1) [AS 5656 {}] 64 msec 68 msec 64 msec
9 challenger.darkness.nu (207.246.201.5) [AS 5656 {}] 68 msec 72 msec 72 msec
Exiting.

Routing is definitely assymetrical, I don't really have any methods for testing the return path properly from here. Is there anything you guys can do to pass this on to Level3 and see if they can look into the packet loss? Seems to be starting before your colo networks in their core.

Might be fallout from that Cisco router issue. I think yesterday a lot of the ISPs were scheduling outages for off-hours so they could get the patches in.

Of course there might be some who didn't do anything. The exploit is out I believe.

Worked with the NOC a bit at level3 and got it resolved. Not a Cisco issue, though may have been caused indirectly by it. Routes just got in an unhappy state