Comcast NOC - issues to/from AS13331 (Seattle)

[Michael Loftis <mloftis () wgops com>]

Comcast doesn't appear to have any usable NOC contacts via whois, and
this issue is apparently very widespread.  Comcast obviously has
multiple saturated paths out in this area, so if you're seeing issues
getting to your customers on Comcast...well, it's probably Comcast.
Sort of an ongoing/me too on last months thread about same...


Outbound traffic to Comcast via basically any of our (AS13331)
upstreams except Spectrum (AS11404) is experiencing very high packet
loss once inside Comcast's network, we suspect a router or link very
near to us in Seattle is failing.  I've marked out the destination
customer IPs but they're available to Comcast engineers if they get in
touch with me directly or as noc () metapeer com -- TIA  -- and again,
sorry to the list.

Below are three failing/high loss traces, and then same three traces
from our other DC which is routing out via spectrum.  The loss is
present on Cogent and L3 outgoing paths at the least, maybe others.
As of now I've routed around the problem point for via the good path.

fission:~# mtr -s 900 --report --report-cycles 30 1.2.3.4
fission.myfreecams.com            Snt: 30    Loss%  Last   Avg  Best  Wrst StDev
ve75-gw1-xmr.metapeer.com                     0.0%   0.4   1.4   0.3  11.8   2.9
te0-7-0-15.ccr21.sea02.atlas.cogentco.com     0.0%   0.9   0.6   0.5   0.9   0.1
te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net  6.7%  43.7  42.3  38.8  43.7   1.4
be-13-cr01.seattle.wa.ibone.comcast.net       3.3%  47.7  44.6  41.3  47.7   1.8
pos-0-7-0-0-cr01.denver.co.ibone.comcast.net  3.3%  69.4  68.7  65.0  71.3   1.5
he-5-15-0-0-cr01.350ecermak.il.ibone.comcast  6.7%  92.6  97.3  91.4 102.0   3.8
he-3-15-0-0-ar01.woburn.ma.boston.comcast.ne 10.0% 120.3 119.8 116.3 122.0   1.8
pos-0-1-0-0-ar01.needham.ma.boston.comcast.n  0.0% 122.0 120.5 117.1 123.9   1.8
po-80-ur01.deering.nh.boston.comcast.net      0.0% 121.4 121.3 118.3 133.8   2.8
po-21-ur01.concord.nh.boston.comcast.net      6.7% 119.6 130.1 119.3 272.8  31.8
te-1-0-0-ten01.concord.nh.boston.comcast.net  6.7% 118.7 120.0 117.6 121.4   1.3
???                                          100.0   0.0   0.0   0.0   0.0   0.0
You have new mail in /var/spool/mail/root
fission:~# mtr -s 900 --report --report-cycles 30 5.6.7.8
fission.myfreecams.com            Snt: 30    Loss%  Last   Avg  Best  Wrst StDev
ve75-gw1-xmr.metapeer.com                     0.0%   0.3   0.6   0.3   5.2   0.9
te0-7-0-15.ccr21.sea02.atlas.cogentco.com     0.0%   0.8   0.6   0.5   0.8   0.1
te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net  0.0%  41.4  42.6  38.5  43.8   1.3
be-13-cr01.seattle.wa.ibone.comcast.net       0.0%  45.6  44.1  40.4  46.7   1.8
pos-0-8-0-0-cr01.denver.co.ibone.comcast.net  6.7%  71.2  69.0  65.0  72.1   1.9
he-5-12-0-0-cr01.350ecermak.il.ibone.comcast  3.3%  98.1  96.3  86.4 101.2   3.9
he-0-15-0-0-ar01.pontiac.mi.michigan.comcast 10.0%  93.5  97.3  93.5 100.6   2.3
xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.c  0.0%  97.2  99.1  92.9 141.8  10.4
te-17-10-cdn05.rochestrhlls.mi.michigan.comc  3.3% 118.9 113.2 104.6 118.9   3.5
???                                          100.0   0.0   0.0   0.0   0.0   0.0
fission:~# mtr -s 900 --report --report-cycles 30 a.b.c.d
fission.myfreecams.com            Snt: 30    Loss%  Last   Avg  Best  Wrst StDev
ve75-gw1-xmr.metapeer.com                     0.0%   0.3   2.1   0.3  47.1   8.6
te0-7-0-15.ccr21.sea02.atlas.cogentco.com     0.0%   0.7   0.6   0.5   0.8   0.1
te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net  6.7%  40.6  42.4  39.7  45.2   1.3
be-15-cr01.seattle.wa.ibone.comcast.net       0.0%  47.3  48.2  41.3  54.8   4.3
68.86.92.34                                   3.3%  43.7  45.5  39.5 101.7  11.6
be-18-ur06.bellevue.wa.seattle.comcast.net    0.0%  43.8  43.0  40.2  44.3   1.4
te-3-0-0-ten15.bellevue.wa.seattle.comcast.n  6.7%  42.7  44.1  42.1  45.7   1.1
???                                          100.0   0.0   0.0   0.0   0.0   0.0








mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 1.2.3.4
HOST: phobos                                           Loss%   Snt
Last   Avg  Best  Wrst StDev
  1. 207.229.74.1                                      0.0%    30
0.4   5.1   0.3 132.4  24.1
  2. agg1-sea-t7-8.bb.spectrumnet.us                   0.0%    30
1.5  14.0   1.5 206.4  39.8
  3. 23.30.206.33                                      0.0%    30
1.8   2.0   1.7   3.5   0.3
  4. be-17-cr01.seattle.wa.ibone.comcast.net           0.0%    30
12.6   5.3   2.1  12.6   2.7
  5. pos-0-4-0-0-cr01.denver.co.ibone.comcast.net      0.0%    30
28.5  29.2  27.2  31.4   1.3
  6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net  0.0%    30
49.9  52.2  49.9  56.7   2.8
  7. he-3-5-0-0-ar01.woburn.ma.boston.comcast.net      0.0%    30
78.5  79.5  78.3  81.0   1.0
  8. pos-1-12-0-0-ar01.needham.ma.boston.comcast.net   0.0%    30
80.2  80.1  78.2  81.7   1.1
  9. po-80-ur01.deering.nh.boston.comcast.net          0.0%    30
80.0  80.1  79.9  80.4   0.1
 10. po-21-ur01.concord.nh.boston.comcast.net          0.0%    30
80.7  85.1  80.7 199.3  21.6
 11. te-1-0-0-ten01.concord.nh.boston.comcast.net      0.0%    30
80.5  80.5  80.5  80.7   0.0
 12. ???                                              100.0    30
0.0   0.0   0.0   0.0   0.0
mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 5.6.7.8
HOST: phobos                                                 Loss%
Snt   Last   Avg  Best  Wrst StDev
  1. 207.229.74.1                                            0.0%
30    0.3   3.6   0.3  96.3  17.5
  2. agg1-sea-t7-8.bb.spectrumnet.us                         0.0%
30   10.2  10.4   1.5  58.5  15.1
  3. 23.30.206.33                                            0.0%
30    2.4   2.0   1.8   2.4   0.1
  4. be-14-cr01.seattle.wa.ibone.comcast.net                 0.0%
30    3.6   4.4   1.7  10.6   2.4
  5. pos-0-3-0-0-cr01.denver.co.ibone.comcast.net            0.0%
30   31.1  29.5  27.2  31.6   1.3
  6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net        0.0%
30   52.3  55.4  50.3  61.5   3.6
  7. he-0-14-0-0-ar01.pontiac.mi.michigan.comcast.net        0.0%
30   58.9  58.0  56.1  60.1   1.2
  8. xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.comcast.net  0.0%
30   71.2  58.9  56.6  83.6   6.4
  9. te-17-10-cdn05.rochestrhlls.mi.michigan.comcast.net     0.0%
30   71.2  72.6  67.6  78.0   3.2
 10. ???                                                    100.0
30    0.0   0.0   0.0   0.0   0.0
mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 a.b.c.d
HOST: phobos                                         Loss%   Snt
Last   Avg  Best  Wrst StDev
  1. 207.229.74.1                                    0.0%    30    0.7
  5.1   0.3 143.4  26.1
  2. agg1-sea-t7-8.bb.spectrumnet.us                 0.0%    30    1.5
  3.1   1.5  30.1   5.6
  3. 23.30.206.33                                    0.0%    30    1.9
  2.0   1.9   2.2   0.1
  4. be-12-cr01.seattle.wa.ibone.comcast.net         0.0%    30    6.9
  5.1   1.7  12.3   3.1
  5. 68.86.94.70                                     0.0%    30    1.8
  2.4   1.8  17.3   2.8
  6. be-18-ur06.bellevue.wa.seattle.comcast.net      0.0%    30    3.5
  3.4   3.3   3.5   0.1
  7. te-3-0-0-ten15.bellevue.wa.seattle.comcast.net  0.0%    30    3.3
  3.2   3.2   3.4   0.0
  8. ???                                            100.0    30    0.0
  0.0   0.0   0.0   0.0
mloftis@phobos:~$

--

"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler