Re: Recent NTP pool traffic increase (update)

[Denys Fedoryshchenko <denys () visp net lb>]

Hello,

I'm not sure i should continue to CC nanog, if someone interested to be 
in CC for further updates this story please let me know.

TP-Link not related, it was misunderstanding or wrong customer report.

Tenda routers i believe most of cheap models are affected by this 
problem.
On ISPs i have access i see too many of them sending requests to 
133.100.9.2 (if it is unreachable, repeating each 10 seconds), this 
particular ip seems hardcoded there. I am sure as soon as your server is 
down, way they coded - it will make all this routers to ddos this 
particular ip, repeating NTP queries very often without any 
backoff/protection mechanism.
Particular model i tested - W308R / Wireless N300 Home Router, but i 
believe many models are affected.
Firmware: System Version: 5.0.7.53_en hw version : v1.0

Another possible vendor is LB-Link, but i dont have yet any info from 
customers who own them.

On 2016-12-21 11:00, FUJIMURA Sho wrote:
> Hello.
>
> I'm Sho FUJIMURA.
> Thank you for your information.
> I operate the public NTP Services as 133.100.9.2 and 133.100.11.8.
> I'd like to reduce the traffic because I have trouble with too much
> traffic recently.
> So, I'm interested in the root of the the problem.
> If possible, would you please tell me the model numbers of Tenda and
> TP-Link??
>
> --
> Sho FUJIMURA
> Information Technology Center, Fukuoka University.
> 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan
>
> fujimura () fukuoka-u ac jp
>
> 2016-12-20 5:33 GMT+09:00 Denys Fedoryshchenko <denys () visp net lb>:
>
> > I'm not sure if this issue relevant to discussed topic, Tenda
> > routers here for a while on market, and i think i noticed this issue
> > just now,
> > because NTP servers they are using supposedly for healthcheck went
> > down (or NTP owners blocked ISP's i support, due such routers).
> >
> > At least after checking numerous users, i believe Tenda hardcoded
> > those NTP IPs. What worsen issue, that in Lebanon several times per
> > day, for example at 18pm - short electricity cutoff,
> > and majority of users routers will reboot and surely reconnect, so
> > it will look like a countrywide spike in NTP traffic.
> >
> > I checked for a 10min also this NTP ips in dns responses, none of
> > thousands of users tried to resolve any name with them over any DNS
> > server, so i conclude they are hardcoded somewhere in firmware.
> >
> > Here is traffic of Tenda router after reconnecting (but not full
> > powercycle, i dont have it in my hands). But as you can see, no DNS
> > resolution attempts:
> >
> > 20:15:59.305739 PPPoE  [ses 0x1483] CHAP, Success (0x03), id 1, Msg
> > S=XXXXXX M=Authentication succeeded
> > 20:15:59.306100 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
> > length 12
> > 20:15:59.317840 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
> > length 24
> > 20:15:59.317841 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 1,
> > length 12
> > 20:15:59.317867 PPPoE  [ses 0x1483] IPCP, Conf-Nack (0x03), id 1,
> > length 18
> > 20:15:59.325253 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 2,
> > length 24
> > 20:15:59.325273 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 2,
> > length 24
> > 20:15:59.335589 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
> > 133.100.9.2.123: NTPv3, Client, length 48
> > 20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
> > 192.5.41.41.123: NTPv3, Client, length 48
> > 20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
> > 192.5.41.40.123: NTPv3, Client, length 48
> >
> > Here is example of Tenda traffic if it is unable to reach
> > destination, it repeats request each 10 seconds endlessly, my guess
> > they are using ntp to show
> > status of internet connection.
> > So, now that NTP servers getting quite significant DDoS such way.
> >
> > 19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags
> > [none], proto UDP (17), length 76)
> > 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length
> > 48
> > Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
> > 0 (1s), precision 0
> > Root Delay: 0.000000, Root dispersion: 0.000000,
> > Reference-ID: (unspec)
> > Reference Timestamp:  0.000000000
> > Originator Timestamp: 0.000000000
> > Receive Timestamp:    0.000000000
> > Transmit Timestamp:   3691177063.000000000 (2016/12/19
> > 22:57:43)
> > Originator - Receive Timestamp:  0.000000000
> > Originator - Transmit Timestamp: 3691177063.000000000
> > (2016/12/19 22:57:43)
> > 19:57:52.163277 IP (tos 0x0, ttl 64, id 38516, offset 0, flags
> > [none], proto UDP (17), length 76)
> > 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length
> > 48
> > Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
> > 0 (1s), precision 0
> > Root Delay: 0.000000, Root dispersion: 0.000000,
> > Reference-ID: (unspec)
> > Reference Timestamp:  0.000000000
> > Originator Timestamp: 0.000000000
> > Receive Timestamp:    0.000000000
> > Transmit Timestamp:   3691177063.000000000 (2016/12/19
> > 22:57:43)
> > Originator - Receive Timestamp:  0.000000000
> > Originator - Transmit Timestamp: 3691177063.000000000
> > (2016/12/19 22:57:43)
> > 19:57:52.164435 IP (tos 0x0, ttl 64, id 38517, offset 0, flags
> > [none], proto UDP (17), length 76)
> > 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length
> > 48
> > Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
> > 0 (1s), precision 0
> > Root Delay: 0.000000, Root dispersion: 0.000000,
> > Reference-ID: (unspec)
> > Reference Timestamp:  0.000000000
> > Originator Timestamp: 0.000000000
> > Receive Timestamp:    0.000000000
> > Transmit Timestamp:   3691177063.000000000 (2016/12/19
> > 22:57:43)
> > Originator - Receive Timestamp:  0.000000000
> > Originator - Transmit Timestamp: 3691177063.000000000
> > (2016/12/19 22:57:43)
> > 19:58:02.164781 IP (tos 0x0, ttl 64, id 38518, offset 0, flags
> > [none], proto UDP (17), length 76)
> > 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length
> > 48
> > Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
> > 0 (1s), precision 0
> > Root Delay: 0.000000, Root dispersion: 0.000000,
> > Reference-ID: (unspec)
> > Reference Timestamp:  0.000000000
> > Originator Timestamp: 0.000000000
> > Receive Timestamp:    0.000000000
> > Transmit Timestamp:   3691177073.000000000 (2016/12/19
> > 22:57:53)
> > Originator - Receive Timestamp:  0.000000000
> > Originator - Transmit Timestamp: 3691177073.000000000
> > (2016/12/19 22:57:53)
> > 19:58:02.164884 IP (tos 0x0, ttl 64, id 38519, offset 0, flags
> > [none], proto UDP (17), length 76)
> > 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length
> > 48
> > Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
> > 0 (1s), precision 0
> > Root Delay: 0.000000, Root dispersion: 0.000000,
> > Reference-ID: (unspec)
> > Reference Timestamp:  0.000000000
> > Originator Timestamp: 0.000000000
> > Receive Timestamp:    0.000000000
> > Transmit Timestamp:   3691177073.000000000 (2016/12/19
> > 22:57:53)
> > Originator - Receive Timestamp:  0.000000000
> > Originator - Transmit Timestamp: 3691177073.000000000
> > (2016/12/19 22:57:53)
> > 19:58:02.165061 IP (tos 0x0, ttl 64, id 38520, offset 0, flags
> > [none], proto UDP (17), length 76)
> > 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length
> > 48
> > Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
> > 0 (1s), precision 0
> > Root Delay: 0.000000, Root dispersion: 0.000000,
> > Reference-ID: (unspec)
> > Reference Timestamp:  0.000000000
> > Originator Timestamp: 0.000000000
> > Receive Timestamp:    0.000000000
> > Transmit Timestamp:   3691177073.000000000 (2016/12/19
> > 22:57:53)
> > Originator - Receive Timestamp:  0.000000000
> > Originator - Transmit Timestamp: 3691177073.000000000
> > (2016/12/19 22:57:53)
> >
> > On 2016-12-19 21:40, Roland Dobbins wrote:
> > On 20 Dec 2016, at 2:22, Denys Fedoryshchenko wrote:
> >
> > If it is necessary i can investigate further.
> >
> > Yes, please!
> >
> > -----------------------------------
> > Roland Dobbins <rdobbins () arbor net>