Re: Port 80 SYN flood-like behavior

[Dave Dittrich <dittrich () cac washington edu>]

> What you are describing exactly fits the description of a "midpoint server"
> participating in a new form of Distributed Denial of Service attack. We
> were on the receiving end of such an attack a little over one month ago.
>
> Briefly, the idea is that a spoofed source IP SYN flood is gently spread
> across a LARGE number of TCP servers. Each of the many servers replies with
> SYN/ACK packets ... aimed at the attack's intended target.  Since each
> unacknowledged SYN/ACK will be repeated (generally three times) this
> results in a factor-four bandwidth multiplication.

I have also done some research into this for of reflector attack,
which is an evolutionary variant on several other forms of reflector
attack.  A tool was found recently that fits the bill (all it needs
is a list of reflectors.)

Here is the current writeup I have on this subject:


Some Followup Analysis to the 01/11/02 DDoS Attack on grc.com
=============================================================

Last modified: Thu Jan 17 10:02:23 PST 2002

On the morning of Friday, January 11 2001, Steve Gibson's web site
(grc.com) was hit with a distributed denial of service attack at 2:00
AM PST.

This attack used a variation of a TCP based reflection attack that is
not widely known to exist in the wild.  Steve's early analysis of the
attack in included below (Appendix A).

While there may be a new (D)DoS program "in the wild" to implement this
attack, the risks and methods have been known for two or more years
and some simple modifications to existing tools, and a good list of
high-capacity routers, switches, and servers, could affect an attack
of this type.


Reflector attacks
=================

Reflector attacks (and their implications for making traceback more
difficult) have been know for over a year. Vern Paxson mentioned them
at a traceback workshop in Chicago sponsored by CERIAS in September
2000[1].  Vern then went on to give a formal talk on the subject in June of
2001[2]

In January of 2001, Register.com was taken out with a DNS reflection
attack[3], by a single (or so it seems) system.

The attacker used a year+ old list of DNS servers, and used a program
to spray out forged (with the victim, register.com's source addresses)
DNS requests, making a recursive query for the MX records for the
domain "aol.com." If I recall correctly, the request was about 50
bytes, and the answer some 500 bytes (so it also amplified the flood
by a factor of 10 -- Steve calculated a "FACTOR FOUR" amplification
for his attack.)

Since the requests looked somewhat legitimate (very few of these
servers should have even been responding to recursive DNS queries in
the first place, but were not configured to refuse them), it was very
difficult to trace.  The attackers, however, only used one IP address
for targeting the site, which made it pretty easy to filter in-bound
the inbound DNS traffic.  It still took a WEEK for the major backbone
providers to trace and shut down the host that was originating the
flood of requests.


Traffic reflection off routers
==============================

Back in 1999[4] someone (nobody knows who, or even if there was a
someone) appeared to be sending out packets with forged sources
addresses within various networks and low TTLs.  Routers within the
Internet infrastructure would send back ICMP Time Exceeded in Transit
packets to sites all over the globe.

This one was so low bandwidth that nobody assumed it was an attack
(but it looked to me like a possible covert channel for
command/control, since it returned some 8 bytes of the original packet
within the ICMP Time Exceeded in Transit packet.)


Accessibility of Routing Infrastructure Hosts
=============================================

Rob Thomas has also been discussing vulnerabilities in the
routing infrastructure of the Internet for several months, now,
and has pointed out that many routers shouldn't be accessible from
outside the owner's network anyway.


Analysis and Defenses Proposed by Steve Gibson
==============================================

[Steve's early analysis is found in Appendix A]

Steve claims that it would take a lot of research to come up with a
list of hundreds of routers and servers with open ports that could be
exploited.  He also claims that he saw both SYN|ACK and RST|ACK
packets coming into his network, and the ports being used were for
BGP, SSH, Telnet, DNS, and HTTP.  Since these are all services with a
high probability of being enabled on routers, it is not necessary to
first look for them (just assume they are open), and since Steve was
also seeing RST|ACK packets, it means that some of them were sent
packets to ports that were NOT open.

Getting a list of "hundreds" of routers (as Steve claims were used
against his site) is not at all difficult.

A quickly hacked together Bourne Shell script of a handful of lines,
using a list of 10 popular web sites, results in a list of 50 routers
and servers.  To get "hundreds" of routers/servers, it would only
be necessary to add another 30 to 50 web sites.

Even easier is to go to a web site that lists BGP route peerings, which
gives the addresses of several thousand infrastructure routers.
If you are tempted to say, "so let's make the list private instead",
you need only fall back on the prior simple script to harvest
IP addresses of routers directly.  The answer is, of course, do to
ingress filtering to protect the routers, not to try to hide their
existence in the infrastructure (although backbone providers are quick
to point out that they fear CPU overutilization from filtering,
so they instead leave their routers exposed to outsider access.)

The point that the flooding program is clever enough to send packets
out at a small enough rate to fly beneath the radar, so to speak, is
not as sophisticated as it seems.  The single host that took out
Register.com was able to do this with DNS requests, and most DDoS
attacks today are done with systems that are sufficiently slow so as
to not swamp the high speed networks of today.  This may be purely an
accident, not a clever ploy or feature.

Steve is right about some of the defensive mechanisms, but in his case
his "don't allow privileged port traffic" defense works ONLY in the
case where your site exclusively has servers making no outbound
connections.  Of course this will break things like DNS zone
transfers, NTP time synchronization, outgoing email, etc. from his
site as well.  Being able to simply reject all outbound
privileged port connections is thus not a widely deployable
"solution."  Instead, direct ingress filtering that only allows
incoming connections from those servers you trust (or stateful
filtering as provided by more sophisticated firewalls, filtering
bridges, or host based packet filters like "ipchains", "iptables",
or "ipf".)

He is also correct that detecting this kind of an attack is more
difficult (since many sites don't have methods of sniffing traffic
within their network infrastructure), but many in the security
industry have been saying that for years.

Copyright (C) 2002, All rights reserved.
David Dittrich <dittrich () cac washington edu>


References
==========

[1] CERIAS Traceback Workshop proceedings
        http://www.cerias.purdue.edu/traceback/

[2] Vern Paxson talk on reflector attacks
        http://www.icir.org/vern/papers/reflectors.CCR.01/reflectors.html

[3] Thread on Register.com DNS reflector attack
        http://staff.washington.edu/dittrich/misc/ddos/register.com-unisog.txt

[4] Threads on ICMP Time Exceeded in Transit traffic
        http://lists.insecure.org/incidents/1999/Dec/0005.html
        http://cert.uni-stuttgart.de/archive/incidents/1999/12/msg00005.html



Appendix A: Steve Gibson analysis of 01/11/2002 attack on grc.com
=================================================================

This past Friday morning at 2:00 AM PST, we were hit by a new sort of
massive Distributed Denial of Service (DDoS) attack.  Since this operated
in an intriguing new way that I have never seen reported before, I thought
you folks might want to know what was going on.  Would you please forward
this note to anyone else who should learn of it?  Thanks.


------------------------------------------------------------------------
Here is the report I wrote and posted to my newsgroups after the attack:
------------------------------------------------------------------------

        The "Packet Bounce" Distributed Denial of Service Attack

As many of you know, we were the target of a new sort of attack very early
this morning.  We were restored to operation after four hours of complete
outage when I characterized the attack and was able to filter a large part
of the nonsense at Verio's upstream routers.  However, after 13 hours, well
into the day, the attack continues unabated, though blocked.

We were initially driven off the Internet by a flood of SYN/ACK and RST/ACK
packets -- all having a source port of 179 and apparently random
destination ports.  These flooding packets were apparently (and confirmed)
coming from the Internet's core routers!  Our own ISP's (Verio) routers
were 'attacking' us, as were the routers of other large ISPs, a few of the
main DNS root servers, and many of the web servers belonging to YAHOO.COM.

I placed 'attacking' in quotes above, since, as we'll see below, all of
these machines were innocent bystanders in the attack. However, they were
utilized to swamp our bandwidth and force us off the Net.

Since we do not have any need for port 179 traffic -- which is the Border
Gateway Protocol (BGP) used for inter-router communications -- we were able
to readily block that traffic at Verio's upstream "aggregation" router.

But ... the second we did that, another whole class of additional flooding
attack traffic (which had been unable to compete with the core routers
overwhelming BGP attack) showed itself for the first time. It had been
buried "beneath" the main thrust of the attack.

With the unneeded BGP protocol now blocked, we were getting SYN/ACK and
RST/ACK packet floods coming from ports 22 (SSH), 23 (TELNET), 53 (DNS) and
80 (HTTP). These packets were flooding in from hundreds of actual machines
(including many at YAHOO.COM and even one at NSA.GOV) and widely spread
across the Internet.

This second-tier of traffic was being sourced by well-connected but
similarly innocent servers.

-------------------------------------------------------------------
Here is what someone has done, and why it's rather clever ...
-------------------------------------------------------------------

Someone, or some group of people, have accumulated a large list of
well-connected server IPs and corresponding TCP service ports out on the
Internet.  Among these are several hundred well-connected routers serving
the BGP (port 179) protocol. This list also includes many other servers
listening to other common service ports including SSH, TELNET, DNS, and HTTP.

Using full raw sockets on an unknown platform (presumably UNIX) in order to
spoof the source IP and generate isolated SYN packets, the SYN packets are
"sprayed" at each machine in their list, at that machine's corresponding
listening TCP service port. (BGP for the routers, HTTP for YAHOO. etc.)

In each case, the source IP being spoofed is the attack's target. For this
attack it was the IP of the grc.com server.  The high bandwidth of the
attack, crossing our Internet connection, saturated our available bandwidth
many MANY times over.

The bandwidth of the "spray" is such that no one of these "intermediary"
target servers receives a "flooding-level" of SYN packets. In other words,
the SYN packet rate to each server is modest so that the server is never
placed into DoS by the packet originator, and no alarms are raised on that
server locally.

However, for every new SYN packet arriving at one of these servers, that
server, believing that a TCP connection is being requested and initiated,
replies to the apparent sender with a SYN/ACK packet. That SYN/ACK packet
is, of course, aimed at the APPARENT source of the SYN ... in this case
grc.com.

So ... grc.com is consequently subjected to a substantial Distributed
Denial of Service (DDoS) attack from many hundreds of well-meaning and
innocent Internet servers.  And, significantly, none of those servers is
being inconvenienced by its role in this attack.

-------------------------------------------------------------------
Why is this rather clever?
-------------------------------------------------------------------

Unlike traditional DDoS attacks, where a high-bandwidth spoofed-SYN flood
can be traced back up the router chain towards its actual bandwidth source
(because all packets have a common destination address although a spoofed
source IP), the use of a large number of intermediate TCP servers prevents
any single "stream" from having enough bandwidth to sound any alarms.  And
all of these low-bandwidth streams are coming from very different
physically isolated servers.

Conceivably, only a very few malicious and well-connected SYN-generating
machines would be required to "spray" the SYN packets out across a large
population of intermediate servers.  Each of these malicious well-connected
machines may be generating packets at its maximum speed, but since those
packets are so widely and immediately distributed -- rather than all being
aimed at a single traditional DoS target -- none of the intermediate
recipient machines is unduly inconvenienced.  Furthermore, since the
intermediate machines are widely spread across the Internet, the
SYN-source's local flood thins and spreads out quickly, even a few hops
away from its source machine.  This further impedes successful SYN-source
backtracking.

This also creates a potentially more manageable architecture than the
traditional comparatively difficult to establish many-machine multi-tier
DDoS system.

And finally, there is a very good chance that the intermediate machine will
generate MULTIPLE SYN/ACK packets for every SYN packet received. Since
traditional TCP stacks will typically generate four SYN/ACK packets in
response to the receipt of a single unanswered SYN packet (believing that
their responding SYN/ACK might have been lost in transit) this system has
the potential for creating a FACTOR FOUR attack bandwidth multiplication.
(If the receiving target system can get a "RST" packet back to the
originating server, then further SYN/ACKs will be averted. However, since
many commercial enterprise-level firewalls simply drop unsolicited SYN/ACK
packets, the bandwidth multiplication effect tends to rein.)

Therefore, the use of intermediate TCP servers has the effect of
quadrupling the original input bandwidth ... which they are innocently
aiming at the attack target.

(Note that the fact that we were also seeing some RST/ACK packets amid the
SYN/ACK flood tells us that the malicious hacker's IP and Server Port list
is not perfect. Those RST/ASK packets were generated in response to a
spoofed source IP SYN packet hitting a non-open service port.)

However, the list of IPs and Ports was certainly accurate enough, and it is
trivial and maintain such a list.  Every "TraceRoute" across the Internet
yields a list of intermediate, probably very-well-connected routers, and
any large well-known web server farm -- such as YAHOO.COM which was used
here -- provides plenty of web servers for the "packet bounce".

Any site lacking extremely comprehensive and unusually tight upstream
filtering will probably be placed into a sustained Denial of Service by the
flood of SYN/ACK and RST/ACK traffic pouring into it from many hundreds of
well-connected public servers.

This attack will NOT appear on the "radar screens" of the organizations
monitoring traditional spoofed source IP SYN floods and other similar
attacks.  There will be no monitorable "backscatter" from this attack since
every IP used is for that of a valid machine.

For all of these reasons, this new style of attack creates a very potent,
highly effective, difficult to filter distributed denial of service.

-------------------------------------------------------------------
Defending against the "Packet Bounce" DDoS attack:
-------------------------------------------------------------------

The trick to defending against this attack is recognizing the very
different roles of an Internet server and client.  Since TCP packets are
bouncing off an Internet server, the originator and final destination of
the traffic are both presumed to be Internet clients.

In other words, the packets flooding our network necessarily had a SOURCE
port in the "service port" range from 1-1023, since they were "bouncing"
off servers handling and answering well known service ports (22, 23, 53, 80
and 179).  But our server, when it is acting as a server, NEVER NEEDS to
receive packets with a source port less than 1024.  Internet clients which
access our web, news, pop, and other servers always generate packets with
source ports in the client range, typically 1024 to 5000 or more.

Only when our server is acting as a client -- when sending eMail via SMTP
to a remote port 25, or perhaps when querying external DNS servers, is
there any need to receive traffic with source ports in the "service port"
range. Specific rules can usually be created to safely allow traffic in
those special cases from specific authorized IPs, while denying ALL other
traffic with source ports less than 1024.

After adding these rules to our upstream router all, malicious "Packet
Bounce" attack traffic was filtered and we returned to the Internet with a
completely clean connection.

______________________________________________________________________
Steve.
=================================================================
--
Dave Dittrich                           Computing & Communications
dittrich () cac washington edu             University Computing Services
http://staff.washington.edu/dittrich    University of Washington

PGP key      http://staff.washington.edu/dittrich/pgpkey.txt
Fingerprint  FE 97 0C 57 08 43 F3 EB 49 A1 0C D0 8E 0C D0 BE C8 38 CC B5


----------------------------------------------------------------------------
This list is provided by the SecurityFocus ARIS analyzer service.
For more information on this free incident handling, management 
and tracking system please see: http://aris.securityfocus.com