2006.06.06 NANOG-NOTES DDoS attack information collection

["Matthew Petach" <mpetach () netflight com>]

Information collection on DDoS attacks,
Anna Claiborne, Prolexic Technologies.
[slides are at:
http://www.nanog.org/mtg-0606/pdf/anna-claiborne.pdf

DDoS mitigation service.
personal experience mitigating over 150 DDoS
attacks.

Popular topic, but nobody talks about how you
can defend yourself or take legal action;
only thing you can do is collect information.

0.1% of DDoS attacks end in an arrest, that's
out of the reported number to the US Secret
Service, and that's out of the ones that fall
into their jurisdiction.

These are real losses:
A major US corp lost over $2mil in a 20 hour
outage
An offshore gambling comp. lost estimated $4m
in 3 days
Online payment processor lost $400,000 in 72 hours
online retailer lost $20K/day over 3 weeks.

These are directly reported losses; doesn't include
lost PR, etc.

Canadian retailer spend 50K on hardware mitigation,
they got kicked out of 3 datacenters due to the DDoS
attacks, spent 20K on IT and security consultants,
and another $6K on a different mitigation that also
failed.

Basic Information Collection
Get packet captures--either from machine being
attacked, or a span port, or from upstream
device,
tcpdump -n -s0 -C
(get full length of raw packet, limit pcap file
to 5MB or smaller)
take 3 or 4 over 15 minutes, to start, and then
repeat every hour
Determine the type of attack and duration (ex SYN
flood lasting 6 hours)
Obtain as complete a list as possible of source IP
addresses
Save bandwidth graphs, flow data, pps graphs, any and
all visual material relating to the attack
Save any contact with the attacker, email, chat
conversation, phone calls, etc.
Get loss figures from management--downtime, per hour
losses, per day losses, section 18 of some law, have
to substantiate losses over $5k before you can take
legal action against someone.

Recommendations
have a plan!  DDoS is stressful
Put all attack information in a central location
God monitoring doesn't have to be expensive, a simple
fiber card in a 1u box can be a mirror port for a
large volume of traffic
Don't have to have expensive hardware like arbor
 boxes.
 Limit to 100mb to prevent killing your capture box.
Graphs and flow data can be retrieved from upstream

Find the source
Use list of source addresses, find a reputable hosting
company, you may even see a friend's IP
Approach the network with the infected machine, give them
as much information as possible, it can take time
finding someone willing to help
Obtaining information is dependent on who you are dealing
with, be as helpful as possible.
Get information from the infected machine netstat,
tcpdumps, who is logged in, web logs, access logs
Get and save the source code responsible

process can take hours to weeks--prolexic has huge
contact list, and even for them can be really
difficult
And SAVE all your information to a central location!
and back it up!

Examine the source code
scripts are best, you know exactly what's going on
compiled code, run strings on it
best case, you can get a name or identification for who
wrote it, passwords, domain names, port usage
worst case you can obtain information that doesn't make
sense...yet
(it may fit into a bigger context later)

Locate controlling server
Examine TCP connection table or source code to find
the controlling server
verify your information, scan or connect to the suspect
machine
contact abuse where the server is hosted, explain the
situation
have as much information possible to verify your
conclusion and validate your identity
Good luck, most abuse contacts are less than helpful
Raises a good question: how to improve awareness and
legitimate requests answered.
(may be able to get FBI to provide warrants to seize
machines that are being used to control attacks against
you, but takes time and documentation)

Hunting the attacker (not for the faint of heart!)
Review all information gathered so far on the attack
contact the attacker, establish a report
save all information and/or conversations (important
note, if conversations aren't on a public server,
they can't be used)
Piecing the information together to form a high level
view of the exploit, attack, and attacker
A long process, most attackers are highly motivated
and skilled, you usuallly have to wait for them to
slip up!

Resources:
local FBI field office department of cybercrime
department of homeland security
CERT
Cymru--great guys, if they have to help you
NHTCU--EU, cyber crime divisions in local offices
Local US secret service--division of electronic crimes
DDoSDB.org -- under development at the moment.
 how to identify/recognize different types of attacks
 may be able to put their attack database open to the
  public up there.

A success story
The tracking of x3m1st/eXe
responsible for hundreds of extortion based DDoS
attacks
tracked for months
eventually lead to his arrest.

hid behind four levels of compromised servers.

eXe and his group only talked on private IRC
servers; made the mistake of connecting from
his home domain, from a machine registered to
his real name; that was his slip up, Ivan
arrested in Russia.

Tracking Pkeglhema/aaabaa
targetted redhat linux boxes for his zombies
they generally sat on higher bandwidth links.
PHP/cross scripting vulnerability; insert the
script without validity checking.
Used cpanel holes, mySQL holes, he browsed
zeroday, modified code in a few hours to use
new holes,

The result: synflood over 10G, knocked upstreams
off, and got them null routed, bunch of outbound
networks also null routed.

some conversations recorded, he was paid by an
employer, he'd done this before for other employers.

He eventually got away.
English as a second language, always from hacked
university,
attacking six other sites that also sold similar
items as the client under protection.
They'd had phone calls from competitors trying to
push them out of business, and was during the
busiest time of year for them.

He was most professional attacker she's dealt with,
he never slipped up, he'd been doing this for years.
Logged in from China or Japan.

She turned over info to FBI, let them pursue things
further.


Matters to address in community
Better abuse contacts, specific to DDoS
Centralized repository specifically for DDoS profiling
Information gathering is extremely resource intensive,
but worth it.
Null routing IP space is not a good idea from either
side
DDDoS is everyone's problem.

fix your open recursive DNS servers!!

NHTCU--Mike Hughes, rolled into SOCA, serious
organized crimes something--DDoS is way down on the list,
they're more into big crimes.  Watch for more
developments in that space though.
NHTCU was more approachable,

Q: Bill Woodcock--could she talk more about public vs
private IRC servers---what is the legal issue?
A: private IRC server is any run that is not publically
accessible, is only open to the group.
Any machine that is hacked is a private IRC server, since
it is not intended for public access.
public--a machine run so that anyone can connect to
it, and intended as such.
You can assert the conversation, but it is hearsay;
it can help in court, but it is itself not admissable
as evidence.

Q: Tony Kapella, 5nines--what does Prolexic suggest
customers do to make sure their host hasn't been
compromised to the point where netstat and other
utilities are affected?
A: Well, you have to trust the people you work with
to be able to verify that the information they're
seeing is accurate.  But for boxes that neither side
has access to, like colocation boxes, you could just
be out of luck.

Q: Gene Kim? what if the server is located outside the
US?
A: fine as long as it's publically accessible.
Q: What about private messages?
A: fine as long as it's a public IRC server

Q: Louis Lee, equinix--suggest mirror switch port to
address Tony's issue; capture unaffected traffic
to a virgin machine when possible.

Q: Rob seastrom, bluetrust--what is the incidence of
encrypted communication, and multiple C&C hosts?
A: This only works for easiest case scenario of
non-spoofed attack with centralized C&C attack.
Peer to peer, proxy servers, etc. you need to go
to an expert.

Q: Stuart Phillips, New Metra...he's cut off...raise
it at the security BOF.

Announcements--if you've not picked up your shirt,
pick it up, JD Frazer, userfriendly did the image.

A few short of goal of six for lightning talks;
sign up, or we'll have Randy sing at you.

PGP signing during this break too.

Be back at 10 after.