Re: Snort Anomaly

[Kevin Ross <kevross33 () googlemail com>]

Hi,

Yes Mr Smith security onion is great, I built my setup from scratch but
that was mostly for the learning potential and keeping things minimal.
Security onion can get you started really quick. I think they key thing
about detecting anomalies (and the unknown) is actually gather as much data
as you can. While this is obviously useful for analysis following detection
of a breach to have all these things as the vast majority of traffic in an
organization is legitimate if you have ways to analyse the data at a high
level the threats can come to the surface. It is all about collecting the
right data (HTTP and DNS are good ones to collect and analyse) and once you
have enough legit stuff the volume in distances to the malicious  Two
examples of this based on my stuff above is:

1) While it is getting rarer malware sometimes uses odd HTTP useragents
when communicating. However you can assume a few machines infected with
malware, the malware will make a lot less requests vs all the other traffic
in your network (especially on a large network). Now using ELSA you can
basically do a search when you search for all HTTP traffic in a time
period, group it by HTTP user agent and then if need be increase the limit
to show more than 100 results (i.e limit:1000). This will give you a big
list of each unique user agent and how many times it was seen.

Based on this you can just basically scroll to the bottom of that list
assuming anything with significant numbers of use will be legit and look at
the least used user agents. While there is odd looking stuff not used much
but legit you will quickly be able to click on them and few the connections
to see what they are so you will learn quick. Then the unknown stands out
to you. There are obvious ones where it is something completely odd and it
is malware but even the legit looking ones can be bad. In one case there
was a Windows looking useragent that looked like most of the others except
having lots of requests it was near the bottom, looking at the requests
revealed them to be suspicious. It was a typo of a space in the wrong place
so while it looked legit if you saw it in traffic you might not have looked
twice but because it stood out as unique among all the other legit traffic
it led to a detection.

2) Another case is the PassiveDNS tool I mentioned. While I have only been
running this for about 4 months now the data is excellent. Now after 4
months generally I decided most day-to-day Domain to IP mappings have been
seen that occur. So I automatically run a query at the end of each day
basically which looks for all domains where their first seen date is that
day, then I negate it a bit to remove some FPs and I am left with a list of
daily interesting domains. While this is quite simple and nothing compared
to the scoring, reputation & analysis that is potentially possible it is a
start and while I am only beginning to use this data I have seen it
identify exploit kit domains, malware domains etc seen that day in other
logs so it seems promising although tuning and some automatic tying
together of other logs may be more useful. While also there is FPs and
likely False negatives in this approach the value of it as data is good and
it is an idea in progress. Also other things like Alexa top websites and
other whitelists could be used to filter down FP cases to make it more
useful.

Why these ways work and is useful and done by detecting anomalies in larger
amounts of data based on that idea most stuff is useful and by the time you
have enough data about what is known it tends to dwarf that which is
not.Hope that gives you some ideas for anomaly detection (although really
it is more data analysis). Really both these things are simple but it is
about asking the right question and you can end up with some interesting
results. And while not released yet I am hoping
http://www.amazon.co.uk/Data-Driven-Security-Visualization-Dashboards/dp/1118793722/ref=pd_sim_sbs_b_1/276-1335909-1181044and
http://www.amazon.co.uk/Network-Security-Through-Data-Analysis/dp/1449357903/ref=sr_1_2?ie=UTF8&qid=1381098350&sr=8-2&keywords=million+logsmay
give more more skills and ideas for this way of dealing with security
analysis.

Kind Regards,
Kevin Ross


On 9 January 2014 12:21, Doug Burks <doug.burks () gmail com> wrote:

> Hi Mr Smith,
>
> Kevin provided some great recommendations and you can have many of
> them up and running in about 15 minutes with Security Onion:
> http://www.securityonion.net/
>
> Security Onion gives you the following:
> - Snort and Bro (with PF_RING)
> - ELSA
> - Full packet capture
> - OSSEC HIDS
> (and much more!)
>
> We released an update yesterday that especially helps in finding the
> anomalies in your network:
>
> http://blog.securityonion.net/2014/01/new-securityonion-web-page-package.html
>
> Hope that helps!
>
> On Wed, Jan 8, 2014 at 11:00 AM, Kevin Ross <kevross33 () googlemail com>
> wrote:
> > It depends what you mean by anomaly. These days "anomaly" to me means odd
> > HTTP communications, useragents, geolocation patterns, traffic anomalies
> > like bad fields for DNS or hosts talking on protocols they shouldn't be
> like
> > non-DNS servers trying to contact external DNS etc. To be more capable of
> > detecting these things and other anomalies I suggest taking a network
> > security monitoring approach with multiple levels of tools. This means
> > collecting various data from IDS, network etc and applying detection to
> it.
> > An excellent recently released book on this is this which while I am not
> too
> > far into it the book is truly excellent; especially as it covers snort,
> > anomaly detection, BRO (which very nicely complements things like Snort).
> http://www.amazon.co.uk/Applied-Network-Security-Monitoring-Collection/dp/0124172083/ref=sr_1_1?ie=UTF8&qid=1389194990&sr=8-1&keywords=applied+network+security+monitoring
> > Obviously though you don't need a book to learn this as you can read
> > documentation on each of these bits. To get to a good detection level I
> > would suggest looking into the following things:
> > - Make sure you have Snort tuned so you aren't overwhelmed and the rules
> and
> > preprocessors are setup as you want them. Read the Snort documentation on
> > this, a lot of rules and preprocessor settings will highlight traffic
> > anomalies anyway.
> >
> > - Install BRO http://www.bro.org/. It can detect other anomalies and
> also
> > generates very detailed logs on HTTP traffic, file hashes, tunnels, DNS,
> > other protocols that will complement any alerts you get from Snort etc. I
> > then feed those logs and IDS logs and things into ELSA
> > http://code.google.com/p/enterprise-log-search-and-archive/ which
> allows me
> > to do querying on all events surrounding a snort alert and also a lot of
> > hunting (i.e show me all unique useragents in my traffic and it will
> count
> > them up and display that, show me all executables from certain countries
> > etc). With snort I also have Snorby setup and full packet capture with
> > openfpc so it can be queried easily from Snorby from alerts. It can also
> > extract files from the network (which Snort 2.9.6 can do too) but the
> > advantage is also hashing of all files in protocols. So executables, HTML
> > pages, Java files, PDFs everything is getting hashed so even if you don't
> > have a file you can search for the hashes on things like Virustotal.
> >
> > - Setup full packet capture solution like OpenFPC, Moloch or StreamDB (I
> use
> > OpenFPC due to it being integrated into Snorby and it is less intense
> than
> > say Moloch which indexes network traffic for my sensors). This allows
> you to
> > analyse the traffic in depth depending how far you can go back (1 day
> min 3
> > days ideal but you may find it is only hours. Still some FPC for as long
> as
> > your disk space allows (and you can ignore hosts, protocols etc with BPF
> > filters to increase that time) is better than none.
> >
> > - Other types of anomaly detection can be implemented in other things
> such
> > as if you have a SIEM with your firewall logs going into it if you
> create a
> > correlation rule for high port numbers (above 1024 but not well known
> high
> > port numbers like SIP ports etc) and then log for UDP and TCP firewall
> > denies for so many in a certain time like a minute period you will
> actually
> > pick out P2P protocols with no knowedge of the protocol itself. I.e Using
> > this logic and some negation for my enviroment I reliably have detected
> > (although it may not have been the only alert) BitTorrent Traffic, Zeus
> > trojan P2P protocol and other protocols for malware etc. This will be
> very
> > useful as P2P is used increasingly in malware families.
> >
> > - Another good thing is PassiveDNS ideas which you can get going with
> > https://github.com/gamelinux/passivedns. Just logging in with NXDOMAINs
> into
> > a database with the web interface is good and for instance you can
> create a
> > lookup in Snorby so that when you have an IDS alert you can quickly
> lookup
> > the IP in your PassiveDNS database for domains which can very quickly
> help
> > you determine a false positive or a true positive and even when the
> incident
> > first appeared. I.e I have had alerts for exploit kits but through DNS
> for
> > the other names resolved to the IP I have found previously used domains
> and
> > when they were seen and am then able to look back and other logs at those
> > times. Also using regular expressions, blacklists and other methods in
> SIEM
> > for NXDOMAINs for instance I can detect malicious or suspect domains: i.e
> > alerts for domain generation algorithm domains
> > (https://blog.damballa.com/archives/1504), bad domains, supect domains
> such
> > as each day I extract with a script all new domains queried (and also
> cases
> > where new IPs mapped to a name) that day and then with some negation and
> > other things. The logic being if that is the first time ever it has
> appeared
> > within your enterprise and it looks kind of suspicious it just might be.
> >
> > While no one thing here is a silver bullet the combination of all the
> > combined tools and methods is basically provided lots of ability to
> detect
> > intrusions, properly analyse them, hunt for the unknown, detect anomalies
> > etc. With this you will end up with:
> >
> > - Snort alerting you to all kinds of intrusions and anomalies. For
> anomalies
> > though protocol rules and the preprocessors which you can read about in
> the
> > documentation is where you should look.
> > - BRO IDS providing detailed logging and if fed into something like ELSA,
> > SPLUNK, Logstash etc analytics. Also actual on disk BRO logs compress to
> > very little space automatically so essentially you have a historical
> record
> > of all flows, IRC chats, FTP traffic, HTTP records, file hashes and so on
> > for a long time of perhaps many months or even years.
> > - Full packet capture. Useful for short term but high detail analysis
> > - File extraction for analysis if you implement in BRO/Snort. You can
> then
> > do other analysis like running tools on them, checking the file hashes on
> > Virustotal frmo BRO etc
> > - PassiveDNS will allow you to analyse URLs and IPs for their
> relationships
> > and it will provide a long term historical analysis (i.e a partner
> > organsiation says they have malware which talks to badguys.com. Have you
> > been hit? You can go to that, type it in and if you get results you will
> > have a first and last time to begin hunting through other logs and BRO
> would
> > have even more detail. Also with regex you can detect all kind of
> anomalies
> > and if you look at research like  http://labs.umbrella.com/
> > http://www.lastline.com/papers/dns.pdf and
> > https://www.damballa.com/damballa-labs/publications.php you might get
> more
> > ideas on things in DNS to look for to detect malicious activity (or
> simply
> > feeding in blacklists of known bad ones).
> >
> > Hope that helps,
> > Kevin
> >
> >
> > On 7 January 2014 18:38, Mr Smith <engineer.demo2020 () gmail com> wrote:
> > > Hi
> > > I Have a question about Snort:
> > > What is the best solution to improve Snort performance in terms of
> > > "Anomaly Detection" Capability?
> > > What is the best solution to add "Anomaly Detection" capability into
> > > Snort?
> > > 1. Using a Host-Based IDS(like what?) in conjunction with Snort(NIDS)?
> > > 2. Adding anomaly based plugins(like what) into Snort?
> > > 3....?
> > >
> > > Thanks
> ------------------------------------------------------------------------------
> > > Rapidly troubleshoot problems before they affect your business. Most IT
> > > organizations don't have a clear picture of how application performance
> > > affects their revenue. With AppDynamics, you get 100% visibility into
> your
> > > Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
> AppDynamics
> > > Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
> > > _______________________________________________
> > > Snort-users mailing list
> > > Snort-users () lists sourceforge net
> > > Go to this URL to change user options or unsubscribe:
> > > https://lists.sourceforge.net/lists/listinfo/snort-users
> > > Snort-users list archive:
> > > http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
> > >
> > > Please visit http://blog.snort.org to stay current on all the latest
> Snort
> > > news!
> ------------------------------------------------------------------------------
> > Rapidly troubleshoot problems before they affect your business. Most IT
> > organizations don't have a clear picture of how application performance
> > affects their revenue. With AppDynamics, you get 100% visibility into
> your
> > Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
> > Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Snort-users mailing list
> > Snort-users () lists sourceforge net
> > Go to this URL to change user options or unsubscribe:
> > https://lists.sourceforge.net/lists/listinfo/snort-users
> > Snort-users list archive:
> > http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
> >
> > Please visit http://blog.snort.org to stay current on all the latest
> Snort
> > news!
>
>
>
> --
> Doug Burks
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today. 
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!