Re: Making my own IDS... how to detect packet loss ?

[Martin Roesch <roesch () sourcefire com>]

You should be calculating the next expected ack number based on the
packets that you observe coming from a transmitter.  If the returning
ack numbers are higher than the calculated next expected ack you've
missed something (barring spoofing, of course).  Once you've missed
something in a state-machine-based traffic analyzer you have a couple
choices:

1) If you're tracking flows you can mark the flow as desynchronized
and ignore it until it times out or goes through a shutdown sequence.

2) You can try to resynchronize on the protocol based on known
sequences that are intrinsic and recognizable to that protocol.

3) You can just log everything (the packets) and post-process the data
if you think that it could be important.

If you get packets out of order the acks will still come in order
(assuming no drops), your job will be to queue them until the point
where you're confident you can do a proper reassembly and processing
pass.  The acks tell the story and if you're in purely passive-mode
they're your sole indicator of what each end-point has actually seen.
If the packets are coming out of order and there are large windows
involved (and even when there are not) you're going to have to have
fail-safe strategies for saving yourself from memory exhaustion and
computational complexity attacks against your engine.

TCP stream analyzers in pure-passive technologies are complex animals,
don't be surprised at the complexity required to do it properly.

Marty




On Thu, Jul 16, 2009 at 2:49 PM, Francis,
Shannon<Shannon.Francis () jetblue com> wrote:
> I actually think that's one of the shortcomings of TCP. I could be wrong but I don't think there is a way to tell if 
> a packet has been dropped if they are sent out of order, because the whole reason it's a TCP connection is that it's 
> error-checking, meaning it will wait until either it has received all packets and reassembled them or the connection 
> times out. Please correct me if I'm wrong though :-D.
>
> Regards,
>
> Shannon Francis
>
> -----Original Message-----
> From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Jonathon
> Sent: Thursday, July 16, 2009 1:44 PM
> To: focus-ids () securityfocus com
> Subject: Re: Making my own IDS... how to detect packet loss ?
>
> Thanks all for your replies.  I think the general consensus was to
> look at the ACKs.  However the problem with that solution is that I
> could get packets in any order.  So, for example, I could receive an
> ACK for packet N, which doesn't necessarily mean that I've dropped
> packet N.  It could possibly mean that packet N has not come to me.
> If I did get the packets though in order, that would definitely make
> things much easier :)
>
> Thanks
>
> J
>
>
> On Thu, Jul 16, 2009 at 9:15 AM, Aaron Turner<synfinatic () gmail com> wrote:
> > On Wed, Jul 15, 2009 at 6:09 PM, Jonathon<thejunkjon () gmail com> wrote:
> > > I am writing my own IDS.  I have a packet sniffer + TCP reassembler
> > > that I've written.  For each stream I detect, I have to keep some
> > > state + a buffer of the packets for the stream.  However, a problem
> > > that I've run into is that sometimes packets could get lost (meaning I
> > > do not capture them, but they were actually sent/received between two
> > > hosts on the network).  If I do drop the packets, my current
> > > implementation gets all messed up since I never end up freeing the
> > > resources I've allocated for that stream.  So, lost packets seem to be
> > > the bain of my current implementation.
> > >
> > > My question is (assuming all TCP streams), how can I determine whether
> > > a packet is lost by just looking at packets that I currently have in
> > > my buffer?
> > >
> > > The packets could come in any order so just because I receive a packet
> > > with seq #N, doesn't mean that a packet with seq #N-1 was sent.
> > >
> > > I hope this is the appropriate mailing list for my question.
> >
> > If you track the values of the Ack's in packets then you'll know when
> > you've dropped one or more TCP segments which were received by the
> > other end.
> >
> > Example:
> >
> > Assume server sends 3 packets with Sequence numbers of 10, 50 and 100.
> >  But you only see 10 & 100.  If the client only Ack's 10, then you
> > know it didn't see 50 either.  But if it Ack's 100 then you know it
> > saw data you missed.
> >
> > --
> > Aaron Turner
> > http://synfin.net/
> > http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
> > Those who would give up essential Liberty, to purchase a little temporary
> > Safety, deserve neither Liberty nor Safety.
> >    -- Benjamin Franklin
> >
> > -----------------------------------------------------------------
> > Securing Your Online Data Transfer with SSL.
> > A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL 
> > certificate on your web server, you can securely collect sensitive information online, and increase business by 
> > giving your customers confidence that their transactions are safe.
> > http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194
>
> -----------------------------------------------------------------
> Securing Your Online Data Transfer with SSL.
> A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL 
> certificate on your web server, you can securely collect sensitive information online, and increase business by 
> giving your customers confidence that their transactions are safe.
> http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194
>
>
>
> -----------------------------------------------------------------
> Securing Your Online Data Transfer with SSL.
> A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL 
> certificate on your web server, you can securely collect sensitive information online, and increase business by 
> giving your customers confidence that their transactions are safe.
> http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194



-- 
Martin Roesch - Founder/CTO, Sourcefire Inc. - +1-410-290-1616
Sourcefire - Security for the Real World - http://www.sourcefire.com
Snort: Open Source IDP - http://www.snort.org

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate 
on your web server, you can securely collect sensitive information online, and increase business by giving your 
customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a17f194