[PATCH] Do not cut established TCP connections and VoIP calls with Tcpslice

[Sebastien Raveau <sebastien.raveau () epita fr>]

Hi list,


I sent this email weeks ago but it is still pending approval from the 
moderators because of its size, so I'm sending it again, this time without 
attachment. The patch itself is available at this address:
http://www.epita.fr/~raveau_s/tcpslice_follow_tcp_and_voip.patch

The following is lengthy, but you only have to read it if you have any 
questions about the patch ;)


Thanks

===========================================================

Half a year ago I needed a solution to extract slices from huge PCAP files
according to given temporal boundaries: for that Tcpslice proved to be an
excellent tool... but I also needed the TCP connections that were
established before the "end-time" to be kept in full instead of being brutally 
cut at the end of the destination PCAP file.

I was really surprised to find out that Tcpslice, despite its name, had no
support for the TCP protocol whatsoever, so I went and fixed that :) In the 
process, I also added support for the two most-used VoIP protocol families: 
SIP and H.323, so that Tcpslice does not brutally cut VoIP calls when 
end-time is reached.

As a result, it is now possible to do the following:
tcpslice -s tcp -w slice.pcap 2005y12m16d17h00m \
2005y12m16d18h30m capture.pcap

The file slice.pcap will then contain all packets from capture.pcap that were 
transmitted between 5pm and 6.30pm on December 16th 2005, plus the packets of 
TCP connections that were established before 6.30pm until the last one of 
these closes, say, at 7.12pm; so slice.pcap will really end at 7.12pm, but 
between 6.30pm and 7.12pm there will only be packets belonging to these TCP 
connections, and no other packets (UDP, ICMP, etc.. or even TCP packets from 
other/new connections).

You will probably see an output like that:
Session #335 (TCP) closed at 2005y12m16d18h30m12s829057u (active sessions
total: 7)
Session #333 (TCP) closed at 2005y12m16d18h34m03s406480u (active sessions
total: 6)
Session #336 (TCP) closed at 2005y12m16d18h39m11s647317u (active sessions
total: 5)
Session #334 (TCP) closed at 2005y12m16d18h49m40s236182u (active sessions
total: 4)
Session #306 (TCP) closed at 2005y12m16d19h57m23s325096u (active sessions
total: 3)
3 unclosed sessions (id, type, last, source, destination, bytes):
#314    TCP     2005y12m16d18h28m04s785102u         172.16.5.17:3639
84.96.146.199:80              1798
#313    TCP     2005y12m16d18h28m04s786416u         172.16.5.17:3638
84.96.146.199:80              1797
#108    TCP     2005y12m16d18h02m13s121889u         172.16.5.17:3422
66.218.77.68:80            168019

The first five lines tell you when the TCP connections going beyond end-time
terminated. As you can see in the last three lines, sometimes sessions (a 
generic term I use for both TCP connections and VoIP calls) do not close 
properly or go beyond the end of the original PCAP file; to prevent having to 
track too many of them (which happens quite often if you process a capture of 
peer2peer traffic: many connections just disappear without being properly 
closed), you can set a timeout of seconds after which idle sessions will be 
considered as closed, with the "-e" flag, used here to solve the problem 
above, with a timeout of 5 minutes:
tcpslice -e 300 -s tcp -w slice.pcap 2005y12m16d17h00m \
2005y12m16d18h30m capture.pcap

You can see the effect of that if you use the "-v" flag to show the openings 
and closings of all connections, from start-time to the end of processing:
[...]
Session #108 (TCP) closed at 2005y12m16d18h07m16s717756u (active sessions
total: 6)
[...]
Session #314 (TCP) closed at 2005y12m16d18h24m45s144423u (active sessions
total: 6)
Session #313 (TCP) closed at 2005y12m16d18h24m45s144423u (active sessions
total: 5)
[...]

As I said earlier, I also implemented VoIP support, so you can do:
tcpslice -vv -s sip,h323 -w /dev/null -f 'slice-%s-%.6u.pcap' \
2005y11m03d10h00m 2005y11m03d10h45m capture.pcap

This will track SIP and H.323 sessions, including the TCP connections 
initiated by these VoIP calls (like H.225.0-CS TCP connections triggered by 
H.225.0-RAS UDP datagrams) but not other TCP connections that might be in 
capture.pcap unless you specify "-s sip,h323,tcp".
Notice also that I specified /dev/null as the parameter to the "-w" flag... 
Why? Because I also added an option to extract each VoIP call or TCP 
connection into a separate capture file :D In this example it will create 
files with names that look like slice-h323-000105.pcap, 
slice-sip-000011.pcap, etc... But if you also want the normal destination 
file with all packets between 10am and 10.45am and the VoIP packets of the 
calls continuing past 10.45am, just specify a normal argument to "-w" instead 
of /dev/null :)
Using the "-f" flag is what triggers the splitting into separate files; the %s 
in the format string passed as argument to this flag will be replaced by the 
session name in lowercase (currently either "tcp", "sip" or "h323") and the 
%u by the session ID number (hence it is convenient to enforce padding with 
%.6u for example so that the files remain sorted in directory listings). Also 
notice that in this example I used the "-v" flag twice: if you use it only 
once you will see each VoIP call starting and ending (like in the TCP example 
with TCP connections), but if you specify it twice, you will also see the 
openings and closings of subsessions, that is, sessions that were triggered 
by these VoIP calls: H.225.0 Call Signaling as cited earlier, RTP and RTCP 
streams, etc. These subsessions will of course end up in the same PCAP file 
as their "parent" session.

Regarding the implementation:
I tried to modify tcpslice.c as little as possible and put everything in a new 
file called sessions.c; I commented it enough for everybody to understand my 
code I believe, even though there are still parts of it you might find weird, 
but that's because some behaviors in SIP and H.323 are just... weird.

And since SIP and H.323 are weird and complicated protocols, I used the 
Libosip2 and Libooh323c libraries in order to parse their packets instead of 
writing the parser myself, but that required hacking the libs anyway because 
they weren't designed to be used outside the normal client/server model. By 
"hacking" in this case I only mean reading their source code in full, 
understand it and see how I can manage to use it in a packet capture
context. However, in the case of Libnids (the library I used to track TCP
connections in a really stateful way, as opposed to what Mark Allman's 
Tcpsplit does with only a list of source IP & port, destination IP & port 
tuples) it required heavy modifications which Rafal Wojtczuk kindly agreed to 
release in a new Libnids version. I waited for a while before sending my 
Tcpslice patch here so that Linux & BSD distributions can bring Libnids-1.21 
to their package systems. Now Libnids-1.21 has been out since May 10th and 
it's reached Gentoo's portage soon after, but I don't know about the other 
distributions, so if yours doesn't have it you can always get it at:
http://libnids.sourceforge.net/

Libosip2 is available on every distro I tested on, but it's sometimes called
just Libosip like on Gentoo for example. Anyway, if you need, you can find it 
at: http://www.gnu.org/software/osip/

For Libooh323c it's less simple though: it's the only H.323 library I could
find written in C (otherwise using OpenH323 would have meant turning Tcpslice 
into a C++ application) and I apparently I installed it from source... Their 
website: http://www.obj-sys.com/open/index.shtml says that it's under the GPL 
license (so no worries about that) and that it is now part of Asterisk, so I 
suppose distros ship it with Asterisk packages like Gentoo's asterisk-oh323 I 
guess...

Now I know many people won't like the fact that this adds 3 dependancies to
the Tcpslice package, so - even though I'll never understand these people 
considering today's price of the gigabyte - I didn't make these dependancies 
mandatory: the ./configure script will check for the presence of each of 
these three libraries and will activate support for TCP tracking if Libnids 
1.21 was found, support for SIP tracking if Libosip2 _and_ Libnids 1.21 were 
found, and/or support for H.323 if Libooh323c _and_ Libnids 1.21 were found. 
Otherwise, if none of these three libraries were found, Tcpslice should 
remain unchanged compared to the current version.


Thank you for reading me til the end, sequentially ;)
and to those who will review my patch and hopefully merge it.

-- 
Sébastien Raveau
computer and network security student
head of the hawKeye network monitor project
http://hawkeye.sourceforge.net/

-
This is the tcpdump-workers list.
Visit https://lists.sandelman.ca/ to unsubscribe.