Re: Adding pcap-ng pipe support to dumpcap

[Anthony Coddington <Anthony.Coddington () endace com>]

On Aug 31, 2017, at 1:31 PM, Guy Harris <guy () alum mit edu> wrote:
>  On Aug 30, 2017, at 6:00 PM, Ed Beroset <beroset () mindspring com> wrote:
>  > There are some limitations.  Specifically, pipes don't allow random access, so any file formats that currently 
> require that would need to either be rewritten

>  Which, for at least one capture file format (Network Monitor format), would be impossible, as we don't define it, 
> Microsoft does (and they're probably not very amenable to changing it, not least because they've deprecated NetMon in 
> favor of Message Analyzer).  The only file formats *we* control to any degree are pcap and pcapng, neither of which 
> require random access in order to read them sequentially.

>  If we make extcap programs work like dumpcap, the only pipe involved is the control pipe between *shark and the 
> program - the packets are written directly to a file - but that wouldn't make any difference for those file formats, 
> as you can't, for example, read a NetMon file until it's *completely* written, with a frame table, and you can't do 
> that until you've written all the packets to it.

Wtap filetypes could indicate that they do/do not support sequential only access with a flag in dump_open_table. There 
is already a flag for writing_must_seek. I'm sure at some point I saw some support in wtap internals for trying to read 
from pipes/stdin without random seeking support? If necessary wtap could be extended to have wtap filetypes return a 
'need more data' rather than WTAP_ERR_SHORT_READ or blocking if they don't have the rest of the packet but have not 
reached EOF. There is also nothing stopping having an interface for extcap that requires dumping to an intermediate 
file for files that require seeking.

Also note that with the extcap toolbar support there is now optionally already a control pipe to and from extcap 
applications. One option would be to extend and use that interface to indicate when packets are available. In dumpcap 
as well ideally (replacing the capchild control pipe). Not sure if it is feasible to suggest extcap applications flush 
the output pipe only on packet boundaries?

Regards,
Anthony


From: Wireshark-dev <wireshark-dev-bounces () wireshark org> on behalf of Guy Harris <guy () alum mit edu>
Sent: Thursday, 31 August 2017 1:31 p.m.
To: Developer support list for Wireshark
Subject: Re: [Wireshark-dev] Adding pcap-ng pipe support to dumpcap
    
On Aug 30, 2017, at 6:00 PM, Ed Beroset <beroset () mindspring com> wrote:

> One problem is that as dumpcap is currently written, it treats files and pipes very differently.

*Files* and pipes, or *capture devices* and pipes?

> but I can't help but think that the general approach you describe is the better long term strategy.

Probably.  It means that the interface between *shark and extcap programs would be different - but, while having extcap 
programs behave like dumpcap might complicate the extcap programs (although some of the code to do that could be in a 
library used by dumpcap  and by extcap programs), it might simplify the Wireshark capture code path.

> There are some limitations.  Specifically, pipes don't allow random access, so any file formats that currently 
> require that would need to either be rewritten

Which, for at least one capture file format (Network Monitor format), would be impossible, as we don't define it, 
Microsoft does (and they're probably not very amenable to changing it, not least because they've deprecated NetMon in 
favor of Message Analyzer).   The only file formats *we* control to any degree are pcap and pcapng, neither of which 
require random access in order to read them sequentially.

If we make extcap programs work like dumpcap, the only pipe involved is the control pipe between *shark and the program 
- the packets are written directly to a file - but that wouldn't make any difference for those file formats, as you 
can't, for example, read  a NetMon file until it's *completely* written, with a frame table, and you can't do that 
until you've written all the packets to it.

However, I suspect Stephen is thinking of ERF format, which should be writable purely sequentially, so it shouldn't be 
a problem, whether you're writing to a pipe or to a file that's being read incrementally.
___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
Archives:    https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
              mailto:wireshark-dev-request () wireshark org?subject=unsubscribe    
___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
Archives:    https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request () wireshark org?subject=unsubscribe