Re: [the-tcpdump-group/tcpdump] Color support: WIP with code+screenshots (#686)

[Ryan Doyle <ryan () doylenet net>]

Hi Michael

On Thu, 21 Jun 2018, at 12:14, Michael Richardson wrote:
> Ryan Doyle <notifications () github com> wrote:
>     > Anyhow, this is the email I sent. I'm keen to open up a discussion!
>
> I never saw the email on the list...
>
>     > 
> -----------------------------------------------------------------------------
>
>     > Hi there
>
>     > I have started to implement color/colour support in tcpdump and 
> wanted to get
>     > some feedback before I continue. It was mentioned in
>     > http://seclists.org/tcpdump/2012/q2/25 a few years back but can't 
> find
>     > mentions since. Is this still something that is wanted in tcpdump?
>
>     > COLOR IMPLEMENTATION
>
>     > I have used ANSI escape codes for coloring and have used the 4bit pallet,
>     > having "bright/bold" as well.
>
> Did you use them directly, or are you using ncurses?

I used them directly and printed them to the screen.

>     > APPLYING COLORS
>
>     > I have created two helper macros like the ND_PRINT(...) macro. These are
>     > ND_PRINT_CATEGORY(category, ...) which will print the ANSI escape code,
>     > printf and then reset the terminal back to its normal color.
>
> So, we are doing some rework for tcpdump 5 with some of this stuff.
>
>     > Also, there is a ND_PAINT_CATEGORY_START(category) and 
> ND_PAINT_CATEGORY_END
>     > macros that set the ANSI escape code so existing code can be 
> "painted" in a
>     > color without modifying all the ND_PRINT usages to 
> ND_PRINT_CATEGORY. I'm
>     > still unsure if this is a good idea.
>
> I think not... My preference would be go in this direction.
>
> Instead, I'd prefer to emit HTML marked up output (or maybe JSON or CBOR),
> and then have a program that would read the stream and colourize it using
> some kind of CSS.

I think this could be a good way to go too if it can be retrofitted in the codebase. It's probably less about the 
actual format but building up the semantic meaning of a packet.

We could build XML dynamically by changing ND_PRINT to store the string contents instead of printing it directly and 
then actually print once we've reached the end of the packet. 

Using this line as as example: 

18:35:59.376658 IP localhost.43144 > localhost.domain: 20972+ [1au] SSHFP? monadic.cynic.net. (46). 

Below lists how this could be implemented.

Step1. Wrap all ND_PRINT's in <raw>...</raw> so _all_ output is XMLified. We would build this string (or use an XML 
library) and then print it once we've reached the end of the packet. This first step introduces XML and simple printing 
of it. There would be a simple "printer" that only knows how to unpack and print <raw> tags. End users shouldn't notice 
any difference. 
 
EG:
<raw>18:35:59.376658</raw><raw> IP</raw><raw> localhost.43144 > localhost.domain</raw> etc...

Step2. Expand by adding meaning to XML. The printer could then start to know what a <packet> is and what it's 
attributes mean. At this point we could start adding color to these.
<packet num=1 timestamp="16:25:21.713467">
  <raw>IP</raw> <raw>localhost.43144 > localhost.domain:</raw> <raw>20972+</raw>...
</packet>
 
Step3. Expand further, protocol by protocol adding printers that know about the contents of the XML. Also, IMO any 
dropping of printed information (eg, depending on the -vvvv flags) should be done when parsing the XML, not when 
generating it. The parser/printer would be the one that knows about -v flags and or timestamp options.
<packet num=1 timestamp="16:25:21.713467">
  <protocol name="ip">
    <srcIP>10.0.0.1</srcIP>
    <destIP>10.0.0.2</destIP>
    <raw>something not done via new method yet</raw>
    ...
  </protocol>
  <raw>localhost.43144 > localhost.domain: 20972+ [1au] SSHFP? monadic.cynic.net. (46)</raw>
</packet>

Step 4. Eventually remove _all_ uses of ND_PRINT
<packet num=1 timestamp="16:25:21.713467">
  <protocol name="ip">
    <srcIP>10.0.0.1</srcIP>
    <destIP>10.0.0.2</destIP>
    ...
  </protocol>
  <protocol name="udp">
    <srcPort>53</srcPort>
    <destPort>43144</destPort>
    ...
  </protocol>
  <protocol name="dns">
    <query>SSHFP<>
    <host>monadic.cynic.net</host>
    ...
  </protocol>
</packet>

The print-$protocol.c files would then really just construct the XML. Then there would need to be a series of printers 
that can print the XML out to the screen. I think this method would allow the best incremental approach and give the 
most backwards compatibility.

Disclaimer though: I really don't know how easy/hard this would be to do. Also it might introduce a dependency on 
libxml or otherwise unless we were to parse XML ourselves or bring it into the tcpdump codebase.

>     > I've tried to not tie the idea of color to the printing and 
> instead defined
>     > different categories. At the moment I have:
>
> Conceptually, I like this very much.  Let's give semantic meaning to the
> different parts, and then, if colourization makes sense to the user, given
> the context they are in, do that.
>
>     > I've chosen colors that aim to work on both light and dark 
> terminals. I'm not
>     > sure if these colors have issues with people with color-blindness 
> so it would
>     > be good to validate that too.
>
> The right answer is to always set the foreground and background colours.
> Never assume white or black for background.
> There will be hundreds of problems with any colour scheme you pick.

I'm not sure if users would like their background color changed? I think there could be a set of colors that work well 
for dark/light but further down the line there would need to be a config file. If a user has trouble with the coloring, 
it could always be turned off.

>     > DEMO
>
>     > You can see the code at:
>
> I very much like the output, it's just going to be extremely invasive
> to add this everywhere.  Right now, we need to get a version out to deal with
> CVEs, and there are some substantial changes occuring which are hard to
> incrementally release without releasing the attacks as well.
>
> So my suggestion is to put this work on hold for a couple of months,
> and then let's see how we can make this work.

No worries.

> --
> ]               Never tell me the odds!                 | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works        | network architect  [
> ]     mcr () sandelman ca  http://www.sandelman.ca/        |   ruby on rails    [
>
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)
_______________________________________________
tcpdump-workers mailing list
tcpdump-workers () lists tcpdump org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers