Re: Open source Netflow analysis for monitoring AS-to-AS traffic

[Nick Plunkett <nplunkett () cenic org>]

In the same vein, if you can get your devices exporting sFlow, or for
others reading that do have sFlow capable devices: the sFlow-RT team has
built ready to deploy, all in one docker containers using Grafana and
Prometheus that you can stand up within minutes to start visualizing and
easily querying/processing sFlow data from your routers, with no prior
experience with the underlying software needed.

https://blog.sflow.com/2023/07/deploy-real-time-network-dashboards.html
https://github.com/sflow-rt/prometheus-grafana

On Wed, Mar 27, 2024 at 12:00 PM Peter Phaal <peter.phaal () gmail com> wrote:

> Brian, you may want to see if your routers support sFlow (vendors have
> added the feature over the last few years).
>
> In particular, see if it includes support for the sFlow extended_gateway
> structure:
>
> /* Extended Gateway Data */
> /* opaque = flow_data; enterprise = 0; format = 1003 */
>
> struct extended_gateway {
>    next_hop nexthop;           /* Address of the border router that should
>                                   be used for the destination network */
>    unsigned int as;            /* Autonomous system number of router */
>    unsigned int src_as;        /* Autonomous system number of source */
>    unsigned int src_peer_as;   /* Autonomous system number of source peer
> */
>    as_path_type dst_as_path<>; /* Autonomous system path to the
> destination */
>    unsigned int communities<>; /* Communities associated with this route */
>    unsigned int localpref;     /* LocalPref associated with this route */
> }
>
> The dst_as_path field is particularly valuable since it allows you to see
> who your customers are peering with.
>
> While not a complete solution, you might want to take a look at sflowtool,
> https://github.com/sflow/sflowtool, to decode the sFlow records and
> convert them to JSON. It's not hard to write a Python script to calculate
> BGP peering metrics and push the results into a time series database
> (Prometheus, InfluxDB, etc) and build dashboards in Grafana. The following
> article gives a few examples:
>
> https://blog.sflow.com/2018/12/sflow-to-json.html
>
> On Tue, Mar 26, 2024 at 5:06 PM Brian Knight via NANOG <nanog () nanog org>
> wrote:
>
> > What's presently the most commonly used open source toolset for
> > monitoring AS-to-AS traffic?
> >
> > I want to see with which ASes I am exchanging the most traffic across my
> > transits and IX links. I want to look for opportunities to peer so I can
> > better sell expansion of peering to upper management.
> >
> > Our routers are mostly $VENDOR_C_XR so Netflow support is key.
> >
> > In the past, I've used AS-Stats
> > <https://github.com/manuelkasper/AS-Stats> for this purpose. However, it
> > is particularly CPU and disk IO intensive. Also, it has not been actively
> > maintained since 2017.
> >
> > InfluxDB wants to sell me
> > <https://www.influxdata.com/what-are-netflow-and-sflow/> on Telegraf +
> > InfluxDB + Chronograf + Kapacitor, but I can't find any clear guide on what
> > hardware I would need for that, never mind how to set up the software. It
> > does appear to have an open source option, however.
> >
> > pmacct seems to be good at gathering Netflow, but doesn't seem to analyze
> > data. I don't see any concise howto guides for setting this up for my
> > purpose, however.
> >
> > I'm aware Kentik does this very well, but I have no budget at the moment,
> > my testing window is longer than the 30 day trial, and we are not prepared
> > to share our Netflow data with a third party.
> >
> > Elastiflow <https://www.elastiflow.com/> appears to have been open source
> > <https://github.com/robcowart/elastiflow?tab=readme-ov-file> at one time
> > in the past, but no longer. Since it too appears to be hosted, I have the
> > same objections as I do with Kentik above.
> >
> > On-list and off-list replies are welcome.
> >
> > Thanks,
> >
> > -Brian