Re: Open source Netflow analysis for monitoring AS-to-AS traffic

[Brian Knight via NANOG <nanog () nanog org>]

Thanks to all who took the time to comment and make suggestions. 

To summarize the private messages, one respondent suggested Argus as a
collector. Another mentioned that they are still using AS-Stats. 

I'm drawn to Akvorado. I like the self-contained nature of the
application. NF collector, database, and modern web GUI are all bundled
in one docker container. The full-featured demo [5] is fantastic. That
the app can enrich the Netflow data with BMP is an added bonus. 

The best part is, the GUI has the report viz I need, and it is actually
the default visualization in the demo. It also has the graph types that
I didn't know I needed, like the Sankey graph. 

FlowViewer looks interesting as well. I suspect getting the reports
right may take some time, given the amount of GUI filtering options. 

pmacct and Argus seem to be capable tools that have been around for a
long time, but I haven't seen a concise stack building guide to get
Netflow data into a good GUI using these. Looks like there are some
older Docker images available for both. I could write my own SQL or roll
my own stack, but I'd much rather spend my time on other things. 

I appreciate the conversation around sFlow. I actually wasn't aware that
XR supported it. AS path probably doesn't add a whole lot of value given
that I'm focused on flows across our IP transit circuits. I'm able to
determine my next AS hop simply by looking at the flow's associated
tuple of (flow exporter, interface). I can use other tools like
RouteViews or RIPE's RIS to determine the destination AS's upstreams if
needed. The rest of the path is probably not too helpful for determining
peering opportunities. 

I think I'm going to get Akvorado running in my environment. If that
doesn't pan out, I'll likely go back to AS-Stats. 

Can those running Akvorado comment on their system specs? The only spec
I've seen is a mention in this blog post [6]: "Akvorado is performant
enough to handle 100 000 flows per second with 64 GB of RAM and 24 vCPU.
With 2 TB of disk, you should expect to keep data for a few years." 

Thanks again all, 

-Brian 

On 2024-03-26 19:04, Brian Knight via NANOG wrote:

> What's presently the most commonly used open source toolset for monitoring AS-to-AS traffic?
>
> I want to see with which ASes I am exchanging the most traffic across my transits and IX links. I want to look for 
> opportunities to peer so I can better sell expansion of peering to upper management. 
>
> Our routers are mostly $VENDOR_C_XR so Netflow support is key.
>
> In the past, I've used AS-Stats [1] for this purpose. However, it is particularly CPU and disk IO intensive. Also, it 
> has not been actively maintained since 2017.
>
> InfluxDB wants to sell me [2] on Telegraf + InfluxDB + Chronograf + Kapacitor, but I can't find any clear guide on 
> what hardware I would need for that, never mind how to set up the software. It does appear to have an open source 
> option, however. 
>
> pmacct seems to be good at gathering Netflow, but doesn't seem to analyze data. I don't see any concise howto guides 
> for setting this up for my purpose, however. 
>
> I'm aware Kentik does this very well, but I have no budget at the moment, my testing window is longer than the 30 day 
> trial, and we are not prepared to share our Netflow data with a third party. 
>
> Elastiflow [3] appears to have been open source [4] at one time in the past, but no longer. Since it too appears to 
> be hosted, I have the same objections as I do with Kentik above. 
>
> On-list and off-list replies are welcome. 
>
> Thanks, 
>
> -Brian

 

Links:
------
[1] https://github.com/manuelkasper/AS-Stats
[2] https://www.influxdata.com/what-are-netflow-and-sflow/
[3] https://www.elastiflow.com/
[4] https://github.com/robcowart/elastiflow?tab=readme-ov-file
[5] https://demo.akvorado.net/
[6] https://vincent.bernat.ch/en/blog/2022-akvorado-flow-collector