Re: Service Provider NetFlow Collectors

[Tim Raphael <raphael.timothy () gmail com>]

That’s a much better cardinality (AS based) but it’s not the general case. Even if you want per-prefix information I’d 
argue that Influx would still not handle the load (~700k ^ 2 cardinality). For limited tag-sets it would do the trick.

I never did attempt to push it to Influx with some foresight that it’d be suboptimal for my ultimate use cases. I 
wanted a solution that could handle a wide range of use cases without having to worry about limits on tag-sets.

I found Clickhouse able to do what I wanted in a performant way. 

- Tim

> On 2 Jan 2019, at 20:37, H I Baysal <hibaysal () gmail com> wrote:
>
> Hi Tim,
>
> That absolutely depends on the amount of TAGs you use, and how you aggregate, etc.
> I am collecting DSTAS, SRCAS, en DST AS per IP. And influx is not even sweating a single drop....
>
> We have a 4 Tbps of traffic during peak, and as well as pmacct and influxdb or running very very smooth.
>
> (With the mentioned aggregations I can see what a single customer costs with Transit, Peering and IX (per IP even if 
> needed) )
> And dst AS per port/description/ethernet name
>
> From your mail i derive that you just pushed everything to influx from flows, you have to be a bit smarter with the 
> layout, aggregations and continuous queries.
> (collect what you need)
>
>
>
> > On 02-01-19 13:08, Tim Raphael wrote:
> > I would advise against InfluxDB in this case - flow data has a very high (and open) tag cardinality which is not 
> > suited to Influx (although their recently new index format has improved this).
> >
> > I’m currently pushing sFlow through Pmacct —> Kafka —> Clickhouse (columnar store) with a summing merge tree 
> > database engine.
> > Clickhouse is very fast for queries across columns as well as aggregating down them (e.g. summing number of bytes).
> >
> > For example this is the results of a query of nearly a year’s worth of MAC-to-MAC flows (7-tuple) queried for the 
> > last 7 days between two given sets of MACs:
> >
> > 2016 rows in set. Elapsed: 0.208 sec. Processed 17.56 million rows, 1.03 GB (84.51 million rows/s., 4.97 GB/s.)
> >
> > There is also a Grafana datasource plugin for Clickhouse :)
> >
> > - Tim
> >
> >
> > > On 2 Jan 2019, at 7:50 pm, H I Baysal <hibaysal () gmail com> wrote:
> > >
> > > PMACCT (Works Awesome)
> > > push to influxdb ( Works awesome)
> > >
> > > With some custom scripts to add/match interface descriptions. And you can query whatever you want in grafana :D
> > > And grafana has a nice API for rendering a                 dashboardgraph to a PNG and you can send this png to 
> > > whatever chat/bot or mail you want.
> > >
> > > And all for free with 99% of accuracy.
> > >
> > > (Mucho gracias to Paulo :D )
> > >
> > >
> > > > On 01-01-19 05:56, Avi Freedman wrote:
> > > > We do have a minimum for commercial service that's more like $1500/mo but we are coming out with a free tier in Q1 
> > > > with lower retention (among other deltas, but including fully slice and dice flow analytics +BGP that it sounded 
> > > > like Erik might be looking for).
> > > >
> > > > Feel free to ping me if anyone would like to help us test the free tier in January.
> > > >
> > > > Thanks,
> > > >
> > > > Avi Freedman
> > > > CEO, Kentik
> > > >
> > > > > Doesn't Kentik cost like $2000 a month minimum?
> > > > >
> > > > >
> > > > > On Mon, Dec 31, 2018 at 11:57 AM Matthew Crocker <matthew () corp crocker com>
> > > > > wrote:
> > > > >
> > > > > >  +1 Kentik as well,  DDoS, RTBH, Netflow.  Cloud based so I don't have to
> > > > > > worry about it.
> > > > > >
> > > > > > On 12/31/18, 11:37 AM, "NANOG on behalf of Bryan Holloway" <
> > > > > > nanog-bounces () nanog org on behalf of bryan () shout net> wrote:
> > > > > >
> > > > > >     +1 Kentik ...
> > > > > >
> > > > > >     We've been using their DDoS/RTBH mitigation with good success.
> > > > > >
> > > > > >
> > > > > >     On 12/31/18 3:52 AM, Eric Lindsjö wrote:
> > > > > >     > Hi,
> > > > > >     >
> > > > > >     > We use kentik and we're very happy. Works great, tons of new
> > > > > > features
> > > > > >     > coming along all the time. Going to start looking into ddos
> > > > > > detection
> > > > > >     > and mitigation soon.
> > > > > >     >
> > > > > >     > Would recommend.
> > > > > >     >
> > > > > >     > Kind regards,
> > > > > >     > Eric Lindsjö
> > > > > >     >
> > > > > >     >
> > > > > >     > On 12/31/2018 04:29 AM, Erik Sundberg wrote:
> > > > > >     >>
> > > > > >     >> Hi Nanog….
> > > > > >     >>
> > > > > >     >> We are looking at replacing our Netflow collector. I am wonder what
> > > > > >     >> other service providers are using to collect netflow data off their
> > > > > >     >> Core and Edge Routers. Pros/Cons… What to watch out for any info
> > > > > > would
> > > > > >     >> help.
> > > > > >     >>
> > > > > >     >> We are mainly looking to analyze the netflow data. Bonus if it does
> > > > > >     >> ddos detection and mitigation.
> > > > > >     >>
> > > > > >     >> We are looking at
> > > > > >     >>
> > > > > >     >> ManageEngine Netflow Analyzer
> > > > > >     >>
> > > > > >     >> PRTG
> > > > > >     >>
> > > > > >     >> Plixer – Scrutinizer
> > > > > >     >>
> > > > > >     >> PeakFlow
> > > > > >     >>
> > > > > >     >> Kentik
> > > > > >     >>
> > > > > >     >> Solarwinds NTA
> > > > > >     >>
> > > > > >     >> Thanks in advance…
> > > > > >     >>
> > > > > >     >> Erik
> > > > > >     >>
> > > > > >     >>
> > > > > >     >>
> > > > > > ------------------------------------------------------------------------
> > > > > >     >>
> > > > > >     >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any
> > > > > > documents,
> > > > > >     >> files or previous e-mail messages attached to it may contain
> > > > > >     >> confidential information that is legally privileged. If you are not
> > > > > >     >> the intended recipient, or a person                       responsible for delivering it
> > > > > > to
> > > > > >     >> the intended recipient, you are hereby notified that any
> > > > > > disclosure,
> > > > > >     >> copying, distribution or use of any of the information contained in
> > > > > > or
> > > > > >     >> attached to this transmission is STRICTLY PROHIBITED. If you have
> > > > > >     >> received this transmission in error please notify the sender
> > > > > >     >> immediately by replying to this e-mail. You must destroy the
> > > > > > original
> > > > > >     >> transmission and its attachments without reading or saving in any
> > > > > >     >> manner. Thank you.
> > > > > >     >