Re: Service Provider NetFlow Collectors

[Nick Peelman <npeelman () ETC1 net>]

We rolled a large(ish) ElasticSearch cluster last year out of SuperMicro Microclouds (3U, 8 nodes per chassis, Xeon-D 
based processors), mostly 32GB of RAM per node, and M.2 PCIe SSDs as well as HDD storage.  ES is a finicky beast to 
maintain. It can handle a node completely dying or disappearing from the network, but not when one runs out of space 
(at least not gracefully).  Maintaining retention and rotation is tedious at best (yay curator).  We’re dumping a 
boatload of log data there, as well as Flow data using Elastiflow, which provides the necessary collector bits as well 
as all the pretty Kibana graphs and stuff.  Probably overbuilt, but I can pretty much keep whatever logs we want in 
perpetuity, we have plenty of headroom, and searching is incredibly fast.

ELK is an awesome set of tools, but be warned, there be dragons.  Admin’ing even a small cluster can be time consuming 
and frustrating, and requires a pretty stout linux and server background, or at least some really good troubleshooting 
skills and an ability to turn to the code when the docs fall short.  Doing a larger cluster could easily be a full time 
job.  Still, all in all, I’m happy with the cost of ours, including my time building it and continued time maintaining 
it, compared to what the yearly outlay was going to be for Kentik.

-nick

On 31 Dec 2018, at 11:40, Mike Hammett <nanog () ics-il net<mailto:nanog () ics-il net>> wrote:

I just recently rolled out Elastiflow. Lots of great information.



-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com<http://www.ics-il.com/>

Midwest-IX
http://www.midwest-ix.com<http://www.midwest-ix.com/>

________________________________
From: "Michel 'ic' Luczak" <lists () benappy com<mailto:lists () benappy com>>
To: "Erik Sundberg" <ESundberg () nitelusa com<mailto:ESundberg () nitelusa com>>
Cc: nanog () nanog org<mailto:nanog () nanog org>
Sent: Monday, December 31, 2018 3:40:40 AM
Subject: Re: Service Provider NetFlow Collectors

Don’t underestimate good old ELK
https://www.elastic.co/guide/en/logstash/current/netflow-module.html
+ https://github.com/robcowart/elastiflow

BR, ic

On 31 Dec 2018, at 04:29, Erik Sundberg <ESundberg () nitelusa com<mailto:ESundberg () nitelusa com>> wrote:

Hi Nanog….

We are looking at replacing our Netflow collector. I am wonder what other service providers are using to collect 
netflow data off their Core and Edge Routers. Pros/Cons… What to watch out for any info would help.

We are mainly looking to analyze the netflow data. Bonus if it does ddos detection and mitigation.

We are looking at
ManageEngine Netflow Analyzer
PRTG
Plixer – Scrutinizer
PeakFlow
Kentik
Solarwinds NTA


Thanks in advance…

Erik


________________________________

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it 
may contain confidential information that is legally privileged. If you are not the intended recipient, or a person 
responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, 
distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If 
you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must 
destroy the original transmission and its attachments without reading or saving in any manner. Thank you.