Re: vFlow :: IPFIX, sFlow and Netflow collector

[i mawsog via NANOG <nanog () nanog org>]

A few questions and  comments. 
1. Is there any  good  open repository of netwflow data ?     
2. How about open repository of raw packet capture ? 3. There are many companies that help collect  raw packet -  
Gigamon, BigSwitch, ... .  Do folks on this list have any experiences  with these vendors ? 3. xFLows are  apparently 
the only   detailed metric collected on a wider scale.  I heard even that is often considered a nuisance for the value 
it  provides .  What are the experiences of the the folks on this list  ?   Where and how netflow is usually collected 
? 
SG

      From: Mehrdad Arshad Rad <arshad.rad () gmail com>
 To: Vitaly Nikolaev <nvitaly () gmail com> 
Cc: nanog () nanog org
 Sent: Wednesday, May 17, 2017 7:01 AM
 Subject: Re: vFlow :: IPFIX, sFlow and Netflow collector
   
I tried w/ standalone MemSQL w/ 100K IPFIX samples per second and it works.
if you pay MemSQL license you can have more than one node (cluster).
another solution is ClickHouse https://clickhouse.yandex/ but I'm gonna to
test it soon :-)
The MemSQL's nice feature is it has built in Kafka consumer w/ transform
feature.

On Tue, May 16, 2017 at 8:04 AM, Vitaly Nikolaev <nvitaly () gmail com> wrote:

> Hello,
>
> Interesting, what receives and where do you keep flows at the other end of
> messaging bus ?
>
>
> PS: in my case I am talking about hundreds of kilo flows/s that I would
> like to keep for at least few weeks, so MemSQL or any other SQLs are out of
> the picture.
>
> Thank you
>
>
> On Mon, May 15, 2017 at 2:31 PM, Mehrdad Arshad Rad <arshad.rad () gmail com>
> wrote:
>
> > Hi all,
> >
> > I just wanted to share the vFlow - IPFIX, sFlow and Netflow collector,
> > it's
> > scalable and reliable, written by pure Golang!
> > It doesn't have any library dependency and works w/ Kafka and NSQ (you can
> > write your own MQ plugin).
> >
> > https://github.com/VerizonDigital/vflow
> >
> > For more information
> > https://www.linkedin.com/pulse/high-performance-scalable-
> > reliable-ipfix-sflow-open-arshad-rad
> >
> > It can be able to integrate w/ MemSQL easy and you can have kind of below
> > SQL query:
> >
> > memsql> select * from samples order by bytes desc limit 20;
> > +----------------+-----------------+-----------------+------
> > --+--------+-------+---------+---------+----------+--------+
> > ---------------------+
> > | device        | src            | dst            | srcASN | dstASN
> > | proto | srcPort | dstPort | tcpFlags | bytes  | datetime
> > |
> > +----------------+-----------------+-----------------+------
> > --+--------+-------+---------+---------+----------+--------+
> > ---------------------+
> > | 192.129.230.0  | 87.11.81.121    | 61.231.215.18  | 131780 |  21773
> > |    6 |      80 |  64670 | 0x10    | 342000 | 2017-04-27 22:05:55
> > |
> > | 52.20.79.116  | 87.11.81.100    | 216.38.140.154  |  41171 |  7994
> > |    6 |    443 |  26798 | 0x18    | 283364 | 2017-04-27 22:06:00
> > |
> > | 52.20.79.116  | 192.229.211.70  | 50.240.197.150  |  41171 |  33651
> > |    6 |      80 |  23397 | 0x10    | 216000 | 2017-04-27 22:05:55
> > |
> > | 108.161.249.16 | 152.125.33.113  | 74.121.78.10    |  13768 |  9551
> > |    6 |      80 |  49217 | 0x18    | 196500 | 2017-04-27 22:05:59
> > |
> > | 192.229.130.0  | 87.21.81.254    | 94.56.54.135    | 132780 |  21773
> > |    6 |      80 |  52853 | 0x18    | 165000 | 2017-04-27 22:05:55
> > |
> > | 108.161.229.96 | 93.184.215.169  | 152.157.32.200  |  12768 |  11430
> > |    6 |    443 |  50488 | 0x18    |  86400 | 2017-04-27 22:06:01
> > |
> > | 52.22.49.106  | 122.229.210.189 | 99.31.208.183  |  22171 |  8018
> > |    6 |    443 |  33059 | 0x18    |  73500 | 2017-04-27 22:05:55
> > |
> > | 52.22.49.126  | 81.21.81.131    | 66.215.169.120  |  22171 |  20115
> > |    6 |      80 |  57468 | 0x10    |  66000 | 2017-04-27 22:05:59
> > |
> > | 108.160.149.96 | 94.184.215.151  | 123.90.233.120  |  16768 |  14476
> > |    6 |      80 |  63905 | 0x18    |  65540 | 2017-04-27 22:05:57
> > |
> > | 52.22.79.116  | 162.129.210.181 | 60.180.253.156  |  21271 |  31651
> > |    6 |    443 |  59652 | 0x18    |  64805 | 2017-04-27 22:06:00
> > |
> > | 108.161.149.90 | 93.184.215.169  | 80.96.58.146    |  13868 |  22394
> > |    6 |    443 |    1151 | 0x18    |  59976 | 2017-04-27 22:05:54
> > |
> > | 102.232.179.20 | 111.18.232.131  | 121.62.44.149  |  24658 |  4771
> > |    6 |      80 |  61076 | 0x10    |  59532 | 2017-04-27 22:05:54
> > |
> > | 102.232.179.20 | 192.129.145.6  | 110.49.221.232  |  24658 |  4804
> > |    6 |    443 |  50002 | 0x10    |  58500 | 2017-04-27 22:05:55
> > |
> > | 102.232.179.20 | 192.129.232.112 | 124.132.217.101 |  24658 |  43124
> > |    6 |    443 |  37686 | 0x10    |  57000 | 2017-04-27 22:06:00
> > |
> > | 192.229.230.0  | 87.11.81.253    | 219.147.144.22  | 132380 |  2900
> > |    6 |      80 |  25202 | 0x18    |  56120 | 2017-04-27 22:05:58
> > |
> > | 192.129.130.0  | 87.21.11.200    | 180.239.187.151 | 132380 |  8151
> > |    6 |    443 |  55062 | 0x18    |  52220 | 2017-04-27 22:05:59
> > |
> > | 52.12.79.126  | 87.21.11.254    | 64.30.125.221  |  21071 |  14051
> > |    6 |      80 |  57072 | 0x10    |  51000 | 2017-04-27 22:05:54
> > |
> > | 192.229.110.1  | 150.195.33.40  | 98.171.170.51  | 132980 |  28773
> > |    6 |      80 |  53270 | 0x18    |  51000 | 2017-04-27 22:05:57
> > |
> > | 192.229.110.1  | 87.21.81.254    | 68.96.162.21    | 132980 |  28773
> > |    6 |      80 |  46727 | 0x18    |  49500 | 2017-04-27 22:06:01
> > |
> > | 52.22.59.110  | 192.129.210.181 | 151.203.130.228 |  21271 |  12452
> > |    6 |      80 |  43720 | 0x18    |  49500 | 2017-04-27 22:05:55
> > |
> > +----------------+-----------------+-----------------+------
> > --+--------+-------+---------+---------+----------+--------+
> > ---------------------+
> > 20 rows in set (0.06 sec)
> >
> >
> > Please let me know if you have any questions.
> >
> > Thanks,
> > Mehrdad
> >
> > --
> > *M*ehrdad Arshad Rad
> > *P*rincipal Software Engineer
> > https://www.linkedin.com/in/mehrdadrad
>
>
>
> --
> --
> Vitaly Nikolaev



-- 
*M*ehrdad Arshad Rad
*P*rincipal Software Engineer
https://www.linkedin.com/in/mehrdadrad