Re: Next Generation Firewalls

["Ferguson, Michael" <mferguson () CHAPMAN EDU>]

Brandon,

I can say the lesson we learned from our POC and the Breaking Point tests that we did is that you can't rely on the 
data sheet of any vendor.  If Mark Twain were around today, he might say there are "Lies, Statistics and Vendor 
Performance Metrics".  When you see performance numbers of any Data Sheet and think how impressive they are, there 
could be some truth to it.  But it's also because they're testing against optimal traffic conditions to get the 
best-possible numbers.  So similar to Statistics, you can convincingly argue a point with Vendor Performance Metrics 
that don't match up to real life.  As you start adding a mix of real-world TCP traffic and turn on more than just one 
NextGen firewall feature, that's when performance significantly drops compared to the data sheet (and I mean 
significantly).  Some vendors also talk about IMIX traffic and give separate metrics for it, but the results of our 
stress tests didn't match vendor claims for IMIX traffic when tested for real-world traffic.  Moreover, when you turn 
on two or more NextGen firewall features (ie. Application Awareness/Control, URL Filtering, Anti-Virus, and Logging), 
you're completely without bearing on how to spec out equipment as the numbers aren't there to tell you.  I can think of 
2 vendors that give distinct performance metrics with at least 2 features turned on:  Palo Alto with AppID + IPS and 
Checkpoint with Production IPS (IPS + Logging).  But most vendors don't state the cumulative impact of turning on 2 
nextgen firewall features, and none tell you the impact of turning on 3 features or more.

The most reliable performance metrics I've seen for real-world traffic mix are from NSS Labs.  For the Fortigate 1500D, 
NSS Labs rated it capable of 9,597 Mbps in Sept 2014.  This rating is calculated as an average of 5 different tests--4 
protocol mix tests (Enterprise Perimeter, Financial, Education and Datacenter) and a test for 21KB Http response rates. 
 Fortinet's data sheet claims the 1500D is capable of 11G of IPS.  So on the face of it, the vendor's data sheet number 
is pretty close to the likely real-world performance tested by NSS Labs, only over-estimating performance by 1.4G (the 
Financial test that NSS Labs does can really bring down the data sheet numbers).  However, the NSS Labs rating is only 
for IPS inspection.  It doesn't include any performance impact when turning on other features like logging, App 
Control, etc, which will certainly drop the performance to much less than 9.6G depending on what features you turn on.  
It's too bad that NSS Labs only rated IPS performance in the NGFW report, which is a report that should let you know 
the performance impact of more features than just IPS.  As a matter of fact, Application Inspection is discussed in 
their NextGen Firewall reports as an important criteria.  But in 2014 and earlier, the rated performance by NSS Labs 
for NGFW tests is only for IPS.

We asked NSS Labs about this discrepancy during our POC and they recognized this as a limitation in their report.  They 
mentioned they're looking to add cumulative performance numbers for Application Control + IPS in the next set of NGFW 
tests, which are expected to be released in late Summer/early Fall.  However, I don't believe their NGFW tests will 
include the cumulative impact of turning on other features like URL Filtering, A/V or Logging.  This is why the 
Breaking Point tests were so valuable to us.  We could cut skip worrying about the data sheet and really see what the 
cumulative impact of turning on additional features was.  We tested for IPS, AppControl and Logging, then added URL 
Filtering and A/V (we weren't able to fully test SSL decryption because of cipher problems with the Breaking Point).  
Also, for our tests, we ran them against a Fortigate 3700D; so I couldn't tell you what to expect for a 1500D.

But what I can say is that for relatively, similar priced models from different vendors (and I use relatively in a 
general sense as some vendors were well more than others), the Fortinet 3700D really stood out in performance compared 
to others we tested in our POC, recognizing that the results of our Breaking Point tests didn't match up to the data 
sheet numbers from Fortinet.  Also, one other thing to be mindful about is that performance numbers are based on 
aggregate bandwidth--adding inbound + outbound.  This means if you're 1.4G down and 400M up on your overall bandwidth, 
you should test for 1.8G now, then factor in what you expect 3 to 5 years from now as an aggregate.

Last, I'd like to just comment on the word NGFW, or NextGen Firewall.  I admit it is an awkward term, particularly when 
we start discussing firewalls 10-15 years from now and try to distinguish the Nextest Gen Firewalls then against the 
old NextGen Firewalls of today.  I just don't know of any elegant term to use to distinguish the current brand of 
firewalls that have Application Control, which to me is what makes them NextGen Firewalls.  When IDC coined the term 
UTM, the UTM firewalls at that time didn't have Application Control as part of their capability.  But since then, 
they've added it.  And by adding Application Control, I think UTM devices have become more than UTM devices because 
they not only combine network security features in one box, but they have layer7 visibility that is fundamentally new.  
Subconsciously, this is why I think Gartner phrased the term Next-Gen Firewall to highlight that they're firewalls that 
do UTM + Application Control.  Unfortunately, Gartner's kind of confused the discussion by indicating that firewalls 
with Application Control and other security features are UTM for small to middle-sized organizations and firewalls with 
the same features for larger organizations are NextGen firewalls.  Really, there all NextGen firewalls if they have 
Application Control along with other features like URL Inspection, A/V and the optional AntiSpam.  Do we call this new 
brand of firewalls AUTM or ATM devices--Application UTM or Application Threat Management?  Probably not!  But what term 
to refer to our current generation of firewalls is still open for discussion and we use NextGen because we don't have a 
better term.  

It might end up like when our campus had been building a new building a few years ago before it got an official name.  
We called it New Hall during construction, which was all fine and good.  But then New Hall became Named Hall and the 
new New Hall starts getting built without a name yet.  We learned our lesson the second time not to name the new New 
Hall as New Hall, but give it a different name during construction until it becomes Named Hall.  This might be how we 
address the NextGen Firewall question.  When the second generation of NextGen Firewalls comes out with their own 
revolutionary technology (like inline sand-boxing to be ridiculous), we'll actually call that type of firewall by a 
name that distinguishes its capability instead of as NextGen.


--
Mike Ferguson
Chapman University
Network Operations Manager
714-744-7873


-----Original Message-----
From: The EDUCAUSE Security Constituent Group Listserv [mailto:SECURITY () LISTSERV EDUCAUSE EDU] On Behalf Of Brandon 
Dick
Sent: Friday, June 19, 2015 5:32 PM
To: SECURITY () LISTSERV EDUCAUSE EDU
Subject: Re: [SECURITY] Next Generation Firewalls

For you all that went with Fortinet:

When we were doing our bid, we put in requirements that the Firewall be 
10GB capable and someone put in a bid for the Fortinet 1500D. According 
to the specs, the 1500D would put a Palo Alto 5050 to shame, but the 
cost difference wasn't even close, with Fortinet being the clear winner 
in price.  I kept digging into it with the reseller and requesting 
guarantees that this would actually perform as well as they said, and 
eventually the reseller pulled out of the bid.   So I never really got 
the full story on that one.

So my question is, how many of you think the 1500D would've actually 
handled 10GB of traffic?  I noticed in your comparison below, they chose 
the 3600C, which makes me think even more that the 1500D wouldn't have 
performed to spec.  It was just a really odd situation...

On 6/19/2015 2:45 PM, Kumar, Shashank wrote:
> Hello All,
>
> We recently went through a bake-off ourselves and evaluated the Palo Alto 5050s, Fortinet 3600C, ASA5585-SSP-20 and 
> the SRX 3600.
> PA and Fortinet impressed us the most with flexible deployment options (routed mode, transparent mode, tap mode, 
> virtual systems), good hardware specs and throughput, intuitive management consoles and good vendor interaction. 
> Positive feedback from universities that had deployed PA and Fortinet was reassuring.
>
> Our evaluation process involved hitting the test unit with constant 2Gig of traffic and measuring the throughput with 
> various features enabled.  We also planned for virtual systems and doubling our throughput within the next 3 years or 
> so.  Let me know if you would like to see the test template that we used.
>
> Hope this helps.
>
> Best Regards,
> Shashank
> FGCU Network Services |Tel: 239-590-7448
>
> Florida has a very broad public records law.  As a result, any written communication created or received by Florida 
> Gulf Coast University employees is subject to disclosure to the public and the media, upon request, unless otherwise 
> exempt.  Under Florida law, e-mail addresses are public records.  If you do not want your email address released in 
> response to a public records request, do not send electronic mail to this entity.  Instead, contact this office by 
> phone or in writing.
>
> -----Original Message-----
> From: The EDUCAUSE Security Constituent Group Listserv [mailto:SECURITY () LISTSERV EDUCAUSE EDU] On Behalf Of 
> Gramke, Jim
> Sent: Friday, June 19, 2015 2:12 PM
> To: SECURITY () LISTSERV EDUCAUSE EDU
> Subject: Re: [SECURITY] Next Generation Firewalls
>
> I've got to second the Fortigate recommendation.  We've had a High Availability pair of them for a number of 
> iterations now.   Recently PA made a push, and we looked, but just undoable because of cost and even performance 
> differences.  Bang for buck, Fortigate is a formidable competitor.
>
>
> -----Original Message-----
> From: Ferguson, Michael [mailto:mferguson () CHAPMAN EDU]
> Sent: Thursday, June 18, 2015 11:17
> Subject: Re: Next Generation Firewalls
>
> We ourselves just completed a Firewall POC solution.  I would recommend also being open to Fortinet as part of your 
> consideration as this is what we selected.   It’s too early to say how well we like the solution as we’re in the 
> process of implementation.  But at least during our POC,  it distinguished itself the most in a couple key categories 
> with an emphasis on making sure we have a NGFW firewall that not only works well today, but also 4-5 years from now.  
> Like you, we saw our selection of NGFW as a significant investment.  I’ll refrain from mentioning the other solutions 
> we considered, but suffice it to say that we considered all the top solutions that are performing well in NSS Labs’ 
> new Cyber Advanced Warning System.
>
> Our testing included Ixia Breaking Point tests, which I would recommend you consider as part of your evaluation if 
> you have time to do it.  We also captured live traffic from our Production environment and sent it to several other 
> solutions simultaneously using a Gigamon.  This was valuable for seeing the manageability and effectiveness of each 
> of the solutions against each other, but not very useful when considering performance.  We also ran other security 
> tests outside the Breaking Point to look at the effectiveness of each tool.  But by far, the results of the Breaking 
> Point tests revealed the most distinction of the products we evaluated.
>
> I know Ixia offers Test Consulting for a relatively modest fee, as well as some security consulting firms.  There 
> might be other ways to get a Breaking Point or a different stress-testing tool from Spirent or others.  But needless 
> to say, it was very enlightening to see the differences between each of the solutions under heavy stress when all 
> inspection (including Application awareness) and logging was turned on.
>
> --
> Mike Ferguson
> Chapman University
> Network Operations Manager
> 714-744-7873
>
>
> -----Original Message-----
> From: The EDUCAUSE Security Constituent Group Listserv [mailto:SECURITY () LISTSERV EDUCAUSE EDU] On Behalf Of 
> Carroll, Tim
> Sent: Thursday, June 18, 2015 7:00 AM
> To: SECURITY () LISTSERV EDUCAUSE EDU
> Subject: [SECURITY] Next Generation Firewalls
>
> All,
>
> Roane State Community College is in the process of reviewing next generation firewalls.  Since this is a significant 
> investment, I would be interested in hearing from the community what you are using, your experience, why you made the 
> choice and your satisfaction with the vendor chosen.
>
> Thanks in advance for any feedback.
>
> Regards,
>
> Tim Carroll
> Assistant Vice President and Chief Information Officer Information Technology Roane State Community College
>
> ________________________________
>
> This email is intended for the addressee and may contain privileged information. If you are not the addressee, you 
> are not permitted to use or copy this email or its attachments nor may you disclose the same to any third party. If 
> this has been sent to you in error, please delete the email and notify us by replying to this email immediately.
>
> ________________________________
>
> Never give out your username or password to anyone. This includes any accounts you have such as: FGCU, bank and 
> credit card accounts, and other personal accounts.

-- 
Brandon Dick
Network Engineer
Information Systems
Murray State University
Phone: (270) 809-3694
Fax:   (270) 809-3465

  

MSU Information Systems staff will never ask for your password or other confidential information via email.