Re: newbie questions

[bob_walder () mac com]

I will say the same thing to you I sad to Dave Aitel - you need to read our methodology more carefully.

The exploits to which you refer by name are our BASELINE exploits - they are chosen to ensure that EVERYONE has the 
signature in their database - there are about 7 of those ONLY - they are used to do some of the basic evasion stuff. 
Yes, they are old - yes they are simple - that doesn't matter because they are not part of the DETECTION test. 

If you see an explanatory reference such as (i.e. change GOBBLES to GOOBLES") then it is just that - an explanation of 
the TYPE of evasion we are doing there - it is NOT the extent of the testing. Why would we tell everyone (including the 
vendors who will be in future tests) EXACTLY what we are going to do in our evasion tests? We just provide EXAMPLES. 
UNDERSTAND the methodology before making stupid claims.

You don't KNOW what is in our attack library because that is the one part of the methodology we do not detail. People 
who get too hung up on this part of the test don't understand about the realities of IPS devices. You are focussing 
ONLY on the detection and nothing else when you dismiss the rest of our test suite so blithely. As it happens, we have 
some VERY recent stuff in our library and it is updated/changed for every edition of the report

We DO, however, understand that IPS is not IDS - I am not sure you do....

Come up with better tools than fragroute and Whisker/Nikto for testing those PARTICULAR evasion techniques and we will 
gladly use them. Once again you seem to imply that because we say we use "Whisker evasion techniques" in our 
methodology that we are "outdated" because we should be using Nikto. Once again, you do not understand what we are 
testing. 

We do NOT use Nikto/Whisker as a Web scanner - we have better means to test whether IPS can block stuff like that. We 
say quite clearly that we use the Whisker EVASION TECHNIQUES (on exploits of our own, not those in the standard test 
databases you get with these tools). Nikto is based on libwhisker - the same techniques apply. Once again you have 
focussed on a very small point ("Oh my GOD! They don't use Nikto!") without attempting to understand what exactly it is 
we are testing.

FYI We STILL find devices that cannot do TCP seg reassembly properly - and in the latest test, one device even failed 
two of the Whisker test cases. Just because those tools have been around for a while does not make them irrelevant - if 
we did NOT test against such commonly available tools, we would be doing our readers - and the vendors - a disservice. 
FYI - we do far more than JUST fragroute/whisker.... Contrary to what some other misinformed people have claimed, we 
even... Horror of horrors.... Do MSRPC evasion testing.... Yes, really!

I have no problems with people making constructive criticism, but you and Dave Aitel are quite simply playing stupid 
public guessing games and getting it wrong every time

You need to stop making ridiculous claims and get your facts straight before posting in public forums. I am always 
happy to answer questions about our methodologies.

Bob Walder
The NSS Group





On 12/1/05 7:02 pm, "Julius Detritus" <julius.detritus () ifrance com> wrote:

> About NSS tests:
>
> > > "They're not open tests."
>
> > The NSS test methodologies are published in full.
>
> You don't have the details of the tests (not even the "baseline"
> signatures).
>
> > > "They're outdated."
>  
> > The first IPS test was a year ago and the NSS methodology was brand new.
> > You're right that it's mostly the same this year, save for some new
> > exploits, but I would not consider it outdated.  I don't know of a more
> > recent or more comprehensive set of tests for a network IPS.
>
> They are outdated. The most recent exploit tested must be two years old...
> They are copy and paste from IDS tests which are far older. 
>
> And the whole methodology is not appropriate. IPS are not IDS.
>
> For IDS "false alarms" generated by out of session packets (like the one
> snot would raise on snort) are not acceptable as it would confuse
> administrators in their research for effective attacks.
>
> In the case of IPS it is different. OK, it was not a real attack but who
> cares. The purpose of IPS is to block. Who cares if it blocked attacks out
> of session? It was not legitimate anyway.
>
> But to understand that, you need to understand IPS, and to be used to
> security operations (devices management, post-mortem audits, forensics
> analysis and the like...)
>
> > > "They largely test for things you don't care about, such as pushing
> packets down a wire..."
>  
> > My experience shows that organizations DO care about the things that NSS
> > tests for: signature coverage, baseline performance, performance under
> > load, latency, application response times, anti-evasion capabilities,
> > stateful operation, management and configuration.  I already  mentioned my
> > view about "pushing packets down the wire."
>
> Do you really care about the phf exploit? Or maybe the old sshutupteo from
> gobbles? Are you talking about organizations or museums?
>
> > Bob Walder from NSS can chime in here, but my understanding is that the NSS
> > signature coverage tests include many RPC-related exploits and their
> > variants, run both "in the clear" and with various evasion techniques,
> > including modified exploit code and RPC fragmentation.
>
> Anti-evasion is Whisker (not nikto, I said whisker) and fragroute 1.2...
>
> Modified exploits are common ones with strings changed (GOBBLES to GOBBLED).
>
> > > "No scientific test should be non-repeatable"
>
> > We've been able to repeat the majority of the NSS tests consistently in our
> > lab.  
>
> So your exploit database must be very old
>
> My 0.02$
>
> Julius
>
>
> _____________________________________________________________________
>
> Envie de discuter gratuitement avec vos amis ?
> Téléchargez Yahoo! Messenger http://yahoo.ifrance.com
>
>
> --------------------------------------------------------------------------
> Test Your IDS
>
> Is your IDS deployed correctly?
> Find out quickly and easily by testing it with real-world attacks from 
> CORE IMPACT.
> Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
> to learn more.
> --------------------------------------------------------------------------


--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
--------------------------------------------------------------------------