Re: Compliance VS Pen-Test and there relative value

[Andrew van der Stock <vanderaj () greebo net>]

Philippe,

Compliance done well is awesome and it works. Compliance done in the  
wrong way ("Don't tell the QSA anything they don't ask for" or "You  
have five days to look at this 45 year old system with 10 million  
lines of COBOL and 150,000 POS terminals"), or with the wrong  
motivation ("We must have the Hack Proof Mark") simply puts back our  
entire industry's move to become trustworthy.

One of the biggest issues with compliance is that the folks who dream  
up the lists simply do not know all potential outcomes, and then being  
wishy-washy about the requirements they require to the point that it  
is possible to honestly say "I am compliant with X" when  
realistically, you've done nothing to fix the real risks. This is a  
problem for all standards, not just PCI, ISO 27000 family or COBIT.

Add in if a firm has a weak assessor who can be talked into  
downgrading the risk of a really dumb idea, or worse, never knew about  
the really dumb idea, you can get certified when you're not really  
secure. This is allegedly how TJ Maxx got done over. The use of  
insecure wireless connections at outlying stores breaks the hard  
outside / soft inside model used by nearly everyone. All security  
folks would have said "hey, that's going to get you into trouble one  
day and it's an easy fix". They also stored track 2 data, and at some  
stage someone told them to encrypt it, for the attackers apparently  
used TJX's own decryption utility to get at the track 2 data. If  
you're encrypting data using reversible encryption to meet a  
compliance requirement, you're probably doing it a bit wrong. For TJX,  
it was a set of really dumb ideas that cost them a bucket load of  
money so far, and way more to come.

Would PCI compliance have helped TJX? Absolutely. If they'd done it  
right instead of chasing the cheapest way to PCI compliance.

First by not capturing any track two info (a key PCI DSS requirement).  
That alone would have eliminated or dramatically reduced the loss  
because they had PCI-compliant POS registers that correctly encrypted  
the PIN and cardholder info before sending it through to their central  
systems. No amount of sniffing would have got around that. Attacking  
CC storage would have been the only option, and if they had done the  
right thing (capturing only the necessary CC info for a limited period  
of time) would have dramatically reduced the attacker's breadth and  
ease of capture without being caught by other mandatory PCI  
requirements, such as host and network monitoring.

You can guess I believe in compliance. But only in the context of  
assurance, and only if it's done well as part of a wider program of  
appropriate corporate governance. Notice that I didn't say "IT  
security" or something silly like that. IT security has nothing to do  
with this - the business wants assurance that their systems and  
processes will cause the minimum loss in the face of motivated attack  
with a reasonable spend to get there. Strong corporate governance is  
way beyond just giving an isolated and powerless group such as IT  
Security a bucket of money to go test stuff. Instead, it's consistent,  
long term actions that count towards assurance.

Assurance comes in many ways. Personally, I don't think penetration  
testing alone helps:

a) the skill of the tester is by far the most important decider if a  
pen test is going to provide assurance or just be a complete waste of  
time. If I find nothing, am I skilled and found nothing, or did I read  
Slashdot the entire day? You can't tell by this one artifact alone.

b) the time generally allotted (one day to several weeks, but  
typically a week to two weeks) is not long enough for how long a  
motivated attacker would spend attacking you. They only need one  
workable attack. Even if you're a good pentester and you have awesome  
tools and pen-test-fu, you may only have a day to do some of the more  
important areas, such as access control. This is simply not long enough.

c) the tools and the skill of the tester in using them are far too  
much of a variable. Some folks will use just the browser, others rely  
heavily upon automated scans, but do not know how to use those results  
to leverage further attacks. See (a).

d) information disclosed prior to the start of the test can affect how  
much is found. Zero knowledge tests are a case in point: essentially a  
rod length check (i.e. a WAFTAM) and not at all useful (and in my  
view, are actually illegal in any country with even partially  
effective anti-hacking laws)

Does that mean that compliance helps? Yes and No. SAS70 certification  
- useless*. COBIT - not so useless. PCI - is a good start. SOX - what  
compliance with SOX exactly means is subject to massive interpretation  
and is the subject of what I consider to be snake oil projects - pet  
projects with no real security outcomes were part of the SOX process  
at many large firms, and will have no bearing on the issue at hand -  
initiator / approver controls for value transactions with effective  
security controls for systems with a financial impact. What really  
happened? A lot of reviews, a lot of useless software was bought and  
installed, and often no improvement to the key systems in point, such  
as SAP or Oracle Financials.

Compliance helps start absolutely insecure folks down the road of  
security. I think every organization needs to work on assurance:

a) Assurance what they're doing is current industry best practice, by  
accessing peer resources such as this list, attending OWASP meetings  
and other industry conferences, and formal training and education, and  
to a certain degree - choosing a standard to comply with (COBIT, ISO  
27000, ISO 27034, OWASP Top 10, etc)

b) Assurance their design is robust against likely threats, via secure  
architecture reviews

c) Assurance their implementation is built to an adequate standard for  
the information they are trying to protect, by peer reviews, help from  
security buddies, and obviously, manual code reviews and automated  
static code analysis. SCA tools are really coming along in leaps and  
bounds now. Let the manual testing be for things SCA tools can never  
figure out - process issues, and deep authorization issues, etc.

d) Assurance their deliverables are robust against a motivated  
attacker, such as a grey box test - a pen test using the code to find  
as much as possible in the time available. This probably means the use  
of both manual testing and automated scans. Automated scanning should  
be considered the low hanging fruit, but it is remarkably useful to  
eliminate.

e) Building up a picture of assurance over time, by having a risk  
register and coming back to it regularly. Security is not a one time  
event.

f) Continuous improvement. 2002 webappsec security is simply not going  
to cut the mustard today, let alone tomorrow. Implement lightweight  
processes which helps cement the gains won by the assurance process.  
It's not enough to get a pen test or code review or SAR - you must  
action the things that will bite you on the derriere.

I think compliance has its place, just as I think SARs, manual code  
reviews, source code analysis, and final verification tests - even  
WAFs, have their place. It's part of the total picture.

There is no "Hack proof" seal of approval - period, only marketing.  
Compliance without thought is marketing. Compliance with risk based  
assurance works best to move insecure folks to a mature risk based  
approach so that money sunk into the assurance process is wisely spent  
on real risks.

thanks,
Andrew

* SAS 70 reviews can only be conducted by a certified accountant and  
are generally interview based. That's exactly like asking me to do  
Microsoft's taxes using Google to learn the tax code - just because  
you're a professional in one field does not make you a professional in  
another.

On Sep 2, 2008, at 3:24 PM, Rivest, Philippe wrote:

> I'm a bit surprised as to all the feedbacks I'm getting. Everyone  
> who answers
> me is reducing the value of a compliant company if you check it  
> security
> wise. But on the other hand you all seem to praise the pen-test a lot.
>
> I know that if a client is compliant with standard XYZ it may still be
> insure, vulnerable to a lot of exploits and so on. I know it could be
> penetrated with ease. But take this picture I'm new to auditing and my
> experience is pretty much book wise (ISACA, Cobit, COSO, SOX, CISA),  
> how ever
> I only see good with compliance, SOX and COBIT/COSO (keeping to IT).  
> Here I
> am out of my books and everyone is taking doing compliance. What did  
> I not
> get?
>
> Are you taking down compliance because its hard to get properly set  
> within a
> company?
>
> Please explain and help me get why compliance has a low reputation and
> pen-test is so great *(I know the difference between them)

thanks,
Andrew

Attachment: smime.p7s
Description: