Re: How does a customer get PCI audited?

[Adriel Desautels <adriel () netragard com>]

Craig,
	Just out of curiosity how are you defining a "Pen Test"? You also say 
that "Pen Testing is sexier", but sexier than what? You say that it is 
not as effective as "other forms of testing and validation", but you 
give no examples of other more effective forms of testing. What other 
forms of testing and validation are you talking about?

>"Actually, common sense is something that fails more than not. Most of 
>science in the 20th C turned against common sense. We are a species 
>that has developed to suit an environment that no longer exists. This 
>is why we measure perception and call it a measurement of security."

I disagree. Common sense doesn't fail more than it succeeds, if it did 
then we would not survive as a species.

>"You state that a firm needs to be "tested at the same or higher 
threat >level as you'd face in the real world." Please answer how?"

Testing at a higher threat level means performing research against the 
target to be assessed. Often times that involves performing a review of 
historical IDS events, a review of the targets policies and procedures, 
and gaining an understanding of what they have that is of value that 
others want. Another helpful aspect is to be able to keep tabs on what 
sorts of 0-day threats exist, which we can do.

>"Real world threats are never tested by red teams. Not even military 
>ones. A cut down attempt is made. You can simulate the script kiddie, 
>but not the Dragon's."

You are wrong and you are blurring the lines. Are you suggesting that a 
talented, financially oriented malicious hacker with intent to extort 
his or her victims is not a real world threat? Or perhaps someone from a 
hostile foreign nation that is talented enough to use his or her talent 
to cause infrastructural damage to our people, power plants, 
communications, etc. (Like what happened in Canada recently, or the 
Aurora Generator Project). Do you consider those to be real world 
threats Craig? Or are they not real because they do not come wielding 
hand grenades and guns? Are they not real because they never get 
physical? Our Air force takes that threat pretty seriously, haven't you 
seen the commercials?

>"In the 90's I did some "real" red team work. With pieces of paper 
from >government agencies in case something went wrong we even did a 
number >of actions that are technically illegal minus the "get out of 
free >card"."

We're not in the 90's anymore, that's 18 years old. In this day and age 
the physical threat is becoming much less than the cyber threat. I see 
many more people suffering because of "hacks" than I do because of 
"kidnappings". If you need to worry about kidnappings then you don't 
need a penetration test, you need a bodyguard and a bullet proof car.

>"When was the last time your red team kidnapped somebody? When did you 
>pay an employee $50k to see how they react? I have seen all this in 
the >"real world". I worked on a case where an Asian bank manager was 
sent >parts of his daughter. Sorry, but controls that added a distress 
code >helped more than a pen test. So did segregation of duties (to 
qualify, >it helped the bank)."

So now you are talking about policies and procedures that are far out of 
the scope of where this discussion started. I do not think that the 
person who started this thread is worried about being chopped up, I 
think that he's concerned about malicious hackers.

>So how DO you test segregation of duties on a database system using a 
>pen test?

>Even from a physical limitation removed perspective with access 
>granted, how common is it to map internal business processes in a pen 
>test?

For my team, very common.

>Security is not just "hacker" style vulnerabilities. It also involves 
>tests of the payroll processes, HR systems, reporting engine, etc. The 
>general vulnerabilities account for less than 1% of real issues and 
>problems I come across in forensic work."

A good team capable of real testing will test those processes and 
procedures.

>Running a scanner is not an audit. If a firm states it is, take them 
to >court for false or misleading trade practices. I would be happy to 
sue >their ass off for such a lie.

Agreed, sue away!

>Yes, I will agree there is a place for pen testing. I am one of the 
>question authors for the SANS GPEN cert so I have to have some support 
>for the concept. It is NOT the be all of security as it is made to be. 
>As noted, Pen Tests are a detective control. These controls are by 
>nature essential, but they are also quantitatively less effective than 
>preventative controls.

You can not prevent something from happening unless you are aware of the 
risk first. Penetration Tests identify those risks if done properly by a 
capable team.


" it is very hard to find good vendors these days."
This I agree with completely.

Awesome.
        

Regards,
        Adriel T. Desautels
        Chief Technology Officer
        Netragard, LLC.
        Office : 617-934-0269
        Mobile : 617-633-3821
        http://www.linkedin.com/pub/1/118/a45

        Join the Netragard, LLC. Linked In Group:
        http://www.linkedin.com/e/gis/48683/0B98E1705142

---------------------------------------------------------------
Netragard, LLC - http://www.netragard.com  -  "We make IT Safe"
Penetration Testing, Vulnerability Assessments, Website Security

Netragard Whitepaper Downloads:
-------------------------------
Choosing the right provider : http://tinyurl.com/2ahk3j
Three Things you must know  : http://tinyurl.com/26pjsn


Craig Wright wrote:
> Hi,
> Actually I did read it. You should also note that I did not state that you do not test. Rather I stated as in the 
> papers I have published in the past that Red teams are less effective than Blue taking a number of assumptions of equal 
> skills etc into effect.
>
> Pen Testing is "sexier". It is not as effective as other forms of testing and validation. It is the easier sell.
>
> "The unfortunate truth is that MOST businesses that provide "auditing"
> type services, or assessing type services do not know what they are doing."
> I agree. The same goes for over 90% of red teams. This argument is a red herring that applies to each argument equally. Most 
> "Pen Testers" are equally clueless. So what is the argument?
>
> Actually, common sense is something that fails more than not. Most of science in the 20th C turned against common 
> sense. We are a species that has developed to suit an environment that no longer exists. This is why we measure 
> perception and call it a measurement of security.
>
> You state that a firm needs to be "tested at the same or higher threat level as you'd face in the real world."
> Please answer how?
>
> Real world threats are never tested by red teams. Not even military ones. A cut down attempt is made. You can simulate the 
> script kiddie, but not the Dragon's.
>
> In the 90's I did some "real" red team work. With pieces of paper from government agencies in case something went wrong we even 
> did a number of actions that are technically illegal minus the "get out of free card".
>
> When was the last time your red team kidnapped somebody? When did you pay an employee $50k to see how they react? I have seen all 
> this in the "real world". I worked on a case where an Asian bank manager was sent parts of his daughter. Sorry, but 
> controls that added a distress code helped more than a pen test. So did segregation of duties (to qualify, it helped the bank).
>
> So how DO you test segregation of duties on a database system using a pen test?
>
> Even from a physical limitation removed perspective with access granted, how common is it to map internal business processes in a 
> pen test? Security is not just "hacker" style vulnerabilities. It also involves tests of the payroll processes, HR 
> systems, reporting engine, etc. The general vulnerabilities account for less than 1% of real issues and problems I come across in 
> forensic work.
>
> Running a scanner is not an audit. If a firm states it is, take them to court for false or misleading trade practices. 
> I would be happy to sue their ass off for such a lie.
>
> Yes, I will agree there is a place for pen testing. I am one of the question authors for the SANS GPEN cert so I have 
> to have some support for the concept. It is NOT the be all of security as it is made to be. As noted, Pen Tests are a 
> detective control. These controls are by nature essential, but they are also quantitatively less effective than 
> preventative controls.
>
> " it is very hard to find good vendors these days."
> This I agree with completely.
>
> Regards,
> Craig Wright GSE LLM GREM CISA... and ...
>
>
> Craig Wright
> Manager, Risk Advisory Services
>
> Direct : +61 2 9286 5497
> Craig.Wright () bdo com au
> +61 417 683 914
>
> BDO Kendalls (NSW-VIC) Pty. Ltd.
> Level 19, 2 Market Street Sydney NSW 2000
> GPO BOX 2551 Sydney NSW 2001
> Fax +61 2 9993 9497
> http://www.bdo.com.au/
>
> The information in this email and any attachments is confidential. If you are not the named addressee you must not 
> read, print, copy, distribute, or use in any way this transmission or any information it contains. If you have received 
> this message in error, please notify the sender by return email, destroy all copies and delete it from your system.
>
> Any views expressed in this message are those of the individual sender and not necessarily endorsed by BDO Kendalls. 
> You may not rely on this message as advice unless subsequently confirmed by fax or letter signed by a Partner or 
> Director of BDO Kendalls. It is your responsibility to scan this communication and any files attached for computer 
> viruses and other defects. BDO Kendalls does not accept liability for any loss or damage however caused which may 
> result from this communication or any files attached. A full version of the BDO Kendalls disclaimer, and our Privacy 
> statement, can be found on the BDO Kendalls website at http://www.bdo.com.au/ or by emailing mailto:administrator () 
> bdo com au.
>
> BDO Kendalls is a national association of separate partnerships and entities. Liability limited by a scheme approved 
> under Professional Standards Legislation.
> -----Original Message-----
>
> From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Adriel Desautels
> Sent: Thursday, 5 June 2008 1:08 AM
> To: Craig Wright
> Cc: Scott Race; security-basics () securityfocus com
> Subject: Re: How does a customer get PCI audited?
>
> Craig,
>         I'm afraid that you didn't read what I wrote carefully enough or that I
> wasn't sufficiently clear in my communications. If you go back and read
> what I said, I explicitly used the words "real" and "good".
>
>         You can be purple in the face with controls and training, but if you
> are never PROPERLY tested by a REAL team then you will never know where
> your REAL risks are. The end solution consists of the PROPER
> implementation of controls mixed with the right amount of training. You
> can't create a GOOD solution without knowing details about the problem.
>
>         The unfortunate truth is that MOST businesses that provide "auditing"
> type services, or assessing type services do not know what they are
> doing. In fact, very sadly most of them run automated scanners and
> produce deliverables that are the product of those scanners. Those
> deliverables should be free in my opinion as they are usually poor quality.
>
>         In order to properly defend a network you must first know what you need
> to defend it against. You must have a strong understanding of the threat
> and how the threat might align with your risk and exposure profile. The
> only way to do that is to either have good threat intelligence, or work
> with a qualified penetration testing team that has REAL threat
> intelligence.
>
>         Once you've identified such a firm your IT Infrastructure, personnel,
> policies, etc. need to be tested at the same or higher threat level as
> you'd face in the real world. That will identify your risks and help you
> to build the proper CONTROLS to counter those risks. Suggesting that
> anyone build controls without first having a GOOD and REAL assessment is
> horrible advice. That would be akin to building defenses against Russia
> during the cold war with no intelligence about their capabilities.
>
>         With respect to your paper, I still need to go read it. That said, even
> if Penetration Testing is 30% of the total solution, it is clearly the
> foundation to building the solution. Else you are building a blind
> defense that most probably won't work.
>
>         Its common sense Craig, know your enemy, know yourself, and then you
> can build a good defense.
>
> Regards,
>         Adriel T. Desautels
>         Chief Technology Officer
>         Netragard, LLC.
>         Office : 617-934-0269
>         Mobile : 617-633-3821
>         http://www.linkedin.com/pub/1/118/a45
>
>         Join the Netragard, LLC. Linked In Group:
>         http://www.linkedin.com/e/gis/48683/0B98E1705142
>
> ---------------------------------------------------------------
> Netragard, LLC - http://www.netragard.com  -  "We make IT Safe"
> Penetration Testing, Vulnerability Assessments, Website Security
>
> Netragard Whitepaper Downloads:
> -------------------------------
> Choosing the right provider : http://tinyurl.com/2ahk3j
> Three Things you must know  : http://tinyurl.com/26pjsn
>
>
> Craig Wright wrote:
> > "You secure your systems by having an IT Security Company perform a real
> > good assessment"
> > No, good controls and training help secure a system. Pen testing is a detective control. No more. In fact, it is not 
> > even close to being the best (bang for buck) detective control.
> >
> > Actually, Pen testing is about a 30% option that ignores many of the vulnerable points of a system. Economic limits 
> > constrain it. The difficulty
> >
> > I did some scientific research into this in the past. I published in the IIA journal and a few others and a paper is on 
> > the sans reading room as well:
> > http://www.sans.org/reading_room/whitepapers/auditing/1801.php
> >
> > The difficultly is that there are very few auditors with a deep technical level of knowledge.
> >
> > Craig
> >
> >
> > Craig Wright
> > Manager, Risk Advisory Services
> >
> > Direct : +61 2 9286 5497
> > Craig.Wright () bdo com au
> > +61 417 683 914
> >
> > BDO Kendalls (NSW-VIC) Pty. Ltd.
> > Level 19, 2 Market Street Sydney NSW 2000
> > GPO BOX 2551 Sydney NSW 2001
> > Fax +61 2 9993 9497
> > http://www.bdo.com.au/
> >
> > The information in this email and any attachments is confidential. If you are not the named addressee you must not 
> > read, print, copy, distribute, or use in any way this transmission or any information it contains. If you have received 
> > this message in error, please notify the sender by return email, destroy all copies and delete it from your system.
> >
> > Any views expressed in this message are those of the individual sender and not necessarily endorsed by BDO Kendalls. 
> > You may not rely on this message as advice unless subsequently confirmed by fax or letter signed by a Partner or 
> > Director of BDO Kendalls. It is your responsibility to scan this communication and any files attached for computer 
> > viruses and other defects. BDO Kendalls does not accept liability for any loss or damage however caused which may 
> > result from this communication or any files attached. A full version of the BDO Kendalls disclaimer, and our Privacy 
> > statement, can be found on the BDO Kendalls website at http://www.bdo.com.au/ or by emailing mailto:administrator () 
> > bdo com au.
> >
> > BDO Kendalls is a national association of separate partnerships and entities. Liability limited by a scheme approved 
> > under Professional Standards Legislation.
> > -----Original Message-----
> >
> > From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Adriel Desautels
> > Sent: Wednesday, 4 June 2008 7:25 AM
> > To: Craig Wright
> > Cc: Scott Race; security-basics () securityfocus com
> > Subject: Re: How does a customer get PCI audited?
> >
> > Craig,
> >         You secure your systems by having an IT Security Company perform a real
> > good assessment. It should be either a Penetration Test or a
> > Vulnerability Assessment. Penetration Tests are more in-depth,
> > Vulnerability Assessments are more safe. If you want information on both
> > of those services I've got a few white-papers that we've written that
> > you can download. Unfortunately, it is very hard to find good vendors
> > these days.
> >
> > Regards,
> >         Adriel T. Desautels
> >         Chief Technology Officer
> >         Netragard, LLC.
> >         Office : 617-934-0269
> >         Mobile : 617-633-3821
> >         http://www.linkedin.com/pub/1/118/a45
> >
> >         Join the Netragard, LLC. Linked In Group:
> >         http://www.linkedin.com/e/gis/48683/0B98E1705142
> >
> > ---------------------------------------------------------------
> > Netragard, LLC - http://www.netragard.com  -  "We make IT Safe"
> > Penetration Testing, Vulnerability Assessments, Website Security
> >
> > Netragard Whitepaper Downloads:
> > -------------------------------
> > Choosing the right provider : http://tinyurl.com/2ahk3j
> > Three Things you must know  : http://tinyurl.com/26pjsn
> >
> >
> > Craig Wright wrote:
> > > "What are the chances of them getting audited?"
> > > PCI Top level = 100%
> > > A break-in occurs = 99.5%
> > >
> > > A better question is how do I pass an audit and better yet, how do I secure my systems.
> > >
> > > PCI-DSS is just the tip of the iceberg for potential issues. In fact it is the low risk consideration. Think class 
> > > action. PCI-DSS compliance is at least a small defence. That is being ACTUALLY compliant and not just doing enough to 
> > > fool the QSA.
> > >
> > > Regards,
> > > CSW GSE LLM
> > >
> > >
> > > Craig Wright
> > > Manager, Risk Advisory Services
> > >
> > > Direct : +61 2 9286 5497
> > > Craig.Wright () bdo com au
> > > +61 417 683 914
> > >
> > > BDO Kendalls (NSW-VIC) Pty. Ltd.
> > > Level 19, 2 Market Street Sydney NSW 2000
> > > GPO BOX 2551 Sydney NSW 2001
> > > Fax +61 2 9993 9497
> > > http://www.bdo.com.au/
> > >
> > > The information in this email and any attachments is confidential. If you are not the named addressee you must not 
> > > read, print, copy, distribute, or use in any way this transmission or any information it contains. If you have received 
> > > this message in error, please notify the sender by return email, destroy all copies and delete it from your system.
> > >
> > > Any views expressed in this message are those of the individual sender and not necessarily endorsed by BDO Kendalls. 
> > > You may not rely on this message as advice unless subsequently confirmed by fax or letter signed by a Partner or 
> > > Director of BDO Kendalls. It is your responsibility to scan this communication and any files attached for computer 
> > > viruses and other defects. BDO Kendalls does not accept liability for any loss or damage however caused which may 
> > > result from this communication or any files attached. A full version of the BDO Kendalls disclaimer, and our Privacy 
> > > statement, can be found on the BDO Kendalls website at http://www.bdo.com.au/ or by emailing mailto:administrator () 
> > > bdo com au.
> > >
> > > BDO Kendalls is a national association of separate partnerships and entities. Liability limited by a scheme approved 
> > > under Professional Standards Legislation.
> > > -----Original Message-----
> > >
> > > From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Scott Race
> > > Sent: Tuesday, 3 June 2008 8:37 AM
> > > To: security-basics () securityfocus com
> > > Subject: How does a customer get PCI audited?
> > >
> > > I have a client (same one from a previous post) who has some pretty
> > > serious security issues on their network (unsecured .mdb file with
> > > credit card into, etc).  I will be fixing the major security holes in
> > > their network, but they still have PCI compliance issues, and I'm
> > > assuming they need to have a quarterly scan done.
> > >
> > > They've had this setup for about a year, they knows nothing about PCI
> > > and compliance (myself included, I am not a QSA and still learning about
> > > the compliance procedure).
> > >
> > > What are the chances of them getting audited?  How does all that work?
> > > Could they potentially fly under the radar for years?  I thought there
> > > was something they had to report quarterly to show they're working on
> > > compliance, or something.
> > >
> > > I want to be able to tell they company "Listen, here's what could happen
> > > if you get audited, and here's the chances of you getting audited" in
> > > hopes they would take it seriously.  I don't want to scare them without
> > > knowing the facts, first I want to know the facts, then I will scare
> > > them.  Thanks.
> > > Scott Race
> > > Technology Manager
> > >
> > > JD+A NETWORK SERVICES
> > > 1264 Hawks Flight Court, Suite 200
> > >
> > > El Dorado Hills, CA 95762
> > > P:  916.941.3700  |  F:  916.941.3777
> > >
> > >
> > > ethingHo????