Penetration Testing mailing list archives
Re: Evaluating Pen Testers
From: Daniel Kennedy <danielkennedy74 () gmail com>
Date: Thu, 15 Apr 2010 12:49:46 -0700
At this point you're resorting to ad hominem attacks or misdirections rather then addressing anything, and thus we've entered into a spiral of the kind described by Godwin's Law. Some of your ham-fisted insults are fairly direct, and since you have the fortitude to be direct and make them under your own name, I'll take a second to respond:
If you're not an addicted self-learner,
Suffice to say just pasting links in a response without any reference as to what point they are supporting is a waste. I tried to say that more diplomatically. "Go read this and come back" responses on the Internet are worse than useless.
You don't seem like an economist or a business-level decision-maker to me.
Are you sure? I check out who I'm speaking to usually. Positions like CISO and Vice President of Application Security usually have some decision making authority.
Robert Auger recently said...
Generic quotes about the problems of consulting in general, or citing the obvious, well identified, limitations of penetration testing doesn't really advance any discussion. It seems to every question your answer is some manner of application security mitigation, along the lines of when you have a hammer everything starts looking like a nail.
Actually the best penetration-testers come from other fields
The origin of the penetration tester is not related to having a resource, with other job responsibilities, also attempt to be a good penetration tester. There is a time and self learning tract that must be undertaken to be successful once the switch to becoming a security tester is made.
You want Ivan Arce and HD Moore?
I can ask him, but I think Rapid7 is keeping him pretty busy.
You don't have to be a member of OWASP. There are no dues and no fees. OWASP is the opposite of ISSA
I've been a member of both at various times, but I wasn't talking about myself I was talking about evaluating a potential penetration testing partner. I was suggesting that not every capable tester is necessarily also a member of OWASP.
Why? Isn't an alert box or !exploitable output (especially peer/tool> reviewed) enough for you?
In web application security testing, both of those things are proof of exploitation, falling under what would count as supplying a proof of exploitation.
What I'm trying to say is that there are plenty of people who work for Core and do penetration-testing (or write Impact). Just because they use Impact doesn't mean that they only use the RPT module. Get over it already -- you were being presumptuous and I called you on it.
I've used Core in a professional setting as well as a capture the flag, I understand its capabilities and limitations and am impressed with it. But you "called me out" on a point you made up in your head, there was no presumption on my part. Go back and read what I wrote, you'll see you were mistaken. So you'll excuse me if I "don't get over it" when you suggest I insulted someone when I most certainly did not.
Somehow I'm sure we don't disagree on much. You just seem to be new and I just feel like I'm over-educating you for free.
A search on linkedin, Google, or whatever takes about two seconds and saves you from making asinine statements like this one. But if I need more help I'll be sure to give Gotham Digital Science a call.
Background check companies (you definitely want one that is listed on napbs.com) are notoriously expensive
Not really, but if you've never been involved in having background checks done for a company, I could see why someone might think that.
Maybe they are too busy working to be talking on mailing-lists?
Its good we don't suffer from that problem.
Define "usually"? Most people just don't want to be bothered with industry punditry.
Your leadership seems to understand the value of such punditry: http://www.darkreading.com/security/attacks/showArticle.jhtml?articleID=212400450. Most of the company and researcher names you dropped below also have a history of understanding basic marketing.
Doesn't sound like you read it to me,
Forrester corporate memberships are a great thing. Later and done, Dan Kennedy On Thu, Apr 15, 2010 at 1:37 AM, Andre Gironda <andreg () gmail com> wrote:
On Tue, Apr 13, 2010 at 1:59 PM, Daniel Kennedy <danielkennedy74 () gmail com> wrote:Are you referring to CHECK? They are still verifyingDon't recall, it was a presentation years ago at a conference. Doesn'tIt would be nice if the CHECK people would respond here about what they offer and why it's worth anyone's time.I sympathize with the security consumer when trying to find someone competent to perform a test.I don't necessarily sympathize. There are better activities to perform, such as threat-modeling or code-assisted app assessments. Pen-testing is not everything that everyone makes it out to be.I do not like the words "manual" or "automated". Not allWhether you like the terms or not is not really material. A good penetration test likely has some automated tasks for time savings (these are time boxed tests) and some hand, or manual, or custom testing, whatever you'd like to call it. That said there are some penetration testers out there who use no well known tools and are at the top of the game. We are in complete agreement that fully automated vulnerability scanning is not effective and that human involvement (using knowledgeable humans) is a key component in a successful vulnerability management program.Define "well-known" tool. I think everybody uses Burp Suite Pro, unless there are specific circumstances that require using UHooker or Echo Mirage before using Burp Suite Pro. Certain people like Jared DeMott or Charlie Miller gravitate towards EFS or the Pedram Amini PaiMei tool. Microsofties gravitate towards IOActive, Leviathan Security, and Casaba Security tools. The security blogging community gravitates towards Matasano and Gotham Digital Science tools (which rely a lot on Burp, btw). And, yes, Immunity Security and Core Security guys like their toolchains built into their commercial products. The AttackResearch/Offensive-Security guys are totally into Metasploit (although there was a recent blog post about Burp, which shows up often).If it makes sense, then yes. With an in house team, there are all kinds of company policies affecting the type of software that can be used. But if your point is that a knowledgeable person must be equipped with adequate tools that the person requests, then sure.If an internal penetration-testing team can't walk over to the exceptions-management team and make an exception, then there is some sort of breakdown in that particular InfoSec/Risk-Mgmt department. Or it's a government agency that doesn't hire state-supported random kids.Not taking reading assignments that aren't linked as a reference to a pointIf you're not an addicted self-learner, then you will probably fail as a penetration-tester, or even finding a good one. Why wouldn't you just take my word for it?So, to me, I expect to see results in a penetration test that show, at a high level, what was attempted, what is believed to be exploitable, and what was exploited, with exploitation within the penetration test's time frame being the end goal of the testing team.Yeah asking for that stuff usually requires money. It's probably better to find the developer and ask him to show you where the code is obviously secure then to hire a penetration-tester at 150-300 US dollars per hour per person to exploit a target that could take hours, days, or weeks to write an exploit that is now unusable unless you plan on selling it or using it to an adversarial advantage. Another reason why penetration-testing is flawed.What you're proposing could be interpreted as being handed a report of possible vulnerabilities (an incomplete one at that since you're stopping testing at something 'believed to be capable of exploitation'. That's probably useful information, but for me not useful enough to warrant spending money on a penetration test over having someone do a vulnerability scan which will show me all possible or believed routes of exploitation.What? Ok, look, man. You can pay for whatever you want to pay for -- it's your money. I'll just say that I don't agree with your approach.Actual exploitation, which involves finding a vulnerability or chaining vulnerabilities together, in a custom environment, to achieve a proof that a system can be exploited, is difficult and what I'm looking to have be attempted in a penetration test.Why? Isn't an alert box or !exploitable output (especially peer/tool reviewed) enough for you? Isn't an obvious lack of input validation combined with improper coding practices enough to say -- let's make this obviously secure in the code instead of spending time on penetration-testing?Almost all RFQ analysis is followed by Case Study analysis and extremely high-quality References before hiring an application security consulting company.To assume that all security companies/consultants are hired after aI didn't assume anything. You seemed to have assumed that I assumed something. Re-read what I wrote.In house is valuable because you retain available talent and generally can spend more time testing more things. That said it does not have the economy of scale of hiring outside consulting help, and manyRobert Auger recently said, "Many consultants don't seem to understand practical business risk management (or often aren't around long enough to get good at these activities) and instead are used to providing generic advice for solving a problem with little understanding on how to accomplish this in the real world (at both a technical level and business ). An advantage of doing appsec full time is the ability to develop real solutions and see how they can be improved based on real world experience rather than educated guesses".companies (especially in this economy) are not of the shape and scale to justify maintaining a full time penetration testing team. Having aYou don't seem like an economist or a business-level decision-maker to me.single resource runs the risk of having that resource leave at any time (no coverage overlap), and robs the penetration testing team of the benefit of collaborating during testing (most testers are notThey could always collaborate with the consultants on deck.experts in every system or type of system they encounter). Having a resource that does other things, but sometimes tries to do penetration tests, leaves you with a party not fully committed or immersed in the infosec industry doing your testing.Actually the best penetration-testers come from other fields. Inciting others inside the organization to take up the pen-testing torch is a very wise move. And, boom, you've got even more collaboration.#Certification there are a great many security luminaries without any. The CEH is gaining some traction, not sure if that's a good or bad thing yet.I think it's a bad thing. What does China use to certify their penetration-testing talent?Not sure why what China is doing or not doing is important. But IOops. I think I forgot to mention that the US DoD is now requiring the CEH for specific roles.other words, the first question in the interview should be "Which BackTrack tool did you write or contribute to?" and the second question should be "When was the last time that you spoke at an OWASP local chapter meeting?"What is they're not a member of OWASP, or just don't want to speak atYou don't have to be a member of OWASP. There are no dues and no fees. OWASP is the opposite of ISSA and ISACA. You don't require a background check to come to meetings like InfraGuard. You just show up. You just ask the chapter leader if you can present at a future meeting. You just send the OWASP Board an email and start your own chapter if you don't have one, or if yours seems dead. It's actually easier than what I'm describing.OWASP meetings. OWASP is a great outfit, but not everyone is a member.OWASP does need more sponsorship and money in the form of memberships. They honestly do. But that has nothing to do with this conversation.What if its not an application penetration test? What if they don't use BackTrack (which is a great tool as an aside)?If they don't like BackTrack, then I hope they have another answer that would respect the purpose of that question. A bad answer might be "What's BackTrack?", unless it's followed up with "Oh yeah, I don't use that garbage; I built my own pen-test OS platform"Further you're making my point below for me, that a company or person with verifiable talent is a better hire then one without.Somehow I'm sure we don't disagree on much. You just seem to be new and I just feel like I'm over-educating you for free.I don't think I said anything about anyone at Core Impact. Core Impact is a tool that has reached a point in maturity where even a fairly non-technical person can know an IP address or range, run a scan, run a set of vulnerabilities based on that scan, and install the Core Impact backdoor on the target which would meet the definition of most penetration tests, but which is probably not worth paying someone to do. Its moronic to think such a statement was an insult.No, Core is much more than the RPT module -- I don't think you understand that. Recently, Core added support to drop into Metasploit. Wait until all of the web application scanners add that same sort of drop-in support for Burp. Or Metasploitburpuby. What I'm trying to say is that there are plenty of people who work for Core and do penetration-testing (or write Impact). Just because they use Impact doesn't mean that they only use the RPT module. Get over it already -- you were being presumptuous and I called you on it.In the hands of an experienced person, Core Impact is a powerful tool and one that can be a help during a penetration test. So is Metasploit. The point is that if I'm paying the money to bring someone in, I want that experienced person. The point I made above is that I'dYou want Ivan Arce and HD Moore? I think they already have day jobs that keep them busy...rather have an experienced person with Metasploit then someone with no experience using Core Impact. You can write it the other way too, I'dYou forgot that I told you that you should let people run their own tools.rather have an experienced person with Core Impact then an inexperienced person with Metasploit.Ok, I think you're starting to understand now!This all leads to not using "the person lists Metasploit as a tool" as a way to eliminate candidate companies or persons for doing your penetration testing.The best way to eliminate someone from your list of candidates is to not know them personally or what they are capable of. If you don't know anybody -- go to a local OWASP chapter meeting, or perhaps a CitySec event (e.g. ChiSec), or maybe a Hackers Anonymous (e.g. AHA). Or go to a cheaper, regional conference such as Toorcon/Toorcamp, Shmoocon/SOURCE, or a SecurityBSides event. Also -- be an addicted self-learner and post stuff to mailing-lists, read blogs/twitter, and make friends and influence people by reading books.Insurance, especially with limitations in coverage, may protect the security company in cases of legal liability but provides a small amount of protection to the hiring company. In most cases, if a penetration tester went rogue with information from a penetration test, the resulting reputation damage and bad publicity would be of greater value then the insurance settlement.When insurance fails, litigation is quick to follow... BTW IANALSo I stand by checking people out, both from a legal protection standpoint, but also because you want the engagement to be successful in your environment and therefore should check out the backgrounds of the people involved with the test. I don't view possessing insurance as an end all indicator of anything.I suggest criminalsearches.com (it's free and it works). Also good to do an SSN check -- http://www.ssa.gov/employer/ssnv.htm Verify their business license, do those case studies, and verify a reference if you really want to do more. Track their parcel addresses back and make sure you know where they live/work and zoom in on it from Google Maps if you are really paranoid. Background check companies (you definitely want one that is listed on napbs.com) are notoriously expensive and difficult to deal with -- so best of luck with that strategy. Perhaps it's best to build your own background check system. Even LexisNexis and ChoicePoint are usually a total failure.With respect to their personal wishes, one would immediately ask why they want to keep a low profile. Assuming there is nothing untowardMaybe they are too busy working to be talking on mailing-lists?there, those folks should understand that there abilities have to be known to someone in order for a demand to be there for them. EvenKnown, yes. By Google? No!folks with pseudonyms usually leave a trail to find them, they just don't want to be identified trivially and sent nonsense correspondence by people who don't understand the information security industry.Define "usually"? Most people just don't want to be bothered with industry punditry.In reality, the decision to hire one company or individual over another is based on a range of factors (that could include an RFQ or RFP) some more legitimate factors than others. But if I wanted to hire someone, and one candidate had something like this online: And the next guy had no information I could verify, then I would probably look more favorably on the skills of the first candidate.Isn't the Internet great? I think we agree on these points ;>http://www.forrester.com/rb/Research/techradar%26trade%3B_for_srm_professionals_application_security%2C_q3/q/id/48394/t/2Good example of how RFP processes can be rife with document templates filled with boilerplate language.Doesn't sound like you read it to me, but it's not free information for probably a damn good reason. Cheers, Andre
------------------------------------------------------------------------ This list is sponsored by: Information Assurance Certification Review Board Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT and CEPT certs require a full practical examination in order to become certified. http://www.iacertification.org ------------------------------------------------------------------------
Current thread:
- Evaluating Pen Testers Daniel Kennedy (Apr 12)
- Re: Evaluating Pen Testers Stephen Mullins (Apr 14)
- Re: Evaluating Pen Testers Andre Gironda (Apr 14)
- Re: Evaluating Pen Testers Daniel Kennedy (Apr 14)
- Re: Evaluating Pen Testers Andre Gironda (Apr 15)
- Re: Evaluating Pen Testers Daniel Kennedy (Apr 15)
- Re: Evaluating Pen Testers Andre Gironda (Apr 19)
- Re: Evaluating Pen Testers Nathan Sportsman (Apr 20)
- Re: Evaluating Pen Testers Pete Herzog (Apr 22)
- Re: Evaluating Pen Testers van van (Apr 22)
- Re: Evaluating Pen Testers Daniel Kennedy (Apr 14)