Full Disclosure mailing list archives
[Professional IT Security Providers - Exposed] Redspin, Inc. (C+)
From: secreview <secreview () hushmail com>
Date: Mon, 10 Aug 2009 21:11:46 -0700 (PDT)
We received 22 requests from different people to perform a review of Redspin! Their website can be found at http://www.redspin.com. We haven’t done a review of anyone in quite a while, the last review that we did was for Pivot Point Security who got an A (we still recommend them). We apologize for this long delay but we have been very busy traveling (yes we still have jobs doing consulting work sometimes). As you can see from the comments that we received in other posts we have a lot of catch up work to do, but to be honest we are not sure that we will be able to do it. This review might be our final and last review depending on how much more travel we have. (We have lives, some of us have families, and we can’t keep doing this for free even though we feel that this is a great service). We did a lot of research on Redspin and we managed to get a copy of two reports that they did for two different customers. We won’t share those reports with you because that would be unethical, don’t ask. Redspin claims that it is a “pure penetration testing firm”. What they mean by “pure penetration testing” is that they do not resell third party software or hardware. They also say that “don't find problems on your network so that [they] can make more money; [their] penetration testing services reveal vulnerabilities, [that] will help you become more secure.” We verified their claim with our own research. Redspin will not try to sell you software or hardware… but they might try to sell you software as a service. (see their www.jetmetric.com website). Redspin takes it a step further and is brutally honest about their methodology for delivering penetration-testing services. They openly admit that their services rely on automated vulnerability scanners (Nessus) and are enhanced by manual testing. In fact, Redspin says that automated scanners “can miss about 40% of the security risk so they alone do not adequately assess risk. Furthermore, about half of the findings from a vulnerability scan are false positives”. Any security company that relies on automated scanners can weed out false positives, but doing that doesn’t really increase the depth and accuracy of testing. A false positive, also known as an error of the first kind, or a Type I Error, is the rejection of a null hypothesis when it is in fact true. In more simple terms, this is the error of observing a difference when in fact there isn’t one. Identifying false positives is fairly easily done, as it only requires inspecting the results produced by a scanner. But what about False Negatives? A False Negative, also called a Type II Error, or an error of the second kind, is the error of failing to reject a null hypothesis when it is in fact not true. More simply, a False Negative is the error of failing to observe a difference when in truth there is one. So, if an automated vulnerability scanner tests a vulnerable service (a known vulnerability) but the scanner doesn’t detect the vulnerability then the vulnerability is excluded from the report. If this is the case then Redspin’s methodology will break down because there will be no result in the report for Redspin to manually test. That vulnerability will fly under the Redspin radar but might not be missed by a hacker. So how many vulnerabilities does Redspin miss? It’s a question worth asking. Redspin does say that “vulnerability scanning is not suitable on its own as a complete or billable service offering, it does provides some value in the early reconnaissance phase of a more comprehensive External Network Security Assessment”. They have a typo in that sentence, but other than that, they are right. Vulnerability scanning does have a position in the industry and is a huge time saver, especially when testing large numbers of systems. Just don’t rely on one vulnerability scanner like Redspin does, use two or more like the OSSTMM proposes. Redspin says “manual analysis is at the heart of all of [their] assessments which not only gives you confidence that you have a complete view of your security risk, but provides tailored reporting and recommendations enabling simple work-arounds and cost-effective mitigation strategies for most security issues.” Based on our research Redspin’s “manual analysis” isn’t what we expected it to be. It is not based on vulnerability research and is strictly based on the inspection and verification of scanner output. What we can say is that their “manual analysis” doesn’t produce the highest quality reports that ever we’ve seen, but it does produce reports that are higher than average quality. The Redspin reports have very few, if any, False Positives but will contain more False Negatives than a report that is centered on solid (vulnerability) research. One thing that Redspin does that we really don’t like is to ask their customers to lower their defenses before they do testing. That’s right, they ask their customers to white list their scanner’s IP addresses so that the customer’s Intrusion Prevention System doesn’t block the scanner. We verified this during 3 different interviews on three different dates. We even talked to one Redspin customer to confirm it, and they did. We think that a security testing company should be able to test around a customer Intrusion Prevention System. If they can’t then that really brings their capabilities into question. We feel this way because Intrusion Prevention Systems are a part of the networks defenses and they should be tested. Disabling them for a security test prevents them from being tested. If they aren’t tested then how does one know how effective they are? It just doesn’t’ make sense. On top of that, the test won’t properly reflect the actual security level of the network being tested. Something that Redspin claims is that they’ve done is “ground breaking security research”. We’ve searched high and low for this “ground breaking security research” but haven’t found it anywhere, so we’re not sure what they are talking about. When looking at the research page on their website we see white papers that might make good blog entries, but we don’t see any “ground breaking security research”. When we’re told that a company does “ground breaking security research” we expect to see things like them finding security bugs in critical systems, or publishing professional security advisories, and maybe even publishing proof of concept code. Redspin doesn’t do any of that. The only thing that we were able to find was an “Ultr@ VNC 1.0.1 viwer PoC” (and what’s the point of that?). In conclusion, Redspin’s services are slightly better than average. Their manual testing isn’t true manual testing at all; it’s the inspection of output from scanners and the elimination of false positives. We don’t like the fact that Redspin asks its customers to disable their IPS before being tested, and Redspin doesn’t seem to have any Vulnerability Research capability. Its not all bad, Redspin is very honest about their methodology, they are focused on quality, and they are passionate about what they do. We’d recommend Redspin to people with testing requirements that do not require extreme depth and that can afford some False Negatives. By no means is Redspin a company that we’d suggest you stay clear of, but they’re certainly not the best in the industry. As normal, if there are any issues with this review and its truthfulness please let us know and please provide proof. We will make changes if we need to and we strive to be as honest and fair as we can be. Thanks for reading! Score Card (Click to Enlarge) -- Posted By secreview to Professional IT Security Providers - Exposed at 8/10/2009 08:51:00 PM
_______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/
Current thread:
- [Professional IT Security Providers - Exposed] Redspin, Inc. (C+) secreview (Aug 12)
- Re: [Professional IT Security Providers - Exposed] Redspin, Inc. (C+) Gichuki John Chuksjonia (Aug 13)