Penetration Testing mailing list archives
RE: Vulnerability Assessment vs. PenTest
From: "Omar A. Herrera" <omar.herrera () oissg org>
Date: Mon, 7 Aug 2006 15:28:05 +0100
Hi Bob, I believe that even if we can't agree on a unique definition for both we should be able to at least distinguish between them in general terms. I've taken into account many of the comments from other people in this discussion and have come with these general definitions that try to include the most relevant aspects of both concepts; comments are appreciated. So, in general terms: 1) Vulnerability assessment A series of tasks consisting of the identification and classification of vulnerabilities, as well as their assessment in terms of technical and/or business impact. 2) Penetration Testing An activity used to assess the feasibility and the technical and/or business impact of intrusions within an organization. Here, intrusion refers to the capability of an attacker to illegally extract, modify, destroy or create information and information processing functions. Notes: * Scope is left out. You can have Vas and PTs at network, O.S. or application levels for instance, comprising any number of systems. The scope is particular to the engagement. Also, we have internal and external assessments, and there are a number of additional scope requirements such as 0-knowledge restrictions (and these restrictions are more or less useful depending on the context). * Depth (i.e. level of detail) is left out. I don't think that a general definition should describe the level of detail of the tests nor the exact methods to perform them. This also means that I don't manual testing for any of them. In fact, the only way to perform an effective VA on a proprietary application might be through manual testing. This also means that manual testing (although I personally encourage and support it) is not strictly required for a task to qualify as a penetration test. From the definition above it is clear that using automated tools to prove that intrusion is possible and then analysing the results would be enough. If we would like to refer to a particular level of detail we can always name the particular type of test (e.g. "manual penetration testing" or "tool based, automated, vulnerability assessment"). * I'm also using a couple 'and/or's in the definitions. Both assessments can be executed in two dimensions: the technical and the business dimensions. I personally recommend considering both dimensions if possible, but it would be naïve to think that this is always required. For example, a company might only require a tool based, automated and technical, vulnerability assessment while considering performing the business assessment on their own. * The quality of the report is not included in the definitions. This is a matter of the quality of the assessment (good or bad) and not part of the description itself. We deduce from experience (many past posts in this forum seem to back this) that the analysis of results should involve some manual work to produce quality assessments (i.e. putting the results in context to produce more accurate evaluations), but stating that any assessment will be good just because some manual analysis was performed is not true. Stating that the automated analysis included in the reports of tools is always bad for any given situation is also false. * There is a big issue of whether one assessment is part of the other. From the definition above I would say that VAs are, in general, independent from PTs, although that wouldn't prevent an assessor from stating the theoretical impact of an intrusion based on VA results. In the case of PTs it is implied that at least some tasks from the VA need to be performed (before or during the PT), for instance, the vulnerability identification (the only way to get in "illegally" is through a vulnerability, which does not always means a technical vulnerability; procedural vulnerabilities also count). * General definitions like these should only serve to distinguish between both concepts and shouldn't be used alone to refer to a specific/required type of assessment. Defining additional scope and quality requirements is essential to give the assessor a clear idea of what is being requested (or for the client to understand what is being proposed in a proposal). Good examples of 'not sufficiently well defined' assessments (in my opinion) are included in the PCI DSS standard (I hope they will take care of this in future revisions), for instance: - 11.2 specifies when and what (a VA) but it never states clearly the level of detail required. However, it gives a few examples of what to include and states that qualified vendors must perform these scans (which implies that those qualify vendors would know exactly the level of detail and the scope that is required). The Security Scanning Procedures document gives some more information. - 11.3 Again, it states what (a pentest) and gives a more or less clear idea of the scope (network infrastructure and applications) but fails to describe the level of detail (must this include manual testing? Are there any time restrictions?). - 11.1 This is not a PT and I wouldn't exactly call it a VA because you dont search specifically for vulnerabilities. This requires a series of tests to assess the effectiveness of security controls and measures to prevent unauthorized accesses (of course, ineffective controls/measures make you vulnerable, but the goal of the test is different). It gives a few examples of what to include in the scope but is again too general. In the case of standards I would expect to find detailed requirements for the scope and measurable results for the QA function (after all, that is what a standard is all about: "a required level of quality or proficiency"). This is what should have happened in PCI DSS with respect to VAs and Pentests, but it looks more like a set of guidelines than a standard. I think that the opposite is causing the confusion that reigns here. We are all trying to define our own standards for what we consider good/acceptable/best VAs and PTs, but we are forgetting to extract the relevant elements of each concept and set aside the aspects that define a specific type of assessment. Maybe it is time to create the "Securipedia" ;-). As StyleWar previously stated, being in part an art is not necessarily a bad thing, and I would say that we require some degree of it in both types of assessment (probably more with pentests, where creativity is quite useful). But there is definitely a lack of agreement on key things in our industry. I also agree with the sentiment of Daniel that it is surprising that this lack of agreement exists in relatively basic concepts. To put it in other words I would say that we are not sustaining and developing our profession with adequate levels of formality. Regards, Omar Herrera
-----Original Message----- From: Bob Radvanovsky [mailto:rsradvan () unixworks net] ----- Original Message ----- From: Daniel Accioly Rosa [mailto:listas.accioly () terra com br] To: pen-test () securityfocus com Subject: RE: Vulnerability Assessment vs. PenTestWhat I find most interesting in these discussions is that even tough weareall seasoned professionals, we can't agree 100% on a definition neithertoVulnerability Assessment or Pen Testing. What lesson should we take from this? I'm not saying that we don't knowwhatwe are doing (most of use here are very good professionals), but maybethereis too much "art" in this job... Each day that goes by I believe moreandmore that we need to agree on common grounds on how we perform ourduties... I think much of it comes from the "human factor" in that we view things differently; thus, the method of approach we take will always be different. Even the so-called "security standards" are variable in of themselves, too. So, don't just be pointing fingers at the individuals. Organizations that are representative of this industry, too, have issues with agreeing upon a set of standards or guidelines. ISO is one way, SANS is another, DoD yet another, NSA even still another, etc., etc. To me, the overall schema of things is how we (as an industry) perceive threats and their vulnerabilities, and what are associated to them. Once that is agreed on, the rest of it might fall better into place. ;))You are right StyleWar, coffee now would be nice.. :)
------------------------------------------------------------------------------ This List Sponsored by: Cenzic Concerned about Web Application Security? Why not go with the #1 solution - Cenzic, the only one to win the Analyst's Choice Award from eWeek. As attacks through web applications continue to rise, you need to proactively protect your applications from hackers. Cenzic has the most comprehensive solutions to meet your application security penetration testing and vulnerability management needs. You have an option to go with a managed service (Cenzic ClickToSecure) or an enterprise software (Cenzic Hailstorm). Download FREE whitepaper on how a managed service can help you: http://www.cenzic.com/news_events/wpappsec.php And, now for a limited time we can do a FREE audit for you to confirm your results from other product. Contact us at request () cenzic com for details. ------------------------------------------------------------------------------
Current thread:
- Re: Vulnerability Assessment vs. PenTest, (continued)
- Re: Vulnerability Assessment vs. PenTest Gareth Davies (Aug 07)
- Re: Vulnerability Assessment vs. PenTest Bob Radvanovsky (Aug 04)
- Re: Vulnerability Assessment vs. PenTest Alice Bryson <abryson () bytefocus com> (Aug 05)
- Re: Vulnerability Assessment vs. PenTest Arkem Paul (Aug 05)
- Re: Vulnerability Assessment vs. PenTest Christine Kronberg (Aug 06)
- RE: Vulnerability Assessment vs. PenTest Richard Feist (Aug 07)
- Re: Vulnerability Assessment vs. PenTest xelerated (Aug 07)
- Re: Vulnerability Assessment vs. PenTest Magdelin Tey (Aug 07)
- Re: Vulnerability Assessment vs. PenTest Alice Bryson <abryson () bytefocus com> (Aug 05)
- RE: Vulnerability Assessment vs. PenTest StyleWar (Aug 08)
- RE: Vulnerability Assessment vs. PenTest Omar A. Herrera (Aug 07)
- Port Listening Chris Esezobor (Aug 10)
- RE: Port Listening Luke Walsh (Aug 10)