Penetration Testing mailing list archives
Re: Evaluating Pen Testers
From: Andre Gironda <andreg () gmail com>
Date: Thu, 15 Apr 2010 03:37:51 -0500
On Tue, Apr 13, 2010 at 1:59 PM, Daniel Kennedy <danielkennedy74 () gmail com> wrote:
Are you referring to CHECK? They are still verifyingDon't recall, it was a presentation years ago at a conference. Doesn't
It would be nice if the CHECK people would respond here about what they offer and why it's worth anyone's time.
I sympathize with the security consumer when trying to find someone competent to perform a test.
I don't necessarily sympathize. There are better activities to perform, such as threat-modeling or code-assisted app assessments. Pen-testing is not everything that everyone makes it out to be.
I do not like the words "manual" or "automated". Not allWhether you like the terms or not is not really material. A good penetration test likely has some automated tasks for time savings (these are time boxed tests) and some hand, or manual, or custom testing, whatever you'd like to call it. That said there are some penetration testers out there who use no well known tools and are at the top of the game. We are in complete agreement that fully automated vulnerability scanning is not effective and that human involvement (using knowledgeable humans) is a key component in a successful vulnerability management program.
Define "well-known" tool. I think everybody uses Burp Suite Pro, unless there are specific circumstances that require using UHooker or Echo Mirage before using Burp Suite Pro. Certain people like Jared DeMott or Charlie Miller gravitate towards EFS or the Pedram Amini PaiMei tool. Microsofties gravitate towards IOActive, Leviathan Security, and Casaba Security tools. The security blogging community gravitates towards Matasano and Gotham Digital Science tools (which rely a lot on Burp, btw). And, yes, Immunity Security and Core Security guys like their toolchains built into their commercial products. The AttackResearch/Offensive-Security guys are totally into Metasploit (although there was a recent blog post about Burp, which shows up often).
If it makes sense, then yes. With an in house team, there are all kinds of company policies affecting the type of software that can be used. But if your point is that a knowledgeable person must be equipped with adequate tools that the person requests, then sure.
If an internal penetration-testing team can't walk over to the exceptions-management team and make an exception, then there is some sort of breakdown in that particular InfoSec/Risk-Mgmt department. Or it's a government agency that doesn't hire state-supported random kids.
Not taking reading assignments that aren't linked as a reference to a point
If you're not an addicted self-learner, then you will probably fail as a penetration-tester, or even finding a good one. Why wouldn't you just take my word for it?
So, to me, I expect to see results in a penetration test that show, at a high level, what was attempted, what is believed to be exploitable, and what was exploited, with exploitation within the penetration test's time frame being the end goal of the testing team.
Yeah asking for that stuff usually requires money. It's probably better to find the developer and ask him to show you where the code is obviously secure then to hire a penetration-tester at 150-300 US dollars per hour per person to exploit a target that could take hours, days, or weeks to write an exploit that is now unusable unless you plan on selling it or using it to an adversarial advantage. Another reason why penetration-testing is flawed.
What you're proposing could be interpreted as being handed a report of possible vulnerabilities (an incomplete one at that since you're stopping testing at something 'believed to be capable of exploitation'. That's probably useful information, but for me not useful enough to warrant spending money on a penetration test over having someone do a vulnerability scan which will show me all possible or believed routes of exploitation.
What? Ok, look, man. You can pay for whatever you want to pay for -- it's your money. I'll just say that I don't agree with your approach.
Actual exploitation, which involves finding a vulnerability or chaining vulnerabilities together, in a custom environment, to achieve a proof that a system can be exploited, is difficult and what I'm looking to have be attempted in a penetration test.
Why? Isn't an alert box or !exploitable output (especially peer/tool reviewed) enough for you? Isn't an obvious lack of input validation combined with improper coding practices enough to say -- let's make this obviously secure in the code instead of spending time on penetration-testing?
Almost all RFQ analysis is followed by Case Study analysis and extremely high-quality References before hiring an application security consulting company.To assume that all security companies/consultants are hired after a
I didn't assume anything. You seemed to have assumed that I assumed something. Re-read what I wrote.
In house is valuable because you retain available talent and generally can spend more time testing more things. That said it does not have the economy of scale of hiring outside consulting help, and many
Robert Auger recently said, "Many consultants don't seem to understand practical business risk management (or often aren't around long enough to get good at these activities) and instead are used to providing generic advice for solving a problem with little understanding on how to accomplish this in the real world (at both a technical level and business ). An advantage of doing appsec full time is the ability to develop real solutions and see how they can be improved based on real world experience rather than educated guesses".
companies (especially in this economy) are not of the shape and scale to justify maintaining a full time penetration testing team. Having a
You don't seem like an economist or a business-level decision-maker to me.
single resource runs the risk of having that resource leave at any time (no coverage overlap), and robs the penetration testing team of the benefit of collaborating during testing (most testers are not
They could always collaborate with the consultants on deck.
experts in every system or type of system they encounter). Having a resource that does other things, but sometimes tries to do penetration tests, leaves you with a party not fully committed or immersed in the infosec industry doing your testing.
Actually the best penetration-testers come from other fields. Inciting others inside the organization to take up the pen-testing torch is a very wise move. And, boom, you've got even more collaboration.
#Certification there are a great many security luminaries without any. The CEH is gaining some traction, not sure if that's a good or bad thing yet.I think it's a bad thing. What does China use to certify their penetration-testing talent?Not sure why what China is doing or not doing is important. But I
Oops. I think I forgot to mention that the US DoD is now requiring the CEH for specific roles.
other words, the first question in the interview should be "Which BackTrack tool did you write or contribute to?" and the second question should be "When was the last time that you spoke at an OWASP local chapter meeting?"What is they're not a member of OWASP, or just don't want to speak at
You don't have to be a member of OWASP. There are no dues and no fees. OWASP is the opposite of ISSA and ISACA. You don't require a background check to come to meetings like InfraGuard. You just show up. You just ask the chapter leader if you can present at a future meeting. You just send the OWASP Board an email and start your own chapter if you don't have one, or if yours seems dead. It's actually easier than what I'm describing.
OWASP meetings. OWASP is a great outfit, but not everyone is a member.
OWASP does need more sponsorship and money in the form of memberships. They honestly do. But that has nothing to do with this conversation.
What if its not an application penetration test? What if they don't use BackTrack (which is a great tool as an aside)?
If they don't like BackTrack, then I hope they have another answer that would respect the purpose of that question. A bad answer might be "What's BackTrack?", unless it's followed up with "Oh yeah, I don't use that garbage; I built my own pen-test OS platform"
Further you're making my point below for me, that a company or person with verifiable talent is a better hire then one without.
Somehow I'm sure we don't disagree on much. You just seem to be new and I just feel like I'm over-educating you for free.
I don't think I said anything about anyone at Core Impact. Core Impact is a tool that has reached a point in maturity where even a fairly non-technical person can know an IP address or range, run a scan, run a set of vulnerabilities based on that scan, and install the Core Impact backdoor on the target which would meet the definition of most penetration tests, but which is probably not worth paying someone to do. Its moronic to think such a statement was an insult.
No, Core is much more than the RPT module -- I don't think you understand that. Recently, Core added support to drop into Metasploit. Wait until all of the web application scanners add that same sort of drop-in support for Burp. Or Metasploitburpuby. What I'm trying to say is that there are plenty of people who work for Core and do penetration-testing (or write Impact). Just because they use Impact doesn't mean that they only use the RPT module. Get over it already -- you were being presumptuous and I called you on it.
In the hands of an experienced person, Core Impact is a powerful tool and one that can be a help during a penetration test. So is Metasploit. The point is that if I'm paying the money to bring someone in, I want that experienced person. The point I made above is that I'd
You want Ivan Arce and HD Moore? I think they already have day jobs that keep them busy...
rather have an experienced person with Metasploit then someone with no experience using Core Impact. You can write it the other way too, I'd
You forgot that I told you that you should let people run their own tools.
rather have an experienced person with Core Impact then an inexperienced person with Metasploit.
Ok, I think you're starting to understand now!
This all leads to not using "the person lists Metasploit as a tool" as a way to eliminate candidate companies or persons for doing your penetration testing.
The best way to eliminate someone from your list of candidates is to not know them personally or what they are capable of. If you don't know anybody -- go to a local OWASP chapter meeting, or perhaps a CitySec event (e.g. ChiSec), or maybe a Hackers Anonymous (e.g. AHA). Or go to a cheaper, regional conference such as Toorcon/Toorcamp, Shmoocon/SOURCE, or a SecurityBSides event. Also -- be an addicted self-learner and post stuff to mailing-lists, read blogs/twitter, and make friends and influence people by reading books.
Insurance, especially with limitations in coverage, may protect the security company in cases of legal liability but provides a small amount of protection to the hiring company. In most cases, if a penetration tester went rogue with information from a penetration test, the resulting reputation damage and bad publicity would be of greater value then the insurance settlement.
When insurance fails, litigation is quick to follow... BTW IANAL
So I stand by checking people out, both from a legal protection standpoint, but also because you want the engagement to be successful in your environment and therefore should check out the backgrounds of the people involved with the test. I don't view possessing insurance as an end all indicator of anything.
I suggest criminalsearches.com (it's free and it works). Also good to do an SSN check -- http://www.ssa.gov/employer/ssnv.htm Verify their business license, do those case studies, and verify a reference if you really want to do more. Track their parcel addresses back and make sure you know where they live/work and zoom in on it from Google Maps if you are really paranoid. Background check companies (you definitely want one that is listed on napbs.com) are notoriously expensive and difficult to deal with -- so best of luck with that strategy. Perhaps it's best to build your own background check system. Even LexisNexis and ChoicePoint are usually a total failure.
With respect to their personal wishes, one would immediately ask why they want to keep a low profile. Assuming there is nothing untoward
Maybe they are too busy working to be talking on mailing-lists?
there, those folks should understand that there abilities have to be known to someone in order for a demand to be there for them. Even
Known, yes. By Google? No!
folks with pseudonyms usually leave a trail to find them, they just don't want to be identified trivially and sent nonsense correspondence by people who don't understand the information security industry.
Define "usually"? Most people just don't want to be bothered with industry punditry.
In reality, the decision to hire one company or individual over another is based on a range of factors (that could include an RFQ or RFP) some more legitimate factors than others. But if I wanted to hire someone, and one candidate had something like this online: And the next guy had no information I could verify, then I would probably look more favorably on the skills of the first candidate.
Isn't the Internet great? I think we agree on these points ;>
http://www.forrester.com/rb/Research/techradar%26trade%3B_for_srm_professionals_application_security%2C_q3/q/id/48394/t/2Good example of how RFP processes can be rife with document templates filled with boilerplate language.
Doesn't sound like you read it to me, but it's not free information for probably a damn good reason. Cheers, Andre ------------------------------------------------------------------------ This list is sponsored by: Information Assurance Certification Review Board Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT and CEPT certs require a full practical examination in order to become certified. http://www.iacertification.org ------------------------------------------------------------------------
Current thread:
- Evaluating Pen Testers Daniel Kennedy (Apr 12)
- Re: Evaluating Pen Testers Stephen Mullins (Apr 14)
- Re: Evaluating Pen Testers Andre Gironda (Apr 14)
- Re: Evaluating Pen Testers Daniel Kennedy (Apr 14)
- Re: Evaluating Pen Testers Andre Gironda (Apr 15)
- Re: Evaluating Pen Testers Daniel Kennedy (Apr 15)
- Re: Evaluating Pen Testers Andre Gironda (Apr 19)
- Re: Evaluating Pen Testers Nathan Sportsman (Apr 20)
- Re: Evaluating Pen Testers Pete Herzog (Apr 22)
- Re: Evaluating Pen Testers van van (Apr 22)
- Re: Evaluating Pen Testers Daniel Kennedy (Apr 14)