Penetration Testing mailing list archives
Re: [PEN-TEST] Evaluating Auditors Abilities
From: Deri Jones <Deri.Jones () NTA-MONITOR COM>
Date: Fri, 8 Sep 2000 09:40:28 +0100
At 10:56 07/09/00 -0700, Mark wrote:
1. Management got ripped off by what you describe
Yup! We've heard stories like this so often - poor testing at inflated prices from 'big name' suppliers... It's the norm, not the exception, I would say :<(
2. Many auditing firms unless they present their credentials will just run the typical commercially available toolsuite plus a couple of hobbled together tools, produce a nice report and never validate the results
Yup
...snipped 9. No there are no certifications or Industry Groups that monitor or endorse the auditors.
Actually this is not quite true - in Europe, the UK Government seems to be unique in having a certification scheme for testers - maned by the govt body 'Communications Electronics Security Group' (www.cesg.gov.uk) - the scheme is called the CHECK scheme. We were one of the founding members back in January 1999. (Here at NTA Monitor we do more pen testing than any other company in Europe (eg over 520 test assignments in 1999, over 200 clients on current quarterly or monthly test contracts - from all 5 continents).) Now whether the CHECK scheme is a high enough quality standard- humm?... It was quite a debate in the set-up phase - just how high to set the bar. We think it's too low...but we're just one voice... But, using a CHECK member *does* provide some level of assurance. But the *most* important thing you can do ----------------------------------------- - is ask the vendor for a *list* of customers - if they can't show they've done work for say 30 or 50 companies - then have they really got the experience? Once you've got the list, you can select the references *you* want to speak to - not just call the 2 or 3 names the vendor offers - it's easy to have 2 happt customers! One thing that differentiaites us from the 'Big 5's of this, is we have customer lists on our web site and in our literature - and we invite new prospects to choice their choice of who they want to call. Maybe that's why we are picking up a rush of new clients who are dropping the Big 5..., and the like :<) Deri Jones NTA Monitor
ANyone with enough money and political saavy can open up shop (whether you are name or not), invest some money in a fancy web site, claim to have all the vulnerabilities and exploits, and provide cruddy service, but are backed by large VC.. /mark At 12:46 AM 9/7/00 -0400, Derrick wrote:Dear Pen-Testers, Recently I underwent something that had me thinking about Security Auditing companies and others (Big accounting firms that offer a side service of auditing). Management decided that we needed to be audited by an outside firm, which I am in full favor of. The problem came about in what an un-named auditor did. Firewalls tend to cause false positives in some tests and other anomalies that many auditors may not be aware of. So they performed this audit which we did pick up and were aware of. What happened next is what baffles me. The auditors did not understand the results that nmap and other tools gave them. Near the end of the business day they contact management proclaiming they have found numerous security issues and even some backdoors in our network. After a long couple of days of testing we found none of these issues were correct, and we then spent many hours and several meetings explaining that the firm hired didn't seem to know what they were doing. Management made the default comment of "We are paying them a lot so they must be right, fix these problems". After several days of explaining why they results were wrong and verifying the network we came out to show that the auditors did in fact improperly interpret the results. The end result is management walks away wondering if they got ripped off or if we were just trying to cover problems. It also caused a lot of overtime and extra work for us to explain and prove the network to management. So the end questions are these. How can companies decide which auditors really do a decent job and are worth their value ? Are there any certifications or Industry groups out there or on the horizon that will evaluate and endorse auditors ? What is the best approach from a Network Admin position to counter end results delivered by auditors if they seem to be in error ? Has anyone else been through this, and is destined to get worse before getting better ? Thanks for any thoughts or comments, Derrick
Current thread:
- Re: [PEN-TEST] Firewall identification and penetration Mike Ireton (Sep 02)
- Re: [PEN-TEST] Firewall identification and penetration Ben Lull (Sep 06)
- [PEN-TEST] Evaluating Auditors Abilities Derrick (Sep 07)
- Re: [PEN-TEST] Evaluating Auditors Abilities Steve (Sep 07)
- Re: [PEN-TEST] Evaluating Auditors Abilities Domenico De Vitto (Sep 07)
- Re: [PEN-TEST] Evaluating Auditors Abilities Teicher, Mark (Sep 07)
- Re: [PEN-TEST] Evaluating Auditors Abilities Max Vision (Sep 08)
- Re: [PEN-TEST] Evaluating Auditors Abilities Deri Jones (Sep 08)
- [PEN-TEST] Evaluating Auditors Abilities Derrick (Sep 07)
- Re: [PEN-TEST] Firewall identification and penetration Jeffrey Denton (Sep 07)
- Re: [PEN-TEST] Firewall identification and penetration Gary E. Miller (Sep 07)
- Re: [PEN-TEST] Firewall identification and penetration Ben Lull (Sep 06)