Darkreading: Secure Coding Certification

[pmeunier at cerias.net (pmeunier)]

I think this discussion ignores the common human failing of
ignoring recommendations and advice on things to do for safety and
security, unless one has already suffered the consequences, or if the
consequences are obviously incoming.  This is why there had to be a law
on the wearing of seatbelts (the needed car analogy ;), and in part why 
exploits have to be created to "prove" to reluctant vendors that 
something really is an issue.  People tend to "think positively" (try to 
ignore possible bad consequences) and not bother with things that "won't 
happen to them".

This is why a white list approach to software development, presenting 
only what you should do, is insufficient by itself.  People need to know 
why they should adopt recommended practices, and sometimes that isn't 
enough still.  In the case of seatbelts, just telling people why wasn't 
enough;  it had to be legislated.

People learn from mistakes, and from bugs. In addition, the capability 
to find bugs and know how serious they are and why is valuable, for 
example in code reviews and in deciding what to do about the alerts that 
code scanning software generates.  It's true that bug finding is only a 
small part of the secure programming landscape.  However, it's a good 
place to start and I think it's a good thing to test.

As I look at the exam blueprints for the GSSP, I see not only low-level 
bugs, but also design-level subject matter, such as privacy and 
encryption, identification, authentication, and session management.  The 
Java/J2EE bluprints in particular contains higher-level issues.  So, 
whereas there are no high-level design or architecture-related 
questions, I think that's just fine because that's not the aim of the 
GSSP.  Of course, the actual questions may be bad -- I don't know.

Having used multiple choice questions for years on secure programming, I 
think they are apropriate tools for some of the knowledge.  The grades I 
give are based 50% on labs and 50% on exams, which are mostly multiple 
choice.  There is also at least one "real code" question where students 
have to explain what is going on, and what should have been done 
instead.  This combination seems to provide a good assessment of the 
learning outcomes.  Unfortunately I haven't tracked how well the 
students did after they graduated -- that would be interesting.

In conclusion, I think dismissing this effort is too harsh, and I 
disagree with much of the criticism voiced so far.  I haven't yet, but I 
intend to write a few questions for the exam, because I think it's a 
worthwhile thing to do.  I won't begrudge SANS for being able to make a 
profit with it.  I think they deserve it for demonstrating leadership 
and having the initiative of creating the certification (and no, I don't 
have links to them or get paid by them -- although there's a small 
reward for exam questions).

Regards,
Pascal Meunier



Gary McGraw wrote:
> Hi Yo (and everyone else),
>
> I'm afraid that the current test focuses all of its attention on BUGS (in C/C++ and Java).  While we certainly need 
> to erradicate simple security bugs, there is much more to software security than the bug parade.  Plus when you look 
> into the material, the multiple choice format makes determining the correct answer impossible at times.
>
> I would rather move away from learning about bugs to learning about defensive programming to avoid bugs in the first 
> place.  The SANS material focuses entirely on the negative as far as I can tell.  Here's a bug, there's a bug, 
> everywhere a bug bug.  Better than nothing?  Maybe.
>
> SANS is very good an soliciting everyone's opinion, piling it all up in a nice package, and then charging users for 
> the result.  SANS is a for profit entity, not a university or a non-profit.  Please don't forget that.
>
> As much as I would love to see a way to determine whether a random coder has security clue, I'm afraid all we will 
> get out of this effort is perhaps a bit more awareness.
>
> gem
>
> company www.cigital.com
> podcast www.cigital.com/silverbullet
> blog www.cigital.com/justiceleague
> book www.swsec.com
>
>
> -----Original Message-----
> From: sc-l-bounces at securecoding.org [mailto:sc-l-bounces at securecoding.org] On Behalf Of Johan Peeters
> Sent: Saturday, May 12, 2007 6:11 AM
> To: SC-L at securecoding.org
> Subject: Re: [SC-L] Darkreading: Secure Coding Certification
>
> I agree that multiple choice alone is inadequate to test the true
> breadth and depth of someone's security knowledge. Having contributed
> a few questions to the SANS pool, I take issue with Gary's article
> when it implies that you can pass the GSSP test while clueless.
>
> There is indeed a body of knowledge that is being tested. SANS has
> been soliciting comments on the document.
>
> kr,
>
> Yo
>
> On 5/11/07, Gary McGraw <gem at cigital.com> wrote:
> > Hi all,
> >
> > As readers of the list know, SANS recently announced a certification scheme for secure programming.  Many vendors 
> > and consultants jumped on the bandwagon.  I'm not so sure the bandwagon is going anywhere.  I explain why in my 
> > latest darkreading column:
> >
> > http://www.darkreading.com/document.asp?doc_id=123606
> >
> > What do you think?  Can we test someone's software security knowledge with a multiple choice test?  Anybody seen the 
> > body of knowledge behind the test?
> >
> > gem
> >
> > company www.cigital.com
> > podcast www.cigital.com/silverbullet
> > blog www.cigital.com/justiceleague
> > book www.swsec.com
> >
> > _______________________________________________
> > Secure Coding mailing list (SC-L) SC-L at securecoding.org
> > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> > as a free, non-commercial service to the software security community.
> > _______________________________________________
>
>
> --
> Johan Peeters
> http://johanpeeters.com
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L at securecoding.org
> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> _______________________________________________
>
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L at securecoding.org
> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> _______________________________________________