BSIMM update (informIT)

[arian.evans at anachronic.com (Arian J. Evans)]

Hola Gary, inline:


On Wed, Feb 3, 2010 at 12:05 PM, Gary McGraw <gem at cigital.com> wrote:
> > Strategic folks (VP, CxO) ...Initially ...ask for descriptive information, but once they get
> > going they need strategic prescriptions.
>
> Please see my response to Kevin. ?I hope it's clear what the BSIMM is for.
> ?It's for measuring your initiative and comparing it to others. ?Given some
> solid BSIMM data, I believe you can do a superior job with strategy...and
> results measurement. ?It is a tool for strategic people to use to build an initiative that works.


My response was regarding what people need today. I think BSIMM is too
much for most organization's needs and interests.


> > Tactical folks tend to ask:
> > + What should we fix first? (prescriptive)
> > + What steps can I take to reduce XSS attack surface by 80%?
>
> The BSIMM is not for tactical folks.

That's too bad. Security is largely tactical, like it or not.


> ?But should you base your decision regarding "what to fix first" on goat sacrifice?
> ?What should drive that decision? ?Moon phase?


It doesn't take much thinking to move beyond "moon phase" to pragmatic
things like:

+ What is being attacked? (the most | or | targeting you)
+ What do I have the most of?
+ What issues present the most risk of impact or loss?
+ etc.

Definitely doesn't take Feynman. Or "moon phase" melodrama.


> > Implementation level folks ask:
> > + What do I do about this specific attack/weakness?
> > + How do I make my compensating control (WAF, IPS) block this specific attack?
>
> BSIMM != code review tool, top-n list, book, coding experience, ...

Sure. Again, I was sharing with folks on SC-L what people out in IRL
at what layers of an organization actually care about.


> > BSIMM is probably useful for government agencies, or some large
> > organizations. But the vast majority of clients I work with don't have
> > the time or need or ability to take advantage of BSIMM. Nor should
> > they. They don't need a software security group.
>
> Where to start. ?All I can say about BSIMM so far is that is appears
> to be useful for 30 large commercial organizations carrying out real
> software security initiatives.


BSIMM might be useful. I don't think it's necessary. More power to
BSIMM though. I think everyone on SC-L would appreciate more good
data, and BSIMM certainly can collect some interesting data.


> But what about SMB (small to medium sized business)?

I don't deal a lot with SMB, but certainly they don't need BSIMM. They
might make use of the metrics (?) though I doubt it. They want, and
probably need, Top(n) lists and prescriptive guidance.


> Arian, who are your clients?

Mostly fortune-listed (100/500/2000, etc.), but including a broad
spectrum from small online startups to east coast financial
institutions. Mostly people who do business on the Internet, and care
about that business, and security (to try and put them all in a
singular bucket).


> ?How many developers do they have?

> From a handful to thousands, to tens of thousands. Why?


> ?Who do you report to as a consultant?

I haven't done consulting in years.


> ?How do you help them make business decisions?

With Math, mostly, and pragmatic prioritization so they can move on
and focus on their business, and get security out of the way as much
as possible.


> Regarding the existence of an SSG, see this article
> <http://www.informit.com/articles/article.aspx?p=1434903>.
> ?Are your customers too small to have an SSG? ?Are YOU the SSG?
> ?Are your customers not mature enough for an SSG? ?Data would be great.

Not many organizations need an SSG today, unless they have a TON of
developers and are an ISV, or a SaaS version of an old-school ISV
(Salesforce.com).

I do think they benefit highly from a developer-turned-SSP. But I
don't think there are enough of those to go around. So the network and
widget security folks, and even the policy wanks, are going to
probably play a role in software security.


> > But, as should be no surprise, I cateogrically disagree with the
> > entire concluding paragraph of the article. Sadly it's just more faith
> > and magic from Gary's end. We all can do better than that.
>
> You guys and your personal attacks. ?Yeesh.

Gary -- you've been a bit preachy and didactic lately; maybe Obama's
demagoguery has been inspiring you. So be prepared to duck. I'll
define my tomatoes below. Alternately you might consider ending your
articles with "Amen". :)


> ?I am pretty sure you meant the "next to last" paragraph

You are correct.


> "As I have said before, the time has come to put away the bug parade boogeyman
> <http://www.informit.com/articles/article.aspx?p=1248057>,
> the top 25 tea leaves <http://www.informit.com/articles/article.aspx?p=1322398>,
> black box web app goat sacrifice, and the occult reading of pen testing entrails.
> It's science time. ?And the more descriptive and data driven we are, the better."
>
> Can you be more specific about your disagreements please?


Yes, I think, quite simply: that paragraph has a sign swinging over it
that says "out to lunch".


1. Bug Parades are great. They can include design flaws and such as
well. (Don't need a semantic debate about bug vs. flaw, please; we all
get it.)

It's time to refine our bug parades with more real world data and make
sure they reflect people's needs. If flawed design patterns need to be
in there, they can be.


2. Top25 has a very valuable place. It gets things done. It moves the
bar. It gets seatbelts installed.


3. Black Box testing is super valuable. It gives you a run-time
measuring stick to evaluate what works and what doesn't. Is developer
training working? Is your WAF working? Is your source code scanning
properly including the right libraries?

Not to mention BB gives you an immediate, essential attack surface you
need to know. And yes, I mean Need to Know.


4. Pen Testing is very valuable. It tells you what you absolutely,
positively, need to care about, at a minimum. There are many reasons
pen testing is valuable, and still sought-after, but that's for a
longer discussion.

I've been picking up Marcus Ranum detritus for over a decade by
helping people confused with his rants about how "penetrate and patch"
doesn't work, and how we need to start from the ground up and "build
secure networks" and "write secure code". Maybe Ranum sandbagged you
with one of his rants and it stuck?

Anyway -- I can help you out here if you want to discuss further.


5. What world of science do you live in?

Modern science is driven by statistics. I provide my customers math
and stats, and constantly work to improve this. I think, in fact, we
provide more stats than anyone on the planet today in the field of
webappsec.

The fundamental premise of science is that a hypothesis becomes a
theory when you have tests that can be performed by two or more
people, and publicly verified. So we're doing all of that above.
That's definitely science.

I'm not sure where the "It's science time" comes in. Is there a dance
that goes with that?



> We'll just ignore the Nader > Feynman stuff.

I did not say Nader > Feynman.

I said Nader fundamentally improved society by changing business SOP
and promoting safety controls that affected millions, through use of
bug parades and Top(n) lists, and awareness campaigns.

Feynman, not so much.

I'm not sure what your goal is.

Advance SoA: If it's to advance the state-of-the-art in software
security, then BSIMM may be a worthy, lofty goal, and rambling about
Feynman and "science" may be related.


Improve Immediate Quality: If it's to pragmatically improve the
quality of software security, then that's a different thing.

We could do some basic work on improving quantitative vs. qualitative
metrics definitions in software security, and improve focus on finding
out what is really attacked, and what attack surface is most
immediately at risk of compromise, and move the bar a meaningful
amount.

I guess I'm just not a fan of huge GW Bush style programs where you
mobilize a special task force and invade another country to count WMDs
before you can identify that you have a basic problem and take steps
to solve it. I'm not even sure big programs like that improve
security.

I think a couple of guns in a couple of hands of a couple of pilots
and, wow...we might not be in Afghanistan or Iraq.

I tend to lean more towards pragmatic solutions like that in software
security. I know most executives I deal with seem to lean in a similar
fashion.

I hear ESAPI makes a good gun these days. Whadda they call that thing?
ESAPI(waf)?

---
Arian J. Evans
"When a strong man, fully armed, guards his own homestead, his
possessions are undisturbed." Luke 11:21