Re: Vulnerability statistics

["Steven M. Christey" <coley () mitre org>]

All,

As a part of some ongoing informal work I've done in vulnerability
classification, specific flaw types have been recorded in CVE data for
the past couple years.  This breaks down into about 20 or 25
categories, with approximately 25% being uncategorizable due to (1)
lack of information regarding the issue (thus Michael Howard's
difficulties), (2) flaw complexity, or (3) the flaw is outside the
given set of categories.

For those who are concerned with web application security itself, note
that some web-specific flaw types aren't yet captured in CVE's
internal flaw data, such as PHP code injection (as I've said, this has
been an informal project).  It is also interesting to note that the
stats for software overall can vary widely from those stats related to
web apps specifically; consider buffer overflows, which are
effectively nonexistent in applications that use scripted languages
(except where such scripts interface with other components).

Adam Shostack is correct in his commentary regarding some of the bias
in CVE, especially regarding CVE's focus on widely deployed software
(we have been working on improving our comprehensiveness to include
all software).

Another source of fairly unpreventable bias is in the researcher
community itself.  There are sometimes research "fads" in which people
focus on one or two new or interesting bug types; and each researcher
has his/her own skill set for finding particular kinds of bugs
(witness the results of DJ Bernstein's class, which found a mostly
narrow set of bugs).  Consider the top researchers, who often focus on
finding new classes of vulnerabilities, or in studying software that
has already been through stringent security development and testing
(relatively speaking, at least).

There are also terminological difficulties that can further bias the
data.  For example, integer overflows are often labeled "buffer
overflows;" where available, we've tried to make this distinction in
CVE descriptions by discussing "integer overflows that lead to buffer
overflows," and doing similar things with other bug types that use
buffer overflow attacks for exploitation (so Michael, double-check
your script ;-)

The lack of terminological precision can also skew the results.

Some examples:

  - "malformed input" is often blamed, without any further details;
    but there are many ways that input can be malformed.  Thus this is
    a very broad category.  Similar is "denial of service
    vulnerability" (which is a misnomer since DoS is the *result* not
    the *cause*)

  - Microsoft's security bulletins sometimes mention an "unchecked
    buffer" (MS04-045/CAN-2004-0567), which might technically be a
    buffer overflow, but not a classic style overflow if you're doing
    things like tampering with length fields; an open source example
    is CAN-2003-0695, which describes "buffer management errors."
    This might not be a big deal if you only care about broad
    categories, but it's important if you want to statistically
    demonstrate whether classic overflows are declining in popularity
    or not (I strongly suspect they are, but can't prove it because
    lots of reports just use "buffer overflow").

Finally, since this project has been informal, our internal processes
have not enforced the entry of a flaw type into each new CVE item
(although it's common practice).

These are some of the reasons why I haven't published any more
specific papers or descriptions of flaw trends based on CVE data,
although there were some interesting results that I described in a
presentation at the "Open Source Security Summit" in 2002.  That
presentation also showed some differences in flaw types between open
and closed source software, perhaps related to the use of static
vs. dynamic auditing methods and tools.

All that said - and with major disclaimers that this data is only
SUGGESTIVE and not AUTHORITATIVE, especially with respect to JUST web
bugs, and with the aforementioned BIASES - below are some more precise
statistics based on the past few years of CVE data.  People who've
seen CVE-based flaw stats in the past will notice the addition of
several new categories in this round.


Regards,
Steve Christey
CVE Editor



                         2002        2003        2004        TOTAL    
                        (1495)      ( 906)      (1194)      (3595)   
                      ----------  ----------  ----------  ---------- 
[ 1] buf              22.0% ( 1)  22.5% ( 1)  17.8% ( 1)  20.8% ( 1)
[ 2] CSS              07.9% ( 2)  06.1% ( 2)  05.7% ( 2)  06.7% ( 2)
[ 3] dos-malform      05.8% ( 3)  01.5% ( 7)  03.2% ( 4)  03.8% ( 3)
[ 4] dot              05.6% ( 4)  02.2% ( 6)  02.4% ( 8)  03.7% ( 4)
[ 5] link             02.5% ( 8)  03.2% ( 4)  04.2% ( 3)  03.3% ( 5)
[ 6] infoleak         03.7% ( 5)  01.5% ( 8)  02.3% ( 9)  02.7% ( 6)
[ 7] format-string    02.0% (10)  03.2% ( 3)  02.8% ( 5)  02.6% ( 7)
[ 8] sql-inject       01.7% (11)  02.8% ( 5)  02.5% ( 7)  02.3% ( 8)
[ 9] priv             02.7% ( 7)  00.7% (13)  01.3% (10)  01.7% ( 9)
[10] metachar         02.8% ( 6)  00.7% (14)  00.9% (12)  01.6% (10)
[11] int-overflow     00.5% (20)  01.4% ( 9)  02.8% ( 6)  01.5% (11)
[12] crypt            02.4% ( 9)  01.1% (10)  00.5% (15)  01.4% (12)
[13] perm             01.3% (15)  00.6% (15)  00.9% (11)  01.0% (13)
[14] auth             01.3% (14)  00.6% (16)  00.3% (20)  00.8% (14)
[15] dos-flood        01.3% (13)  00.2% (22)  00.3% (21)  00.7% (15)
[16] sandbox          01.4% (12)  00.0%  N/A  00.2% (24)  00.6% (16)
[17] pass             00.8% (17)  00.0%  N/A  00.8% (13)  00.6% (17)
[18] signedness       00.4% (23)  00.9% (11)  00.4% (19)  00.5% (18)
[19] relpath          00.3% (26)  00.4% (17)  00.8% (14)  00.5% (19)
[20] form-field       00.9% (16)  00.0%  N/A  00.3% (22)  00.4% (20)
[21] msdos-device     00.5% (21)  00.8% (12)  00.2% (25)  00.4% (21)
[22] race             00.4% (24)  00.3% (20)  00.4% (18)  00.4% (22)
[23] memleak          00.2% (29)  00.4% (18)  00.5% (16)  00.4% (23)
[24] double-free      00.1% (31)  00.4% (19)  00.5% (17)  00.3% (24)
[25] rand             00.6% (18)  00.1% (23)  00.1% (26)  00.3% (25)
[26] spoof            00.5% (22)  00.1% (24)  00.0%  N/A  00.2% (26)
[27] type-check       00.5% (19)  00.0%  N/A  00.0%  N/A  00.2% (27)
[28] default          00.4% (25)  00.1% (27)  00.0%  N/A  00.2% (28)
[29] dos-release      00.2% (30)  00.2% (21)  00.2% (23)  00.2% (29)
[30] CF               00.3% (27)  00.1% (25)  00.0%  N/A  00.2% (30)
[31] design           00.3% (28)  00.0%  N/A  00.0%  N/A  00.1% (31)
[32] path             00.0%  N/A  00.1% (26)  00.0%  N/A  00.0% (32)


UNKNOWN/UNSPECIFIED ITEMS
------------------------
unk              06.8%  N/A  02.3%  N/A  04.6%  N/A  05.0%  N/A
other            15.7%  N/A  02.8%  N/A  14.0%  N/A  11.9%  N/A
not-specified    06.1%  N/A  42.6%  N/A  29.0%  N/A  22.9%  N/A






Flaw Terminology
-------------------
Type: CF
Rank: [1]
Total vulns: 6
Desc:

General configuration problem

-------------------------------------
Type: dos-malform
Rank: [2]
Total vulns: 138
Desc:

DoS caused by malformed input

-------------------------------------
Type: dos-flood
Rank: [3]
Total vulns: 26
Desc:

DoS caused by flooding with a large number of *legitimately formatted*
requests/etc.; normally DoS is a crash, or spending a lot more time on
a task than it "should"

-------------------------------------
Type: pass
Rank: [4]
Total vulns: 21
Desc:

Default password

-------------------------------------
Type: sandbox
Rank: [5]
Total vulns: 23
Desc:

Java/etc. sandbox escape - NOT BY DOT-DOT!

-------------------------------------
Type: signedness
Rank: [6]
Total vulns: 19
Desc:

Signedness error; a numeric value in one format/representation is
improperly handled when it is used as if it were another
format/representation.  Overlaps integer overflows and array index
errors.

-------------------------------------
Type: metachar
Rank: [7]
Total vulns: 59
Desc:

unescaped shell metacharacters or other unquoted "special" char's;
currently includes SQL injection but not XSS.

-------------------------------------
Type: double-free
Rank: [8]
Total vulns: 12
Desc:

Double-free vulnerability

-------------------------------------
Type: other
Rank: [N/A]
Total vulns: 427
Desc:

Other vulnerability; issue could not be described in version of
taxonomy that was available at the time the flaw type was determined.

-------------------------------------
Type: spoof
Rank: [9]
Total vulns: 8
Desc:

Product is vulnerable to spoofing attacks, generally by not properly
verifying authenticity.

-------------------------------------
Type: path
Rank: [10]
Total vulns: 1
Desc:

OBSOLETE.  Reveals real pathname for files/etc.

-------------------------------------
Type: design
Rank: [11]
Total vulns: 4
Desc:

design problem, generally in protocols or programming languages

-------------------------------------
Type: sql-inject
Rank: [12]
Total vulns: 81
Desc:

SQL injection vulnerability

-------------------------------------
Type: infoleak
Rank: [13]
Total vulns: 98
Desc:

"intentional" information leak by product, i.e. not as the result of
another vulnerability; typically by design or by producing different
"answers" that suggest the state; often related to configuration /
permissions or error reporting/handling.

-------------------------------------
Type: form-field
Rank: [14]
Total vulns: 16
Desc:

CGI program inherently trusts form field that should not be modified
(i.e. stored locally)

-------------------------------------
Type: dos-release
Rank: [15]
Total vulns: 7
Desc:

DoS because system does not properly release resources

-------------------------------------
Type: CSS
Rank: [16]
Total vulns: 241
Desc:

Cross-site scripting (aka XSS or CSS)

-------------------------------------
Type: priv
Rank: [17]
Total vulns: 62
Desc:

Bad privilege assignment, or privileged process/action is
unprotected/unauthenticated.

-------------------------------------
Type: unk
Rank: [N/A]
Total vulns: 178
Desc:

Unknown vulnerability; report is too vague, or issue could not be
described in version of taxonomy that was available at the time the
flaw type was determined.

-------------------------------------
Type: msdos-device
Rank: [18]
Total vulns: 16
Desc:

Problem due to file names with MS-DOS device names.

-------------------------------------
Type: int-overflow
Rank: [19]
Total vulns: 54
Desc:

a numeric value can be incremented to the point where it overflows and
begins at the minimum value, with security implications.  Overlaps
signedness errors.

-------------------------------------
Type: memleak
Rank: [20]
Total vulns: 13
Desc:

memory leak (doesn't free memory when it should); use this instead of
dos-release

-------------------------------------
Type: default
Rank: [21]
Total vulns: 7
Desc:

Insecure default configuration, e.g. passwords or permissions

-------------------------------------
Type: rand
Rank: [22]
Total vulns: 11
Desc:

Generation of insufficiently random numbers, typically by using easily
guessable sources of "random" data

-------------------------------------
Type: relpath
Rank: [23]
Total vulns: 18
Desc:

relies on search paths to find other executable programs or files,
opening up to Trojan horse attacks, e.g. PATH environment variable in
Unix.

-------------------------------------
Type: type-check
Rank: [24]
Total vulns: 8
Desc:

Product incorrectly identifies the type of an input parameter or file,
then dispatches the wrong "executable" (possibly itself) to process
the input, or otherwise misrepresents the input in a security-critical
way.

-------------------------------------
Type: link
Rank: [25]
Total vulns: 117
Desc:

symbolic link following

-------------------------------------
Type: auth
Rank: [26]
Total vulns: 29
Desc:

Weak/bad authentication problem

-------------------------------------
Type: buf
Rank: [27]
Total vulns: 746
Desc:

buffer overflow

-------------------------------------
Type: format-string
Rank: [28]
Total vulns: 93
Desc:

Format string vulnerability; user can inject format specifiers during
string processing.

-------------------------------------
Type: race
Rank: [29]
Total vulns: 14
Desc:

general race condition (NOT SYMBOLIC LINK FOLLOWING (link)!)

-------------------------------------
Type: crypt
Rank: [30]
Total vulns: 52
Desc:

Cryptographic error (poor design or implementation)

-------------------------------------
Type: dot
Rank: [31]
Total vulns: 132
Desc:

directory traversal (file access via ".." or variants)

-------------------------------------
Type: perm
Rank: [32]
Total vulns: 35
Desc:

assigns bad permissions, improperly calculates permissions, or
improperly checks permissions

-------------------------------------
Type: not-specified
Rank: [N/A]
Total vulns: 823
Desc:

The analyst has not assigned a flaw type to the issue.

-------------------------------------