Re: Image forensics

["Dr. Neal Krawetz" <hf () hackerfactor com>]

Hi Dan,

> Forensics aren't a game. People live and die over the determinations  
> we make. There have...been issues, with bite mark analysis, and with  
> arson determination, that have thoroughly destroyed lives, up to and  
> including the death penalty.  This stuff is really important, way more  
> than anything on this list.

I fully agree.  That's another reason I have not publicly released the
code.  (My Gender Guesser is public, and it is consistently used incorrectly
by regular netizens.  Dude -- it is only 60-70% accurate and it needs
large samples.)


> What I would like to do is actually give you the hundred images as  
> described, and receive:

[snip]

I'm currently doing 1-2 blind test each year, and each has the same
requirements as you.

The most notable was from the Department of Defence Cyber Crime Center (DC3).
(For other people who don't know, the DC3 runs the world's largest
cyber crime forensics lab.  If any methods pass their evaluation, then it
is considered "blessed".)

The DC3 gave me 51 pictures, with 6 that were "unknown" to the DC3.
(I didn't learn about the 6 unknowns until after the test ended.)
I ended up being 86% accurate for the known images.  Of the 6 I missed,
none were known-real (no false positives).

Moreover, 4 of the 6 misses were CG Society Award winners. These were
pictures by professional artists who have the time, skill, and incentive
to do it right.  And in every image where I failed, I had commented that
it was a difficult image to analyze (various algorithms gave borderline
results).  Fortunately, most people (including professional graphic
designers) don't have this level of skills and/or time.

Of the ones I missed, two were CG Society Award Winners by the same artist
who spent more than two years working on his wireframe, coloring, and
rendering.  Even after being told that they were CG, I didn't believe it
until I saw the wireframes.  (Really amazing.)  It took me a year to
develop a few algorithms that can detect these as being CG and not real.

Today I have many more algorithms.  (It may be easy to get an inconclusive
result from one algorithm, but it is very hard to beat a suite of them.)
In my last two blind tests, I was 100% accurate (25 images) and about 96%
accurate (80 images, missed 3 -- each miss denoted as "hard to tell", and
only 3 total identified as "hard to tell").
In every image with a strong result, me+tool was 100% accurate.

My tool recently went through a external review (hoping for certification).
In this review, they used my tool with no interaction from me.  The only
things I've been told so far: (1) the evaluation is completed, (2) there is
a steep learning curve (good thing I have 118 pages of technical
documentation!), and (3) the tool "does what it claims to do."  Hopefully I
will see the final report before my presentation at the end of January.
I'd love to make a formal announcement at The DoD Cyber Crime Conference
(http://www.dodcybercrime.com/10CC/).

If it fails to get certification, then I'm really looking forward to
addressing the limitations.  I've already done some work to improve the
usability and shorten the learning curve.  Someone who isn't me should be
able to be at least 80% accurate with no training (and near 100% for
pictures by amateur artists) after a few hours of playing.  Someone with
training should be in the upper 90% range.  (Even though I created the
program, I have two associates who are much better at evaluating images
than me.  I think like a programmer and not a graphics designer.)


Dan: I'm certainly not offended by your request for hard numbers. (I fully
agree with the need!)  But I'd rather leave the blind tests to formal
forensic organizations.  Otherwise I'll end up spending all of my time
doing blind tests and not improving the software.

                                        -Neal
--
Neal Krawetz, Ph.D.
Hacker Factor Solutions
http://www.hackerfactor.com/
Author of "Introduction to Network Security" (Charles River Media, 2006)
and "Hacking Ubuntu" (Wiley, 2007)

_______________________________________________
Fun and Misc security discussion for OT posts.
https://linuxbox.org/cgi-bin/mailman/listinfo/funsec
Note: funsec is a public and open mailing list.