Re: newbie quetsions

[Rainer Duffner <rainer () ultra-secure de>]

Julius Detritus wrote:

> They are outdated. The most recent exploit tested must be two years old...
> They are copy and paste from IDS tests which are far older. 
>
>  

[...]

> Do you really care about the phf exploit? Or maybe the old sshutupteo from
> gobbles? Are you talking about organizations or museums?
>
>  

[,,,]

> So your exploit database must be very old
>  

This raises the question: when testing IDSs - are the exploits used 
important ? Do you have to use the latest ? Are the results invalidated 
by old exploits ?
Back when I tested IDSs myself (that's now 4 years ago...), I didn't 
think it was very important - I made sure, though, that I had various 
"classes" of attacks (directory-traversal, shellcode/buffer-overflow).

Reproducability is a big problem, too. NSS probably wants test from 
Q1/04 to be comparable with test from Q1/05, at least to some degree. So 
I'd say it is - under this objective - mandatory to change as little 
factors as possible - and that surely includes the exploits and 
evasion-techniques (though they seem to vary nonetheless).

People flaming NSS should also keep in mind that testing IDSs today is 
really a battle of materials. Back in late 2000 (when I did the work for 
my thesis), I could use our training-lab with a dozen (well equipped) 
PCs and some hubs to get a good picture of the capabilities of half a 
dozen products.
Today, you need a big lab with switches, taps, packet-generators, big 
servers - and you still can't simulate what it will look like in the 
real-world with dozens of sensors and remote-locations - costs are 
probably skyrocketing anyway.

I don't think NSS's test are the Holy Grail - but it cannot be disputed 
that they have at least a methodically correct approach and re-test old 
and new products continuously.


If anybody can do better - please stand up now ;-)



cheers,
Rainer

--
===================================================
~     Rainer Duffner - rainer () ultra-secure de     ~
~           Freising - Munich - Germany           ~
~    Unix - Linux - BSD - OpenSource - Security   ~
~  http://www.ultra-secure.de/~rainer/pubkey.pgp  ~
===================================================


--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from 
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
--------------------------------------------------------------------------