Dailydave mailing list archives
Continuous Integration (was Re: Coverage and a recent paper by L. Suto)
From: "Andre Gironda" <andreg () gmail com>
Date: Tue, 30 Oct 2007 02:33:36 -0400
On 10/29/07, J.M. Seitz <lists () bughunter ca> wrote:
Developers can also improve automation through continuous system tests (e.g. using Canoo WebTest, Jameleon, staf.sf.net, etc).Continuous integration is a definite plus, if you can couple a per-build smoke test with a nightly build robustness test (fuzzing), you are sitting pretty. And generally it's a great first warning that something is up. The real problem is in designing an automation framework that is extensible enough to not only allow custom fuzzers, but is able to adapt to changes in the test targets, network parameters, protocol/file format changes, etc.
Wow, somebody is talking my language! Smoke test? Nightly build? Nobody uses those words! Can you add intake testing with something like ct-eclipse.tigris.org, clover, fortifysoftware sca, and test-ng all integrated? if the developer gets up from their seat, can it run findbugs with extra security checks, too? and a pony?!111?
Design and code review is difficult to automate, but we can "automate"! processes through workflow toolsAtlassian's JIRA also has some excellent workflow tools that can extend into any one of their products and SVN, etc. It is easily automatable through build tools like Tinderbox, Hudson, etc.
Seriously, yes. You really are mirroring my words. It almost feels like you are mocking me (pun intended there).
More advanced operations/maintenance security testing can be done using EFS and/or WI/AppScan/HailstormHow do you know where your weak areas of code are? Is EFS going to successfully dissect a binary protocol and give you metrics that are useable? Not really. In my experience, and I love Jared like a brother, EFS works very well against ASCII protocols, with a known set of seeds. If you turn it loose against an unknown binary protocol, it doesn't do so well. This is where we need to advance the whole notion of automated binary protocol dissection, and then couple that will a block-based (or genetic if you want) fuzzer that can then utilize the results that your dissection yielded. Look for something like this in December, by some goofy looking Canadian dude :)
I saw Charlie Miller talk at Toorcon 9 last weekend and he showed the audience how to "do it" better with EFS. By "do it", I sure hope you know what I'm talking about ;> If not, you can search for the full pornographic details via Google Video... Even giving into your point (which I wasn't arguing against in the first place) - existing binary protocols can be Matasano'd, Sulley'd, and Canadian-guy'd to keep those methods viable. Protocol dissection work is so tedious. I wish the people who wrote the protocols just ran them through a model-checker like SPIN (or the code through modern-day equivalents like PREVENT or Java PathFinder). Better yet would be to do a very good job at design review... see: Justin Schuh (TAOSSA), Brenda Larcom (Octotrike), the SwA Acquisition WG, NIST SAMATE, and the SATC group at NASA.
Unfortunately, the largest problem we're facing with the above strategies is the growing rate of code vs. the growing rate of qualified experts. Make sure to check out Felix Linder (FX)'s talk from HITB 2007, available here - http://conference.hackinthebox.org/hitbsecconf2007kl/materials /D2T1%20-%20Felix%20Lindner%20-%20%20%09%20Attack%20Surface%20 of%20Modern%20Applications.pdfYou got that right, in a development shop you are lucky to have one person in the whole company that can actually test the corner cases, use a debugger to track down problems (without source), demonstrate attack surface, yadayada. Problem is there are all kinds of jobs for security engineers, pentesters, app-pen testers, etc. But how often do you see a job posting like: "Hax0r In the QA Department Needed". That's where the demand _has_ to start moving towards. And please folks, pay your QA hackers well :)
Were you there for my talk at Toorcon? They don't have the slides up yet, but you can get a similar copy of the talk here - https://www.owasp.org/index.php/Image:Andreg-cpt-owasp-msp.pdf What you're saying about "lucky to have one person" is almost exactly what I said. I also included the opposite: you're unlucky to have one specific person: the ignorant and lazy guy/gal who checks in all their code at the last minute of the project, completely untested, and then management makes you go live (and probably doesn't even fire the jerk). SQA is dead. Reasons: 1) Security testing has evolved to be as important or more important than performance testing. I read this on a blog somewhere lately and whoever wrote it has a good idea of the landscape going on right now. So do some of the people at the VERIFY conference that I am attending right now in DC (Security / Automated / Quality testing con) 2) Performance testing has evolved to be the "new" quality testing (and very automated - look at tools like YSlow as an example) 3) Anyone who is still so good at quality testing that anyone would skill want/need their skills is really a developer-tester. See: Jason Huggins (ex-ThoughtWorks developer of Selenium who now works for Google with the title of software quality engineer) The remaining title "SQE" is a misnomer today, just as SQA is. Developer-testers should be involved in writing functional testing tools, system testing tools, automated anything, unit test integration with the above, continuous integration with the lot, and anything-everything test-driven development (TDD) and design for [test|operations|security]. Other SQE's and quality testers should consider careers in performance or security testing before AOL fires another 2k of you little buggers (or should I say, former-bug-finders)? dre _______________________________________________ Dailydave mailing list Dailydave () lists immunitysec com http://lists.immunitysec.com/mailman/listinfo/dailydave
Current thread:
- Continuous Integration (was Re: Coverage and a recent paper by L. Suto) Andre Gironda (Oct 30)