Nmap Development mailing list archives
Transcript of Nmap/GSoC Planning Meeting
From: Fyodor <fyodor () nmap org>
Date: Mon, 25 Mar 2013 14:03:49 -0700
Hi folks! Last Friday's Nmap/GSoC planning meeting was a success, and I'm planning to update our GSoC web pages and apply as an organization by the deadline this Friday. For those who missed the meeting, here is a transcript: <fyodor> Alright, it's showtime <fyodor> So there are a million people in the channel, or 83 anyway, mostly probably people just idling, and I'm not sure who's here for the actual Summer of Code meeting <fyodor> so maybe those of us here for the meeting can introduce ourselves, particularly since I don't always remember which IRC nick corresponds to who <fyodor> I'll start: I'm Fyodor :) <yyzfp> I'm David Fifield. <fyodor> That makes 2 of us, hopefully we're not the only two <Singhabhinavds> I am Singh Abhianv D S :) new to the nmap.org <fyodor> Welcome, Singh! <Cipher-0> I don't develop, so I'll just sit here and fan-boy the channel. XD <j0k3r_> I am Prakash Gamit and I'm also new to nmap <Bzitka_Yaknotz> Peter, or your local equivalent. Also new. <fyodor> Well, this meeting isn't just for developers, anyone can have great ideas of where to take Nmap this year and especially as part of SoC this summer * unknown_had another fanboy. <hatchee> (I'm henri doreau, hi all) <fyodor> I see hatchee, iago-x86, kroosec, kost, batrick, etc. in the channel list, but maybe you guys aren't actually watching your screens? <fyodor> I'm glad you could make it, Henri <fyodor> OK, well, let's get started with just us and maybe more will join later * fyodor notes maybe he should have given more than 8 hours notice before the meeting <yyzfp> It's okay, people will notice as we make noise in the channel. * fyodor and maybe Friday night in Europe isn't he ideal time. Perhaps some Nmap developers (except me) do have social lives to plan around <fyodor> Alright, maybe let's start by briefly discussing some high level directions for Nmap and then <fyodor> we can get down more to the nitty gritty of potential GSoC roles and tasks for this Summer <fyodor> so in recent years I think we've done some pretty awesome stuff, including all the new IPv6 support, including the new IPv6 OS detection system, and <Bzitka_Yaknotz> Sounds good <fyodor> of course the Nmap Scripting Engine has been a major priority, and that exploded to 435 scripts! <fyodor> You could run a different one every day of the year <fyodor> and of course we've continued to focus on portability, making sure Nmap works great with newer systems like Windows 8, the latest Mac releases, and <fyodor> Linux too of course, though that hasn't been as much of a problem <fyodor> and we've made some good infrastructure improvements that helped the system greatly, but aren't as visible to users, like Henri's work with Nsock Engines <fyodor> and of course we've kept up the release schedule, with Nmap 6, and 6.01, and 6.25 and the various test releases in the middle. So that users actually are able to play with and use these new features, and they're not just limited to us devs who mostly all run from svn version anyway <fyodor> Also, we've kept up and improved the ancillary tools, ncat, nping, ndiff, zenmap <fyodor> so those are some of the places we've been from the top of my head, and so the next thing is where we want to go <fyodor> a second version of Nmap Network Scanning is a very big priority, but that's not coding (I don't think Google counts DocBook XML), so we can't get GSoC help on that * hatchee has quit (Ping timeout: 264 seconds) <fyodor> Nmap Scripting Engine definitely is a high priority again, and we had great success last year, I think, with * henri has joined #Nmap <fyodor> 3 students: one focused on discovery, one on web-related scripts, and one on exploitation related <fyodor> and so we could consider doing something similar this year, or at least put such on the task list and then <fyodor> we might not necessarily find a great person for each task, but we can look at what people apply for and in part decide which roles to fulfill based on which ones we have great applicants for <fyodor> Also, David and I were discussing SoC a bit last week and came up with a couple ideas: <fyodor> - Test automation specialize <fyodor> - Parallelize Ncat tests <fyodor> - let Ncat listen on port 0 to choose a random open port (and print <fyodor> it on the debug output). Then tests can start multiple instances <fyodor> at once without port conflicts. <fyodor> - Create Nmap test suite <fyodor> - Performance/Optimization specialist <fyodor> - Optimize Nmap performance and resource usage <fyodor> - Maybe do some of our large-scale scanning research too, like <fyodor> keeping top ports stats up to date. <fyodor> - Winpcap Specialist <fyodor> - WinPcap support for NDIS 6 <fyodor> - WinPcap privileges <fyodor> - No-install DLL support <fyodor> - Driver signing <fyodor> - Maybe release it with new name and function entry points and <fyodor> maintain it ourselves <fyodor> - Possibly this person could do other low-level Windows tasks too <fyodor> - Feature creeper <fyodor> - Zenmap GUI developer <fyodor> And then the NSE folks: <fyodor> - Nmap Scripting Engine Script Developer <fyodor> - Web scanning specialist <fyodor> - Discovery scanning specialist <fyodor> - Vuln/exploitation specialist <yyzfp> Maybe we should wikify this list. <fyodor> Yeah, that would be a great idea <yyzfp> Stand by. <bonsaiviking> Under the "feature creeper" heading, I'd like to see a Lua engine for new port scan types. <fyodor> Welcome, Daniel! <bonsaiviking> yeah, sorry, I was afk for the starting intros (reading scrollback) <fyodor> that's a good idea. So this would be a new script type, but which runs for portscanning stage? <henri> that'd be nice, it's one of the items on https://secwiki.org/w/GSoC_community_ideas <fyodor> Thanks for remdinding us of that page <fyodor> I had forgotten about that <bonsaiviking> That'd be my guess, though I'd not like to restrict possiblities until we've looked at code and prototyped * hatchee has joined #Nmap <fyodor> David, are you adding these ideas to that community ideas page or another page? <fyodor> maybe we should merge them into community ideas <yyzfp> I'll add to the community ideas page. <Singhabhinavds> BTW is that page updated ? <henri> I had a working proto (see on the wiki page I gave) <fyodor> the community ideas page hasn't been updated since last year, AFAIK, though David's adding these newer ideas now <Singhabhinavds> good <henri> it's probably not fully functional anymore, and would need some polishing, but it made me believe that this approach is right * henri is now known as henri__ <yyzfp> https://secwiki.org/w/GSoC_community_ideas#Ideas_for_project_roles <fyodor> Manybe Daniel or Henri can add the port scan engine idea to the page? <bonsaiviking> I'll get it up there. <fyodor> Thanks <fyodor> So the WinPcap role may seem a little strange, but the unfortunate reality is that Riverbed seems to have mostly dropped the ball on WinPcap (currently -- I hope they get back into it) <fyodor> For months I've been receiving mails from people who wanted to use Nmap's Winpcap instead of the official one with their software because we had better Windows 8 support * henri has joined #Nmap <fyodor> and as you can see from the list above, there's a lot of stuff we could potentially add in our own NPcap or whatever we'd call it <fyodor> Also, I heard that one of the main WinPcap developers, Loris Degioanni has left Riverbed <fyodor> still, they did eventually do a release where they fixed the Win 8 stuff, I think * henri__ has quit (Ping timeout: 272 seconds) <fyodor> So did anyone else have other ideas they wanted to mention for SoC tasks? <fyodor> or roles <henri> I especially like the large scale research, as well as the performance work and regression suite <bonsaiviking> henri: I agree. I'm adding a link to the Carna botnet info to that bullet, since it could be useful. <fyodor> I'm excited about those too. And now we have a large dataset which could be used for some of the research. I finally finished downloaded that 583 gig torrent <henri> I actually believe that they can be pretty tightly linked <fyodor> Now I just need to find space to actually decompress the Carna botnet stuff <henri> heh :) <henri> and setup a hadoop cluster to go extract information from that <fyodor> Also, we set up a machine in the Netherlands with a gigabit connection to use for scanning research, and apparently also for torrenting of giant datasets <iago-x86> fyodor: Nope, I don't watch my screen. :) <fyodor> Haha, welcome Ron! <fyodor> The party doesn't really start until Ron arrives <fyodor> henri: when you say tightly linked, you are referring to <fyodor> the performance and regression suite work? or the research? Or all 3? <henri> well a regression suite would allow us to improve the performances <fyodor> ah, that's a good point <henri> and to track and reveal issues that would only appear on large scans <bonsaiviking> hear hear <fyodor> Henri, can you add a note to the regression role on the wiki noting this? <henri> sure <fyodor> thanks <yyzfp> The main benefit to having a regression test, to me, is having something to run before releases to check that simple things aren't broken. <fyodor> Yeah, right now David and I basically just do a bunch of testing on our own once we do the final builds and try to find any problems, but <fyodor> it's not really structured. I just open up Zenmap on Windows and get to work, and meanwhile start a bunch of scans in terminals on Linux <fyodor> So speaking of the wiki, maybe we should go through the ideas which are already on there and figure out what to do with them <brain> Jumping in late: +1 performance/optimization work. I'm interesting in working on large scale heterogeneous scans where infrastructure performance varies. <fyodor> So the top one is XML parser for NSE, which still sounds like an appropriate thing to me <brain> er, seeing work on, not working not - not so much a developer :) <fyodor> Ah, thanks brain <fyodor> Then "Moving packet.lua from lua to C++" which says "For efficiency (both runtime and developers' productivity), it might make sense for NSE to leverage the existing packet crafting classes." <fyodor> I wonder if Patrick added that? <yyzfp> That doesn't make sense to me. <henri> IIRC I did, though he emitted the original idea... <henri> yyzfp: why so? <yyzfp> My guess is that the effect on runtime efficiency would be negligibly positive, and on developer efficiency negative. <Singhabhinavds> I had like to work on NSE by writing various scipts and would like to know which scripts are left from this page https://secwiki.org/w/Nmap/Script_Ideas#GSoC <bonsaiviking> I can see that unification would be helpful, especially for the NSE portscanning. <yyzfp> I'm not the biggest fan of packet.lua, but why would we move it to a memory-unsafe language that is harder to debug? I don't think packet parsing is a performance bottleneck for us anywhere. <yyzfp> I already worry somewhat about the packet parsers in Nping's library. <henri> it was also about code "cleanness" and maintaining two separate stacks <bonsaiviking> so the project would be to unify on one or the other language, beginning with a cost/benefit analysis of each? <bonsaiviking> (security risk, existing code, maintainability, extensibility, etc) <henri> I don't think we would want NSE to craft packets for everyone, that'd be a strange layering <henri> everyone = (nmap non-nse modules and nping) <henri> imho <fyodor> ok, so I'm trying to reorganize the wiki page a bit so that <fyodor> we can add some top 2013 ideas to the top and <fyodor> keep some other ideas below <fyodor> like the ones that we think are less likely to do for 2013 SoC <fyodor> ok, I made that change, which should help us sort through the ideas, I think <fyodor> So for the packet.lua language move, it doesn't sound like we have consensus yet on what to do, so maybe we'll add this to the other ideas list for now? <hatchee> alright <fyodor> OK, and next is "Improved port specification" -- "The --top-ports parameter is incredibly valuable but hardly adaptive. It would be nice to extend the port specification syntax to easily add/remove ports from the top-ports lists." <fyodor> I'm not sure how often that would be used <yyzfp> The add part at least I'm totally on board with. <yyzfp> I want -F, --top-ports, and -p to take a union. <fyodor> it seems like a pretty advanced feature, and given that users who want an exact set of ports can already specify one ... <fyodor> that's an interesting idea, taking a union <yyzfp> So I can do, for example, -F -p 61000 if I want to add a single port to the list of -F. <yyzfp> There's a thread about this somewhere we should link to. I'll look for it. <fyodor> ok, if you can add the link if/when you find it, that'd be great <fyodor> then exploring port scanning from within NSE, which we just talked about <fyodor> then Scanning through proxies. <bonsaiviking> yyzfp: http://seclists.org/nmap-dev/2012/q3/336 <fyodor> Don't we have a PoC of that now? <bonsaiviking> henri's nmap-proxies branch does Nsock, not scanning yet. <yyzfp> The idea is to add proxy support to Nsock (henri's branch), and then make connect scan use Nsock. <fyodor> So if Henri is already making great progress, would it still make sense as a GSoC idea? <fyodor> I guess that's a question for Henri <yyzfp> Porting the connect scan engine to Nsock is conceptually a separate task. <yyzfp> Also a big and important one. <fyodor> OK, let's keep it here then <fyodor> so nmap-proxies is almost done, but no work yet on the Nmap connect scan nsock stuff, is that correct? <henri> right <fyodor> ok, I added that note to the page <fyodor> Implement new scan techniques -- maybe I should just merge that one with the explore port scanning from within NSE task description? <fyodor> ok, that's what I'll do <henri> yes, makes sense <fyodor> Bringing lua to ncat <fyodor> "A scripting engine in ncat would allow users to easily design network applications and automatize things (stats, logs...)" <fyodor> Did anyone here add that? <henri> o/ <fyodor> Awesome, can you tell us more about the idea? Like an example script? <henri> I don't think it should have any high priority but I thought it could be a fun "research" project <fyodor> Do you have any script ideas in mind that might work well in Ncat? <henri> for instance the builtin "chat" or a http server or some proxy protocols <fyodor> better in Ncat than Nmap <henri> could be (re-)implemented in lua <yyzfp> Mebbe the WebSocket idea would be a good fit. <henri> yes, maybe too <bonsaiviking> yeah, any of the server-side-type things (dependent on Nsock server-mode?) <fyodor> When you get a chance, Henri, maybe you could write up more about the Ncat scripting engine idea onto the page? <henri> not necessarily dependent, but could leverage it, sure <henri> fyodor: ok <fyodor> Thanks <yyzfp> For ideas like this, we need to have a mentor lined up who knows what he/she wants from the project. <fyodor> yeah, definitely <fyodor> Scanning pipeline is next <fyodor> I think that is a great idea for making Nmap more efficient, though I also agree that it sounds like a lot for a GSoC student <fyodor> I'm wondering if we should move this off this page and onto Nmap TODO list? Or maybe there is a chance we could find someone talented enough to do such a major re-architecture? I'm kind of skeptical of that though <fyodor> even someone amazingly talented would need to get more experience with Nmap first, probably <henri> right, what we could do though is to try to split that huge task into smaller ones <henri> having a roadmap would be a big step forward already <fyodor> Good point. OK, so how about if I add it to Nmap TODO and remove it from this GSoC page, but if we find a way to break it up and make a roadmap, we can definitely add it back? <fyodor> ok, so what I wrote up is this: <fyodor> o Consider re-architecting Nmap to have more of a scanning pipeline <fyodor> approach rather than fixed sets of hosts which start and finish one <fyodor> phase and then move into the next in parallel. This could potentially <fyodor> allow us to add hosts one by one to a phase as other hosts finish that <fyodor> phase and, ideally, the phases could run in parallel too. <fyodor> Is that sort of what you had in mind? <henri> absolutely <fyodor> Great. OK, I'm putting that in the Nmap TODO and taking out of GSoC ideas for now <fyodor> next is nsock server mode <fyodor> is that done? <henri> I have working code on nmapexp <henri> (I know I have a lot there...) <henri> but I want to rework some parts <Bzitka_Yaknotz> On the pipeline project, could we use that as a multi-person project and put two people on it. That way it's a little more feasable but still only needs 1 mentor <fyodor> That's a good idea, Peter, but Google is somewhat strict about "no teams" because <henri> I'd prefer my other branches (especially the proxy one) to be merged first <fyodor> they are worried about if the project isn't completed then one student blames the other for not finishing and who do you pass and who do you fail, however, <fyodor> if it can be divided up into several discrete tasks where they can be assigned to individual students and evaluated on their own even if one of the other students doesn't deliver, then that is allowed <Bzitka_Yaknotz> Understood <fyodor> OK, so I want to finish this meeting in the next 12 minutes, so ... <fyodor> maybe we will have to skip the rest of the ideas (going through them) on this page for now, and let me check what's left on the agenda... <fyodor> So one thing we wanted to see is who is interested in mentoring this year? <fyodor> and if so, are there any particular tasks/roles which would be your top choices to mentor? <fyodor> I'm hoping to mentor at least one student <fyodor> Are you up for it again, David? <yyzfp> Me. <yyzfp> I want to do the Ncat and Nmap testing, at least. <fyodor> Good :). And Henri? Daniel? <henri> I unfortunately don't think I can mentor again this year, I might lack time <fyodor> OK, if that changes, Henri, definitely let us know <henri> (which is a pity, because I loved it!) <fyodor> and you did a great job <henri> thanks, maybe as a backup mentor <fyodor> that's a good idea <fyodor> OK, so the last thing I wanted to cover was <bonsaiviking> I would love to be able to help, but I'm expecting family commitments beginning in June to eat up my time <fyodor> promotion of Nmap GSoC. I think one of the most important aspects to whether GSoC is a success for us is of course whether we get great applicants <fyodor> and we're competing with like 150 other projects, not to mention all the other things that top students can do over the summer <bonsaiviking> fyodor: is there ever any concern about *too many* applicants? <fyodor> so I'm trying to think of ideas for how we can promote Nmap SoC so that more people know about it, and also how to make Nmap SoC particularly enticing to these students <bonsaiviking> Just wondering. <yyzfp> bonsaiviking: not really. <fyodor> I think too many applicants is a problem we could handle :). Actually, in the very first year or two, <fyodor> we really did get an absolute ton of great applicants, because I think there were only like 20 orgs at first, and the program was new so it got tons of press <fyodor> but in recent years, it has been more of a struggle. Particularly, I think, as the security market has grown and as awesome as GSoC is, it doesn't really pay competitively to internships with U.S. software and security companies, in general <brain> Does GSoC stipulate any restrictions on organizations offering incentives? I don't know what that would be for nmap, just curious. <fyodor> but of course it has many great points. Folks can work from home or anywhere else in the World they want to be <fyodor> That's a good point, brain, we have thought about possible incentives. I want to discuss it with the GSoC organizers first to make sure they're cool with it though <fyodor> we don't want to get on their bad side, for sure <brain> *nod* <bonsaiviking> you could DM @securitytwits on twitter to promote it. They've got 21K followers (about same as @nmap) <fyodor> Oh, great idea! I'm writing that down <fyodor> And wow, Nmap has 21K followers?! <bonsaiviking> very close to 22k <brain> They also have an IRC channel, which will get less exposure but perhaps a higher possibility of response. <fyodor> too bad my last post was last November :) <brain> securitytwits, that is. <bonsaiviking> fyodor: may be a little late, but you should probably tweet about Carna, since it's related to Nmap <fyodor> That's a good point <fyodor> anyone else have ideas for good promotion channels? <fyodor> so of course there are the Nmap channels: <fyodor> Nmap facebook and twitter <fyodor> Insecure.Org, Nmap.Org <brain> Higher ed infosec practioners, some of them have connections to their academic counterparts. <fyodor> nmap-hackers, nmap-dev mailing list <brain> Can put the word out to some of those folks (educause, for example). <fyodor> That would be great <fyodor> A good time, I think, is shortly after we're accepted (assuming we are), and we should have the ideas page and such finished by then too <bonsaiviking> I'll send emails to my local university contacts. Small potatoes, but personal contact is often helpful. <fyodor> BTW, you didn't introduce yourself brain <fyodor> Thanks Daniel <brain> Doh, sorry, I came in after the intros - Brian, I work on NYU's security team. <fyodor> Awesome, welcome! <fyodor> We should encourage folks to join the official Google gsoc channel(s) when they have time too, because <fyodor> I think students come in there to chat and it's a good way to answer questions and can perhaps mention to them that we'd love to have them apply for Nmap <fyodor> if it sounds like something fitting. If they come in looking for a game related project, then maybe Nmap wouldn't be so great, unless they want to implement pong in NSE <yyzfp> We already have ping, why not? <fyodor> which would mean we'd have to add a new 'games' category <fyodor> good point, David! <Singhabhinavds> haha <fyodor> Alright, well I said I'd finish this meeting in an hour or 1.5 so I don't want to let it go on any longer, but <fyodor> I want to thank everyone for coming, and if there is any last thing you want to mention, do speak up <yyzfp> http://www.google-melange.com/gsoc/events/google/gsoc2013 <Singhabhinavds> just the page of script ideas to get updated <fyodor> and maybe we'll have another meeting later about this stuff <Singhabhinavds> since i want to contribute by writing scripts <yyzfp> That's the timeline for potentially interested mentors. <Bzitka_Yaknotz> Nope, thanks a lot. <fyodor> I think the script ideas page is mostly up to date, lert me check <Singhabhinavds> yup just wanted to confirm if its updated or not <Singhabhinavds> heres the link https://secwiki.org/w/Nmap/Script_Ideas <fyodor> Yeah, I think it mostly is. If you find one in there which we have already finished, it is great if you can remove it <fyodor> or if you get new ideas, feel free to add them <Singhabhinavds> alright <Singhabhinavds> nothing else from my side <fyodor> Great folks, thanks again for coming and I'll TTYL Mar 22 11:36:16 * Disconnected (). Cheers, Fyodor _______________________________________________ Sent through the dev mailing list http://nmap.org/mailman/listinfo/dev Archived at http://seclists.org/nmap-dev/
Current thread:
- Transcript of Nmap/GSoC Planning Meeting Fyodor (Mar 25)