Transcript of Nmap/GSoC Planning Meeting

[Fyodor <fyodor () nmap org>]

Hi folks!  Last Friday's Nmap/GSoC planning meeting was a success, and I'm
planning to update our GSoC web pages and apply as an organization by the
deadline this Friday.  For those who missed the meeting, here is a
transcript:

<fyodor> Alright, it's showtime
<fyodor> So there are a million people in the channel, or 83 anyway, mostly
probably people just idling, and I'm not sure who's here for the actual
Summer of Code meeting
<fyodor> so maybe those of us here for the meeting can introduce ourselves,
particularly since I don't always remember which IRC nick corresponds to who
<fyodor> I'll start: I'm Fyodor :)
<yyzfp> I'm David Fifield.
<fyodor> That makes 2 of us, hopefully we're not the only two
<Singhabhinavds> I am Singh Abhianv D S  :) new to the nmap.org
<fyodor> Welcome, Singh!
<Cipher-0> I don't develop, so I'll just sit here and fan-boy the channel.
XD
<j0k3r_> I am Prakash Gamit and I'm also new to nmap
<Bzitka_Yaknotz> Peter, or your local equivalent. Also new.
<fyodor> Well, this meeting isn't just for developers, anyone can have
great ideas of where to take Nmap this year and especially as part of SoC
this summer
* unknown_had another fanboy.
<hatchee> (I'm henri doreau, hi all)
<fyodor> I see hatchee, iago-x86, kroosec, kost, batrick, etc. in the
channel list, but maybe you guys aren't actually watching your screens?
<fyodor> I'm glad you could make it, Henri
<fyodor> OK, well, let's get started with just us and maybe more will join
later
* fyodor notes maybe he should have given more than 8 hours notice before
the meeting
<yyzfp> It's okay, people will notice as we make noise in the channel.
* fyodor and maybe Friday night in Europe isn't he ideal time.  Perhaps
some Nmap developers (except me) do have social lives to plan around
<fyodor> Alright, maybe let's start by briefly discussing some high level
directions for Nmap and then
<fyodor> we can get down more to the nitty gritty of potential GSoC roles
and tasks for this Summer
<fyodor> so in recent years I think we've done some pretty awesome stuff,
including all the new IPv6 support, including the new IPv6 OS detection
system, and
<Bzitka_Yaknotz> Sounds good
<fyodor> of course the Nmap Scripting Engine has been a major priority, and
that exploded to 435 scripts!
<fyodor> You could run a different one every day of the year
<fyodor> and of course we've continued to focus on portability, making sure
Nmap works great with newer systems like Windows 8, the latest Mac
releases, and
<fyodor> Linux too of course, though that hasn't been as much of a problem
<fyodor> and we've made some good infrastructure improvements that helped
the system greatly, but aren't as visible to users, like Henri's work with
Nsock Engines
<fyodor> and of course we've kept up the release schedule, with Nmap 6, and
6.01, and 6.25 and the various test releases in the middle.  So that users
actually are able to play with and use these new features, and they're not
just limited to us devs who mostly all run from svn version anyway
<fyodor> Also, we've kept up and improved the ancillary tools, ncat, nping,
ndiff, zenmap
<fyodor> so those are some of the places we've been from the top of my
head, and so the next thing is where we want to go
<fyodor> a second version of Nmap Network Scanning is a very big priority,
but that's not coding (I don't think Google counts DocBook XML), so we
can't get GSoC help on that
* hatchee has quit (Ping timeout: 264 seconds)
<fyodor> Nmap Scripting Engine definitely is a high priority again, and we
had great success last year, I think, with
* henri has joined #Nmap
<fyodor> 3 students: one focused on discovery, one on web-related scripts,
and one on exploitation related
<fyodor> and so we could consider doing something similar this year, or at
least put such on the task list and then
<fyodor> we might not necessarily find a great person for each task, but we
can look at what people apply for and in part decide which roles to fulfill
based on which ones we have great applicants for
<fyodor> Also, David and I were discussing SoC a bit last week and came up
with a couple ideas:
<fyodor> - Test automation specialize
<fyodor> - Parallelize Ncat tests
<fyodor>   - let Ncat listen on port 0 to choose a random open port (and
print
<fyodor>     it on the debug output).  Then tests can start multiple
instances
<fyodor>     at once without port conflicts.
<fyodor> - Create Nmap test suite
<fyodor> - Performance/Optimization specialist
<fyodor> - Optimize Nmap performance and resource usage
<fyodor> - Maybe do some of our large-scale scanning research too, like
<fyodor>   keeping top ports stats up to date.
<fyodor> - Winpcap Specialist
<fyodor> - WinPcap support for NDIS 6
<fyodor> - WinPcap privileges
<fyodor> - No-install DLL support
<fyodor> - Driver signing
<fyodor> - Maybe release it with new name and function entry points and
<fyodor>   maintain it ourselves
<fyodor> - Possibly this person could do other low-level Windows tasks too
<fyodor> - Feature creeper
<fyodor> - Zenmap GUI developer
<fyodor> And then the NSE folks:
<fyodor> - Nmap Scripting Engine Script Developer
<fyodor> - Web scanning specialist
<fyodor> - Discovery scanning specialist
<fyodor> - Vuln/exploitation specialist
<yyzfp> Maybe we should wikify this list.
<fyodor> Yeah, that would be a great idea
<yyzfp> Stand by.
<bonsaiviking> Under the "feature creeper" heading, I'd like to see a Lua
engine for new port scan types.
<fyodor> Welcome, Daniel!
<bonsaiviking> yeah, sorry, I was afk for the starting intros (reading
scrollback)
<fyodor> that's a good idea.  So this would be a new script type, but which
runs for portscanning stage?
<henri> that'd be nice, it's one of the items on
https://secwiki.org/w/GSoC_community_ideas
<fyodor> Thanks for remdinding us of that page
<fyodor> I had forgotten about that
<bonsaiviking> That'd be my guess, though I'd not like to restrict
possiblities until we've looked at code and prototyped
* hatchee has joined #Nmap
<fyodor> David, are you adding these ideas to that community ideas page or
another page?
<fyodor> maybe we should merge them into community ideas
<yyzfp> I'll add to the community ideas page.
<Singhabhinavds> BTW is that page updated ?
<henri> I had a working proto (see on the wiki page I gave)
<fyodor> the community ideas page hasn't been updated since last year,
AFAIK, though David's adding these newer ideas now
<Singhabhinavds> good
<henri> it's probably not fully functional anymore, and would need some
polishing, but it made me believe that this approach is right
* henri is now known as henri__
<yyzfp> https://secwiki.org/w/GSoC_community_ideas#Ideas_for_project_roles
<fyodor> Manybe Daniel or Henri can add the port scan engine idea to the
page?
<bonsaiviking> I'll get it up there.
<fyodor> Thanks
<fyodor> So the WinPcap role may seem a little strange, but the unfortunate
reality is that Riverbed seems to have mostly dropped the ball on WinPcap
(currently -- I hope they get back into it)
<fyodor> For months I've been receiving mails from people who wanted to use
Nmap's Winpcap instead of the official one with their software because we
had better Windows 8 support
* henri has joined #Nmap
<fyodor> and as you can see from the list above, there's a lot of stuff we
could potentially add in our own NPcap or whatever we'd call it
<fyodor> Also, I heard that one of the main WinPcap developers, Loris
Degioanni has left Riverbed
<fyodor> still, they did eventually do a release where they fixed the Win 8
stuff, I think
* henri__ has quit (Ping timeout: 272 seconds)
<fyodor> So did anyone else have other ideas they wanted to mention for SoC
tasks?
<fyodor> or roles
<henri> I especially like the large scale research, as well as the
performance work and regression suite
<bonsaiviking> henri: I agree. I'm adding a link to the Carna botnet info
to that bullet, since it could be useful.
<fyodor> I'm excited about those too.  And now we have a large dataset
which could be used for some of the research.  I finally finished
downloaded that 583 gig torrent
<henri> I actually believe that they can be pretty tightly linked
<fyodor> Now I just need to find space to actually decompress the Carna
botnet stuff
<henri> heh :)
<henri> and setup a hadoop cluster to go extract information from that
<fyodor> Also, we set up a machine in the Netherlands with a gigabit
connection to use for scanning research, and apparently also for torrenting
of giant datasets
<iago-x86> fyodor: Nope, I don't watch my screen. :)
<fyodor> Haha, welcome Ron!
<fyodor> The party doesn't really start until Ron arrives
<fyodor> henri: when you say tightly linked, you are referring to
<fyodor> the performance and regression suite work?  or the research?  Or
all 3?
<henri> well a regression suite would allow us to improve the performances
<fyodor> ah, that's a good point
<henri> and to track and reveal  issues that would only appear on large
scans
<bonsaiviking> hear hear
<fyodor> Henri, can you add a note to the regression role on the wiki
noting this?
<henri> sure
<fyodor> thanks
<yyzfp> The main benefit to having a regression test, to me, is having
something to run before releases to check that simple things aren't broken.
<fyodor> Yeah, right now David and I basically just do a bunch of testing
on our own once we do the final builds and try to find any problems, but
<fyodor> it's not really structured.  I just open up Zenmap on Windows and
get to work, and meanwhile start a bunch of scans in terminals on Linux
<fyodor> So speaking of the wiki, maybe we should go through the ideas
which are already on there and figure out what to do with them
<brain> Jumping in late: +1 performance/optimization work.  I'm interesting
in working on large scale heterogeneous scans where  infrastructure
performance varies.
<fyodor> So the top one is XML parser for NSE, which still sounds like an
appropriate thing to me
<brain> er, seeing work on, not working not - not so much a developer :)
<fyodor> Ah, thanks brain
<fyodor> Then "Moving packet.lua from lua to C++" which says "For
efficiency (both runtime and developers' productivity), it might make sense
for NSE to leverage the existing packet crafting classes."
<fyodor> I wonder if Patrick added that?
<yyzfp> That doesn't make sense to me.
<henri> IIRC I did, though he emitted the original idea...
<henri> yyzfp: why so?
<yyzfp> My guess is that the effect on runtime efficiency would be
negligibly positive, and on developer efficiency negative.
<Singhabhinavds> I had like to work on NSE by writing various scipts and
would like to know which scripts are left from this page
https://secwiki.org/w/Nmap/Script_Ideas#GSoC
<bonsaiviking> I can see that unification would be helpful, especially for
the NSE portscanning.
<yyzfp> I'm not the biggest fan of packet.lua, but why would we move it to
a memory-unsafe language that is harder to debug? I don't think packet
parsing is a performance bottleneck for us anywhere.
<yyzfp> I already worry somewhat about the packet parsers in Nping's
library.
<henri> it was also about code "cleanness" and maintaining two separate
stacks
<bonsaiviking> so the project would be to unify on one or the other
language, beginning with a cost/benefit analysis of each?
<bonsaiviking> (security risk, existing code, maintainability,
extensibility, etc)
<henri> I don't think we would want NSE to craft packets for everyone,
that'd be a strange layering
<henri> everyone = (nmap non-nse modules and nping)
<henri> imho
<fyodor> ok, so I'm trying to reorganize the wiki page a bit so that
<fyodor> we can add some top 2013 ideas to the top and
<fyodor> keep some other ideas below
<fyodor> like the ones that we think are less likely to do for 2013 SoC
<fyodor> ok, I made that change, which should help us sort through the
ideas, I think
<fyodor> So for the packet.lua language move, it doesn't sound like we have
consensus yet on what to do, so maybe we'll add this to the other ideas
list for now?
<hatchee> alright
<fyodor> OK, and next is "Improved port specification" -- "The --top-ports
parameter is incredibly valuable but hardly adaptive. It would be nice to
extend the port specification syntax to easily add/remove ports from the
top-ports lists."
<fyodor> I'm not sure how often that would be used
<yyzfp> The add part at least I'm totally on board with.
<yyzfp> I want -F, --top-ports, and -p to take a union.
<fyodor> it seems like a pretty advanced feature, and given that users who
want an exact set of ports can already specify one ...
<fyodor> that's an interesting idea, taking a union
<yyzfp> So I can do, for example, -F -p 61000 if I want to add a single
port to the list of -F.
<yyzfp> There's a thread about this somewhere we should link to. I'll look
for it.
<fyodor> ok, if you can add the link if/when you find it, that'd be great
<fyodor> then exploring port scanning from within NSE, which we just talked
about
<fyodor> then Scanning through proxies.
<bonsaiviking> yyzfp: http://seclists.org/nmap-dev/2012/q3/336
<fyodor> Don't we have a PoC of that now?
<bonsaiviking> henri's nmap-proxies branch does Nsock, not scanning yet.
<yyzfp> The idea is to add proxy support to Nsock (henri's branch), and
then make connect scan use Nsock.
<fyodor> So if Henri is already making great progress, would it still make
sense as a GSoC idea?
<fyodor> I guess that's a question for Henri
<yyzfp> Porting the connect scan engine to Nsock is conceptually a separate
task.
<yyzfp> Also a big and important one.
<fyodor> OK, let's keep it here then
<fyodor> so nmap-proxies is almost done, but no work yet on the Nmap
connect scan nsock stuff, is that correct?
<henri> right
<fyodor> ok, I added that note to the page
<fyodor> Implement new scan techniques -- maybe I should just merge that
one with the explore port scanning from within NSE task description?
<fyodor> ok, that's what I'll do
<henri> yes, makes sense
<fyodor> Bringing lua to ncat
<fyodor> "A scripting engine in ncat would allow users to easily design
network applications and automatize things (stats, logs...)"
<fyodor> Did anyone here add that?
<henri> o/
<fyodor> Awesome, can you tell us more about the idea?  Like an example
script?
<henri> I don't think it should have any high priority but I thought it
could be a fun "research" project
<fyodor> Do you have any script ideas in mind that might work well in Ncat?
<henri> for instance the builtin "chat" or a http server or some proxy
protocols
<fyodor> better in Ncat than Nmap
<henri> could be (re-)implemented in lua
<yyzfp> Mebbe the WebSocket idea would be a good fit.
<henri> yes, maybe too
<bonsaiviking> yeah, any of the server-side-type things (dependent on Nsock
server-mode?)
<fyodor> When you get a chance, Henri, maybe you could write up more about
the Ncat scripting engine idea onto the page?
<henri> not necessarily dependent, but could leverage it, sure
<henri> fyodor: ok
<fyodor> Thanks
<yyzfp> For ideas like this, we need to have a mentor lined up who knows
what he/she wants from the project.
<fyodor> yeah, definitely
<fyodor> Scanning pipeline is next
<fyodor> I think that is a great idea for making Nmap more efficient,
though I also agree that it sounds like a lot for a GSoC student
<fyodor> I'm wondering if we should move this off this page and onto Nmap
TODO list?  Or maybe there is a chance we could find someone talented
enough to do such a major re-architecture?  I'm kind of skeptical of that
though
<fyodor> even someone amazingly talented would need to get more experience
with Nmap first, probably
<henri> right, what we could do though is to try to split that huge task
into smaller ones
<henri> having a roadmap would be a big step forward already
<fyodor> Good point.  OK, so how about if I add it to Nmap TODO and remove
it from this GSoC page, but if we find a way to break it up and make a
roadmap, we can definitely add it back?
<fyodor> ok, so what I wrote up is this:
<fyodor> o Consider re-architecting Nmap to have more of a scanning pipeline
<fyodor> approach rather than fixed sets of hosts which start and finish one
<fyodor> phase and then move into the next in parallel.  This could
potentially
<fyodor> allow us to add hosts one by one to a phase as other hosts finish
that
<fyodor> phase and, ideally, the phases could run in parallel too.
<fyodor> Is that sort of what you had in mind?
<henri> absolutely
<fyodor> Great.  OK, I'm putting that in the Nmap TODO and taking out of
GSoC ideas for now
<fyodor> next is nsock server mode
<fyodor> is that done?
<henri> I have working code on nmapexp
<henri> (I know I have a lot there...)
<henri> but I want to rework some parts
<Bzitka_Yaknotz> On the pipeline project, could we use that as a
multi-person project and put two people on it. That way it's a little more
feasable but still only needs 1 mentor
<fyodor> That's a good idea, Peter, but Google is somewhat strict about "no
teams" because
<henri> I'd prefer my other branches (especially the proxy one) to be
merged first
<fyodor> they are worried about if the project isn't completed then one
student blames the other for not finishing and who do you pass and who do
you fail, however,
<fyodor> if it can be divided up into several discrete tasks where they can
be assigned to individual students and evaluated on their own even if one
of the other students doesn't deliver, then that is allowed
<Bzitka_Yaknotz> Understood
<fyodor> OK, so I want to finish this meeting in the next 12 minutes, so ...
<fyodor> maybe we will have to skip the rest of the ideas (going through
them) on this page for now, and let me check what's left on the agenda...
<fyodor> So one thing we wanted to see is who is interested in mentoring
this year?
<fyodor> and if so, are there any particular tasks/roles which would be
your top choices to mentor?
<fyodor> I'm hoping to mentor at least one student
<fyodor> Are you up for it again, David?
<yyzfp> Me.
<yyzfp> I want to do the Ncat and Nmap testing, at least.
<fyodor> Good :).  And Henri?  Daniel?
<henri> I unfortunately don't think I can mentor again this year, I might
lack time
<fyodor> OK, if that changes, Henri, definitely let us know
<henri> (which is a pity, because I loved it!)
<fyodor> and you did a great job
<henri> thanks, maybe as a backup mentor
<fyodor> that's a good idea
<fyodor> OK, so the last thing I wanted to cover was
<bonsaiviking> I would love to be able to help, but I'm expecting family
commitments beginning in June to eat up my time
<fyodor> promotion of Nmap GSoC.  I think one of the most important aspects
to whether GSoC is a success for us is of course whether we get great
applicants
<fyodor> and we're competing with like 150 other projects, not to mention
all the other things that top students can do over the summer
<bonsaiviking> fyodor: is there ever any concern about *too many*
applicants?
<fyodor> so I'm trying to think of ideas for how we can promote Nmap SoC so
that more people know about it, and also how to make Nmap SoC particularly
enticing to these students
<bonsaiviking> Just wondering.
<yyzfp> bonsaiviking: not really.
<fyodor> I think too many applicants is a problem we could handle :).
 Actually, in the very first year or two,
<fyodor> we really did get an absolute ton of great applicants, because I
think there were only like 20 orgs at first, and the program was new so it
got tons of press
<fyodor> but in recent years, it has been more of a struggle.
 Particularly, I think, as the security market has grown and as awesome as
GSoC is, it doesn't really pay competitively to internships with U.S.
software and security companies, in general
<brain> Does GSoC stipulate any restrictions on organizations offering
incentives?  I don't know what that would be for nmap, just curious.
<fyodor> but of course it has many great points.  Folks can work from home
or anywhere else in the World they want to be
<fyodor> That's a good point, brain, we have thought about possible
incentives.  I want to discuss it with the GSoC organizers first to make
sure they're cool with it though
<fyodor> we don't want to get on their bad side, for sure
<brain> *nod*
<bonsaiviking> you could DM @securitytwits on twitter to promote it.
They've got 21K followers (about same as @nmap)
<fyodor> Oh, great idea!  I'm writing that down
<fyodor> And wow, Nmap has 21K followers?!
<bonsaiviking> very close to 22k
<brain> They also have an IRC channel, which will get less exposure but
perhaps a higher possibility of response.
<fyodor> too bad my last post was last November :)
<brain> securitytwits, that is.
<bonsaiviking> fyodor: may be a little late, but you should probably tweet
about Carna, since it's related to Nmap
<fyodor> That's a good point
<fyodor> anyone else have ideas for good promotion channels?
<fyodor> so of course there are the Nmap channels:
<fyodor> Nmap facebook and twitter
<fyodor> Insecure.Org, Nmap.Org
<brain> Higher ed infosec practioners, some of them have connections to
their academic counterparts.
<fyodor> nmap-hackers, nmap-dev mailing list
<brain> Can put the word out to some of those folks (educause, for
example).
<fyodor> That would be great
<fyodor> A good time, I think, is shortly after we're accepted (assuming we
are), and we should have the ideas page and such finished by then too
<bonsaiviking> I'll send emails to my local university contacts. Small
potatoes, but personal contact is often helpful.
<fyodor> BTW, you didn't introduce yourself brain
<fyodor> Thanks Daniel
<brain> Doh, sorry, I came in after the intros - Brian, I work on NYU's
security team.
<fyodor> Awesome, welcome!
<fyodor> We should encourage folks to join the official Google gsoc
channel(s) when they have time too, because
<fyodor> I think students come in there to chat and it's a good way to
answer questions and can perhaps mention to them that we'd love to have
them apply for Nmap
<fyodor> if it sounds like something fitting.  If they come in looking for
a game related project, then maybe Nmap wouldn't be so great, unless they
want to implement pong in NSE
<yyzfp> We already have ping, why not?
<fyodor> which would mean we'd have to add a new 'games' category
<fyodor> good point, David!
<Singhabhinavds> haha
<fyodor> Alright, well I said I'd finish this meeting in an hour or 1.5 so
I don't want to let it go on any longer, but
<fyodor> I want to thank everyone for coming, and if there is any last
thing you want to mention, do speak up
<yyzfp> http://www.google-melange.com/gsoc/events/google/gsoc2013
<Singhabhinavds> just the page of script ideas to get updated
<fyodor> and maybe we'll have another meeting later about this stuff
<Singhabhinavds> since i want to contribute by writing scripts
<yyzfp> That's the timeline for potentially interested mentors.
<Bzitka_Yaknotz> Nope, thanks a lot.
<fyodor> I think the script ideas page is mostly up to date, lert me check
<Singhabhinavds> yup just wanted to confirm if its updated or not
<Singhabhinavds> heres the link https://secwiki.org/w/Nmap/Script_Ideas
<fyodor> Yeah, I think it mostly is.  If you find one in there which we
have already finished, it is great if you can remove it
<fyodor> or if you get new ideas, feel free to add them
<Singhabhinavds> alright
<Singhabhinavds> nothing else from my side
<fyodor> Great folks, thanks again for coming and I'll TTYL
Mar 22 11:36:16 * Disconnected ().



Cheers,
Fyodor
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/