Re: [Snort-devel] Re: RFC: Forking Snort

[Martin Roesch <roesch () sourcefire com>]

On 7/4/02 3:50 AM, "Jed Pickel" <jed () pickel net> wrote:

> First off: I sent the last mail based on what I've witnessed as a
> growing dis-satisfaction and concern over the management of Snort over
> the past year. My leaving the Snort project 6 months ago was part of
> that and I recently have received an increase in private communication
> about the topic. Being in a decent position to stick my neck out I
> crafted the last message in an attempt to get a gauge on the community.

Interesting how there can be "growing dissatisfaction" in the user base
without any of the core developers hearing of it.  When you left the project
back in January the only reason that you gave was "I don't have time for the
project any more", not that it was somehow being mismanaged by myself.  Who
are you in "private communication" with about the management of the project?
Do they have any commercial interest in Snort?  Seeing as your neck is out
instead of theirs is pretty interesting, why can't they speak for
themselves?

> I will state for the record at this point that I believe a fork in Snort
> is not the best route for the community at this time; however, I believe
> some changes in management philosophy and process for Snort are
> necessary to prevent a fork. The changes I am proposing are in the best
> interest of users, developers, third party businesses, Sourcefire, and
> the projects current leadership in my opinion.

Have you ever managed a large open source project before?  What is the basis
of your opinion?

> Mixing the "benevolent dictator" model with business
> ------------------------------------------------------------------
> I believe Snort would benefit from a change in management philosophy and
> that the current leadership is capable of implementing and following
> through on these changes.
>
> To explain the reasons lets consider what would happen if Linus Torvalds
> were the CTO of RedHat.

Alan Cox is the lead developer on the production kernels and works for
RedHat, so you can probably say that the situation exists.  Has Linux's
development suffered as a result of it's developers being able to work on it
full time and for pay?

> * Would Linus accept contributions to the Linux kernel from the Debian
> or Suse folks?

Yes, just as I continue to take contributions from a number of people from
the community (Mike Fisk, Jed Haile, Glenn Mansfield Keeni, Roman Danyliw,
Phil Wood, etc.).  The reason that things get into or out of Snort is based
on what they do (functionality) and how well they do it and not a whole lot
else.  That's always been the case, performance, portability and stability
are the criteria against which contributions are judged.

If code doesn't make it into the distro, people still have the option of
maintaining it as a patch against the core distro, such as the "ac" patches
for Linux or the SnortSAM module for Snort.  Because it doesn't make it in
doesn't mean you can't maintain it separately.

This brings up a whole new discussion: who maintains the code.  For example,
if you have walked off 6 months ago and hadn't had Roman Danyliw there to
pick up maintenance of the code, who would have maintained it?  What if
nobody but me was left to do it?  Should I be required to?  Would I be an
"evil corporate guy out for his own interests" if I were to remove the code
from the distro because I didn't want to maintain it?

If we're going to go down the road of hypotheticals there are all sorts of
bizarre scenarios we can come up with I'm sure.  Why are we bothering to
talk about things that  haven't come to pass as debating points for whether
we should take drastic action?

> * Would existing third party code contributions be "scrubbed" if it
> meant solidifying RedHats market share and increasing sales?

Scrubbing means nothing more than bug fixing and sanity checking.  For
example, does it make more sense to use automatic variables (like a char
array) in functions or is it better to malloc to a pointer, use the
allocated memory space, then free that pointer at the end of the function?
Who's responsibility is it to fix the code if it's doing the latter instead
of the former due to the inexperience of the coder?  What if the advice is
given and not heeded?  When can I as maintainer take the initiative to make
a contributor's work better?  Since it's "my" codebase, shouldn't I be able
to do what I want with it?  Linus certainly seems to feel free to run the
Linux project and manage patches/contributions anyway he feels fit.

> * (Or -- following on a previous post) What if a contribution would
> be a huge leap in Linux kernel technology but it would mean rewriting
> the proprietary pieces of RedHat Linux all while dealing with customers
> demanding this feature. Would that contribution to the "open-source"
> Linux kernel be delayed or ignored?

Like the pre-emptive kernel patches that Linus hasn't allowed into the
kernel because they violate his submission guidelines?  There's a good
example.  He's not working for a company with a commercial interest in the
development of Linux, yet something along those same lines is happening
because the change is as yet unproven and the patch is so big that Linus
rejected it for violating his submission guidelines.

You don't have to have commercial interest in the project to reject things.
I'm not sure that allowing the community to play Monday morning quarterback
for the inclusion of code is a good thing.  I've found over and over again
that many contributors to Snort have little concept of writing high
performance code or effective testing and validation techniques.  Is it
better to have a committee of people with no focus on the real needs of the
project or a bunch of like minded individuals who have a similar level of
experience and perspective of what's important?

> * What would happen to the Linux kernel if the RedHat investors sold
> the company to Microsoft for some ridiculous sum of money and Linus
> had signed a non-compete? The "benevolent dictator" would be out of
> commission and the Linux community as a whole would suffer,
> particularly if the intention of Microsoft was to disrupt their main
> competitor. Would the fallout halt the progress of Linux while a new
> management structure forms out of the chaos?

That's fantasy land stuff, I won't even address it.  I'm not going to
discuss Sourcefire's corporate strategy where all my competitors can see it,
Snort will always be free and I'm going to be the guy heading it until and
unless I hand it off to another highly qualified individual with the skills
and the personality to handle the project.

> * Or would the Linux kernel either pro-actively fork or be forced into
> a different style of management because of community pressure to
> avoid the above problems?

If people want to fork Snort they can do as they please, but without the
support of the core developers all you've got is 64,000+ lines of legacy
code to start maintaining.  I'm only seeing "pressure" from an exceedingly
small group of the overall Snort community, so I don't see any great cry to
address the specters that you raise here.  Quite frankly, the fact that this
was done "Pearl Harbor" style continues to have me scratching my head as to
the identity of the silent 3rd parties that are advocating this course of
action.

> Sourcefire can continue as is for a while but conflicts of interest
> are going to come up regardless of the promises from Marty. Walking
> the fine line of preventing a fork and maximizing profits can be
> avoided all together by pro-actively changing the management style and
> philosophy of open-source Snort.

You state as fact things that you can't possibly know.  Why are there
suddenly going to be conflicts of interest where there have been none
before?  Have there been any conflicts of interest?  Why am I going to go
back on the one key promise that is essential to the success of my corporate
endeavor?  You state as an inevitability something that hasn't happened in
18 months of Sourcefire's history, indeed Snort's development has
accelerated rapidly since the founding of Sourcefire and I think few people
disagree that Snort is vastly improved over the last pre-Sourcefire version
(version 1.7).
 
> I'm a big fan of the "benevolent dictator" model and I think it works
> beautifully when profit is not one of the primary motivators of the
> dictator. When profit motivations are thrown into the mix measures have
> to be put in place to retain the public confidence that these conflicts
> of interest can not manifest. No doubt there are times when business and
> open source goals run in parallel. The above hypothetical analogy of
> Linus as the CTO of RedHat are a few examples where they do not.

"Profit" means different things to different people.  Is profit the
notoriety associated with running a successful open source project?  It sure
as hell is!  Up until Sourcefire was started, I made *nothing* directly on
Snort but quite a bit on being the author of Snort.  I didn't get paid for
developing Snort, but I gained some amount of fame which in turn led to
better jobs and opportunities to get paid for speaking.

You seem to ignore my stated position that Sourcefire can only succeed by
making Snort as good as possible, do you doubt that that is the case?

> One of certainly many possible ways to address this issue could be for
> Sourcefire to create a non-profit corporation to manage open-source
> snort. A possible way to structure such an organization is how the
> Apache folks have. Take a look at this url
> (http://www.apache.org/foundation/roles.html) for an example. The ideal
> would be to ensure that a diverse group is represented on the Board Of
> Directors for this non-profit corp including users, developers, and
> businesses that provide Snort based solutions.
>
> Take significant steps now to prohibit conflict of interest issues and I
> believe that Snort will unify and get stronger than any of us ever
> imagined. I believe this would ensure the goals for Snort mentioned in
> Marty's message would not only be met, but significantly exceeded.

I doubt that.  There are people out there who have their commercial
aspirations placed *way* ahead of the good of Snort and the community, some
of whom I used to trust implicitly.  If such an organization were
established they'd be first in line to get a leading position in such an
organization.  I would refute that I have mismanaged the project, and I
doubt that signing up a predatory committee to run the development process
is going to help anything.  I've been screwed way too many times over the
past 3 years by supposed friends and partners to turn Snort over to you when
you're acting as a proxy for a as-yet-unknown 3rd party with an interest in
controlling Snort.

> Well defined processes for code contributions and "scrubbing" code
> ------------------------------------------------------------------
> One way to increase the number and quality of contributions is to have a
> well defined process for contributing code. Why do I mention this?
> Because shortly after I sent the last message half of the private
> responses I received came from people or organizations that tried to
> contribute code or bugfixes. They never received any response whatsoever
> from the Snort leadership.

How many total people is that?  One?  One thousand?  What significant
patches/contributions ended up not being used?

> Obviously, certain requirements have to be met for code contributions.
> When a contribution does not meet those requirements tell the
> contributer the decision and if possible point out the deficiencies in
> the contribution. Contributers improve the quality flexibility of Snort.
> Snort would not be what it is today without its contributers.
>
> As for scrubbing --- scrub away, but tell the maintainer of that code
> why their code is being scrubbed or what they can do to address the
> problems to prevent it from being scrubbed. Many contributers have often
> made a large investment in Snorts success and they deserve at least that
> much.

How much do you think they'd complain about me being a big meanie when I
told them that their code was slow, crashed constantly, had terrible memory
management and introduced DoS conditions into Snort?  Do we need to
introduce political correctness into the process as well?  That'd certainly
help people keep from getting their feelings hurt, we wouldn't want that
either...

> As a disclaimer --- My past code contributions are not my motivation for
> bringing up these issues. I personally don't care what is done with any
> code I've contributed in the past. Scrub it, delete it, throw darts at
> it. I won't loose any sleep over it. I am aware of a number of
> organizations that use some of that code in their production snort
> installations and couple companies and organizations that have built
> products and services around some of that code. They might loose sleep!

You stated in your original message that the treatment that you had received
at my hands was part of your motivation....

People who build products on top of Snort should probably try to make sure
that they stay in close communications with the core development team to
make sure that they're always apprised of the development trends.  As a
group we're pretty easy to get a hold of.

BTW, the sense of entitlement that people have for free code is pretty
interesting, I'm constantly amazed at what people feel are my obligations
towards them because they use something that I develop for free...

> Now for a few responses....
> ------------------------------------------------------------------
> Alfred Huger wrote...
> > You don't happen to have a business plan in hand that requires Snort
> > by chance do you? Or better yet are you working somewhere that does?
>
> No! My motivations are solely to protect open-source snort, see its
> continued growth and improvement, and continued increase in market
> share. To fully disclose, my only business in the info-sec industry is
> occasional independent consulting (Incident Response and Code
> Development). My past, present, or future role in the Snort community is
> not relevant to that effort. The majority of my current business
> ventures are in other industries.

But you also claim to be acting on behalf of unknown third parties as well,
what is their interest in Snort?

> Martin Roesh wrote (these 2 comments are from different messages)...
> > Having code accepted for inclusion in the base Snort distro is a
> > privilege, not a right, you aren't entitled to seeing your code go
> > into Snort just because you ship it over.
> ..
> > I get a lot of patches and contributions otherwise and only review
> > what immediately piques my interest or what I can get to.
>
> You have every right to make that decision. I believe that Snort would
> find more success if contributions are encouraged and if those that have
> made the investment to contribute have a well defined process, including
> guidelines and requirements. Contributors also deserve a response and a
> reason if their code is not accepted. Or not... But the consequence is
> that pressure will continue to build for a fork.

If you want to fork the code, be my guest, have fun maintaining it.
Contributions are and always have been encouraged, what leads you to believe
otherwise?

I guess I'm confused.  If people don't like the current cycle of major bug
fixing and feature additions from myself and the rest of the core developer
group, I'd like to hear about what directions they think we should be going
in.  More instability?  Slower code?  Their own pet features?  The fact of
the matter is that Snort's development has never been more vibrant and
active than since Sourcefire was started and, more recently, brought on
Andrew Baker and Chris Green.  You seem to imply that this insurmountable
pressure from this as-of-yet unheard from group of fellow dissenters is
going to force us to do something or risk a fork.  That sounds a lot like an
ultimatum, I really hate those...

> Andrew Baker wrote...
> > It was assumed the developers of the existing output plugins would
> > port over their existing code to Barnyard.  However, this has not ever
> > come to pass and thus to bridge the gap
>
> Here are two examples of how this statement is false.
>
> * Database plugin: After discussing porting my Snort contributed db
> plugin to Barnyard on the snort-admin list and completing 90% of the
> port you released an incomplete plugin that only included MySql
> support. Had I known that your intentions were to replace the
> database plugin, I would have never bothered investing my time porting.

This completely flies in the face of the Cathedral and the Bazaar, if your
code was better it would have quickly edged out Andrews code.  As it was, we
never saw anything from you and you quit the project 3 months later.
Andrew's intentions (as I recall) was to solve a problem that we were facing
(not having DB support in BY with no port in sight from you) and he did
that.  Why couldn't you keep going?  We had spp_unidecode and
spp_http_decode for a long time before we merged the two functionalities,
why does your plugin need to have an exclusive place in the hierarchy of
contributions for you to continue with it?

This doesn't make any sense, it seems like you're ignoring the entire
ideology of open source development when you make statements like this.

> * A Barnyard port of the XML plugin was submitted 3 months ago yet I
> still have not seen it in the distribution.

Due to the license changes (GPL->QPL) the lawyers at CMU/CERT needed to leap
into action, we're (well, Andrew and Roman) waiting to see what they say.
Funny you haven't discussed this with Roman at all...

> So exactly what is required here to "bridge the gap"? I don't understand
> your underlying motivations or the logic behind criticizing for not
> contributing code while at the same time not acknowledging or accepting
> contributions. Another mystery about Barnyard is the reason for the
> license changed from GPL to QPL.

A commercial entity was using it with modifications and selling it as part
of a product without open sourcing those mods and refusing to contribute
them back.  The QPL allows Andrew and myself to force compliance with the
open source license (everyone using the license has a perpetual nonexclusive
license to code developed on top of it) and it's OSI approved.  Go to
http://www.opensource.com for more info.

> Ryan Russel wrote...
> > And there's nothing to stop them. Even if your worst case comes
> > true, why would it stop the independents? The worst monopoly in our
> > industry, Microsoft, still leaves tons of room for others to make
> > money. And you're a whole lot better of with Snort, because it's
> > open source and you CAN fork if SourceFire goes nuts and releases
> > the next version only to paying customers.
>
> Very true. It is a nice insurance policy. Although, I would not like to
> see a fragmented community. Hopefully Sourcefire will do the right thing
> to prevent that from happening.

The "right thing" being what you recommend?  I would refute that that is the
right thing at all.  Just because you say it is doesn't make it so.

> Jed Haile -- the other Jed :) wrote his one...
> > Another thing I should point out, as one of the developers of hogwash, is
> > that if you want to do your own thing with Snort, have at it. Marty and the
> > rest of the Snort team have been fully supportive of the hogwash team's
> > efforts. Hogwash is in many aspects a fork of Snort, it is also a project
> > that I hope to one day see merged with Snort. Marty has expressed that he
> > would like to see that also.
>
> Obviously anyone can fork at any time they want. There is also the
> snort-adapter fork. The fewer forks the better IMHO. The reason this has
> not happened more often is that this requires significant investment of
> both time and money, not because people have not wanted to. I'd like to

Additionally, forking sounds a lot better on paper than it works in reality.
How many people are going to sign up to get a few hundred emails to their
inbox a day with no expectation of getting compensated in any way for their
time and effort and relentless criticism when they don't reply in a timely
and courteous fashion?  Who has the patience to answer the same FAQ for the
25347th time from people who can't be bothered to read the docs or don't
speak english or that have a grudge against them?  Who is going to work a
full 8-16 hour day at their job to come home and handle this stuff?  Who is
going to coordinate the efforts of developers from all over the world and
make the determination what gets in and what doesn't for the code base as
well as arbitrate the inevitable disputes?  Who has the personality to do
this successfully without becoming a raving lunatic or non-benevolent
dictator?  How long will this go on before the forkers get bored and go onto
things where they can get paid to be smart and personable and manage a herd
of cats?

Who is going to dedicate time away from friends and family for the sake of a
bunch of people who are going to turn around someday and suggest that
because they'd like to get some compensation by devising a nice
complementary way of solving the problems that large users face that they no
longer have the best interests of the users at heart any more?

Really Jed, your line of reasoning is *really* starting to make me scratch
my head, you don't have the *any* idea why Sourcefire exists, what it's
principles are, what my intentions are for Snort, or anything even remotely
approaching an idea of what reality looks like with regards to Snort these
days.

If you would like to find out, here's what I'll do.  If you want to drive
out to the Sourcefire office in Columbia MD (it's about 3 hours drive from
you in Pittsburg) I'll personally take you on a guided tour of what we're up
to and how it relates to Snort.  If you walk away from that meeting feeling
like we're about to make Snort proprietary or that we have anything less
than the interests of the community in mind, then you'll have a basis for
your complaint.  Otherwise this is all just idle speculation and hardly
anything to take drastic action on.  Once again, my office phone number is
410-290-1616, feel free to call and schedule a meeting, I'll give you a full
day of my time if necessary.

> see people unify around one version of Snort and I believe some changes
> in management style would facilitate this.

I believe that ultimately you have no idea if this would solve anything and
you're looking for change because you felt slighted by me in public and
you'd like me to not have anywhere near the voice that I do, or at least
that seemed to be the theme of your first message.  This one seems to be
different, however, so I'm not sure what you're really looking for.

The fact that you still haven't talked to any of the core developers of
Snort or myself but yet you continue to speak to and coordinate your message
with this cabal of disgruntled 3rd parties really has me asking a lot of
questions about your motivations and real intentions.  Even the thought of
somehow developing a committee with this group of people who are sitting in
the shadows is a non-starter, I don't work with people I can't trust and
there's way too much subterfuge going on here.

> My Concluding Thoughts
> ------------------------------------------------------------------
>
> I want to thank everyone for their opinions. This has helped clarify
> many issues to me and hopefully the users and developers on these lists.
> This discussion has led me to believe that a fork is not the best route
> at this time. Nevertheless, I believe we need to encourage the current
> leadership to put measures in place to avoid conflicts of interest and
> have a well defined and open process for code contributions.

I encourage you to take the time to understand everything that Sourcefire is
doing, you have been offered unprecedented access to our technical and
management staff and if you're truly interested in what's best for the Snort
project you'll get active in it again instead of presenting wild conjecture
and hypotheticals as truth and recommending courses of action without full
knowledge of why you're doing it.

I would like to suggest that your involvement with these "unknown 3rd
parties" has only given you half of the actual story and that if you're
really interested in Snort's future you'll take me up on this.

     -Marty

-- 
Martin Roesch - Founder/CTO Sourcefire Inc. - (410) 290-1616
Sourcefire: Professional Snort Sensor and Management Console appliances
roesch () sourcefire com - http://www.sourcefire.com
Snort: Open Source Network IDS - http://www.snort.org



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Caffeinated soap. No kidding.
http://thinkgeek.com/sf
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users