Snort mailing list archives

RE: [Snort-devel] Re: RFC: Forking Snort


From: "Bob Walder" <bwalder () spamcop net>
Date: Fri, 5 Jul 2002 09:49:18 +0100

As an outsider with no vested interests here I will chip in with my 0.02
(whatever that is worth at current USD/GBP exchange rates...)

Version 1.8.1 was the first version of Snort we included in our IDS Group
test - it was not the first version we had "looked at". However, even from
1.8.1 to 1.8.6 (the version we most recently tested - see www.nss.co.uk/ids)
we noted huge improvements in many aspects of the product. Improved
resistance to most IDS evasion techniques, some pretty useful stateful
reassembly with a session resync feature that is missing from a lot of
commercial products, some useful-looking protocol analysis preprocessors,
improved stability. And, of course, a huge improvement in the signature set.
From what I have seen on the mailing lists, there has been some considerable
effort put into tidying up the code "under the hood".

All of these improvements seem to have come about since the incorporation of
Sourcefire. Taking a purely cynical view: prior to that, Snort was shambolic
in places (witness different methods of passing parameters to preprocessors
in snort.conf - this is the sort of thing that needs to be made consistent
in a commercial-grade product) - having a major commercial product with some
serious VC funding behind it being based on Snort, and with the happy
coincidence that the creator of Snort heads up that commercial operation, it
is hardly surprising that some serious effort with some full-time developers
has been put it. It HAD to. Prior to Sourcefire, Snort was not really
suitable to use as the basis for a major commercial product - now it is very
close. And - in contrast to some of the claims in recent posts - this has
all happened pretty quickly as far as I can see (i.e. 1.8.1 to 1.8.6 took a
little over 6 months)

In other words, from the outside looking it I can only see GOOD things
having come out of the Sourcefire link so far.

Can it improve further? Of course it can - performance needs looking at now,
but we are promised good things in the next major release. With commercial
pressure driving the development, I am sure we will not be waiting as long
for that release as we would if it were being developed by committee!

Regards,

Bob Walder
Director
The NSS Group






-----Original Message-----
From: snort-devel-admin () lists sourceforge net
[mailto:snort-devel-admin () lists sourceforge net]On Behalf Of Martin
Roesch
Sent: 05 July 2002 05:00
To: Jed Pickel; snort-devel () lists sourceforge net
Cc: snort-users () lists sourceforge net; focus-ids () securityfocus com
Subject: Re: [Snort-devel] Re: RFC: Forking Snort


On 7/4/02 3:50 AM, "Jed Pickel" <jed () pickel net> wrote:

First off: I sent the last mail based on what I've witnessed as a
growing dis-satisfaction and concern over the management
of Snort over
the past year. My leaving the Snort project 6 months ago
was part of
that and I recently have received an increase in private
communication
about the topic. Being in a decent position to stick my neck out I
crafted the last message in an attempt to get a gauge on
the community.

Interesting how there can be "growing dissatisfaction" in
the user base
without any of the core developers hearing of it.  When you
left the project
back in January the only reason that you gave was "I don't
have time for the
project any more", not that it was somehow being mismanaged
by myself.  Who
are you in "private communication" with about the management
of the project?
Do they have any commercial interest in Snort?  Seeing as
your neck is out
instead of theirs is pretty interesting, why can't they speak for
themselves?

I will state for the record at this point that I believe a
fork in Snort
is not the best route for the community at this time;
however, I believe
some changes in management philosophy and process for Snort are
necessary to prevent a fork. The changes I am proposing
are in the best
interest of users, developers, third party businesses,
Sourcefire, and
the projects current leadership in my opinion.

Have you ever managed a large open source project before?
What is the basis
of your opinion?

Mixing the "benevolent dictator" model with business
------------------------------------------------------------------
I believe Snort would benefit from a change in management
philosophy and
that the current leadership is capable of implementing and
following
through on these changes.

To explain the reasons lets consider what would happen if
Linus Torvalds
were the CTO of RedHat.

Alan Cox is the lead developer on the production kernels and
works for
RedHat, so you can probably say that the situation exists.
Has Linux's
development suffered as a result of it's developers being
able to work on it
full time and for pay?

* Would Linus accept contributions to the Linux kernel
from the Debian
or Suse folks?

Yes, just as I continue to take contributions from a number
of people from
the community (Mike Fisk, Jed Haile, Glenn Mansfield Keeni,
Roman Danyliw,
Phil Wood, etc.).  The reason that things get into or out of
Snort is based
on what they do (functionality) and how well they do it and
not a whole lot
else.  That's always been the case, performance, portability
and stability
are the criteria against which contributions are judged.

If code doesn't make it into the distro, people still have
the option of
maintaining it as a patch against the core distro, such as
the "ac" patches
for Linux or the SnortSAM module for Snort.  Because it
doesn't make it in
doesn't mean you can't maintain it separately.

This brings up a whole new discussion: who maintains the
code.  For example,
if you have walked off 6 months ago and hadn't had Roman
Danyliw there to
pick up maintenance of the code, who would have maintained
it?  What if
nobody but me was left to do it?  Should I be required to?
Would I be an
"evil corporate guy out for his own interests" if I were to
remove the code
from the distro because I didn't want to maintain it?

If we're going to go down the road of hypotheticals there
are all sorts of
bizarre scenarios we can come up with I'm sure.  Why are we
bothering to
talk about things that  haven't come to pass as debating
points for whether
we should take drastic action?

* Would existing third party code contributions be "scrubbed" if it
meant solidifying RedHats market share and increasing sales?

Scrubbing means nothing more than bug fixing and sanity
checking.  For
example, does it make more sense to use automatic variables
(like a char
array) in functions or is it better to malloc to a pointer, use the
allocated memory space, then free that pointer at the end of
the function?
Who's responsibility is it to fix the code if it's doing the
latter instead
of the former due to the inexperience of the coder?  What if
the advice is
given and not heeded?  When can I as maintainer take the
initiative to make
a contributor's work better?  Since it's "my" codebase,
shouldn't I be able
to do what I want with it?  Linus certainly seems to feel
free to run the
Linux project and manage patches/contributions anyway he feels fit.

* (Or -- following on a previous post) What if a contribution would
be a huge leap in Linux kernel technology but it would
mean rewriting
the proprietary pieces of RedHat Linux all while dealing
with customers
demanding this feature. Would that contribution to the
"open-source"
Linux kernel be delayed or ignored?

Like the pre-emptive kernel patches that Linus hasn't
allowed into the
kernel because they violate his submission guidelines?
There's a good
example.  He's not working for a company with a commercial
interest in the
development of Linux, yet something along those same lines
is happening
because the change is as yet unproven and the patch is so
big that Linus
rejected it for violating his submission guidelines.

You don't have to have commercial interest in the project to
reject things.
I'm not sure that allowing the community to play Monday
morning quarterback
for the inclusion of code is a good thing.  I've found over
and over again
that many contributors to Snort have little concept of writing high
performance code or effective testing and validation
techniques.  Is it
better to have a committee of people with no focus on the
real needs of the
project or a bunch of like minded individuals who have a
similar level of
experience and perspective of what's important?

* What would happen to the Linux kernel if the RedHat
investors sold
the company to Microsoft for some ridiculous sum of money and Linus
had signed a non-compete? The "benevolent dictator" would be out of
commission and the Linux community as a whole would suffer,
particularly if the intention of Microsoft was to disrupt
their main
competitor. Would the fallout halt the progress of Linux
while a new
management structure forms out of the chaos?

That's fantasy land stuff, I won't even address it.  I'm not going to
discuss Sourcefire's corporate strategy where all my
competitors can see it,
Snort will always be free and I'm going to be the guy
heading it until and
unless I hand it off to another highly qualified individual
with the skills
and the personality to handle the project.

* Or would the Linux kernel either pro-actively fork or be
forced into
a different style of management because of community pressure to
avoid the above problems?

If people want to fork Snort they can do as they please, but
without the
support of the core developers all you've got is 64,000+
lines of legacy
code to start maintaining.  I'm only seeing "pressure" from
an exceedingly
small group of the overall Snort community, so I don't see
any great cry to
address the specters that you raise here.  Quite frankly,
the fact that this
was done "Pearl Harbor" style continues to have me
scratching my head as to
the identity of the silent 3rd parties that are advocating
this course of
action.

Sourcefire can continue as is for a while but conflicts of interest
are going to come up regardless of the promises from Marty. Walking
the fine line of preventing a fork and maximizing profits can be
avoided all together by pro-actively changing the
management style and
philosophy of open-source Snort.

You state as fact things that you can't possibly know.  Why are there
suddenly going to be conflicts of interest where there have been none
before?  Have there been any conflicts of interest?  Why am
I going to go
back on the one key promise that is essential to the success
of my corporate
endeavor?  You state as an inevitability something that
hasn't happened in
18 months of Sourcefire's history, indeed Snort's development has
accelerated rapidly since the founding of Sourcefire and I
think few people
disagree that Snort is vastly improved over the last
pre-Sourcefire version
(version 1.7).

I'm a big fan of the "benevolent dictator" model and I
think it works
beautifully when profit is not one of the primary motivators of the
dictator. When profit motivations are thrown into the mix
measures have
to be put in place to retain the public confidence that
these conflicts
of interest can not manifest. No doubt there are times
when business and
open source goals run in parallel. The above hypothetical
analogy of
Linus as the CTO of RedHat are a few examples where they do not.

"Profit" means different things to different people.  Is profit the
notoriety associated with running a successful open source
project?  It sure
as hell is!  Up until Sourcefire was started, I made
*nothing* directly on
Snort but quite a bit on being the author of Snort.  I
didn't get paid for
developing Snort, but I gained some amount of fame which in
turn led to
better jobs and opportunities to get paid for speaking.

You seem to ignore my stated position that Sourcefire can
only succeed by
making Snort as good as possible, do you doubt that that is the case?

One of certainly many possible ways to address this issue
could be for
Sourcefire to create a non-profit corporation to manage open-source
snort. A possible way to structure such an organization is how the
Apache folks have. Take a look at this url
(http://www.apache.org/foundation/roles.html) for an
example. The ideal
would be to ensure that a diverse group is represented on
the Board Of
Directors for this non-profit corp including users, developers, and
businesses that provide Snort based solutions.

Take significant steps now to prohibit conflict of
interest issues and I
believe that Snort will unify and get stronger than any of us ever
imagined. I believe this would ensure the goals for Snort
mentioned in
Marty's message would not only be met, but significantly exceeded.

I doubt that.  There are people out there who have their commercial
aspirations placed *way* ahead of the good of Snort and the
community, some
of whom I used to trust implicitly.  If such an organization were
established they'd be first in line to get a leading
position in such an
organization.  I would refute that I have mismanaged the
project, and I
doubt that signing up a predatory committee to run the
development process
is going to help anything.  I've been screwed way too many
times over the
past 3 years by supposed friends and partners to turn Snort
over to you when
you're acting as a proxy for a as-yet-unknown 3rd party with
an interest in
controlling Snort.

Well defined processes for code contributions and "scrubbing" code
------------------------------------------------------------------
One way to increase the number and quality of
contributions is to have a
well defined process for contributing code. Why do I mention this?
Because shortly after I sent the last message half of the private
responses I received came from people or organizations
that tried to
contribute code or bugfixes. They never received any
response whatsoever
from the Snort leadership.

How many total people is that?  One?  One thousand?  What significant
patches/contributions ended up not being used?

Obviously, certain requirements have to be met for code
contributions.
When a contribution does not meet those requirements tell the
contributer the decision and if possible point out the
deficiencies in
the contribution. Contributers improve the quality
flexibility of Snort.
Snort would not be what it is today without its contributers.

As for scrubbing --- scrub away, but tell the maintainer
of that code
why their code is being scrubbed or what they can do to address the
problems to prevent it from being scrubbed. Many
contributers have often
made a large investment in Snorts success and they deserve
at least that
much.

How much do you think they'd complain about me being a big
meanie when I
told them that their code was slow, crashed constantly, had
terrible memory
management and introduced DoS conditions into Snort?  Do we need to
introduce political correctness into the process as well?
That'd certainly
help people keep from getting their feelings hurt, we
wouldn't want that
either...

As a disclaimer --- My past code contributions are not my
motivation for
bringing up these issues. I personally don't care what is
done with any
code I've contributed in the past. Scrub it, delete it,
throw darts at
it. I won't loose any sleep over it. I am aware of a number of
organizations that use some of that code in their production snort
installations and couple companies and organizations that
have built
products and services around some of that code. They might
loose sleep!

You stated in your original message that the treatment that
you had received
at my hands was part of your motivation....

People who build products on top of Snort should probably
try to make sure
that they stay in close communications with the core
development team to
make sure that they're always apprised of the development
trends.  As a
group we're pretty easy to get a hold of.

BTW, the sense of entitlement that people have for free code
is pretty
interesting, I'm constantly amazed at what people feel are
my obligations
towards them because they use something that I develop for free...

Now for a few responses....
------------------------------------------------------------------
Alfred Huger wrote...
You don't happen to have a business plan in hand that
requires Snort
by chance do you? Or better yet are you working somewhere
that does?

No! My motivations are solely to protect open-source snort, see its
continued growth and improvement, and continued increase in market
share. To fully disclose, my only business in the info-sec
industry is
occasional independent consulting (Incident Response and Code
Development). My past, present, or future role in the
Snort community is
not relevant to that effort. The majority of my current business
ventures are in other industries.

But you also claim to be acting on behalf of unknown third
parties as well,
what is their interest in Snort?

Martin Roesh wrote (these 2 comments are from different
messages)...
Having code accepted for inclusion in the base Snort distro is a
privilege, not a right, you aren't entitled to seeing your code go
into Snort just because you ship it over.
..
I get a lot of patches and contributions otherwise and only review
what immediately piques my interest or what I can get to.

You have every right to make that decision. I believe that
Snort would
find more success if contributions are encouraged and if
those that have
made the investment to contribute have a well defined
process, including
guidelines and requirements. Contributors also deserve a
response and a
reason if their code is not accepted. Or not... But the
consequence is
that pressure will continue to build for a fork.

If you want to fork the code, be my guest, have fun maintaining it.
Contributions are and always have been encouraged, what
leads you to believe
otherwise?

I guess I'm confused.  If people don't like the current
cycle of major bug
fixing and feature additions from myself and the rest of the
core developer
group, I'd like to hear about what directions they think we
should be going
in.  More instability?  Slower code?  Their own pet
features?  The fact of
the matter is that Snort's development has never been more
vibrant and
active than since Sourcefire was started and, more recently,
brought on
Andrew Baker and Chris Green.  You seem to imply that this
insurmountable
pressure from this as-of-yet unheard from group of fellow
dissenters is
going to force us to do something or risk a fork.  That
sounds a lot like an
ultimatum, I really hate those...

Andrew Baker wrote...
It was assumed the developers of the existing output plugins would
port over their existing code to Barnyard.  However, this
has not ever
come to pass and thus to bridge the gap

Here are two examples of how this statement is false.

* Database plugin: After discussing porting my Snort contributed db
plugin to Barnyard on the snort-admin list and completing
90% of the
port you released an incomplete plugin that only included MySql
support. Had I known that your intentions were to replace the
database plugin, I would have never bothered investing my
time porting.

This completely flies in the face of the Cathedral and the
Bazaar, if your
code was better it would have quickly edged out Andrews
code.  As it was, we
never saw anything from you and you quit the project 3 months later.
Andrew's intentions (as I recall) was to solve a problem
that we were facing
(not having DB support in BY with no port in sight from you)
and he did
that.  Why couldn't you keep going?  We had spp_unidecode and
spp_http_decode for a long time before we merged the two
functionalities,
why does your plugin need to have an exclusive place in the
hierarchy of
contributions for you to continue with it?

This doesn't make any sense, it seems like you're ignoring the entire
ideology of open source development when you make statements
like this.

* A Barnyard port of the XML plugin was submitted 3 months
ago yet I
still have not seen it in the distribution.

Due to the license changes (GPL->QPL) the lawyers at
CMU/CERT needed to leap
into action, we're (well, Andrew and Roman) waiting to see
what they say.
Funny you haven't discussed this with Roman at all...

So exactly what is required here to "bridge the gap"? I
don't understand
your underlying motivations or the logic behind criticizing for not
contributing code while at the same time not acknowledging
or accepting
contributions. Another mystery about Barnyard is the reason for the
license changed from GPL to QPL.

A commercial entity was using it with modifications and
selling it as part
of a product without open sourcing those mods and refusing
to contribute
them back.  The QPL allows Andrew and myself to force
compliance with the
open source license (everyone using the license has a
perpetual nonexclusive
license to code developed on top of it) and it's OSI approved.  Go to
http://www.opensource.com for more info.

Ryan Russel wrote...
And there's nothing to stop them. Even if your worst case comes
true, why would it stop the independents? The worst
monopoly in our
industry, Microsoft, still leaves tons of room for others to make
money. And you're a whole lot better of with Snort, because it's
open source and you CAN fork if SourceFire goes nuts and releases
the next version only to paying customers.

Very true. It is a nice insurance policy. Although, I
would not like to
see a fragmented community. Hopefully Sourcefire will do
the right thing
to prevent that from happening.

The "right thing" being what you recommend?  I would refute
that that is the
right thing at all.  Just because you say it is doesn't make it so.

Jed Haile -- the other Jed :) wrote his one...
Another thing I should point out, as one of the
developers of hogwash, is
that if you want to do your own thing with Snort, have at
it. Marty and the
rest of the Snort team have been fully supportive of the
hogwash team's
efforts. Hogwash is in many aspects a fork of Snort, it
is also a project
that I hope to one day see merged with Snort. Marty has
expressed that he
would like to see that also.

Obviously anyone can fork at any time they want. There is also the
snort-adapter fork. The fewer forks the better IMHO. The
reason this has
not happened more often is that this requires significant
investment of
both time and money, not because people have not wanted
to. I'd like to

Additionally, forking sounds a lot better on paper than it
works in reality.
How many people are going to sign up to get a few hundred
emails to their
inbox a day with no expectation of getting compensated in
any way for their
time and effort and relentless criticism when they don't
reply in a timely
and courteous fashion?  Who has the patience to answer the
same FAQ for the
25347th time from people who can't be bothered to read the
docs or don't
speak english or that have a grudge against them?  Who is
going to work a
full 8-16 hour day at their job to come home and handle this
stuff?  Who is
going to coordinate the efforts of developers from all over
the world and
make the determination what gets in and what doesn't for the
code base as
well as arbitrate the inevitable disputes?  Who has the
personality to do
this successfully without becoming a raving lunatic or non-benevolent
dictator?  How long will this go on before the forkers get
bored and go onto
things where they can get paid to be smart and personable
and manage a herd
of cats?

Who is going to dedicate time away from friends and family
for the sake of a
bunch of people who are going to turn around someday and suggest that
because they'd like to get some compensation by devising a nice
complementary way of solving the problems that large users
face that they no
longer have the best interests of the users at heart any more?

Really Jed, your line of reasoning is *really* starting to
make me scratch
my head, you don't have the *any* idea why Sourcefire
exists, what it's
principles are, what my intentions are for Snort, or
anything even remotely
approaching an idea of what reality looks like with regards
to Snort these
days.

If you would like to find out, here's what I'll do.  If you
want to drive
out to the Sourcefire office in Columbia MD (it's about 3
hours drive from
you in Pittsburg) I'll personally take you on a guided tour
of what we're up
to and how it relates to Snort.  If you walk away from that
meeting feeling
like we're about to make Snort proprietary or that we have
anything less
than the interests of the community in mind, then you'll
have a basis for
your complaint.  Otherwise this is all just idle speculation
and hardly
anything to take drastic action on.  Once again, my office
phone number is
410-290-1616, feel free to call and schedule a meeting, I'll
give you a full
day of my time if necessary.

see people unify around one version of Snort and I believe
some changes
in management style would facilitate this.

I believe that ultimately you have no idea if this would
solve anything and
you're looking for change because you felt slighted by me in
public and
you'd like me to not have anywhere near the voice that I do,
or at least
that seemed to be the theme of your first message.  This one
seems to be
different, however, so I'm not sure what you're really looking for.

The fact that you still haven't talked to any of the core
developers of
Snort or myself but yet you continue to speak to and
coordinate your message
with this cabal of disgruntled 3rd parties really has me
asking a lot of
questions about your motivations and real intentions.  Even
the thought of
somehow developing a committee with this group of people who
are sitting in
the shadows is a non-starter, I don't work with people I
can't trust and
there's way too much subterfuge going on here.

My Concluding Thoughts
------------------------------------------------------------------

I want to thank everyone for their opinions. This has
helped clarify
many issues to me and hopefully the users and developers
on these lists.
This discussion has led me to believe that a fork is not
the best route
at this time. Nevertheless, I believe we need to encourage
the current
leadership to put measures in place to avoid conflicts of
interest and
have a well defined and open process for code contributions.

I encourage you to take the time to understand everything
that Sourcefire is
doing, you have been offered unprecedented access to our
technical and
management staff and if you're truly interested in what's
best for the Snort
project you'll get active in it again instead of presenting
wild conjecture
and hypotheticals as truth and recommending courses of
action without full
knowledge of why you're doing it.

I would like to suggest that your involvement with these "unknown 3rd
parties" has only given you half of the actual story and
that if you're
really interested in Snort's future you'll take me up on this.

     -Marty

--
Martin Roesch - Founder/CTO Sourcefire Inc. - (410) 290-1616
Sourcefire: Professional Snort Sensor and Management Console
appliances
roesch () sourcefire com - http://www.sourcefire.com
Snort: Open Source Network IDS - http://www.snort.org



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Caffeinated soap. No kidding.
http://thinkgeek.com/sf
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel





-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Bringing you mounds of caffeinated joy.
http://thinkgeek.com/sf
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users


Current thread: