Spam blocking and the sorry state of e-mail

[Dave Farber <dave () farber net>]

Delivered-To: dfarber+ () ux13 sp cs cmu edu
Date: Wed, 08 Oct 2003 14:46:21 -0700 (PDT)
From: Lauren Weinstein <lauren () vortex com>
Subject: Spam blocking and the sorry state of e-mail
To: dave () farber net


Dave,

This is a long message, but I need to reply to Brad's comments publicly,
especially since there are more checks in the system than he may have
realized.  And this *does* point out the urgent need for Tripoli or
something like it.

First, why reject on bad or forged reverse DNS in the first place?  I
avoided doing this for years, and only reluctantly implemented it after
considering a range of related issues.  The positive effect was immediate
and dramatic, with very little relative downside.  Currently, I see about
one spam input attempt on the mail server here every two seconds (this is
moving toward one per second rapidly).  Each may represent multiple mailbox
deliveries.

For those of us with relatively limited bandwidth and processing
resources, the load of dealing with such trash is most definitely
non-trivial.  In particular, the processing cost of running so
many messages through the *other* spam filters that do text-based
and other analysis is prohibitive -- load goes through the roof
and all manner of things start to break down.

My own stats show that on the order of 80% of spam attempts have
bad or forged reverse DNS info (at least those arriving here).
By rejecting that mail immediately at the SMTP reply level, rather
than attempting to generate full-blown bounces involving new e-mail
messages (that will probably bounce back anyway due to forged
addresses), the load issues can at least be kept to mildly crazy levels
without everything grinding to a halt.

As Brad noted, my system delivers an *immediate* reply, warning the sender
that the mail has been rejected and clearly directing them to a Web page
section that gives a detailed explanation.  This includes links to pages
with other contact information (like a phone number), and a form link for
sending in a request for whitelisting (that is, exceptions to the block) or
to report other problems.  This is critical -- I refuse to run spam control
systems that do not provide immediate feedback to the sender -- too many
critical messages can be lost without the sender having any idea that there
was even a problem -- inexcusable and dangerous.

Of course, I can't force force people to read the explanatory page if they
don't want to.  In practice though, this is mostly academic.  In the year
I've been running this anti-spam system, I've needed to put maybe 14 names
on the whitelist (and probably half of those were proactive on my part, not
the result of problems).  That's out of a rather incredibly vast pile of
legit e-mail that gets through just fine.

The process of adding Brad to that whitelist did not "fail" -- he simply
didn't wait long enough for the related database to be updated
('cause that machine is slow because of the load due to all
the #!@#@! spam!)

Nor did Brad realize that there was an additional check to the system that I
strongly recommend for implementing DNS-based e-mail controls of this sort.
When using temporary errors for rejects in these situations as is my
standard in most cases, the retry pattern from typical "real" senders is
easily distinguishable from that of typical spambots.  My system scans the
logs at intervals to detect those patterns and report them for attention.
That's only actually come up a couple of times, and in both cases upon
whitelisting the blocked mail came right through.

So bottom line as far as Brad is concerned... I'm sorry that he had a little
extra hassle in reaching me quickly, but I've provided plenty of ways for
people to deal with the blocking system if they have a problem, and in fact
this whole issue comes up once in a blue moon for me.

Now, would I recommend this approach for a large ISP?  Probably not, given
that it needs to be managed carefully and a major ISP could see a very
different sort of mail load qualitatively than I do.  For me it not only
works but has been absolutely essential to keep things running, until we can
*really* fix e-mail.

BUT all the above having been said, the fact remains that the need
to hassle with all this stuff is intolerable.  The situation
with e-mail today is sick and perverted -- and I'm talking about the
technology of our mail systems, not just the content of so many
messages!  We play dueling software between filters and spammers,
with the users caught in the middle.  Garbage mail gets through
and clogs inboxes galore.  Important mail gets blocked.  Challenge-
response systems generate even more traffic and cause more problems.

E-mail needs to be *fundamentally* changed or it will become
utterly useless.  Our current e-mail framework was *not* designed
for the environment we now must deal with.  And frankly, I'm
extremely concerned that some of the approaches being proposed
by major ISPs, for example, could impose draconian controls that
would leave users as little more than slaves to the whims of their
Internet providers.

That's why my Tripoli proposal ( http://www.pfir.org/tripoli-overview )
attempts to deal with many of these issues while giving ultimate
control over mail decisions to users, rather than ISPs.  Nor does
Tripoli (unlike some other proposals) impose on all users a requirement
that they *only* accept authenticated mail -- there are situations
where being able to receive "anonymous" e-mail can be very
important.  Brad fears that most people won't want to receive
anonymous mail and so in practice it won't exist in most situations.
I say leave that decision up to the individual mail recipients, rather
than trying to impose a requirement that refuses to allow users to
block anonymous mail if they wish.

Frankly, I'm tired of all the mucking around.  The Internet community needs
to get together and solve these problems -- with the help of Tripoli or
something else -- Right Now.  Let's stop the purely academic chit chat and
arguing, and get this done, before the VeriSigns and their ilk take
us all to the cleaners.

[ I STRONGLY AGREE. IT IS TIME djf]


One thing I'd like to do ASAP is get a significant number of folks
working on these issues together for an intensive meeting to hash
out a way forward -- before we're all steamrollered down to pancakes
by the "dark side."  I'm in L.A., but such a meeting would probably
be most appropriately held in the San Francisco Bay Area.

There would almost certainly be limits on the number of attendees, but if you
or your organization might be interested in such a meeting, please drop me a
line at lauren () pfir org.

It's time to get off the dime.  Thanks.

--Lauren--
Lauren Weinstein
lauren () pfir org or lauren () vortex com or lauren () privacyforum org
Tel: +1 (818) 225-2800
http://www.pfir.org/lauren
Co-Founder, PFIR - People For Internet Responsibility - http://www.pfir.org
Co-Founder, Fact Squad - http://www.factsquad.org
Co-Founder, URIICA - Union for Representative International Internet
                     Cooperation and Analysis - http://www.uriica.org
Moderator, PRIVACY Forum - http://www.vortex.com
Member, ACM Committee on Computers and Public Policy

-------------------------------------
You are subscribed as interesting-people () lists elistx com
To manage your subscription, go to
 http://v2.listbox.com/member/?listname=ip

Archives at: http://www.interesting-people.org/archives/interesting-people/