Re: SecurityFocus: Botnets Likely Behind Jump in Spam

[Nick FitzGerald <nick () virus-l demon co uk>]

Fergie wrote:

> Via SecurityFocus.
>
> [snip]
>
> A significant rise in the global volume of spam in the past two months
> has security analysts worried that bot nets are increasingly being used
> by spammers to stymie network defenses erected to curtail bulk e-mail.
>
> Estimates of the magnitude of the increase in junk e-mail vary, but
> experts agree that an uncommon surge in spam is occurring. On the low
> side, Symantec, the owner of SecurityFocus, has found that average spam
> volume has increased almost 30 percent for its 35,000 clients in the
> last two months. Others have seen much more significant jumps: Spam
> black list maintainer Total Quality Management Cubed has seen a 450
> percent increase in spam in two months, ...

What's that line about lies, damn lies and statistics??

The TQM chart:

   http://tqmcube.com/tide.php

is uninterpretable _in the terms used in the SF article_.  Note that it 
charts volume in "units" of standard deviation _in the total data set_. 
> From _eyeballing_ their chart we see three to five (depending on your 
own gut feel for such things) modest-term _plateaus_) in about the 
first year of data (June 05 to June 06).  However, each of these 
platueas are themselves only slightly higher than the one before -- in 
short, there is very little variablilty in the level of spam for that 
first year, then in the next four months we see a large and erratic 
growth in spam, _measured in terms of the small variability of the 
preceding year's worth of data.  In reality this may actually only be a 
small overall increase in spam coupled with unusual week-to-week 
_variability_ in the volume.

Someone needs to give Rob a lesson in basic statistics -- I suspect 
that this particular, odd, choice of graphing has been chosen because 
it shows the _most dramatic_ effect (I doubt that they chose this 
because it's also about the _least informative_ approach they could 
have taken, but who knows??  It certainly has the feel that the techies 
may have produced a bunch of different graphs and the marketing folk 
made the decision as to which to use...).

I'd like to see their data normalized for "inboxes/addresses protected" 
or some similar vaguely meaningful simulacrum of a "typical Email 
recipient", as that is surely what "spam is increasing" means to 
ordinary folk -- "I used to get X spams per day but now get Y".  Who do 
you know, even amongst the geekiest of the math grads, who says 
something like "I'm now getting about 4.5 standard deviations, 
calculated over the last year's data, more spam compared to what I got 
a year ago"?

In case you still don't get this and think the TQM graph is strong 
evidence of a large increase in spam, a carefully created data set 
could probably be devised to produce an almost identical graph with 
perhaps as little as a 5-10% increase in total spams/recipient over a 
16 month period. 

> ... and the amount of spam filtered
> out every week by security software maker Sunbelt Software has more
> than tripled compared to six months ago.

And what rate of client spam-filtering growth  has Sunbelt had in that 
time?  400%, so the amount of spam per recipient has reduced by about a 
third?  Or what improvements has Sunbelt made in its spam blocking 
technology in that time?  Improved from 7% to 97% detection, so 
_actual_ spam per recipient has dropped to about 20% of what it was?

When will journalists learn that bald statements like "we blocked twice 
as much spam as last month" don't actually mean anything _meaningful_ 
without getting often a great deal of extra information from the 
speaker.  Of course, that doesn't make for anything like as sexy a 
sound bite, and we all know that you're in the news _business_ rather 
than having education or enligthement as your objectives...

FWIW, I'll add my own observation on the recent, reputedly large, 
increase in spam.  I get a lot of spam but don't systematically count 
it.  My gut tells me that over the last few months my steady-state spam 
rate has gone from proabbaly around 80-85% of all received Email 
messages to probably about 85-90% _and_ that is coupled with a probably 
about 30% increase in total Email (because of my work, my non-spam 
Email tends to track up as spam does).


Regards,

Nick FitzGerald

_______________________________________________
Fun and Misc security discussion for OT posts.
https://linuxbox.org/cgi-bin/mailman/listinfo/funsec
Note: funsec is a public and open mailing list.