The Future of Internet Immune Systems

[David Farber <dfarber () cs cmu edu>]

Begin forwarded message:

From: dewayne () warpspeed com (Dewayne Hendricks)
Date: November 20, 2007 6:56:29 PM EST
To: Dewayne-Net Technology List <xyzzy () warpspeed com>
Subject: [Dewayne-Net] The Future of Internet Immune Systems

The Future of Internet Immune Systems
Written by Cory Doctorow
11/19/2007
<http://www.internetevolution.com/author.asp?section_id=479&doc_id=139358&; 
>

Bunhill Cemetery is just down the road from my flat in London. It’s a  
handsome old boneyard, a former plague pit (“Bone hill” -- as in,  
there are so many bones under there that the ground is actually kind  
of humped up into a hill). There are plenty of luminaries buried there  
-- John “Pilgrim’s Progress” Bunyan, William Blake, Daniel Defoe, and  
assorted Cromwells. But my favorite tomb is that of Thomas Bayes, the  
18th-century statistician for whom Bayesian filtering is named.

Bayesian filtering is plenty useful. Here’s a simple example of how  
you might use a Bayesian filter. First, get a giant load of non-spam  
emails and feed them into a Bayesian program that counts how many  
times each word in their vocabulary appears, producing a statistical  
breakdown of the word-frequency in good emails.

Then, point the filter at a giant load of spam (if you’re having a  
hard time getting a hold of one, I have plenty to spare), and count  
the words in it. Now, for each new message that arrives in your inbox,  
have the filter count the relative word-frequencies and make a  
statistical prediction about whether the new message is spam or not  
(there are plenty of wrinkles in this formula, but this is the general  
idea).

The beauty of this approach is that you needn’t dream up “The Big  
Exhaustive List of Words and Phrases That Indicate a Message Is/Is Not  
Spam.” The filter naively calculates a statistical fingerprint for  
spam and not-spam, and checks the new messages against them.

This approach -- and similar ones -- are evolving into an immune  
system for the Internet, and like all immune systems, a little bit  
goes a long way, and too much makes you break out in hives.

ISPs are loading up their network centers with intrusion detection  
systems and tripwires that are supposed to stop attacks before they  
happen. For example, there’s the filter at the hotel I once stayed at  
in Jacksonville, Fla. Five minutes after I logged in, the network  
locked me out again. After an hour on the phone with tech support, it  
transpired that the network had noticed that the videogame I was  
playing systematically polled the other hosts on the network to check  
if they were running servers that I could join and play on. The  
network decided that this was a malicious port-scan and that it had  
better kick me off before I did anything naughty.

It only took five minutes for the software to lock me out, but it took  
well over an hour to find someone in tech support who understood what  
had happened and could reset the router so that I could get back online.

[snip]

-------------------------------------------
Archives: http://v2.listbox.com/member/archive/247/=now
RSS Feed: http://v2.listbox.com/member/archive/rss/247/
Powered by Listbox: http://www.listbox.com