Fighting Spam

Free Speech And Spam Free

Spam is not just a nusance. It's become a threat to internet freedom in general, because congressional legislative action is looming. The sollution is to have your internet service provider (ISP) install filtering technology, or find an ISP that already has such. Read on for more information, and how you might proceed.

Internet freedom is at stake - specifically, independent news sites on the internet are the only alternative to mainstream media. These independent news sites are vitally important to not only the letter of our free speech rights, but more specifically, the spirit of our free speech rights - because free speech is useless without a voice, and when ALL mainstream media (TV, radio, newspapers, magazines) ownership is centralized in 6-8 large corporations, the internet is that voice.

Folks are so fed up with spam, they're asking congress to legislate the issue. This is likely to result in nothing more than free speech intrusions, because while congress can do things that'll result in internet restrictiveness, they cannot control spam coming from outside the United States, and this is exactly how spammers will get around any legislation congress comes up with.

The good news is, there is really, and truly no need for anti-spam legislation, because there are plenty of anti-spam filtering programs for every Internet Service Provider (ISP) type through which you might want to connect to the internet. ISPs can provide such services (and many already do) without any intervention by you, the end user. For your convenience, they can also provide interfaces you can access with your internet browser, through which you can configure filtering strength, so you don't have to worry about accidently filtering mail you actually want.

Demand that your ISP provide spam filtering services, including a filtering configuration interface. If they won't, there are plenty of ISPs already providing such so you don't have to settle for less. If you are an ISP, installing filtering software is easy, and with so much at stake, there's no reason not to :).

If your ISP does not provide filtering, ask them to do so, and direct them to spam filtering program information - for instance, this page. Most ISPs are running either a Unix operating system type, or a Windows operating system. For information on free spam filtering programs for the Windows ISP operating system:

Web Attack's Free Windows ISP Anti-Spam Programs

, and below are descriptions of three Unix spam filtering programs.

If you want to complain about spam, you are not alone. But don't complain to the government, because you'll get an ineffecitve response resulting only in liberty limitations, and they'll charge you for the privilidge with higher taxes in the bargain. No. Complain to your ISP, and if they won't listen, then take you business elsewhere:

spam.abuse.net maintains a list of anti-spam ISPs.

There's no reason your ISP should not provide some sort of spam filtering. Not doing so is a threat to internet freedom.

Finally, if going the ISP filtering route simply does not work in your situation, then you can install spam filtering software on your PC at home, if only as a tempory measure. This is slightly less convenient and inefficient, though it will solve your spam problem, and it does have the advantage of increased flexibility, and control. There's more information, and links below, to personal anti-spam software for Windows, and various email programs.

Technical Considerations - What You'll Need To Know

Now, don't be confused by the details here. If you run in to anything that seems contradictory to what you thought you knew, chances are that what you thought you knew, you indeed did know, and there's no actual contradiction :).

The most convenient approach to the filtering software question is to leave it up to your ISP, or moving to an ISP that solves the problem, if your present ISP does not. Failing either of these, you can take matters into your own hands and install filtering software on your PC at home. In any case, if you need to take a more active role in getting spam filtering software in place whether at your ISP or on your home PC, then the following tips, information armament, will help.

Two important considerations with regards to spam filtering software involve, first, where your filtering software will be installed - at your ISP's mail server or your PC, and second, operating system - basically, are you/your ISP running Windows, or Unix?. And a third bit - if you choose to install filtering at home, your filtering software needs to be compatible with your email software, though SpamBuster, listed below, is a unique exception as it works independently of your particular email program, and it runs on most Windows versions.

Filitering Location

1. At the ISP

This method has a couple of advantages.

One, you don't have to do anything, as your ISP takes care of the filtering - though ideally, your ISP should provide an easy means by which you can configure the filtering if you want to.

Two, when filtering software is installed on your home PC, you still end up downloading spam from your ISP connection, before your PC installed filter removes it from you inbox. This wastes time, and computer resources. Installation at the ISP avoids this problem - no downloading spam from the internet ISP to your home computer, and no filtering processing on your computer. It's all done at the ISP before it even gets to your mailbox.

2. At your PC

This is a good alternative if your ISP won't install filtering software, and you are not able to switch to an ISP that will, or as a tempory measure as you persue an ISP sollution.

The main disadvantage is that, as described above, you must download any spam from your ISP before your filter gets at it. Though the filter will do the most important job - that of preventing you from seeing or having to delete any spam yourself.

And installing at home is not without advantages. It gives you more flexibility - for instance you get to choose exactly which filter software you're going to use; you get direct, and thus perhaps greater control over the filtering configuration options and such; etcetera.

If you opt for the home filtering installation route, in addition to making sure your filtering software is compatible with your operating system as discussed below, you also need to make sure your filtering software is compatible with your email program (note this is not a consideration if your ISP does the filtering) - be it, for instance, Netscape Mail, Eudora, or Microsoft Outlook. Here are some links to home filtering software relative to various popular email programs (a growing list):

Operating System

This is a simple issue. Once you decide which location is best for you, you'll need to make sure you get a filtering program that's compatible with your's or your ISP's operating system.

Most home PCs are running Windows, and an outfit called Deersoft has home filtering software that works with MS Outlook. The Deersoft filtering software is based on the popular (the one I use as well) open source Unix ISP filtering software, SpamAssassin (described below).

Most ISPs are running either a version of Windows, or some flavor of Unix. The link above points to Windows ISP filtering software, and below are descriptions, and links to home pages of three very good Unix type ISP filtering programs.

Feel free to ask me, if you have any questions: bhoover@wecs.com, I'd be happy to help.

Read on for information on Unix spam filtering systems, and more importantly, additional insight into spam filtering technology concepts in general.

Unix Spam Filtering

I was fed up with spam - had been for several months.  It was just getting out of hand.  There was so much of it (about 50/day) that my good mail was getting lost in it.  But not anymore!

I searched Google and easily found links to several anti-spam software tools.  The tools I list here, are ones I installed, and tried on my Unix web site account.  Only one of them (SpamAssassin) is compatible (potentially) with a non-Unix type ISP system such as Windows NT, etc, about which, in the present context, I know nothing, as my only web site, and such experience is Unix related.  To use any of the programs listed here, it is assumed that you have a server account that allows you more than simple web browser access, and configuration - shell account access, and such as ftp, and telnet for instance.

Tools Summary (Anti-Spam State Of The Art)

The tools here, are, in a word, Cool!  They block spam (of course) through artificial intelligence like algorithms that analyze incoming mail - text, header information, etc. - for likely spam characteristics.  They can confer with live, on-line spam database services, querying them for known spamers, and known spam IP characteristics.  They can report spam to system administrators.  In case SpamBouncer mistakes an innocent mail for spam, it can be configured to notify a suspected spamer that their email was blocked, giving them a password with which to by-pass SpamBouncer (you can then add them to your list of allowed senders so as to avoid future problems).  They provide configuration of "white lists," and "black lists," of email addresses to always block or allow.  SpamAssassin even supports Vipul's Razor, a very exciting concept and implementation in spam filtering that works with on-line databases to stop spam in its tracks, even as a spamer's mass mailing send program is in progress!  Though SpamAssassin directly/internally supports razor, razor can be used independent of a given mail/spam filtering program.

The Anti-Spam Systems

All systems listed here are extendible, and configurable.  I give a cursory "review", and features list, but don't take my word for it - follow the system's associated link, and read what's there.

SpamBouncer - Consists of a collection of procmail recipes accessed through inclusion in user's procmailrc.  After trying JunkFilter (because I had some trouble downloading SB initially), I installed this one without any problems (just give a careful read to its very easy to follow, and straight forward documentation).  It blocked all spam to my domain over the course of several days, after which I got SpamAssassin (below link) installed.  SpamAssassin let 4 spam through (needs configuration tweaking to make more strict) over the course of about 8 hours, so I went back to SpamBouncer - if it aint broke...  I continue to use SpamBouncer - requires a Unix account, with procmail access to use.

Some will find  SpamBouncer's procmailrc configuration parameters set-up somewhat daunting - there are about 10 parameters for procmail, and 15 or so for SB itself - but SB comes with a sample procmailrc you can use or cut and paste from, and all SB configuration parameters are very well described on the SB documentation page (follow the link above).  In fact, of the three programs listed here, I found SB easiest to install, and that's probably because its documentation is so good - complete, and easy to follow.

Features

White/Black Listing

On-line spam database checking

Spamer reporting to system administrators

Fake "address not found" spam bounce response

Optional alternate mail folder for bulk (mailing lists) mail

Log file of procmail processing

Differentiates between suspected, and definite spam

Notifies suspected spamer in case of mistakes

Reports spamers to system administrators

Mail back-up files

SpamAssassin - Consists of Perl modules.  This one is the most sophisticated.  It's a concerted effort, with lots of people working on it and maintaining it, so it promises to just keep getting better.  It is modularly designed, flexible, and extendible.  It apparently requires some configuration tweaking in order to make it more strict about what it allows into your mailbox, but this is part of its flexibility - you can change the weights assigned to a given filtering heuristic.

I've made increasing use of SA's ecclectic filtering techniques since I started using it in May '02. Since installing, I've installed Razor - a distributed spam checksum database, which SA makes use of if present. As of 11/03, I manually trained - an on-going process if you like, which can be sort of fun - and turned on SA's Bayes implemention, and installed for SA's use, the Distributed Checksum Clearinghouse network client. I'm feeling pretty good about spam - that is, the lack of it. I'd say, if SA can't stop it (spam), it's unlikely any filter can.

SA is easy to install, and if possible, uses basic network spam clearing house checks by default (I had trouble because of a syntax error in my procmailrc, and because I forgot the Unix command line requires ./ to precede commands) - just follow the directions.  Though the documentation is terse, leaving you feeling sort of in the dark about things.  Again, follow (and trust) the instructions - there are few - and you shouldn't have any problems. And while SA works fine out of the box, after maybe a little tweeking of its spam scoring threshold, making use of Bayes, Razor, and DCC requires only a bit more very worthwhile effort for each.

Features

Adjustible filtering sensitivity

Bayes probablity spam learning algorithm with manual, or automatic training/learning

White/Black Listing

Network spam database support - Razor, Pyzor, DCC, and others!

Reports spamers to system administrators (optionally)

Manual or auto network checksum database reporting, checking

Modular design

User extendible filtering rules

Actively maintained (very)

Supports multiple platforms

JunkFilter - Another set of procmail recipes.  This one was easy to install.  It continues to be maintained.  It did not work quite as well as SpamBouncer, letting a few spam slip through over the course of a couple days.

Features

White/Black Listing

Actively maintained

Epilogue

Once you've got spam filtering going, if you opt to save spam to a spam folder instead of deleting outright, you may find it handy to set up some sort of auto archiving system to help conserve disk space. For Procmail users, I've got a set of easy to install procmail routines that do the job nicely - ftp://wecs.com/procmail/archcron-fifo.zip.

This version's better than the first one I cooked up for this problem, because archiving is done with a fifo approach - first in, first out - with a configurable maximum number of archive records, and archives.

The two included top level drivers .caughtarchcronrc, and .bayesarchcronerc - two separate demonstration instantiations - demonstrate how to use the routines. Use these as an example to set up your own driver which may be called from your .procmailrc file with something like:

INCLUDERC=${HOME}/.archcron-stack/.caughtarchcronerc

For archiving spam, the above would assume your .procmailrc spam filtering routine call returned true for spam, and the spam was delivered to spam folder, caughtspam in your home directory (the demo driver assumes $HOME/caughtspam), and that the auto-archiving routines reside in /.archcron, off of your home directory.

See the .caughtarchcronerc demo driver for detailed explaination of what must be set/defined in order to create your own custom auto-archiving driver.

Oh, in case you're wondering why you'd want to archive your spam... For one, you might want to check through it at some point in case of a false positive - spam filtering routines mistakenly marking good mail for spam.

More importantly, a good spam database can be used to train a Baysian filtering algorithm such as that integrated into SpamAssassin. SpamAssassin allows for auto training its Bayes implementation, but I prefer to do it manually, and keeping a good spam database around in case I need to retrain, or just to keep the Bayes spam token database up to date with current spammer techniques, is a good idea.

With 10,000 spams over 10 archives, my auto-archiving routines provide me with a good spam corpus over a period of about the last 20 days or so.

Update Notes

SpamBouncer

It should be noted that SB is lean, and mean, both in terms of memory, and speed, due to its procmail recipe implementation.

SB's support/maintenance is not quite as non-existent as first presumed.  Author Catherine Hampton just released an update.  It includes support for a truck load of live, on-line, spam/abuse databases.  And there appears to be plans to integrate SB with Michel Bouissou's parallel SB development (site in french) additions.

SpamAssassin

I've gotta say, I really like using this program.  I guess I'm just kind of excited by its sophistication, though I'm not really sure saying so is a fair cut relative to SB.  I mean, SB does the job very well, so why go further?  In fact, who says SA goes further?  Am I asking more questions than answering?

It may be partly because it's written in Perl, which means it's portable.  And it's modular.  Both these means SA's got the potential to go places, if not do things, SB does not.  SA project manager, Craig Hughes' partnering to bring SA's technology mainstream, starting with SA, MS Outlook support.

Disclaimer

Keep in mind that the information here is not intended to be particularly authoratative - neither with regards to completness, nor accuracy.  I'm no software critic.  I'm a programmer.  This page is not intended to sway your opinion about anything.  Rather, presented here are simply my own random thoughts on a subject in which I became interested.

In other words, don't read this page, and think, "Mm, Bryan thinks I ought to use..."  If you do, you miss the point.  On this page, I'm just posting my thoughts.  Make your own decisions :).

Copyright (C) 2002, Bryan Hoover, Warren Excellence Computing Systems