NuOnce Networks / Cobalt How To do it?

The mail delivery route on your Cobalt goes through three stages. The Mail Transfer Agent (MTA) is sendmail, and it handles connections between hosts. We dealt with the role sendmail plays in the spam-stopping effort in the previous two steps, using the access control database and using DNS block lists. The remaining two steps are the Mail Delivery Agent (MDA) which gets mail from sendmail to your users' inboxes, and the Mail User Agent (MUA) which is the POP or IMAP client used by the user to get mail from the server. For the anti-spam task, you have two options at this stage. The MDA can examine and discard mail it suspects is spam, or it can tag suspected spam and hand it along to the MUA. Most modern MUAs can then use these tags to discard or file junk mail unread.

Almost universally, the MDA is procmail, which features a very powerful filtering mechanism as well as (in my opinion) some of the most obscure documentation imaginable. There is a mailing list for Procmail as well, but they are very prickly about helping anyone who hasn't fully digested the previously-mentioned obscure documentation. Procmail is already installed on your Cobalt, and is probably already running; you just need to configure it to help. (If you're interested in learning more about why various procmail snippets quoted here work the way they do, start with Nancy McGough's procmail quick start.)

Fortunately, there are "canned" or pre-written filters which you can download and install which save you from re-inventing filters to solve problems like:

While procmail can filter attachments and redirect spam to /dev/null, the best use of procmail in the spam fight is tagging, which adds text to the headers and/or subject line of incoming spam which makes it by your block lists. Your users can use the filtering built in to their MUA on these tags, allowing them to filter spam into a junk box and periodically check that box for false positives.

Here I will describe implementation of one commonly used anti-spam filter on our Qube3, SpamAssassin, and a virus-signature-checking filter, the Sanitizer, and provide links to a number of other procmail-invoked packages which can be used for virus scanning or spam filtering.

Procmail checks a global configuration file, and then also uses per-user configuration files (if they exist) before delivery. We suggest trying out filters on your own account - edit the .procmailrc file in your home directory - before applying them to the /etc/procmailrc file, which is the global configuration.

Downloading and installing SpamAssassin

SpamAssassin is a perl module, with a number of dependencies. You will have to download the SpamAssassin package and the dependencies and install them. It can be done using CPAN from the shell, as outlined in the installation notes, but on Cobalts this has an unfortunate tendency to attempt to upgrade your Perl installation, which can lead to a wide variety of confusing problems. For example, using a newer version of Perl on my Qube3 rendered the LCD non-functional (which keeps one from properly shutting down the machine, among other things). To avoid this, here, we'll download an install the components individually. Fortunately this is relatively easy. Here I follow an outline posted to cobalt-users by Kim Schulz.

Note on newer versions of SpamAssassin: Versions of SpamAssassin newer than about 2.50 (2.55 is current as of this writing) require a newer version of Perl than is installed by default on Cobalts. To run them, you'll need to install an up to date Perl into an alternate path, like /usr/local/bin/perl. You can then install all the modules needed for SA, including SA itself, using "/usr/local/bin/perl Makefile.PL" instead of "perl Makefile.PL" as shown in this document.

Doing this with CPAN, however, is a prescription for screwing up your Cobalt unless you are remarkably careful and able to recover unexpected changes. Following the clue Steve Werby posted on the cobaltfacts list, we can walk through that here.

First you need to download the Perl source.

# wget http://www.cpan.org/src/stable.tar.gz

Undo this with "tar -xzvf stable.tar.gz" and cd into the new Perl directory created by the unpacking. Inside that directory, type this:

perl-5.8.0# ./configure.gnu --prefix=/usr/local

Once this is done, you should be able to go on with the install using "make" and "make install" as usual for perl. There's only one more step, once this is all done.

Your standard Cobalt perl install is still at /usr/bin/perl5.00503 and your new Perl 5.8 will be at /usr/local/bin/perl. However, our install has probably also put a symbolic link at /usr/bin/perl pointing to /usr/local/bin/perl! We need to fix that to avoid all kinds of ugliness with our Cobalt.

cd to /usr/bin and give these commands (don't panic):

bin# ls -l perl*  # Notice the new symlink
bin# rm perl
bin# ln -s perl5.00503 perl
bin# ls -l perl*

What you've done is remove the incorrect symlink and put in a correct one to the "correct" perl (verify with the ls -l perl* that the Cobalt one is at perl5.00503).

If you've really munged your perl, you'll have to re-install from the Cobalt RPM. Steve explained how to get started on the cobaltfacts list.

Now, proceed with your SpamAssassin installation, substituting "/usr/local/bin/perl" for "perl" when running Makefile.PL files for SA modules.

Alternately, Bob Noordam wrote a HOWTO for installing a parallel Perl on a Qube3.

First, SSH to your Cobalt and make yourself root. You should make a directory somewhere which will contain the install files; I used /usr/local/src/spamassassin/. cd to that directory, then:

First, you should check the SpamAssassin website and CPAN to make sure you have the most recent versions of these packages; don't use SpamAssassin 2.43 just because that's what I have in this example. As of 14 July, 2003, the most recent version is 2.55!

Second, In some cases, after the "perl ./Makefile.PL" step, you will be warned about yet another package which is missing. You can search CPAN for that module and then download and install it using the same steps above (wget, tar -xzvf, cd, perl ./Makefile.PL, make, make test, make install). In addition to the four modules listed above, I also needed File-Spec-0.82, HTML-Parser-3.26, and HTML-Tagset-3.03 in order to pass all tests on my Qube3. When I recently upgraded so SpamAssassin 2.52, the Makefile.PL suggested I upgrade ExtUtils::MakeMaker as well. You may need others.

Here are CPAN links to all the modules I've needed or found useful for my SpamAssassin (and Razor) installations. This list is probably not exhaustive, but it should give you an idea how to find any modules I haven't included here. These are up-to-date as of 15 July 2003; check the "Other Releases" drop-down for newer versions!

Third, sometimes there will be older versions of these modules already installed on your Cobalt. You can replace these with the newest, freshest version by replacing "make install" with "make install UNINST=1" (as the last line of the script above shows.) Watch the initial output of "make install" - it will suggest using the UNINST=1 flag if it's really needed.

Now that everything is installed, you can delete many of the files in your install directory (certainly all the .tar.gz files). Don't delete the /Mail-SpamAssassin-2.43 directory just yet.

You may find it useful at this point to check your configuration. The configuration file is at /etc/mail/spamassassin/local.cf; you can learn more about it by giving the command "perldoc Mail::SpamAssassin::Conf" at the shell prompt. You may want to start by setting your required_hits value to 8.0, more tolerant than the default 5.0.

"Verify (using a text viewer, ie. 'less' or 'notepad') that nonspam.out has not been tagged as spam, and that spam.out has. The files should contain the full text and headers of the messages, the "spam.out" message should be annotated with '****SPAM****' in the subject line and a report from SpamAssassin, and there should be no errors when you run the commands."

Now you have most of the useful files for SpamAssassin in place. The next step is to actually filter your mail with it.

Starting SpamAssassin filtering your mail

The SpamAssassin program is already installed on your system, thanks to "make install," but you need to have Procmail call it while delivering your mail for it to be useful. Start by going to your own directory. You should NOT be root at this point. If you don't already have a .procmailrc file, create one with some basic data:

Now, send yourself a few test emails. Better still, let this work on your mail for a day or two. Check the raw headers of the incoming mail to verify that SpamAssassin is tagging incoming mail; adjust the /etc/mail/spamassassin/local.cf to try to catch as much spam as possible without tagging "real" mail. Then come back to this.

Gotcha: Procmail versions newer than 3.14 will not "trust" your .procmailrc if either it or your home directory are world or group writable. Rob Prince notes that both files tend to be world writeable on RaQ4s. Do "ls -al" and see what is at the far left of the line for .procmailrc. It should say "-rw-r-----"; if it says "-rw-rw----" or "-rw-rw-rw-", you're going to need to change permissions.

should do the trick. Go up one directory (cd ..) and do the same check, "ls -al" and check the permissions. If your .procmailrc isn't working, procmail will log errors for every incoming message; check the /var/log/ directory for those logs.

Sitewide SpamAssassin

The simplest thing to do is to change the call from your local .procmailrc file and install it in your system /etc/procmailrc file, which is where global filtering on all mail is done. (Notice that there's no . in /etc/procmailrc.) You should change this a little, so you aren't running SpamAssassin as root, which might cause all kinds of unforseen difficulty:

If you get any kind of mail volume, though, you should look in to the spamc/spamd combo, which is written in C and runs as a daemon, therefore avoiding the need of spawning a new perl instance for every incoming mail message - in other words, running SpamAssassin faster and with less memory. spamc and spamd are already installed by the basic SpamAssassin install. You just need to get the spamd daemon running. You've still got the original /usr/local/src/Mail-SpamAssassin-2.43 directory, right?

Go there, then cd to "spamd". You'll see a slew of .sh shell scripts. These are meant to be copied to the /etc/rc.d/init.d directory, to be run automatically on startup. You want the RedHat script. Copy it to its new home like so:

You can use "restart" if you change the /etc/mail/spamassassin/local.cf file and need to reload the configuration, or "stop" if you just need to shut spamd down. Now, go back to editing your /etc/procmailrc file, and change "spamassassin" to "spamc":

Again, send yourself test mail to make sure this is working, but it should now be tagging mail for your whole site.

Most people should do just fine with the default options for SpamAssassin, but if you find yourself needing to tweak things, there is a site which will generate a configuration file for you.

You should keep an eye on the SpamAssassin site and upgrade to the newest version regularly, since spammers frequently use SpamAssassin to test their messages against! Consequently older versions are notably less effective than newer ones.

Downloading and Installing the Sanitizer

The Sanitizer "defangs" dangerous HTML and removes "dangerous" (i.e. executable, possibly viral) attachments. The Sanitizer comes with some canned procmail recipies for blocking common viruses like Code Red, BadTrans, and Klez, but its general rules often catch "new" viruses before a custom filter is needed.

Log in to your system and su - to root. cd to /etc/ and make a new directory, procmail/ which is where we will install the Sanitizer.

tar -xzvf procmail-sanitizer.tar.gz will undo the tarball. Most of the files you just unloaded are actually HTML documents which document The Sanitizer; the file named sanitizer-configuration.html details the installation of the software.

Copy the sample /etc/procmailrc here into the file you already created to run SpamAssassin. You may want to add a "MANGLE_EXTENSIONS=" line to override the default settings, which are very paranoid. We removed html, doc, xls, and vcf from the default list.

Also, the default settings expect a few files to already exist. First, you need a quarantine mailbox, /var/spool/mail/quarantine (you may want to comment out the per-user quarantine line unless you can set it up as a mailbox in your users' IMAP mail folder; if they use POP, don't bother.) You also need a log file for macro scanner scores and a procmail log...

You should also make sure the file /etc/procmail/local-email-security-policy.txt contains appropriate text for your site; that's what's going to people who send you viruses.

Other procmail-invoked filtering programs

We don't have detailed notes for each filter program, but we will list those we know, what they do, and where to get them.

Installing a procmail-based filter

Deal with viral attachments and tag spam for filtering

Downloading and installing SpamAssassin

Starting SpamAssassin filtering your mail

Sitewide SpamAssassin

Downloading and Installing the Sanitizer

Other procmail-invoked filtering programs