Last update: 2007-02-12

// C O N T R O L - A L T - D E L . O R G

"Fate strikes down the strong man, Everyone weep with me " - O Fortuna
NAVIGATION>>
HOME PICS ABOUT NEWS CODE BOOKS MAIL
:: CODE
The three chief virtues of a programmer are: Laziness, Impatience and Hubris
-- Larry Wall
CONTACT
:. Qmail installation guide

This document explains how to install Qmail with the following features:


- Greylisting

- Adaptive blacklist

- SpamAssassin

- Simscan

- Recipient filtering

- Virus scanning (ClamAV)

- Administration tool for configuring whitelists, blacklists, relay control, etc...

 
:. Document conventions

Commands to run use the Courier New font and are highlighted in orange:

/path/to/a/command/to/run --options

 


File contents are always displayed within a text area field:

 

(sorry firefox folks, copy-paste button only works with IE)


Explanations use Arial – This text.

 


Replace all occurances of yourdomain.com with your domain name. Duh.

 
:. Assumptions

I'm going to assume you've already got a working Linux system.

I personally prefer Slackware Linux, and will eventually get around to posting the exact installation instructions used to setup all of the servers I manage. For now, you'll need:


- Working Linux server

- You need to be root to install these packages

- A public IP address that has reverse DNS setup

- Not currently running an MTA on the server (remove sendmail)

- Knowledge on how to compile applications


This document describes the installation from source of all the applications, so it'll probably work on all the *BSDs as well as Linux.


Note that you shouldn't try to just install the binary packages from your distribution and expect it to work with the instructions in this document (unless the distribution decides to adopt my way of implementing Qmail).


One last thing, this isn't really a setup that can be used stand-alone, as it's designed to rely on a backend server for actually handling the delivery to a mailbox (M$ exchange using Active Directory). I may eventually get around to schlepping together a version for supporting local/virtual users, vpopmail, ldap, etc...

 
:. Architectural considerations

Security

  • The Qmail SMTP server was selected because it was built with security in mind. The version we are using has been in a production state since March 1997, and no security vulnerabilities have been discovered in this software since then.
  • Each component of the system is independent of the others, and makes use of privilege separation and different unprivileged user accounts to accomplish the various tasks which need to be performed.
  • Should one component become compromised, it is unlikely to cascade to all systems or lead to a compromised system.
  • Separate functions are moved into mutually untrusting programs.
  • Qmail is one of the most used SMTP servers in the world (well over 1,000,000 deployments).
  • Linux has a fairly good track record for operating system security and can be hardened to meet military grade security if this is deemed necessary.
  • Enforced memory limits on the SMTP process limits the possibility of buffer overflows.

Speed

  • Filtering operations are layered from least resource intensive to most intensive. A large percentage of spam messages are filtered without needing resource intensive scanning.
  • Where possible, the system avoids reading from the hard drive (temporary files are written to a ramdisk). All scanning engines (virus, spam) are resident in memory.
  • Bounces should be avoided at all costs as they tie up resources needlessly.

Limiting the possibilities of false positives

  • SpamAssassin has a target false positive rate of less than 1/2500 (0.04%) on a corpus of 250,000 mail messages.
  • None of the filtering methods should have an impact on normal mail delivery.
  • There should be an easy way to control whitelists, as no filtering system is foolproof

Adaptive filtering

  • The system should be able to update itself and learn as new spam techniques evolve.

Easy to understand and extend

  • No monolithic application, each process should be small and well documented/understood
  • Extensive logging capabilities
  • Easy to monitor
  • Open source should be favored to avoid vendor lock-in
 

 

:. Features - Greylisting

Greylisting is a spam prevention method which is particularly good at stopping spam generated by zombie bots. It operates on a very simple principle.


Since SMTP is considered an unreliable transport, the possibility of temporary failures is built into the core spec (see RFC 821). As such, any well behaved message transfer agent (MTA) should attempt retries if given an appropriate temporary failure code for a delivery attempt.


The implementation of greylisting which is described in this guide was developped here: http://oss.albawaba.com/cqgreylist.html


How it works is simple:


- The first time a remote mail server or client connects to the server, we give a temporary failure error.

- For the next 60 seconds, we will continue to give this temporary failure should the server or client reconnect.

- After 60 seconds, any further attempts from that same IP will be allowed through


Since most spam software and zombies operate on a fire-and-forget mode of operation, this method is fairly effective at reducing the total volume of spam without impacting valid mail servers that regularly send e-mail to our systems.

 
:. Features - Adaptive blacklist

The adaptive blacklist is something I cooked up. The basic idea is that all spam filtering methods which are implemented should create a feedback loop.


The adaptive blacklist application operates as a multilog logfile post-processor. It analyzes patterns of spam/ham/viruses/invalid recipients and makes an assessment of the relative 'trustability' of an IP address over time.


The current implementation will keep track of all spams, hams, viruses and invalid recipient delivery attempts for an IP address for a 60 minute interval. If a remote IP has tried to deliver e-mail more than 10 times in this interval, and over 50% of these delivery attempts were spam/viruses/invalid recipients, I blacklist the IP for 4 hours.


Although not terribly effective for defending against things like one time spams, it will probably catch 'spam storms' and dictionairy attacks.

 
:. Features - Recipient validation

This system is meant to be integrated into a M$ Exchange environment, and integrates itself with Active Directory.


The recipient validation reduces the amount of bounce emails which are due to spam by verifying that the recipient is valid before accepting the mail for delivery.


To do so, I make use of a qmail patch which can be found here


At this layer, we also check to see if the sender is in our blacklists as implemented using Qmail's badmailfrom list.

 

 
:. Features - Spam/Virus scanning

Content scanning is the last line of defense in this system.


Once the mail has reached this layer, we've eliminated most of the obvious robot generated spam that is in 'fire and forget' and 'carpetbomb' mode.


Informal analysis on the systems I've worked with indicate that at least 80% of the e-mail at this point is still spam/viruses. Luckily, the other layers have already eliminated a large number of spam, so scanning what's left in depth isn't such a big deal.


This part of the system will accomplish the following:

- Scan the messages for viruses

- Perform various DCC (distributed checksum clearinghouse) checks

- Perform multiple realtime blackhole DNS lookups

- Perform rule-based analysis using regular expressions

- Lookup SPF (sender policy framework) DNS records for the envelope sender domain

- Lookup DomainKeys values

- Perform URI dns blacklist lookups on URI's contained in the message

- Use Bayesian analysis and automatic whitelists


If a virus is found, the messages is dropped outright. Otherwise, each of the above tests will be combined to create an overall spam score.


The scores are evaluated against two score thresholds; warning and spam. When a messages is below the warning, a message will be generated and delivered to the user along with the message. When the score is above the warning, it is considered spam and rejected during the SMTP conversation.

 
:. Installation - Daemontools

Download, untar, install daemontools


# cd /usr/src
# wget http://cr.yp.to/daemontools/daemontools-0.76.tar.gz
# tar xvfz daemontools-0.76.tar.gz
# mv admin daemontools-0.76
# wget http://www.linuxfromscratch.org/patches/downloads/daemontools/daemontools-0.76-errno-1.patch
# cd daemontools-0.76/daemontools-0.76/src
# patch -p2 <../../../daemontools-0.76-errno-1.patch
# cd ..
# package/install


Verify that svscan is running


# ps ax | grep sv


Should look something like this:


1

 

:. Installation - CDB

Download/Unpack/Install Package


# cd /usr/src
# wget http://cr.yp.to/cdb/cdb-0.75.tar.gz
# wget http://www.jm-associates.com/admin/downloads/cdb-0.75.errno.patch
# tar xvfz cdb-0.75.tar.gz
# cd cdb-0.75
# patch -p1 <../cdb-0.75.errno.patch
# make setup check

 

 

 
:. Installation - DJBDNS

Download/unpack/install package


# cd /usr/src
# wget http://cr.yp.to/djbdns/djbdns-1.05.tar.gz
# tar xvfz djbdns-1.05.tar.gz
# cd djbdns-1.05
# echo gcc -O2 -include /usr/include/errno.h > conf-cc
# make setup check

 

Create user to run caching DNS server

 

# useradd -s /bin/false -d /nonexistent dnscache

 

Configure dnscache

 

# dnscache-conf dnscache dnscache /etc/dnscache 127.0.0.1
# cd /etc/dnscache
# rm -rf log

 

Edit /etc/dnscache/run:

 



# chmod +x /etc/dnscache/run
# cd /service
# ln -s /etc/dnscache
# echo -e "search yourdomain.com\nnameserver 127.0.0.1\n" >/etc/resolv.conf

 

Test DNS

 

# dnsip www.google.com

 
:. Installation - ClamAV

Create a user to run the daemon process as

 

# groupadd spamd
# useradd -s /bin/false -d /home/spamd -g spamd spamd
# mkdir /home/spamd
# chown spamd:spamd /home/spamd

 

Download/unpack/install

 

# cd /usr/src
# wget http://superb-west.dl.sourceforge.net/sourceforge/clamav/clamav-0.88.6.tar.gz
# tar xvfz clamav-0.88.6.tar.gz
# cd clamav-0.88.6
# ./configure --sysconfdir=/etc --prefix=/usr --with-user=spamd --with-group=spamd
# make install

 

Edit the file /etc/clamd.conf:

 

 

Edit file /etc/freshclam.conf:

 

 

More configuration stuff

 

# mkdir /etc/clamav
# cd /etc/clamav

 

Edit /etc/clamav/run:

 

# chmod +x run
# cd /service
# ln -s /etc/clamav
# crontab -l >/tmp/crontab.tmp
# echo "0 */4 * * * /usr/bin/freshclam --quiet 2>&1 1> /dev/null" >>/tmp/crontab.tmp
# crontab /tmp/crontab.tmp
# /usr/bin/freshclam

 

Verify that everything is working properly

 

# svstat /service/clamav

 

Output should resemble this:

1

 

 

:. Installation - UCSPI-TCP

Download/unpack/install package

# cd /usr/src
# wget http://cr.yp.to/ucspi-tcp/ucspi-tcp-0.88.tar.gz
# tar xvfz ucspi-tcp-0.88.tar.gz
# cd ucspi-tcp-0.88
# patch -p1 <../cdb-0.75.errno.patch
# make setup check

 
:. Installation - Perl support modules + SpamAssassin

If prompted, follow all dependencies. These may need a bit of ‘massaging’. If you're not familiar with installing perl modules, this might seem a bit of a challenge.


When in doubt, if the install of a module fails, replace the 'install' command with 'force install'.


If you've never installed perl modules, you'll have to answer some questions before CPAN will allow you to install the modules below.


Good luck :)


# perl -MCPAN -e shell
cpan> install Bundle::CPAN
cpan> install Archive::Tar
cpan> install Time::HiRes
cpan> install LWP
cpan> install Net::LDAP
cpan> install Digest
cpan> install Digest::SHA1
cpan> install MIME::Base64
cpan> install Net::DNS
cpan> install Net::SMTP
cpan> install File::Spec
cpan> install MIME::Base64
cpan> install Test::Simple
cpan> install Test::Harness
cpan> install URI::Escape
cpan> install Module::Build
cpan> install Net::CIDR
cpan> install Sys::Hostname
cpan> install Getopt::Long
cpan> install File::Copy
cpan> install Test::YAML
cpan> install Text::Glob
cpan> install Text::EntityMap
cpan> install Time::TAI64
cpan> install IP::Country
cpan> install IP::Country::Fast
cpan> install IO::Zlib
cpan> install Convert::ASN1
cpan> install File::GlobMapper
cpan> install File::HomeDir
cpan> install HTML::Tagset
cpan> install Crypt::OpenSSL::Bignum
cpan> install HTML::Parser
cpan> install Mail::Util
cpan> install Mail::Field
cpan> install Mail::Cap
cpan> install Mail::DKIM
cpan> install Mail::DomainKeys
cpan> install Mail::SPF::Query
cpan> install Time::TAI64
cpan> install Mail::SpamAssassin
cpan> install Curses
cpan> install Curses::UI
cpan> install Mail::RFC822::Address
cpan> install Validate::Net
cpan> exit

 


Configure SpamAssassin

 


Edit /etc/mail/spamassassin/local.cf

 

 

Create some configuration files

 

# touch /etc/mail/spamassassin/trusted.cf
# touch /etc/mail/spamassassin/whitelist.cf

 

Create Ramdisk

 

# mkdir /tmp/mailscan
# chown spamd.spamd /tmp/mailscan

 

Figure out what the user spamd’s numerical userid and groupid is:

 

# id spamd

 

It should return output like this:

 

1

 

Edit the file /etc/fstab and add a line like this one:

 

 

Replacing the uid=XXXX,gid=XXX part with the numerical values returned by the ‘id’ command.

 

# mount -a

 

Start SpamAssassin

 

# mkdir /etc/spamassassin
# cd /etc/spamassassin

 

Edit file /etc/spamassassin/run:

 

 

# chmod +x /etc/spamassassin/run
# cd /service/spamassassin
# ln -s /etc/spamassassin

 

Check to make sure SpamAssassin is running

 

# svstat /service/spamassassin

 

1

 

Check to see if configuration is sane

 

# spamassassin --lint

 

(shouldn’t output anything if there are no errors)

 

Configure Automatic rule updates

 

# cd /tmp
# wget http://daryl.dostech.ca/sa-update/sare/GPG.KEY
# sa-update
# sa-update --import GPG.KEY
# crontab -l >/tmp/crontab.tmp
# echo -e "0 1  * * * sa-update --channelfile \
/etc/mail/spamassassin/sare-sa-update-channels.txt \
--gpgkey 856AA88A \
&& svc -t /service/spamassassin" >>/tmp/crontab.tmp
# crontab /tmp/crontab.tmp

 

Edit /etc/mail/spamassassin/sare-sa-update-channels.txt:

 

 

Installing Vipul's Razor2

 

Download/unpack/install

 

# cd /usr/src
# wget 'http://prdownloads.sourceforge.net/razor/razor-agents-2.82.tar.bz2?download'
# tar xvfj razor-agents-2.82.tar.bz2
# cd razor-agents-2.82
# perl Makefile.PL
# make
# make test
# make install

 

Configure Razor2

 

# razor-admin -home=/home/spamd/.razor -register
# razor-admin -home=/home/spamd/.razor -create
# razor-admin -home=/home/spamd/.razor –discover
# chown -R spamd.spamd /home/spamd

 

Edit file /home/spamd/.razor/razor-agent.conf :

 

 

NOTE: make sure that firewall rules allow outbound TCP connections on port 2703

 

Installing DCC

 

Download/unpack/install package

 

# cd /usr/src
# wget http://www.rhyolite.com/anti-spam/dcc/source/dcc.tar.Z
# tar xvfz dcc.tar.Z
# cd dcc-1.3.42/
# ./configure
# make install

 

Configure DCC

 

NOTE: Make sure that the firewall allows inbound and outbound UDP port 6277 (src port >1023 dst port 6277 from the box ip to the internet and from the internet src port any to dst port 6277 on the box’s ip.

 

 
:. Installation - NetQmail

Download/unpack/patch/install package


# cd /usr/src
# wget http://www.qmail.org/netqmail-1.05.tar.gz
# tar xvfz netqmail-1.05.tar.gz
# cd netqmail-1.05
# ./collate.sh
# wget http://qmail.jms1.net/patches/netqmail-1.05-validrcptto.cdb.patch
# cd netqmail-1.05
# patch -p1 <../netqmail-1.05-validrcptto.cdb.patch
# groupadd nofiles
# useradd -g nofiles -d /var/qmail/alias alias
# useradd -g nofiles -d /var/qmail qmaild
# useradd -g nofiles -d /var/qmail qmaill
# useradd -g nofiles -d /var/qmail qmailp
# groupadd qmail
# useradd -g qmail -d /var/qmail qmailq
# useradd -g qmail -d /var/qmail qmailr
# useradd -g qmail -d /var/qmail qmails
# make setup check
# ./config-fast yourhost.yourdomain.com

 


Configure Qmail/Startup Scripts

 


Edit /var/qmail/control/rcpthosts:

 

(This file should contain the full list of all the domains you want to receive mail for)

 

Edit /var/qmail/rc:

 

 

# chmod 755 /var/qmail/rc
# mkdir /var/log/qmail
# echo ./Maildir/ >/var/qmail/control/defaultdelivery

 


Edit /usr/bin/qmailctl:

 

 

# chmod 755 /var/bin/qmailctl
# mkdir -p /var/qmail/supervise/qmail-send/log
# mkdir -p /var/qmail/supervise/qmail-smtpd/log

 

Edit /var/qmail/supervise/qmail-send/run:

 

 

Edit /var/qmail/supervise/qmail-send/log/run:

 

 

Edit /var/qmail/supervise/qmail-smtpd/run:

 

 

Edit /var/qmail/supervise/qmail-smtpd/log/run:

 

 

Edit /etc/tcp.smtp:

 

 

Note: In /etc/tcp.smtp you should list all the networks you want to allow relaying from. For example, if your internal network is 1.2.3.0/24, you'd add the following line:

1.2.3.:allow,RELAYCLIENT=""

 

Edit /var/qmail/bin/adaptive_blacklist.pl:

 

 

More commands to run…

 

# chmod +x /var/qmail/bin/adaptive_blacklist.pl
# chmod 755 /var/qmail/supervise/qmail-send/run
# chmod 755 /var/qmail/supervise/qmail-send/log/run
# chmod 755 /var/qmail/supervise/qmail-smtpd/run
# chmod 755 /var/qmail/supervise/qmail-smtpd/log/run
# echo 120 >/var/qmail/control/concurrencyincoming
# echo 60 >/var/qmail/control/concurrencyremote
# mkdir -p /var/log/qmail/smtpd
# chown qmaill /var/log/qmail /var/log/qmail/smtpd
# cd /service
# ln -s /var/qmail/supervise/qmail-send
# ln -s /var/qmail/supervise/qmail-smtpd
# qmailctl stop
# qmailctl cdb

 

Edit /var/qmail/control/smtproutes:

 

 

Note: /var/qmail/control/smtproutes contains the should contain the full list of all the domains you want to receive mail for, and the IP address of the server to route e-mail received for that domain to. In the above example, mail received for yourdomain.com would be sent to the server 1.2.3.5

 

# crontab -l >/tmp/crontab.tmp
# echo -e "* * * * * /usr/bin/qmailctl cdb\n" >>/tmp/crontab.tmp
# crontab /tmp/crontab.tmp

 
:. Installation - RipMIME

Download/unpack/install package


# cd /usr/src
# wget http://www.pldaniels.com/ripmime/ripmime-1.4.0.6.tar.gz
# tar xvfz ripmime-1.4.0.6.tar.gz
# cd ripmime-1.4.0.6
# make; make install

 
:. Installation - CQGreylist

Download/unpack/install package


# cd /usr/src
# wget http://oss.albawaba.com/files/cqgreylist-0.2.tar.gz
# tar xvfz cqgreylist-0.2.tar.gz
# cd cqgreylist-0.2
# make
# cp cqgreylist /var/qmail/bin
# mkdir /var/qmail/cqgreylist
# chown qmaild: /var/qmail/cqgreylist
# crontab -l >/tmp/crontab.tmp
# echo -e "0 0 * * * rm -rf /var/qmail/cqgreylist/*\n" \
>>/tmp/crontab.tmp
# crontab /tmp/crontab.tmp

 
:. Installation - Simscan

Download/unpack/install package

# cd /usr/src
# wget http://www.inter7.com/simscan/simscan-1.3.1.tar.gz
# tar xvfz simscan-1.3.1.tar.gz
# cd simscan-1.3.1
# ./configure --enable-user=spamd --enable-group=spamd \
--enable-clamav=y --enable-clamdscan=/usr/bin/clamdscan \
--enable-spam=y --enable-spam-passthru=n --enable-spamc=/usr/bin/spamc \
--enable-spamc-user=n --enable-spam-hits=10 \
--enable-ripmime=/usr/local/bin/ripmime \
--enable-workdir=/tmp/mailscan
# make; make install

 
:. Installation - Recipient Checking

Edit /var/qmail/bin/updaterecipients.pl:

# chmod +x /var/qmail/bin/updaterecipients.pl

 

Edit /var/qmail/bin/getadsmtp.pl:

 

 

You'll want to edit this script. Near the beginning, you'll need to plug in your domain controller's IP address, your domain name, and some domain user credentials. Any valid user should work, so you should create a user for this purpose.

 

# crontab -l >/tmp/crontab.tmp
# echo -e "0 */4 * * * /var/qmail/bin/updaterecipients.pl 2>&1 1> /dev/null" >>/tmp/crontab.tmp
# crontab /tmp/crontab.tmp
# chmod +x /var/qmail/bin/getadsmtp.pl
# /var/qmail/bin/updaterecipients.pl
# qmailctl start

 

Edit /usr/bin/QmailAdmin:

 

 

# chmod +x /usr/bin/QmailAdmin

 

And here's where things get a bit tricky. QmailAdmin has a few variables that you'll need to update for it to work properly.

 

First off, we're going to add a safeguard to the admin utility. Search for the line :

 

my $defaultregex = '^(?:127\.|1\.2\.3\.|:).*$';

 

This defines a regular expression that we won't allow people who use QmailAdmin to modify. It safeguards someone accidentally turning your mail server into an open relay (or closed relay). This part of the code modifies the file /etc/tcp.smtp which is what defines relay control using tcprules.


If you followed these instructions, your /etc/tcp.smtp file should look like:

 

127.:allow,RELAYCLIENT=""

1.2.3.:allow,RELAYCLIENT=""
:allow,QMAILQUEUE="/var/qmail/bin/simscan"

 

In this example, we're assuming that you are operating from the 1.2.3.0/24 network and want to allow any host on this network to relay mail through your mail server.

 

Notice the red text above, I hope you see the connection. As a second example, suppose you wanted to add another network (2.2.2.0/24), to the list of trusted networks and wanted to make sure nobody could delete it with QmailAdmin, the default regex line would look like:

 

my $defaultregex = '^(?:127\.|1\.2\.3\.|2\.2\.2\.|:).*$';

 

And your /etc/tcp.smtp would look like:

 

127.:allow,RELAYCLIENT=""

1.2.3.:allow,RELAYCLIENT=""
2.2.2.:allow,RELAYCLIENT=""

:allow,QMAILQUEUE="/var/qmail/bin/simscan"

 

 
:. Closing thoughts

At this point, if you followed these instructions dilligently (and I documented the installation process correctly) you should have a working front-end that you can sit in front of a M$ Exchange server.


I'll leave the configuration necessary in Exchange to do this as an exercise to the reader (hell, if you use M$ and don't know how to do this, call M$ and get them to explain, that's what you pay them for).


Sit back, relax, and enjoy the spam you aren't receiving in your Inbox. With a little tweaking I'm sure this could be adapted to work as a front-end to a Unix system, or could be tweaked to actually deliver mail locally.


Feel free to send any questions may way, or god forbid you should find a typo or mistake/omission.

© copyright 2007 Mark Steele