The first step in troubleshooting the mail systems should be sending an email to an internal account (ex: mark@control-alt-del.org) from an external source (gmail/hotmail/etc…). If you receive the mail within about 5-10 minutes, all is good. Otherwise, read on.

Service Troubleshooting

DNS Cache

Verify that the DNS cache is operational

svstat /service/dnscache

The service should be in an ‘up’ state.

dnsip www.google.ca

Restarting the DNS cache

svc -t /service/dnscache

Clam Anti-Virus

Check that clamd is running

svstat /service/clamav

Check to make sure virus definition updates work

/usr/bin/freshclam

Test scanning a file

clamdscan /usr/doc/Troubleshooting/sample-spam.txt

To restart clamd

svc -t /service/clamav

Make sure the virus definition update scheduled task exists

crontab -l | grep fresh

Network Time Protocol

Check to make sure ntpd is running

ntpq -p

The output should look like the example above.

Starting ntpd

ntpd

SpamAssassin

Check to make sure SpamAssassin is running

svstat /service/spamassassin

Testing SpamAssassin’s configuration file

spamassassin --lint

(shouldn’t output anything)

Testing automatic rule updates

sa-update --channelfile /etc/mail/spamassassin/sare-sa-update-channels.txt \
--gpgkey 856AA88A

(shouldn’t output anything)

Test spam filtering

spamc </usr/doc/Troubleshooting/sample-spam.txt

Take a look at the output to see if spamassassin is working properly. Should look like:

Restarting SpamAssassin

svc -t /service/spamassassin

Qmail - SMTP

Check to see if the service is running

qmailctl stat

The qmail-smtpd lines should be “up”

Check to see if tcpserver is responding properly

telnet localhost 25

If the first line isn’t “220 seaqmail01….” there’s a problem.

mount

The last line is the one that indicates that the ramdisk (tmpfs) is mounted properly.

Check to see if there is sufficient disk space for queuing mail

df -h

The ramdisk has 128 megabytes of space allocated for temporary file storage (files are stored there while the spam/virus scanner scans them).

Verify log files

tail -f /var/log/maillog

This log will indicate problems with SpamAssassin.

tail -f /var/log/qmail/smtpd/current | tai64nlocal

This is the SMTP daemon’s log. It includes results of spam scanning, recipient checks, virus checks, etc..

The status lines indicate the number of processes being used for active SMTP connections. These are currently limited to 120 simultaneous connections.

Qmail – Mail Transfer Agent

Check to see if the service is running

qmailctl stat

The qmail-send lines should be “up”. This also tells us the length of the message queues.

Check the logs

tail -f /var/log/qmail/current | tai64nlocal

The status lines indicate the number of processes allocated for mail delivery. This log should also contain any errors that may be occuring.

Check to see if there is sufficient disk space for queuing mail

df -h

The ramdisk has 128 megabytes of space allocated for temporary file storage (files are stored there while the spam/virus scanner scans them). If the mail system disk fills up, local queuing will no longer be possible (bad).

Recipient Filtering

If the problem is a newly created user not receiving external mail

The scheduled task that updates the recipient list is run every 6 hours and pulls the user list from active directory. If you need to update the database immediately, run:

/var/qmail/bin/updaterecipients.pl

Make sure recipient filtering scheduled task exists

crontab -l | grep updatere

Run the update script, check for errors

/var/qmail/bin/updaterecipients.pl

The update script goes to fairly reasonable lengths to avoid errors and should give some good clues if there is any errors along the way.

Configuration

Clam Anti-Virus

/etc/clamd.conf

LocalSocket /tmp/clamd FixStaleSocket MaxThreads 50 User spamd ExitOnOOM Foreground ScanPE DetectBrokenExecutables ScanOLE2 ScanMail ScanHTML ScanArchive ArchiveMaxFileSize 40M ArchiveMaxRecursion 10 ArchiveMaxFiles 2000 ArchiveMaxCompressionRatio 300 ArchiveBlockEncrypted ArchiveBlockMax

/etc/freshclam.conf

DatabaseOwner spamd DatabaseMirror db.us.clamav.net DatabaseMirror database.clamav.net Checks 12 NotifyClamd /etc/clamd.conf Foreground

/etc/clamav/run

This file is used by daemontools to start ClamAV. It should be executable and contain:
#!/bin/sh exec >/dev/null exec 2>&1 exec clamd

Files, Folders, Permissions

There must be a symbolic link in /service: clamav -> /etc/clamav/
The file /etc/clamav/run must have the executable bit set
/usr/share/clamav must be owned by the user/group spamd:spamd

SpamAssassin

/etc/mail/spamassassin/local.cf

This file contains several configuration directives for SpamAssassin. The noteworthy configuration directives are:

trusted_networks : defines networks for which we won’t do any SpamAssassin filtering
whitelist_from : defines envelope sender addresses or domains for which we won’t do any SpamAssassin filtering.

Example local.cf:
include trusted.cf include whitelist.cf lock_method flock dcc_path /usr/local/bin/dccproc use_razor2 1 razor_config /home/spamd/.razor/razor-agent.conf bayes_path /home/spamd/.spamassassin/bayes use_bayes 1 bayes_auto_learn 1

/etc/spamassassin/run

This file is used by daemontools to start the spamassassin service. It specifies the following variables:

Max children: the maximal number of processes to spawn simultaneously. A value of 40 is pretty much the highest the value should be.
Min children: The minimal number of children to keep around ready to accept new connections.
Max/Min spare: the number of children to keep around idle.

#!/bin/sh exec >/dev/null exec 2>&1 export TMPDIR=/tmp/mailscan exec spamd -u spamd -g spamd --max-children=40 --min-children=20 --min-spare=5 --max-spare=5 --max-conn-per-child=1000

Files, Folders, Permissions

There must be a symbolic link in /service: spamassassin -> /etc/spamassassin/
The file /etc/spamassasin/run must have the executable bit set
/tmp/mailscan must be owned by the user/group spamd:spamd, and it’s a mountpoint for a ramdisk.

Qmail

/var/qmail/control/smtproutes

This file controls artificial SMTP routes (routes that don’t use DNS MX records for mail delivery). All our domains have an artificial SMTP route to transfer the mail to the exchange server, so if we are accepting mail for a domain, it should have an entry in this file. The format is domain:NextHopIPAddress. Ex:

mydomain.com:1.2.3.4 mydmoain2.com:1.2.3.4

/var/qmail/control/rcpthosts

This configuration file defines which domains the SMTP server will accept mail for. The format of the file is one domain per line. It should contain all the domains in smtproutes. Ex:
mydomain.com mydmoain2.com

/var/qmail/control/concurrencyincoming

This controls the number of simultaneous incomming external SMTP sessions. Should the incomming sessions exceed this number, the connections are refused.

/var/qmail/control/concurrencyremote

This controls the number of simultaneous remote SMTP sessions the server will spawn. Currently set to 60.

/var/qmail/control/timeoutsmtpd

Number of seconds qmail-smtpd will wait for each new buffer of data from the remote SMTP client. Currently set to 60 seconds.

/var/qmail/control/validrcptto.cdb

This file contains the list of all the users that the server will accept mail for.

The file is in the CDB (constant database) format, and is rebuilt every 6 hours from the script /var/qmail/bin/updaterecipients.pl.

/var/qmail/control/badmailfrom

Unacceptable envelope sender addresses. qmail-smtpd will reject every recipient address for a message if the envelope sender address is listed in badmailfrom. A line in badmailfrom may be of the form @host, meaning every address at host. One entry per line.

/var/qmail/control/timeoutconnect

Number of seconds qmail-remote will wait for the remote SMTP server to accept a connection. Default: 60. The kernel normally imposes a 75-second upper limit.

/var/qmail/control/timeoutremote

Number of seconds qmail-remote will wait for each response from the remote SMTP server. Default: 1200.

/var/qmail/supervise/qmail-send/run

This file is used by daemontools to start the MTA. It should be executable and contain:
#!/bin/sh exec /var/qmail/rc

/var/qmail/supervise/qmail-smtpd/run

This file is used by daemontools to start the SMTP daemon. It should be executable and contain:
#!/bin/sh QMAILDUID=`id -u qmaild` NOFILESGID=`id -g qmaild` MAXSMTPD=`cat /var/qmail/control/concurrencyincoming` LOCAL=`head -1 /var/qmail/control/me` if [ -z "$QMAILDUID" -o -z "$NOFILESGID" -o -z "$MAXSMTPD" -o -z "$LOCAL" ]; then echo QMAILDUID, NOFILESGID, MAXSMTPD, or LOCAL is unset in echo /var/qmail/supervise/qmail-smtpd/run exit 1 fi if [ ! -f /var/qmail/control/rcpthosts ]; then echo "No /var/qmail/control/rcpthosts!" echo "Refusing to start SMTP listener because it'll create an open relay" exit 1 fi exec /usr/local/bin/softlimit -m 40000000 \ /usr/local/bin/tcpserver -v -R -l "$LOCAL" -x /etc/tcp.smtp.cdb -c "$MAXSMTPD" \ -u "$QMAILDUID" -g "$NOFILESGID" 0 smtp /var/qmail/bin/qmail-smtpd 2>&1

It should be noted that the SMTP daemon is started with a hard memory limit of 40 megabytes. Any messages which exceeds this size will be blocked.

/etc/tcp.smtp.cdb

This file is used by tcpserver and performs access control and sets environment variables which are used by the SMTP daemon. The file data is in CDB format, and is built from the text file /etc/tcp.smtp. The text format is NetworkMask:allow|deny,ENV=val
For example:
127.:allow,RELAYCLIENT="" 1.2.3.:allow,RELAYCLIENT="" 123.234.5.:deny :allow,QMAILQUEUE="/var/qmail/bin/simscan"

The above configuration file would allow from the networks 127.*, 1.2.3.* and deny from 123.234.5.*.

For any other source, the connection is also allowed and the QMAILQUEUE environment variable is set to “/var/qmail/bin/simscan”.

The QMAILQUEUE environment variable specifies an application which is run to place a mail message in the mail queue. Simscan is the content scanner which we use and ties into ClamAV and SpamAssassin.

The RELAYCLIENT environment variable is used by qmail-smtpd to determine if it will relay a mail message. By default, we only want to relay messages which are sent from our internal networks.
To build the cdb file, you need to run the command:

qmailctl cdb

Files, Folders, Permissions

There must be a symbolic link in /service: qmail-send -> /var/qmail/supervise/qmail-send
There must be a symbolic link in /service: qmail-smtpd -> /var/qmail/supervise/qmail-smtpd
Qmail is extermely finicky with file permissions because of it’s security model and privilege seperation. Refer to the qmail installation document if you somehow manage to mess them up.