Converting Mbox to Maildir

MBox is the original and ancient format for storing mail on Unix systems, it consists of a single file per user under /var/spool/mail that has messages concatenated. Obviously performance is very poor when deleting messages from a large mail store as the entire file has to be rewritten. Maildir was invented for Qmail by Dan Bernstein and has a single message per file giving fast deletes among other performance benefits. An ongoing issue over the last 20 years has been converting Mbox systems to Maildir. The various ways of getting IMAP to work with Mbox only made this more complex.

The Dovecot Wiki has a good page about converting Mbox to Maildir [1]. If you want to keep the same message UIDs and the same path separation characters then it will be a complex task. But if you just want to copy a small number of Mbox accounts to an existing server then it’s a bit simpler.

Dovecot has a mb2md.pl script to convert folders [2].

cd /var/spool/mail
mkdir -p /mailstore/example.com
for U in * ; do
  ~/mb2md.pl -s $(pwd)/$U -d /mailstore/example.com/$U
done

To convert the inboxes shell code like the above is needed. If the users don’t have IMAP folders (EG they are just POP users or use local Unix MUAs) then that’s all you need to do.

cd /home
for DIR in */mail ; do
  U=$(echo $DIR| cut -f1 -d/)
  cd /home/$DIR
  for FOLDER in * ; do
    ~/mb2md.pl -s $(pwd)/$FOLDER -d /mailstore/example.com/$U/.$FOLDER
  done
  cp .subscriptions /mailstore/example.com/$U/ subscriptions
done

Some shell code like the above will convert the IMAP folders to Maildir format. The end result is that the users will have to download all the mail again as their MUA will think that every message had been deleted and replaced. But as all servers with significant amounts of mail or important mail were probably converted to Maildir a decade ago this shouldn’t be a problem.

Mail Server Training

Today I ran a hands-on training session on configuring a MTA with Postfix and Dovecot for LUV. I gave each student a virtual machine running Debian/Jessie with full Internet access and instructions on how to configure it as a basic mail server. Here is a slightly modified set of instructions that anyone can do on their own system.

Today I learned that documentation that includes passwords on a command-line should have quotes around the password, one student used a semi-colon character in his password which caused some confusion (it’s the command separator character in BASH). I also discovered that trying to just tell users which virtual server to login to is prone to errors, in future I’ll print out a list of user-names and passwords for virtual servers and tear off one for each student so there’s no possibility of 2 users logging in to the same system.

I gave each student a sub-domain of unixapropos.com (a zone that I use for various random sysadmin type things). I have changed the instructions to use example.com which is the official address for testing things (or you could use any zone that you use). The test VMs that I setup had a user named “auser”, the documentation assumes this account name. You could change “auser” to something else if you wish.

Below are all the instructions for anyone who wants to try it at home or setup virtual machines and run their own training session.

Basic MTA Configuration

  1. Run “apt-get install postfix” to install Postfix, select “Internet Site” for the type of mail configuration and enter the domain name you selected for the mail name.
  2. The main Postfix configuration file is /etc/postfix/main.cf. Change the myhostname setting to the fully qualified name of the system, something like mta.example.com.
    You can edit /etc/postfix/main.cf with vi (or any other editor) or use the postconf command to change it, eg “postconf -e myhostname=mta.example.com“.
  3. Add “home_mailbox=Maildir/” to the Postfix configuration to make it deliver to a Maildir spool in the user’s home directory.
  4. Restart Postfix to apply the changes.
  5. Run “apt-get install swaks libnet-ssleay-perl” to install swaks (a SMTP test tool).
  6. Test delivery by running the command “swaks -f auser@example.com -t auser@example.com -s localhost“. Note that swaks displays the SMTP data so you can see exactly what happens and if something goes wrong you will see everything about the error.
  7. Inspect /var/log/mail.log to see the messages about the delivery. View the message which is in ~auser/Maildir/new.
  8. When other students get to this stage run the same swaks command but with the -t changed to the address in their domain, check the mail.log to see that the messages were transferred and view the mail with less to see the received lines. If you do this on your own specify a recipient address that’s a regular email address of yours (EG a Gmail account).

Basic Pop/IMAP Configuration

  1. Run “apt-get install dovecot-pop3d dovecot-imapd” to install Dovecot POP and IMAP servers.
    Run “netstat -tln” to see the ports that have daemons listening on them, observe that ports 110 and 143 are in use.
  2. Edit /etc/dovecot/conf.d/10-mail.conf and change mail_location to “maildir:~/Maildir“. Then restart Dovecot.
  3. Run the command “nc localhost 110” to connect to POP, then run the following commands to get capabilities, login, and retrieve mail:
    user auser
    pass WHATEVERYOUMADEIT
    capa
    list
    retr 1
    quit
  4. Run the command “nc localhost 143” to connect to IMAP, then run the following commands to list capabilities, login, and logout:
    a capability
    b login auser WHATEVERYOUMADEIT
    c logout
  5. For the above commands make note of the capabilities, we will refer to that later.

Now you have a basically functional mail server on the Internet!

POP/IMAP Over SSL

To avoid password sniffing we need to use SSL. To do it properly requires obtaining a signed key for a DNS address but we can do the technical work with the “snakeoil” certificate that is generated by Debian.

  1. Edit /etc/dovecot/conf.d/10-ssl.conf and change “ssl = no” to “ssl = required“. Then add the following 2 lines:
    ssl_cert = </etc/ssl/certs/ssl-cert-snakeoil.pem
    ssl_key = </etc/ssl/private/ssl-cert-snakeoil.key
    1. Run “netstat -tln” and note that ports 993 and 995 are not in use.
    2. Edit /etc/dovecot/conf.d/10-master.conf and uncomment the following lines:
      port = 993
      ssl = yes
      port = 995
      ssl = yes
    3. Restart Dovecot, run “netstat -tln” and note that ports 993 and 995 are in use.
  2. Run “nc localhost 110” and “nc localhost 143” as before, note that the capabilities have changed to include STLS/STARTTLS respectively.
  3. Run “gnutls-cli --tofu 127.0.0.1 -p 993” to connect to the server via IMAPS and “gnutls-cli --tofu 127.0.0.1 -p 995” to connect via POP3S. The --tofu option means to “Trust On First Use”, it stores the public key in ~/.gnutls and checks it the next time you connect. This allows you to safely use a “snakeoil” certificate if all apps can securely get a copy of the key.

Postfix SSL

  1. Edit /etc/postfix/main.cf and add the following 4 lines:
    smtpd_tls_received_header = yes
    smtpd_tls_loglevel = 1
    smtp_tls_loglevel = 1
    smtp_tls_security_level = may

    Then restart Postfix. This makes Postfix log TLS summary messages to syslog and in the Received header. It also permits Postfix to send with TLS.
  2. Run “nc localhost 25” to connect to your SMTP port and then enter the following commands:
    ehlo test
    quit

    Note that the response to the EHLO command includes 250-STARTTLS, this is because Postfix was configured with the Snakeoil certificate by default.
  3. Run “gnutls-cli --tofu 127.0.0.1 -p 25 -s” and enter the following commands:
    ehlo test
    starttls
    ^D

    After the CTRL-D gnutls-cli will establish a SSL connection.
  4. Run “swaks -tls -f auser@example.com -t auser@example.com -s localhost” to send a message with SSL encryption. Note that swaks doesn’t verify the key.
  5. Try using swaks to send messages to other servers with SSL encryption. Gmail is one example of a mail server that supports SSL which can be used, run “swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com” to send TLS (encapsulated SSL) mail to Gmail via swaks. Also run “swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com -s localhost” to send via your new mail server (which should log that it was a TLS connection from swaks and a TLS connection to Gmail).

SASL

SASL is the system of SMTP authentication for mail relaying. It is needed to permit devices without fixed IP addresses to send mail through a server. The easiest way of configuring Postfix SASL is to have Dovecot provide it’s authentication data to Postfix. Among other things if you change Dovecot to authenticate in another way you won’t need to make any matching changes to Postfix.

  1. Run “mkdir -p /var/spool/postfix/var/spool” and “ln -s ../.. /var/spool/postfix/var/spool/postfix“, this allows parts of Postfix to work with the same configuration regardless of whether they are running in a chroot.
  2. Add the following to /etc/postfix/main.cf and restart Postfix:
    smtpd_sasl_auth_enable = yes
    smtpd_sasl_type = dovecot
    smtpd_sasl_path = /var/spool/postfix/private/auth
    broken_sasl_auth_clients = yes
    smtpd_sasl_authenticated_header = yes
  3. Edit /etc/dovecot/conf.d/10-master.conf, uncomment the following lines, and then restart Dovecot:
    unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    }
  4. Edit /etc/postfix/master.cf, uncomment the line for the submission service, and restart Postfix. This makes Postfix listen on port 587 which is allowed through most firewalls.
  5. From another system (IE not the virtual machine you are working on) run “swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com -s YOURSERVER and note that the message is rejected with “Relay access denied“.
  6. Now run “swaks -tls --auth-user auser --auth-password WHATEVER -f auser@example.com -t YOURREALADDRESS -s YOURSERVER” and observe that the mail is delivered (subject to anti-spam measures at the recipient).
  7. Configuring a MUA

    If every part of the previous 3 sections is complete then you should be able to setup your favourite MUA. Use “auser” as the user-name for SMTP and IMAP, mail.example.com for the SMTP/IMAP server and it should just work! Of course you need to use the same DNS server for your MUA to have this just work. But another possibility for testing is to have the MUA talk to the server by IP address not by name.

Postfwd and Local Only Email

Over a year ago when I was considering my first Android phone purchase I setup a test account on my mail server so that I could test email clients on phones and tablets. I used a short password because I didn’t want to type a lot on small screens and because typing a password into a random system owned by someone else isn’t particularly secure anyway. Then I forgot about the account until I noticed that my mail server was sending out spam.

Next time I setup such a test account I’ll put rules similar to the following in my Postfwd [1] configuration to stop Postfix from sending such messages. That will prevent the test account from receiving mail from outside or sending mail out of the server. The former is optional (getting a few thousand spam messages in an unused test account is no big deal) but the latter is needed to prevent getting my server blacklisted.

id=R_test_recipient ; recipient==test@coker.com.au ; sender!~.*@coker.com.au ; action=REJECT
id=R_test_sender ; sender==test@coker.com.au ; recipient!~.*@coker.com.au ; action=REJECT

SASL Authentication and Debian/Wheezy

After upgrading a mail server to Debian/Unstable (which will soon be released as Wheezy) I started getting SASL errors.

535 5.7.8 Error: authentication failed: no mechanism available

The SMTP protocol gave the above error for both LOGIN and PLAIN methods.

SASL LOGIN authentication failed: no mechanism available

The postfix/smtpd process logged messages like the above in syslog.

It turned out that the “auxprop_plugin: mysql” line had to be removed and replaced with the following two lines due to a change in the way SQL plugins are managed:

auxprop_plugin: sql
sql_engine: mysql

Also the SQL query needed to have “%u” replaced with “%u@%r” because we now have user and realm provided separately.

Some Postfix Scripts for dealing with Outbound Spamming

I’ve just written some small scripts to help me manage spam emergencies on a mail server. I’ve been doing this thing with a bit of manual effort for a while, but after having done it once from a phone I want to optimise it a bit to reduce painful typing.

My observation is that when a system I run is used as an outbound mail relay for spamming I will notice this reasonably quickly by the queues getting big. Some portion of mail that is queued is delayed due to general network issues and anti-spam measures that neither accept nor reject mail cause a typical spam message to be more likely to be queued than a typical non-spam message. So I look for accounts that send large amounts of mail.

The biggest mail server I run (and the only one to have an outbound spam problem) is configured to not allow users to fake their sender address (IE use an address from gmail.com or another server) unless they pay extra. So the spam I deal with tends to have a valid origin email address that I can use.

Queue Counting

The following command provides a sorted list of the accounts that have the most mail in the queue. This usually means a maximum of 2 or 3 spammers at the end of the list:

mailq|grep ^[A-F0-9]|cut -c 42-80|sort |uniq -c|sort -n|tail

The following function declaration makes the command get-top-sender find the sender with the largest number of queue entries (which has been a spammer every time I’ve checked such things) and assign their email address to the environment variable $TOP_SENDER. It has to be a shell function so that the environment variable will be set in the context of the shell and available to child processes.

function get-top-sender() { TOP_SENDER_LINE=$(mailq|grep ^[A-F0-9]|cut -c 42-80|sort |uniq -c|sort -n|tail -1) ; export TOP_SENDER=$(echo $TOP_SENDER_LINE|cut -f2 -d\ ) ; TOP_COUNT=$(echo $TOP_SENDER_LINE|cut -f1 -d\ ); echo "Top Sender is $TOP_SENDER with $TOP_COUNT messages" ; }

Viewing the Queue

The following script uses the first parameter or the environment variable $TOP_SENDER to specify the user who’s mail should be read. In the normal course of events user email won’t be read by the sysadmin, but if we are to determine whether a slew of email is spam or not there’s no other way. So far I haven’t seen a large number of queued messages not be spam, but I expect it’ll happen eventually and the user will be happy that I checked it instead of locking their account and deleting all the queued mail. For readers who don’t do sysadmin work, this is why we always have entries in the Terms of Service about the possibility of your email being read to fix technical problems or to investigate possible breaches of the ToS from your end.

#!/bin/bash
set -e
mkdir -p ~/tmp/$$
cd ~/tmp/$$
if [ "$1" != "" ]; then
  TOP_SENDER="$1"
fi
if [ "$TOP_SENDER" = "" ]; then
  echo "Specify the sender on the command-line or in \$TOP_SENDER"
  exit 1
fi
for n in $(mailq|grep ^[A-F0-9].*$TOP_SENDER| cut -c 1-10) ; do
  FILE=$(echo $n|sed -e "s/^\(.\)/\/var\/spool\/postfix\/deferred\/\1\/\1/" -e "s/ .*$//")
  if [ -f "$FILE" ]; then
    postcat "$FILE" > $n
  else
    echo "File $FILE missing, probably active"
  fi
done
less *
rm -rf ~/tmp/$$

Note that this needs to be done with a single less command so that I can terminate it by pressing q and easily go to the next file with “:n“. It does waste some server resources by running postcat on all queue files relating to the user in question, but that doesn’t matter, servers are supposed to be powerful enough to cope with some inefficiency in processing uncommon operations.

The Final Command

After that I have a script that connects by ssh to all outbound mail relays and deletes bad messages from the queues and then connects to the database server to lock the account that was used for sending the mail. I’m not publishing it because it’s specific to the servers I run.

So when NAGIOS reports that the queue is too large I have one script to find the most likely culprit, one script to view the queued email from that account, and a final script to lock the account and purge the spam.

I’ve also added a reminder of the command names to /etc/motd as I don’t want to be running ls and set on a phone to discover command names that I’ve forgotten.

Any suggestions for improvements to this will be welcome.

Why Cyrus Sucks

I’m in the middle of migrating a mail server away from the Cyrus mail store [1]. Cyrus provides a POP and IMAP server, a local delivery agent (accepting mail via LMTP). It is widely believed that Cyrus will give better performance than other mail stores, but according to a review by linux-magazin.de Dovecot and Courier deliver comparable (and sometimes better) performance [2].

The biggest problem with Cyrus is that it is totally incompatible with the Unix way. This wouldn’t be a problem if it would just work and if it would display reasonable error messages when it failed, but it doesn’t. It often refuses to work as desired, gives no good explanation, and it’s data structures can’t be easily manipulated. Dovecot [3] and Courier [4] use the Maildir++ format [5] (as well as many other programs). I have set up a system with Courier Maildrop and Dovecot for the IMAP server [6] and it works well – it’s good to have a choice! But also Maildir++ is reasonably well documented and is an extension to the well known Maildir format. This means that it’s easy to manipulate things if necessary, I can use mv to rename folders and rm to remove them.

Cyrus starts with a database (Berkeley DB file) of all folders in all mailboxes. Therefore it is not possible to move a user from one back-end server to another by merely copying all the files across and changing the LDAP (or whatever else contains the primary authentication data) to point to the new one. It also makes it impossible to add or remove folders by using maildirmake or rm -rf. The defined way of creating, deleting, and modifying mailboxes is through IMAP. One of the problems with this is that copying a mailbox from one server to another requires writing a program to open IMAP connections to both servers at once (tar piped through netcat is much faster and easier). Also if you need to rename a mailbox that contains many gigabytes of mail then it will be a time consuming process (as opposed to a fraction of a second for mv).

Cyrus has a tendency to break while Dovecot is documented as being self-healing and Cyrus also seems to cope well in the fact of a corrupted mail store. Even manually repairing problems with Cyrus is a painful exercise. The Cyrus mail store is also badly designed – and it’s design was worse for older filesystems (which were common when it was first released) than it is for modern ones. The top level of a Cyrus maildir contains all the messages in the INBOX stored one per file, as well as three files containing Cyrus indexes and sub-directories for each of the sub-folders. So if I want to discover what sub-folders a mailbox has then I can run ls and wait for it to stat every file in the directory or I can use an IMAP client (which takes more configuration time). As opposed to a Maildir++ store where every file that contains a message is stored in a folder subdirectory named “new“, “cur“, or “tmp” which means that I can run ls on the main directory of the mail store and get a short (and quick) result. Using tools such as ls to investigate the operation of a server is standard practice for a sysadmin, it should work well!

A finall disadvantage of Cyrus seems to have many small and annoying bugs (such as the reconstruct program not correctly recursing the sub folders). I guess it’s because not many people use Cyrus that such things don’t get fixed.

One trivial advantage of Cyrus is that by default it splits users into different sub-directories for the first letter of the account name. Dovecot supports using a hash of the user-name this is better than splitting by first-letter for performance (it gives a more equal distribution) but will make it slightly more difficult to manipulate the mail store by script. Ext3 can give decent performance without a two level directory structure for as many as 31,998 sub-directories (the maximum that it will support) due to directory indexing and Linux caching of dentries. There may be some other advantages of Cyrus, but I can’t think of them at the moment.

Here is a script I wrote to convert Cyrus mail boxes to Maildir++. To make this usable for a different site would require substituting a different domain name for example.com (or writing extra code to handle multiple domains) and inserting commands to modify a database or directory with the new server name. There is no chance of directly using this script on another system, but it should give some ideas for people performing similar tasks.
Continue reading “Why Cyrus Sucks”

Maildrop, IMAP, and Postfixadmin

I have recently configured my mail server to use IMAP. I started doing this when I was attending Linux.conf.au so that I could read urgent mail using my EeePC while at the conference and then be able to deal with the more complex stuff using my laptop later on.

The next logical step is to have mail delivered to different folders in the IMAP account. While there are ways of doing this via the Subject and other header fields, my needs are not that great. All I need to do is to support user+extension@example.com going to a folder named extension in the user’s mail store. While changing my mail server I decided to install Postfixadmin at the same time.

My first attempt to use Maildrop was to put the following in the /etc/postfix/main.cf file:
mailbox_command = /usr/bin/maildrop -d mail -f “$SENDER” “$DOMAIN” “$USER” “$EXTENSION”

That seems to only work when you have local accounts, so I ended up setting fallback_transport = maildrop and then putting the following in /etc/postfix/master.cf:

maildrop unix – n n – – pipe flags=DRhu user=vmail argv=/usr/bin/maildrop -d vmail ${nexthop} ${user} ${extension}

Where vmail is a Unix account I created for storing mail. Then I added the following to /etc/postfix/main.cf. Some of these are probably redundant (such as the virtual_mailbox_base). The recipient limit is set to 1 because there are no command-line parameters for maildrop to support two recipients for the same message.
virtual_alias_maps = mysql:/etc/postfix/mysql_virtual_alias_maps.cf
virtual_mailbox_domains = mysql:/etc/postfix/mysql_virtual_domains_maps.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
virtual_gid_maps = static:2000
virtual_uid_maps = static:2000
virtual_mailbox_base = /mail
vmaildir_destination_recipient_limit = 1
virtual_transport = maildrop
maildrop_destination_recipient_limit = 1

The files /etc/postfix/mysql* all have fields user=, password=, hosts=, and dbname=. The queries in each of the files are as follows:
mysql_virtual_alias_maps.cf:query = SELECT goto FROM alias WHERE address='%s' AND active = 1
mysql_virtual_domains_maps.cf:query = SELECT domain FROM domain WHERE domain='%s'
mysql_virtual_mailbox_maps.cf:query = SELECT maildir FROM mailbox WHERE username='%s' AND active = 1

The /etc/courier/maildroprc file has the following contents:

# log the maildrop actions
logfile "/var/log/maildrop.log"
#
# parameters 1, 2, and 3 are the domain, user, and extension
DOMAIN=tolower("$1")
USER=tolower("$2")
EXTENSION=tolower("$3")
DEFAULT="/mail/$DOMAIN/$USER"
#
# try making a backup (cc) copy of the mail but do not abort if it fails
exception {
  cc "$DEFAULT/.backup/"
}
#
# try delivering to the extension folder but do not abort if it fails
exception {
  if(length("$EXTENSION") != 0 && "$EXTENSION" ne "backup")
  {
    to "$DEFAULT/.$EXTENSION/"
  }
}
#
# deliver to the main inbox if there is no folder matching the extension or if no extension is specified
to "$DEFAULT/"

Installing Postfixadmin [1] was another challenge entirely. One of the complications of this is that there is no Debian package for Lenny (it seems that there will be one in Squeeze – Lenny+1).

I found David Goodwin’s tutorial on installing Postfixadmin and lots of other things on Debian/Etch [2] to be a very useful resource. I look forward to seeing a Lenny version of that document.

Please let me know if you can think of any way to improve this.

I need an LMTP server

I am working on a system where a front-end mail server sends mail to what it considers to be a LDA (Local Delivery Agent) which actually sends mail to a back-end server via LMTP. I can’t remove that fake LDA from the design because it does a bunch of business specific processing along the way.

I am working on changing the back-end from Cyrus to Dovecot. Currently the mail goes from the fake LDA to the Cyrus LMTP server. What I would like to do is to have an LMTP server run on the back-end machine that launches the Dovecot deliver program immediately and then returns an appropriate code.

So far I have been experimenting with having Postfix run on the back-end to use deliver as the real LDA. The first problem with this is that the mail will be written to the Postfix queue and then written to the mail store. Doubling the number of writes is a real problem for a system that is going to be write-bottlenecked – it would significantly increase the hardware costs. The second problem is that when an account goes over quota the back-end server would be generating a bounce message. I would prefer the front-end server to generate the bounce on an un-munged message.

Basically all I need is a simple daemon (which could even be launched from inetd) that talks LMTP (a very simple cut-down version of SMTP) and executes a single command to receive the data. It might be necessary to serialise running the delivery process, in which case the mail data would have to be stored in memory and there would need to be a semaphore around executing the delivery program.

Does anyone know of such a program? If not then I’ll have to write it myself (which shouldn’t be difficult) and GPL it. If I have to do that then I need a suitable name for it. Any suggestions would be appreciated.

Do Spammers target Secondary MX Servers

Rumour has it that some types of spammer target the secondary MX servers. The concept is that some people have less control over the secondary MX server and less ability to implement anti-spam measures. Therefore if they accept all mail from the secoondary then a spammer will have more success if they attack the secondary server.

True secondary servers are becoming increasingly uncommon, the lower priority servers listed in MX records tend to have the same configuration as the primary, so the benefit for the spammer in attacking the secondary server is probably minimal. But it would be good to know whether they do this.

I decided to analyse the logs from a mail server that I run to see if I can find evidence of this. I chose a server that I run for a client which has thousands of accounts and tens of thousands of messages delivered per day, my own server doesn’t get enugh traffic to give good results.

I analysed the logs for a week for the primary and secondary MX servers to see if the ratio of spam to ham differed. Now this does have some inherent inaccuracy, some spam will slip past the filters and occasionally a legitimate email will be rejected. But I believe that the accuracy required in a spam filter to avoid making the users scream is vastly greater than that which is required to give a noteworthy result.

I produced totals of the number of messages delivered, the number rejected by SpamAssassin (which has a number of proprietary additions), the number of message delivery attempts that were prevented due to rate limiting (most of which will be due to spammers), and the number of attempts to deliver to unknown accounts (some of which will be due to spammers having bad addresses in their lists).

For each of these rejection criteria I produced a ratio of the number of rejections to the number of delivered messages for each of the servers.

The rate limit count didn’t seem useful. While the primary server had a ratio of 0.75 messages rejected due to rate limiting to every message accepted the secondary had a ratio of 0.08. It seems that the secondary just didn’t get enough traffic to trigger the limits very often. This is an indication that the more aggressive bots might not be targetting the secondary.

The ratio of messages rejected by SpamAssassin to legitimate mail was 0.76:1 on the primary server and on the secondary server it was 1.24:1. The ratio of messages addressed to unknown users to successful deliveries was 3.05:1 on the primary and 7.00:1 on the secondary! This seems like strong evidence to show that some spammers are deliberately targetting the secondary server.

In this case both the primary and secondary servers are in server rooms hosted by the same major ISP in the same region. The traceroute between the two mail servers is only 7 hops, and there is only one hop between the two server rooms. So it seems unlikely that there would be some connectivity issue that prevents spammers from connecting to the primary.

One other factor that may be relevant is that the secondary server has been in service for some years while the primary is only a few months old. Spammers who store the server IP address with the email address (which happens – change the DNS records to send your mail to a different server and you will see some spam go to the old server) will be sending mail to what is now the secondary server. The difference in the rejected mail volume on the secondary server and the amount that would be rejected if it had the same ratio as the primary amounts to 7% of all mail rejected by SpamAssassin and 14% of all mail addressed to unknown users. I think it’s unlikely that any significant fraction of that would be due to spammers caching the server IP address for months after the DNS records were changed. So therefore it seems most likely that something between 7% and 14% of spam is specifically targetted at the secondary server.

While the ratio of spam to ham seems significantly worse on the secondary it is still a relatively small portion of the overall spam sent to the service. I had been considering setting up secondary mail servers with extra-strict anti-spam measures but the small portion of the overall spam that is targetted in such a way indicates to me that it is not going to be worth the effort.

Another thing that has occurred to me (which I have not yet had time to investigate) is the possibility that some spammers will send the same messages to all MX servers. If that happens then the ratio of spam to ham would increase every time the number of MX servers is increased. In that case it would make sense to minimise the number of MX servers to reduce the amount of CPU power devoted to runing SpamAssassin.

Note that I have intentionally not given any numbers for the amount of mail received by the service as it is a commercial secret.

Update: One thing I realised after publishing this post is that the secondary MX server is also the main server for mail sent between local users. While the number of users who send mail to other users on the service is probably a small portion of the overall traffic (it’s not a really big ISP) it will make a difference to the ratios. Therefore the ratio of spam to ham would be even worse on the secondary MX (assuming for the sake of discussion that local users aren’t spamming each other).

Please Turn off Your Spam Protection

Hi, I’d like to send an email from a small domain that you’ve never heard of or from a big ISP that’s known for being slack about spam (*), I can’t send the mail to you because of your anti-spam measures. I think that this is unfair, it’s discrimination, and you are cutting off your nose to spite your face in rejecting my mail.
So please reconfigure your mail server now and accept more spam in your inbox, my message is important enough to justify the extra time you will spend manually deleting mail and the risk of accidentally deleting legitimate mail while deleting heaps of spam (I am in fact more important than you).
By not accepting my mail you are being an asshole. I only receive a dozen spam messages a day and I don’t mind it, without even knowing you or having bothered to do a web search to see how well your email address is known I’m sure that you don’t receive any more spam than me and therefore you too can turn off most anti-spam measures and manually sort through the spam.
You don’t really have a problem with spam, you are just paranoid, I’m sure that you installed your anti-spam measures before receiving any spam and then never bothered to check the hit rates.
My sys-admin knows that one of the DNSBLs has an old entry from when his server was broken, but he won’t request that it be removed – so you can change your server because my sys-admin doesn’t want to click on a URL that you sent him.
The RFC-ignorant.org service is used and run by ignorant people – I know this without even reading their web site to discover how it works.

The above is a summary of a number of complaints that I have received about my anti-spam measures. I’ve para-phrased them so that they make sense, I have not actually had someone directly say “I’m more important than you so you should just accept more spam”, but the implication is quite clear.

Now there are some legitimate reasons for requesting that anti-spam measures be reduced. In the distant past almost everyone had working reverse DNS entries which matched the forward entries and it was common to reject mail from systems that didn’t have valid DNS. Nowadays there are many big ISPs that delegate IP address space without permitting reverse DNS entries, and there are companies that have one department in charge of IP addresses (who don’t have a clue about reverse DNS) and another department running mail servers (who are cluefull). So the environment has changed to make reverse DNS checks a non-viable anti-spam measure. Requesting that people remove such checks is reasonable.

Anti-spam measures that attack innocent third parties are bad. Sending “warnings” about viruses has made no sense for many years as all modern viruses fake the sender address, an employee of one company once admitted to me that they were sending out anti-virus “warning” messages as a way of sending unsolicited advertising to random people (I reported them as a spam source). Some time ago on a routine upgrade of ClamAV I accidentally copied in a default configuration file that made it send such warnings – I was grateful when someone informed me of my mistake so that I could fix it. Challenge-response is another technology that causes harm to others. I think it makes a lot of sense for mailing lists (every modern list server will confirm subscription requests), while it does result in sending unwanted mail to innocent third parties (every time a new virus becomes prevalent I receive a number of confirmation messages from list servers), but it’s not something that I will use on a regular email account (but I am prepared to do paid work implementing CR for other people).

Requesting that manually implemented blocks be removed is quite reasonable. Occasionally I find that mail from one of my servers is blocked because a previous owner of the IP address space did bad things. In such a situation it is quite reasonable to provide an assurance that the new owner takes abuse issues seriously and to request that the block be removed.

Requesting that I make any change to my system without making a minimal effort to get your broken mail server fixed is totally unreasonable. If the system administrator is not prepared to click on a URL to get their system removed from a black-list or if the user is unwilling to report the problem to the sysadmin then I will probably be unwilling to make any change to my system. The only exceptions to this rule are for clients, colleagues, and for people who use mail services that are large and unresponsive to users (IE the users don’t directly pay). I recently made a white-list entry for a large European ISP that is used by a Debian Developer for their work, as the ISP is known to be unresponsive to requests and mail related to Debian work is important to me I added a white-list entry.

One thing I am planning to do is to document my anti-spam measures and then allow people the opportunity of suggesting new anti-spam measures that I haven’t tried yet if they want me to turn off any of my current protections.

(*) I’m not aware of any big ISP that takes strong measures against spamming customers and customers whose computers are trojaned. I am aware of one having done it in the past, but I suspect that they may have ceased doing so after their brush with bankruptcy. I suspect that many ISPs simply rate-limit their customers connections to the outbound mail relays and hope that they don’t get enough infected customers at any time to get themselves listed as a spam source.