I’ve just written some small scripts to help me manage spam emergencies on a mail server. I’ve been doing this thing with a bit of manual effort for a while, but after having done it once from a phone I want to optimise it a bit to reduce painful typing.
My observation is that when a system I run is used as an outbound mail relay for spamming I will notice this reasonably quickly by the queues getting big. Some portion of mail that is queued is delayed due to general network issues and anti-spam measures that neither accept nor reject mail cause a typical spam message to be more likely to be queued than a typical non-spam message. So I look for accounts that send large amounts of mail.
The biggest mail server I run (and the only one to have an outbound spam problem) is configured to not allow users to fake their sender address (IE use an address from gmail.com or another server) unless they pay extra. So the spam I deal with tends to have a valid origin email address that I can use.
Queue Counting
The following command provides a sorted list of the accounts that have the most mail in the queue. This usually means a maximum of 2 or 3 spammers at the end of the list:
mailq|grep ^[A-F0-9]|cut -c 42-80|sort |uniq -c|sort -n|tail
The following function declaration makes the command get-top-sender find the sender with the largest number of queue entries (which has been a spammer every time I’ve checked such things) and assign their email address to the environment variable $TOP_SENDER. It has to be a shell function so that the environment variable will be set in the context of the shell and available to child processes.
function get-top-sender() { TOP_SENDER_LINE=$(mailq|grep ^[A-F0-9]|cut -c 42-80|sort |uniq -c|sort -n|tail -1) ; export TOP_SENDER=$(echo $TOP_SENDER_LINE|cut -f2 -d\ ) ; TOP_COUNT=$(echo $TOP_SENDER_LINE|cut -f1 -d\ ); echo "Top Sender is $TOP_SENDER with $TOP_COUNT messages" ; }
Viewing the Queue
The following script uses the first parameter or the environment variable $TOP_SENDER to specify the user who’s mail should be read. In the normal course of events user email won’t be read by the sysadmin, but if we are to determine whether a slew of email is spam or not there’s no other way. So far I haven’t seen a large number of queued messages not be spam, but I expect it’ll happen eventually and the user will be happy that I checked it instead of locking their account and deleting all the queued mail. For readers who don’t do sysadmin work, this is why we always have entries in the Terms of Service about the possibility of your email being read to fix technical problems or to investigate possible breaches of the ToS from your end.
#!/bin/bash
set -e
mkdir -p ~/tmp/$$
cd ~/tmp/$$
if [ "$1" != "" ]; then
TOP_SENDER="$1"
fi
if [ "$TOP_SENDER" = "" ]; then
echo "Specify the sender on the command-line or in \$TOP_SENDER"
exit 1
fi
for n in $(mailq|grep ^[A-F0-9].*$TOP_SENDER| cut -c 1-10) ; do
FILE=$(echo $n|sed -e "s/^\(.\)/\/var\/spool\/postfix\/deferred\/\1\/\1/" -e "s/ .*$//")
if [ -f "$FILE" ]; then
postcat "$FILE" > $n
else
echo "File $FILE missing, probably active"
fi
done
less *
rm -rf ~/tmp/$$
Note that this needs to be done with a single less command so that I can terminate it by pressing q and easily go to the next file with “:n“. It does waste some server resources by running postcat on all queue files relating to the user in question, but that doesn’t matter, servers are supposed to be powerful enough to cope with some inefficiency in processing uncommon operations.
The Final Command
After that I have a script that connects by ssh to all outbound mail relays and deletes bad messages from the queues and then connects to the database server to lock the account that was used for sending the mail. I’m not publishing it because it’s specific to the servers I run.
So when NAGIOS reports that the queue is too large I have one script to find the most likely culprit, one script to view the queued email from that account, and a final script to lock the account and purge the spam.
I’ve also added a reminder of the command names to /etc/motd as I don’t want to be running ls and set on a phone to discover command names that I’ve forgotten.
Any suggestions for improvements to this will be welcome.
What mail servers do you run that you need to deal with outbound spam more than once?
What actions do you take against spammers when you find them?
Also, why do you block the use of arbitrary From addresses for people who have done SMTP AUTH?
If you run something that is in concept like Gmail then you deal with outbound spam on a regular basis.
When I find spammers I lock the accounts. As I’m running a paid service and the accounts usually get locked fast enough that it’s not an economical way to send spam, so people won’t sign up for the purpose of spamming. The accounts that spam come from are those belonging to people who have their PC compromised, when their account is locked they can contact the help desk to get the problem resolved.
The use of arbitrary From addresses is a premium service that is available to people who pay extra. Very few premium users get compromised and of those most of them are compromised by bots that use the default From field from their MUA.
Seeing this startles me a bit:
“mkdir -p ~/tmp/$$”
Consider that someone somehow managed to create the directory and symlinked a file there to someplace fun, a snippet like “TMPDIR=$(mktemp -d)” is generally better.
Mistotebe: If someone manages to create a symlink in root’s home directory then it’s game-over already.
But I take your point that there are safer ways of doing shell scripting, and concede the possibility that one of the few people with root access might create a symlink there for some strange reason that doesn’t involve hostile intent – recall what they say about stupidity being more dangerous than malice.
Keeping mktemp in mind when creating temporary directories is always a good idea.
Thanks for the suggestion, next time I work on those scripts I’ll probably implement the change you suggest!