Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Masked Pumpkin
May 10, 2008
I've got a server that has to allow multiple a reasonably high smtpd_connection_rate_limit as it's managing (sasl_authenticated) mail for multiple domains, each with many users behind a single (ISP allocated, DHCP) IP. As a result, when an account gets hacked and used to send spam, it can take several minutes for the alert regarding a suddenly increased mailq size to come to our attention and get fixed. While the postfix service could simply be stopped when a high mailq is detected, that's not an elegant solution, and so I decided to try my hand at a bash script to run every minute and fix the problem:

code:
#!/bin/bash

#This script will check for the top senders in X lines of postfix maillog, 
#judging by sasl_username. If a given username appears too often on that list, 
#the script will find the most common originating IP and ban it with iptables. 
#It will then clean the mailq of any emails that apply with that address, and 
#(hopefully) send an email to a given address to alert them of the action

###############################################################################
#User specified variables:

#Maximum number of messages from any one user in the last 1500 lines of code
MAXMESSAGES="50"

#Path to your mail.log file
MAILLOGPATH="/var/log/maillog"

#Email Address to receive notification - must be properly set up in your /etc/msmtp.conf
ADMINEMAIL="youremail@gmail.com"
###############################################################################

if [ -f /tmp/topips ];			#First, some cleanup
then
	rm /tmp/topips
fi
if [ -f /tmp/topqueues ];
then
	rm /tmp/topqueues
fi

NUMMSGS=`tail -n1500 $MAILLOGPATH | grep sasl_username | awk '{print $9}' | sort | uniq -c | sort -n | tail -n1 | awk '{print $1}'`	#I really shouldn't have to do this twice?

if [ "$NUMMSGS" -ge "$MAXMESSAGES" ];
then
	USER=`tail -n1500 $MAILLOGPATH | grep sasl_username | awk '{print $9}' | sort | uniq -c | sort -n | tail -n1 | cut -d= -f2`
	echo $USER
	
	#Find the queue IDs for any offending messages with that username, and feed the IP into /tmp/topips
	mailq | grep $USER | awk '{print $1}'| while read LINE; do
		IP=`postcat -q $LINE | grep -E "^Received:(.)*[.]*\[[0-9]*.[0-9]*.[0-9]*.[0-9]*\]\)" | grep -oE "[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*" | uniq | tail -n1 >> /tmp/topips`;
		echo $LINE >> /tmp/topqueues;
	done

	#Change that user's password
	mysql -u POSTFIXUSER -pPASSWORD -e 'UPDATE postfix.USERS SET password = "ChangedPass!34NotEvenMD5" WHERE username = "$USER";'

	#Find the most regularly occurring IP in that file - if it exists
	if [ -f /tmp/topips ];
	then
		TOPIP=`uniq /tmp/topips -c | sort | tail -n1 | awk '{print $2}'`
		echo IP to ban is $TOPIP
		iptables -I INPUT 1 -p tcp -s $TOPIP -j DROP		#This could be set to ACCEPT while testing, drop later
	
		#Now we can clear the mailq of all the offending messages
		cat /tmp/topqueues | while read LINE; do
			#postsuper -d $LINE
			echo $LINE will be deleted;
		done

		#Now let's send off an email to let us know what's happened
		echo The user $USER was blocked for sending $NUMMSGS emails from $TOPIP | msmtp --file=/etc/msmtp.conf --account=gmail $ADMINEMAIL
	else
		echo $USER has had all emails delivered already, no IP to ban
	fi

	#And lastly, clean up after ourselves
	if [ -f /tmp/topips ];                  #First, some cleanup
	then
        	rm /tmp/topips
	fi
	if [ -f /tmp/topqueues ];
	then
        	rm /tmp/topqueues
	fi
else
	echo Highest messages at $NUMMSGS, going back to sleep
fi


My lack of experience with bash scripting should be evident - it feels like I'm making multiple unnecessary calls (for example, to find the highest user and then the highest number of messages). The use of temporary files instead of arrays also doesn't feel right. On the other hand, the script does work - it's caught a handful of chancers in the last few weeks I've been testing it.

I'd like to try and get this script as polished as possible to aid with my own study of bash, and if anyone else finds it useful they're welcome to it. Otherwise, am I perhaps barking up the wrong tree entirely?

Adbot
ADBOT LOVES YOU

kazoosandthings
Dec 6, 2004

In order to join the pirate crew,
you must prove yourself on the kazoo.
I think your script is not bad at all (especially since you say it works!) but here are some opinionated style points that may be of interest:

I would not personally check for the existence of a file before deleting it unless I wanted to do something differently with the existing file like archive or append. "rm filename" fails (exit status non-zero) when filename does not exist. "rm -f filename" returns zero even if the file exists. A bonus is you can save almost 20 lines total: "rm -f /tmp/topips /tmp/topqueues". The reason you might check first is because you don't want stderr to poo poo up your cron logs or something. You might or might not already be aware of the special variables such as "$?". You can trial in the interactive shell with "echo $?".

You append to a file which you deleted, which causes it to exist even if zero bytes are written, so you don't need to check for it later. (try this at home: "echo -n >> loltestfile ; xxd loltestfile")

I don't use SASL on my mail server so I don't have logs to test with but maybe you could eliminate the use of awk in some places by using sort -k ?

Also as far as I can tell you set IP= for no reason.

I don't know how fail2ban works but maybe you could just tell fail2ban about your count of emails instead of doing the firewall rule yourself (not that there's anything wrong with you doing so but fail2ban already solves a similar problem which is one IP fails too many auths within some time period) the benefit being that fail2ban will unblock after some time.

Masked Pumpkin
May 10, 2008
Thank you so much! Based on your advice, I've updated and tweaked the script somewhat -

Changelist:
  • Mailq is now called instead of trawling through the log file - this still requires multiple calls to mailq and subsequently to postcat, which I'd like to trim down if I can
  • MAXMESSAGES is now also checked with MAXMESSAGESPERUSER, to help avoid blocking a user who happens to have just 5 messages in the queue when the server is very congested for whatever reason
  • The password change query to MySQL wasn't being processed properly thanks to the way the $USER variable was passed to it - fixed
  • Exit codes now used to find bad messages in the queue and remove them, and also to check whether an IP hasn't already been banned
  • Removed checking for the existence of temporary files
  • IP variable actually set and used now - a good thing, since we were previously getting a DNS name, which works but is not ideal for IPTables
  • Found a bad if/then check, fixed and cleaned up indenting to reflect properly
  • Integrated fail2ban for temporary blocking, if it's installed (and the sendmail-auth jail is configured)

Any other comments/suggestions are greatly appreciated - I've learned a lot about bash from this, but I can't help but feel that the biggest thing I've learned is to not use bash if I can use PHP or Python (both of which I know reasonably well and allow for variable management that doesn't drive you to drink).

code:
#!/bin/bash
#This script will check for the top senders in X lines of postfix maillog, judging by sasl_username. 
#If a given username appears too often on that list, the script will find the most common originating 
#IP and ban it with iptables. It will then clean the mailq of any emails that apply with that address, 
#and (hopefully) send an email to a given address to alert them of the action.
#AMENDED - This script will keep an eye on the postfix queue - if it grows too big, it'll look for the 
#highest (sasl_auth) sender in that queue - if that sender is above a threshold, it'll change that 
#users password, block the IP, and remove all of those messages from the mailq

###############################################################################
#User specified variables:
#Maximum number of messages from any one user in the last 1500 lines of code
MAXMESSAGES="50"

#Maximum number of messages from any one user
MAXMESSAGESPERUSER="20"

#Path to your mail.log file
MAILLOGPATH="/var/log/maillog"

#Email Address to receive notification - must be properly set up in your /etc/msmtp.conf
ADMINEMAIL="youremail@gmail.com"

#If you have Fail2Ban installed (with the sendmail-auth jail enabled) then change this to 1 so an IP 
#can be banned but automatically unbanned after whatever bantime you have set in Fail2Ban
FAIL2BAN="1"
###############################################################################

#First, some cleanup
rm /tmp/topips
rm /tmp/topuser

NUMMSGS=`mailq | tail -n1 | awk '{print $5}'`

if [ "$NUMMSGS" -ge "$MAXMESSAGES" ];
then
        echo Number of messages in queue at $NUMMSGS, checking for single high user;
        mailq | grep -P "^[0-9A-f]{10}" | awk '{print $1}' | while read LINE; do
                postcat -q $LINE | grep -E "Authenticated sender: [A-Za-z0-9]*@[A-Za-z0-9\.]*\)" | grep -oE "[a-zA-Z0-9]*@[a-zA-Z0-9\.]*" >> /tmp/topuser;
        done
        USERMSGS=`uniq -c /tmp/topuser |sort -n |tail -n1| awk '{print $1}'`;

        if [ "$USERMSGS" -ge "$MAXMESSAGESPERUSER" ];
        then
                echo Highest user at $USERMSGS, taking action;
                USER=`uniq -c /tmp/topuser |sort -n |tail -n1| awk '{print $2}'`;
                echo User is $USER;

                #Find the queue IDs for any offending messages with that username, and feed the 
		#IP into /tmp/topips - Sadly, mailq will only give us the provided from_address, 
		#not the sasl_sender, so this must be run through postcat
                mailq | grep -P "^[0-9A-f]{10}" | awk '{print $1}' | while read LINE; do
                        postcat -q $LINE | grep -iE "Authenticated sender: $USER";
                        if [ $? -eq 0 ];
                        then
                                postcat -q $LINE | grep -E "^Received:(.)*[.]*\[[0-9]*.[0-9]*.[0-9]*.[0-9]*\]\)" | grep -oE "[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*" | uniq | tail -n1 >> /tmp/topips;
                        fi
                        done

                BADIP=`uniq -c /tmp/topips |sort -n |awk '{print $2}'`
                echo Offending IP is $BADIP

                #Change that user's password
                echo Changing password for $USER
                mysql -u POSTFIXUSER -pPASSWORD -e "UPDATE postfix.mailbox SET password = 'ChangedPass!23324234' WHERE username = \"$USER\";"

                #Ban the IP, first checking it's not being blocked already
                if [ "$FAIL2BAN" -eq "1" ];
                then
                        echo Jailing $BADIP with fail2ban;
                        fail2ban-client sendmail-auth banip $BADIP;
                else
                        iptables -L|grep -E "DROP       tcp  --  $BADIP[ ]*anywhere";
                        if [ $? -eq 1 ];
                        then
                                echo banning IP $BADIP with iptables;
                                iptables -I INPUT 1 -p tcp -s $BADIP -j DROP;
                        fi
                fi

                #Now we can clear the mailq of all the offending messages
                mailq | grep -P "^[0-9A-f]{10}" | awk '{print $1}' | while read LINE; do
                        postcat -q $LINE | grep -iE "Authenticated sender: $USER"
                        if [ $? -eq 0 ];
                        then
                                echo $LINE will be deleted;
                                #postsuper -d $LINE;
                        fi
                        done

                #Now let's send off an email to let us know what's happened
                echo The user $USER was blocked for sending $NUMMSGS emails from $BADIP | msmtp --file=/etc/msmtp.conf --account=gmail $ADMINEMAIL

                #And lastly, clean up after ourselves
                rm /tmp/topips
                rm /tmp/topuser
        else
                echo Not enough messages from one user in queue
        fi
else
        echo Highest messages at $NUMMSGS, going back to sleep
fi

kazoosandthings
Dec 6, 2004

In order to join the pirate crew,
you must prove yourself on the kazoo.
I think this is a good iteration. I think I was not very clear when I mentioned $?. I was referring to rm vs rm -f (but as you discovered it works everywhere!) If the file does not exist rm alone will "fail" and $? is 1 and it will emit something on stderr that may end up in a log file somewhere, but rm -f does not care whether it exists or not and $? is always zero and emits no error (so logs do not get filled with garbage).

You can also $? with grep as you are doing, though if you are only checking success/failure of the match and don't care about capturing the matched line, you can also invoke grep with -q and put the whole expression in the if block e.g:
code:
if postcat | grep -q myexpression myfile ; then
	do some stuff
else
	do different stuff
fi
I also like to use iptables -L with -n because then it will not try to reverse-lookup IPs which can stall. If you are going to be putting in rules with IPs from arbitrary internet hosts it's probably a good idea to not look them up when you run iptables -L, by adding -n.

I learned a lot about bash by second-guessing myself and finding a highly upvoted stack exchange post or obscure TLDP reference about it.

I completely agree with your comment about bash's capacity to drive one to drink. I think it's power can be deceptive -- I learned a lot more about the interactive shell when learning about scripting, and vice-versa, but I still don't always know when trying to map a particular problem into bash if I should call it quits and switch to a more feature-ful language.

One last thing I thought of, some potential for trouble exists if the previous instance of your script has not finished processing before the next one starts. Although when that happens it probably means something else is wrong (processing more than a minute's worth of IPs/maillogs!), you may want to read about mktemp, which can help ensure the temporary files in (accidentally) simultaneous instances don't collide (and may actually be interesting if you are enjoying bash so far...!). When you start worrying about this stuff life can become more complicated. Handling those situations gracefully in a way that works on all shell variants is challenging but it's probably doable for a single shell (e.g. bash) and might even be fun to OCD about when you've got nothing else to keep you busy =)

  • Locked thread