Warning: Hackers Are Using SSH Tunnels to Send Spam

Tired of spammers finding new ways to exploit your server?

Me too.

I spend a lot of time investigating spam incidents for customers.  Typically, hackers use a few common techniques to send their spam.   The most common issues I see are:

• Web application exploits.
• Contact form exploits.
• Compromised user password

With these exploits, you can spot them as they leave clear evidence in the logs.  In the case of PHP, you can enable PHP mail logging and quickly find the script sending the spam.   This makes stopping these types of attacks easy.

This week, I ran into something I’ve not seen before.

An attacker was using an SSH tunnel to send spam.

This attack can be difficult to detect as it leaves little log evidence – just a ssh login entry and the spam itself.

Fortunately, it is easy to stop once you know how it works.

Setting up SSH Tunnels

Sometimes the best way to recognize an attack is to do it yourself.  So once I discovered how the attackers were compromising the server, I tried it myself.

So here’s what I did:

ssh -f [email protected] -L 2000:localhost:25 –N

Then

telnet localhost 2000
Connected to localhost.
Escape character is '^]'.
220 remote.host.com ESMTP Postfix

What just happened?

The ssh command tunneled port 25 on the localhost back to my system on port 2000.

I can now send email through the remote host by connecting locally on port 2000.

The attackers used this technique to inject 10,000’s of emails into the server.

This is a clever approach.  Unlike other attacks, this method leaves few clues.

With some sleuthing, however, you can catch this attack.  You can even prevent it with a simple change to SSH.

Sending Spam with SSH Tunnels

I don’t want to be alarmist, so I want to make it clear:

This SSH spam method requires access to a user account.

In this incident, the attacker had compromised a user account due to a poor quality password.

This is essentially a password compromise, but unlike most attacks, the attackers used a SSH tunnel.  The tunnel made it more difficult to detect and block the exploit.

Here’s a breakdown of how the technique works.

TCP Tunnel to SMTP

SSH, by default, permits TCP port forwarding.  The attackers were using this feature to forward the SMTP port over SSH back to their local system.

As you can see in the diagram, The attackers connects to your server over SSH using a compromised user account.   Then, they setup a SSH tunnel to forward port 25 back to their system.  They can then connect locally to port 2000 (or any port they select) to send spam.  Since most servers trust SMTP connections on localhost, no authentication is required.

With this tunnel in place, they attacker can now send spam via the SSH tunnel.

SMTP AUTH & Localhost

In most spam cases involving exploited password, attackers connect directly to the mail server.  As a result, your mail logs will be filled with SMTP authentication attempts – often from many IP addresses.   This makes it easy to identify the compromised account.

With the SSH tunnel technique, SMTP authentication is not required.  As a result, there’s remarkably little evidence in the logs of an attack.

The only indication of a problem is a high volume of bounces or email being sent via localhost.

The attack works because most servers implicitly trust email from localhost.  For email sent via a localhost host connection to the SMTP server, SMTP AUTH is not required.

Without SMTP authentication, there is no log evidence to identify the compromised account.  You just see a lot of email coming from localhost.

Investigating SSH Tunnel Attacks

 

There are two clues I found with this attack

• Email logs showing SMTP connections from localhost
• Netstat showing SSH connecting to SMTP

 

Email from Localhost

In most attacks, either attackers either use a web application exploit or compromised user account.  These methods produce distinct signatures in the mail logs.

In the case of web application attacks, you can often correlate web logs to email logs to find the site or use PHP mail logging to identify the offending scripts.

For compromised user accounts, SMTP authentication logs will quickly reveal the problem. You will see 100’s of authentications, typically from different IP addresses.  Just change the user’s password and your done.

With the SSH tunnel attack, the logs looked like this:

Mar  3 16:05:16 psa001 postfix/smtpd[13792]: 058D82002A: client=localhost.localdomain[127.0.0.1]
Mar  3 16:05:18 psa001 postfix/cleanup[14128]: 058D82002A: message-id=<[email protected]>
Mar  3 16:05:18 psa001 postfix/qmgr[2459]: 058D82002A: from=<[email protected]>, size=405, nrcpt=1 (queue active)
Mar  3 16:05:20 psa001 postfix/smtp[14129]: 058D82002A: to=<[email protected]>, relay=rackaid.com.inbound10.mxlogic.net[208.65.145.3]:25, delay=9.4, delays=7.7/0/0.39/1.3, dsn=2.0.0,
status=sent (250 Backend Replied [e8ee4135.0.326436.00-1980.504496.p02c12m006.mxlogic.net]:  2.0.0 Ok: queued as 149162059B (Mode: n)
Mar  3 16:05:20 psa001 postfix/qmgr[2459]: 058D82002A: removed

There’s no SMTP authentication happening.  This works because the system trusts emails from localhost.

When you email from scripts (e.g. php’s mail() function) or the command line, the email is sent directly via the servers email binary program.

Message sent directly look like this:

Mar  3 16:04:40 psa001 postfix/pickup[13103]: 886D388003: uid=0 from=

Message sent via SMTP look like this:

Mar  3 16:05:16 psa001 postfix/smtpd[13792]: 058D82002A: client=localhost.localdomain[127.0.0.1]

In the second example, you will see that the email has a client associated with it:  127.0.0.1.  This will also be included in the mail headers.

This was the clue I needed.

I now know that the email is not being sent via a script’s mail function but rather, something or someone is opening a direct connection to the SMTP sever over localhost.

More Clues: Netstat & PS

If you catch the attack in progress, netstat and ps can provide further clues.

During this case, I happened to login while the attackers were sending their spam.  As a result, you could see the localhost connection to port 25 in netstat from sshd.

tcp        0      0 127.0.0.1:46298             127.0.0.1:25                ESTABLISHED 8163/sshd

Note that the remote and local connections are both localhost.  For SMTP, this is a strange connection.

Netstat gives you the process id that has the connection open.   Checking ps on that id returns:

root      8157  0.0  0.2  77680  1256 ?        Ss   14:38   0:00 /usr/sbin/sshd
root      8160  0.1  0.6 109352  4196 ?        Ss   14:38   0:00  _ sshd: jeffh [priv]
jeffh     8163  0.0  0.3 109352  2112 ?        S    14:38   0:00      _ sshd: jeffh

During this attack, we see both an SSH process as well as a local connection from SSH to SMTP.

These are two excellent clues to this type of attack.

Access Logs

I usually use the “last” command on systems to review who’s recently logged into the server.  This usually works well in most cases.  However, there’s a problem with last.

The last command uses /var/log/wtmp and not all logins are recorded in wtmp.

SSH does not log a hit into wtmp if it is a non-interactive session.   In the case of a SSH tunnel, you do not need an interactive session.  So you have to check /var/log/secure to find the logins.

Shells

As it turns out, you also do not need a valid shell to use SSH tunnels.

You can still use SSH tunneling even if the user’s shell is set to /bin/false or /sbin/nologin.  So if you need to restrict SSH use, you have toset the user-level security features in sshd.

 Preventing SSH Tunnels

Fortunately, you can easily block tunneling of ports by changing:

AllowTCPForwarding no

If you set this in SSH then try a tunnel, you can still connect to SSH but will get this result:

channel 2: open failed: administratively prohibited: open failed

So this is a quick an easy way to add a layer of security to SSH  — along with our other recommended SSH hardening changes.

Conclusion

Attackers will always find subtle ways to avoid detection.   Their goal is to use your server as much as possible, so techniques like this one make it harder to identify and fix the issue.

In my research, I only found one mention of a similar case spam using SSH port forwarding and this was in 2009.

Hopefully, this is an isolated incident.   However, I know we will start checking for this type of attack as part of our spam incident services.