Recently, we’ve seen a rise in SSH brute force attacks. This has been causing some outages on SSH as the service overloads and cannot handle any more requests. While there are many tips on the internet on how to block this, some of the more popular ones have denial of service exploits. Currently, we are testing a PAM module for automatically blacklisting abusive hosts.
Rise of SSH Attacks
Over the past few days, our monitoring has detected SSH service failures. SSH service failures are a rare event on the systems we maintain. Typically email or web services alert before you see any issues with SSH. When I dug into a server’s logs, I found brute force attacks:
Jan 17 07:36:39 server sshd[7807]: Failed password for invalid user antica from 210.233.71.110 port 41674 ssh2 Jan 17 13:36:39 server sshd[7808]: Received disconnect from 210.233.71.110: 11: Bye Bye Jan 17 07:36:40 server sshd[7810]: Invalid user cncp from 210.233.71.110 Jan 17 13:36:40 server sshd[7812]: input_userauth_request: invalid user cncp Jan 17 07:36:43 server sshd[7810]: Failed password for invalid user cncp from 210.233.71.110 port 41986 ssh2 Jan 17 13:36:43 server sshd[7812]: Received disconnect from 210.233.71.110: 11: Bye Bye Jan 17 07:36:44 server sshd[7814]: Invalid user contabil from 210.233.71.110 Jan 17 13:36:44 server sshd[7815]: input_userauth_request: invalid user contabil Jan 17 07:36:47 server sshd[7814]: Failed password for invalid user contabil from 210.233.71.110 port 42297 ssh2 Jan 17 13:36:47 server sshd[7815]: Received disconnect from 210.233.71.110: 11: Bye Bye Jan 17 13:36:47 server sshd[7820]: Connection closed by 72.26.201.3
Dshield, a collaborative threat reporting service, also detected a rise in SSH attacks as can be seen in the red line spiking from 30,000 to over 100,000.
Security Threat
The security issue here is two fold:
- Denial of Service to SSH
- Potentially Compromise of System Password
In several cases, I’ve seen SSH become unresponsive. You get intermittent access. Eventually, you can gain access and then block the offending IP at your firewall or use /etc/hosts.deny. Unfortunately, within a few hours or minutes, another bot is scanning and you have to block more IPs. Trying to manually keep up with this can be tedious, especially if you have dozens or even hundreds of servers to maintain.
The major threat is that there is an account with a poor password on your system. By not blocking these scanners, the chance of them finding this poorly secured account increases the more they are permitted to scan. According to DShield, common accounts like “root”, “test”, and “tomcat” were attacked the most. I’ve also seen admin, webmaster, and similar role-based accounts. If an attacker guesses a password, they now have access to your server.
Prevention
The easiest way to prevent these attacks is to block all access to SSH except to designated IPs. This can be easily managed via a firewall or using /etc/hosts.deny. This is highly effective but is not often practical in a shared hosting or multi-tenant server environment. Some sort of automated system must be used.
Log-based scanners, such as fail2ban and denyhosts, are very popular. In fact, I used to use them on some servers until exploits were discovered. Too many log-based tools fail to properly check log entries. As a result, a clever attacker can spoof connection details and result in IPs getting blocked. If you are using DROP rules in your firewall, the attacker can attempt to guess important IPs, such as your DNS resolvers and cause major issues for your server. Daniel Cid has a great article on how to attack log analysis tools.
So what to do? Currently, I am testing pam_abl. Pam auto blacklist maintains a database of hosts and users with failed logins. The nice thing is that it takes advantage of the pluggable authentication mechanism presence on Linux servers. The system does not rely on logs so it is immune from log based attacks. Thus far, the results are encouraging. I’ve yet to deploy it in a busy multi-user server but we’ve a large shared hosting client that will be testing it first. If tests go well, then we will deploy the tool more widely.