From time to time, we have a client get hit with a DDoS attack. Most of the time the attacks are simple and easily blocked, but attacks aimed at applications are harder to defend. We have had good success using Nginx as a proxying filter to block some of these attacks. Here are some tips we used to defend a recent attack as part of our advanced server security services.
Apache DDoS Attack
Attacks against Apache or any other HTTP server do not require a significant amount of traffic. I have seen some servers shut down with as little as 1Mbit of traffic. The right request to the right page can generate huge loads on the server causing it to fail. Application design, apache configuration, and other factors all contribute to a relatively low level of traffic causing failures.
Fortunately, there are some things you can do to help defend against DDoS attacks. One of our favorites is using, Nginx, an alternative HTTP server, to handle of the traffic.
Using Nginx to Fight DDoS
I am not going into how to setup Nginx as a reverse proxy system to fight a DDoS attack. There are plenty of tutorials online if you want to do it yourself. What I want to share are the results of using Nginx as well as some tips to get the most out the technique.
Nginx as Reverse Proxy
Due to some technical internals, Nginx is often better at handling high concurrency than Apache. In many cases, we deploy Nginx as a reverse proxy server in front of the apache system. By tuning variables in Nginx, we can often withstand smaller attacks.
If the attacks are larger, you may be able to filter traffic in Nginx using IP addresses, user agents, country or other data. You can then drop this traffic so it never reaches your web server.
Serve Static Pages
If the attack is targeting a script or database driven page, you server or database may quickly overload. One of the first things you can do is create a static version of this page:
- Make a backup in a safe place
- Move the real script to an alternate name
- Setup a cron to use wget or curl to create a static version of the page at needed intervals.
Even if your cron runs every minute, this will still result in far fewer hits to the database than runing the script. This is a quick and easy way to gain significantly scalability of a page.
Serve Targeted Pages from Nginx
In a recent case, we were seeing over 2000 req/sec to apache for a couple of pages. Even with our static page trick and using Nginx as a reverse proxy, the system was still struggling.
In cases where attackers are hitting specific pages, you can move static copies of these pages to your Nginx proxy. Using the Location directive, you can set Nginx to handle these files. The benefit is now your Nginx proxy is handling the bulk of the load, leaving your Apache server to do its work.
h2. Use a RAM Disk
Recently, we started using /dev/shm (RAM) as the location for our static files. By moving the target files from the primary server to an Nginx reverse proxy and serving them from RAM, we were able to handle 1000’s of req/sec on minimal hardware. This reduces disk IO issues can can really serve things very quickly.
User Agent Blocking
In our most recent attack, we discovered that >60% of the attacks had a specific User-Agent. This User-Agent was largely unique. Very little < .1% legitimate traffic identified itself as this User-Agent.
As a result, we used Nginx’s easy filtering techniques to send a 500 error to anyone with this User-Agent. You could optionally drop the traffic or redirect it. I send 500 errors as I’ve seen some attacks change as soon as the attackers start getting no response or 403 errors.
IP Blocking
Don’t forget iptables or your firewall. In some cases, drastic measures may be needed. For example, in one case we blocked certain countries and just dropped their traffic. This reduced the attack by 75% and made it easy for other measures to keep the site running strong.
Your Tips and Questions
This is just a quick overview of some tricks we use. Please share you own. DDoS against HTTP can be hard to block when you are leasing equipment and don’t have the ability to deploy your own filtering appliances. There are some new DDoS prevention services emerging but I am not sure how well they handle HTTP level attacks.