Bad bots can often overload servers and cause issues with websites, leading to decreased performance and potential disruptions. While various bot lists are available online, it is important to note that many of these lists are outdated and include bots that are not actively causing any problems.
To effectively identify and handle bad bots, relying on web server logs is a preferred approach. By analyzing the server logs, you can generate a sub-log that specifically captures bot hits. If you are using cPanel, the following command can be executed:
grep -hi bot /usr/local/apache/domlogs/*-ssl_log > bot.log
This command will create a new log file, ‘bot.log,’ which will contain all the hits that include the keyword ‘bot.’ The ‘-h’ flag is used to exclude the filename from the output.
In cases where you have multiple sites or larger logs, it might be beneficial to filter your logs and exclude the major well-known good bots. This can be achieved by utilizing the following command:
grep -hi bot /usr/local/apache/domlogs/*-ssl_log | grep -viE "bing|google|apple"
By employing this command, you will obtain a log file that excludes hits from major good bots like Bing, Google, and Apple.
Once you have the ‘bot.log’ file, you can process it using GoAccess, a comprehensive log analyzer. To generate a detailed report of the bots hitting your site, follow these steps:
- Launch GoAccess.
- Press ‘7’ to access the ‘Top Visitors’ section.
- Press ‘Enter’ to generate a detailed report specifically for the bots.
By utilizing this workflow, you can effectively identify and analyze the bot activity on your website, allowing you to take necessary measures to mitigate any negative impact caused by bad bots. If you are running cPanel, checkout our our How to Block Bad Blocks on cPanel post.