Every new site wants to get their piece of Google, Yahoo, & Alexa.  Yes I said it, Alexa 😉 There is a bit of a risk when allowing these bots to traverse your endless universe of pages, posts, and other niceties that you want to get out there an noticed.  Keep in mind, these 3 Search Engines arent the only ones looking for your site and content.  There are millions of bots out there just sniffin around, and while most are harmless “Let’s get your site listed in my directory” variety, there are some that look to do harm.  Trust me when I tell you, you don’t want them finding out you’ve left a massive back-door open to your site’s shopping cart. Here is how to protect your site against these malicious bots. NOTE: As of October 1st, 2015 this will block the baddest of the bads.  I will attempt to update on a monthly basis.

Apache Bad-Bot Blocker

Create a .htaccess file in the root of your site, if one does not already exist, and add the following to it:

nGinx Bad-Bot Blocker

Open up your sites nginx.conf (usually located in /etc/nginx) file and add the following code to it: The following code will download the latest SFS configuration.  I have mine setup to run nightly through cron.  Make sure to change the PROFILE and STORE variables to match what you want.

IIS Bad-Bot Blocker

Open up your site’s web.config file and add the following to the rewrite section: