Every new site wants to get their piece of Google, Yahoo, & Alexa. Yes I said it, Alexa ;) There is a bit of a risk when allowing these bots to traverse your endless universe of pages, posts, and other niceties that you want to get out there an noticed.…