http://majestic12.co.uk/bot.php


https://support.google.com/webmasters/answer/182072


http://www.bing.com/bingbot.htm