Bots vs Browsers

Database and commentary on public bots and user agents.

IP Addresses of Search Engine Spiders

Tracks the IP addresses of search engine spiders and user-agents. This service is helpful to search engine optimizers and site owners.

Robots and Crawlers DB

Listings of robot agent strings with descriptions and homepage links.


Provides information about how robots.txt files work. Includes a robots.txt generator and a robots.txt analyzer.