cobrachen Starting Member
48 Posts
weeweeslap Senior Member
USA 1077 Posts
quote:Originally posted by weeweeslapit's a robot, make a robots.txt file and disallow that bot.
Podge Support Moderator
Ireland 3775 Posts
quote:Originally posted by Podgehttp://www.robotstxt.org/wc/robots.htmlIts a standard exclusion protocol which search engine spiders comply with (sometimes). The pages listed in your robots.txt file will usually be ignored by search engine spiders. If you know the user-agent of the spider you can tell it that all pages are off limits to it.See the link above for specific instructions.