I've included a robots.txt file on a site for the first time and wanted to get ye to throw an eye over it to make sure I haven't disallowed anything I shouldn't have or omitted anything that should be disallowed. Also, do you have any suggestions for other user-agents which should be disallowed completely, whether they read robots.txt or not?
Search is your friend “I was having a mildly paranoid day, mostly due to the fact that the mad priest lady from over the river had taken to nailing weasels to my front door again.”
Maybe a leecher tool like netvampire, but any decent UA can impersonate others, so the only people you'll keep out with a robots.txt tend to be legitimate traffic!
Topic Locked
Posted - 26 March 2008 : 09:22:26
