I've always wondered this, but was a little scared to ask. Perhaps now I can.
Will these forums here ever allow search engines to visit? If I'm reading the robots.txt correctly, the major SEs will not visit:
User-agent: *
Disallow: /forum/ # This is an infinite virtual URL space
To my mind there are many reasons why allowing search engines to spider this content would be good, but of course, if it puts too much of a strain on the servers, it might not be a good idea.
Spiders are not intelligent enough to not follow every little linky they can find - and as the boards cross-link everything, they start going through and through and through, basically through every page and every link every month. That's a lot of page impressions, more than is useful imvho!
Topic Locked
Posted - 18 October 2005 : 15:27:19
