I'd like to change this tag so that the robots only harvest a particular page or pages in my portal mod. This is what it currently looks like: "<meta name=""robots"" content=""all"">" & vbNewline & _
To exclude all files except one
This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "docs", and leave the one file in the level above this directory:
User-agent: *
Disallow: /~joe/docs/
Alternatively you can explicitly disallow all disallowed pages:
User-agent: *
Disallow: /~joe/private.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html
Topic Locked
Posted - 06 November 2003 : 06:08:52
