pdrg ; here's the response I got from Google ; I've already replied that this reply was not satisfactory in my opinion.
quote:
---------- Forwarded message ----------
From: help@google.com <help@google.com>
Date: Jun 14, 2005 4:28 AM
Subject: Re: [#28013268] Googlebot ignoring robots.txt
To: Marcel Göertz
Hi Marcel,
Thank you for bringing this to our attention. We are sorry about any excessive strain Google is causing your web servers. If you would like us to slow down the rate at which Googlebot crawls your site, please send us a copy of your most recent weblog that lists Googlebot, and the URL of the host affected. We can then pass your request on to our engineers.
Regards,
The Google Team
You can see my robots.txt here.
I've changed one thing however ; it used to be this:
Disallow: vid.asp
Disallow: post.asp
Disallow: source.asp
Disallow: login.asp
Disallow: faq.asp
Disallow: post_info.asp
Disallow: pop_profile.asp
Disallow: pop_avatarlegend.asp
Disallow: members.asp
Disallow: active.asp
I changed this to this:
Disallow: /vid.asp
Disallow: /post.asp
Disallow: /source.asp
Disallow: /login.asp
Disallow: /faq.asp
Disallow: /post_info.asp
Disallow: /pop_profile.asp
Disallow: /pop_avatarlegend.asp
Disallow: /members.asp
Disallow: /active.asp
So, I added the / before the filename....perhaps thát was the cause of the problem.
Topic Locked
Posted - 14 June 2005 : 06:15:27
; has anybody tried the 