| Author |
 Topic  |
|
|
chinois
Starting Member
9 Posts |
 Posted - 01 July 2008 : 12:36:53
|
I know there is a slightly related topic in mods, but this has zero to do with mods, so I am putting my questions in this section.
What is the best way to get googlebot to index your topics properly?
I have a forum with tens of thousands of topics, but only 400 indexed in google, and it seems like googlebot is having trouble following the proper threads.... I just looked at google webmaster tools and it shows an error 64,000 urls restricted by robots.txt, and when I look at the details it is googlebot trying to follow all of the "Reply" links for all of my threads.... I restricted Reply in robots because google indexing reply boxes looks terrible for new visitors from google, but for some reason googlebot can't figure out to just index the actual topics, not the reply boxes
has anyone figured out how to fix this? do I need to do a ridiculously large sitemap to sitemap my entire huge forum?
I am sad that only 400 pages are indexed yet it looks like google is trying to follow 64000 pages that I don't want it to index instead< |
|
|
chinois
Starting Member
9 Posts |
Posted - 01 July 2008 : 16:06:36
|
this is what my robots looks like:
User-agent: *
Disallow: /forum/post
Disallow: /forum/register
Disallow: /forum/members
Disallow: /forum/default
Disallow: /forum/faq
Disallow: /forum/search
Disallow: /forum/active
Disallow: /forum/disclaimer.asp
Disallow: /forum/pop_profile
Disallow: /admin/
Disallow: /cgi-bin/
Disallow: /TableEditor/
Disallow: /WebLogs/
Disallow: /archive/
Disallow: /forum/cal.asp
Disallow: /forum/vid.asp
Disallow: /forum/post.asp
Disallow: /forum/source.asp
Disallow: /forum/login.asp
Disallow: /forum/faq.asp
Disallow: /forum/post_info.asp
Disallow: /forum/pop_profile.asp
Disallow: /forum/pop_avatarlegend.asp
Disallow: /forum/members.asp
Disallow: /forum/active.asp
Disallow: /forum/pop_mail.asp
Disallow: /forum/pop*.asp < |
 |
|
|
HuwR
Forum Admin
United Kingdom
20584 Posts |
Posted - 01 July 2008 : 16:47:08
|
| you need to edit the code that writes the reply links and add the rel="nofollow" attribute to them< |
|
|
|
bobby131313
Senior Member
USA
1163 Posts |
Posted - 01 July 2008 : 20:25:21
|
You need to run your robots.txt through a validator, Google it there's dozens of them.
You have default in there... why? That's the main page the spidering would start. It all funnels down from there and you're trying to block it. That's most of your problem, the topics that are indexed are probably linked to from off your site.< |
Switch the order of your title tags |
|
|
|
chinois
Starting Member
9 Posts |
Posted - 01 July 2008 : 21:23:57
|
thanks Hu and bobby
I didn't want default to be indexed or followed, as I have one main topic and it is in /forum/forum.asp?FORUM_ID=1
my domains index.htm is just a doorway page that has one link directly to the above, and all topics can be reached from /forum/forum.asp?FORUM_ID=1, so I don't want google to bother with /forum/default at all
and HuwR, so nofollow is better to use than robots I guess? I'll give that a try, what is the filename that contains that code?< |
|
|
|
HuwR
Forum Admin
United Kingdom
20584 Posts |
Posted - 02 July 2008 : 03:48:56
|
quote:
so nofollow is better to use than robots I guess?
No, they should be used in conjunction with each other, you may want some links on a page followed but not others, robots.txt will only allow you to use no follow on a general basis not a link by link basis< |
|
|
|
Lon2
Junior Member
USA
151 Posts |
Posted - 13 February 2009 : 16:55:47
|
This is similar to this topic:
I noticed Google has crawled and listed our topics as well as the exact same topic but with a REPLY on it right under the listing in Google. Is that because we're using the Quick Reply feature and Google is seeing it as another page somehow? Here's an example.
And if this is the case, can we stop crawlers from adding the second REPLY page by adding "nofollow" somewhere in the code before the Quick Reply feature in topic.asp?
Also, was anyone aware that the Quick Reply feature does not display in Firefox and Safari browsers?< |
Edited by - Lon2 on 13 February 2009 19:22:44 |
|
|
|
AnonJr
Moderator
United States
5768 Posts |
Posted - 13 February 2009 : 20:26:31
|
| ... I'm browsing this now in Firefox, and I'm using the "Quick Reply" box. It worked in Chrome (Webkit) yesterday too. Are you talking about here or your forum? If the latter, start a new topic on that in the appropriate forum.< |
|
|
|
Lon2
Junior Member
USA
151 Posts |
Posted - 17 February 2009 : 14:59:03
|
quote:
Originally posted by Lon2
This is similar to this topic:
I noticed Google has crawled and listed our topics as well as the exact same topic but with a REPLY on it right under the listing in Google. Is that because we're using the Quick Reply feature and Google is seeing it as another page somehow? Here's an example.
Does anyone have any insight to this?
Can someone that is not using Quick Reply please check their site in Google to see if it shows the second 'REPLY' topic under the original topic in the Google results?< |
|
|
|
Lon2
Junior Member
USA
151 Posts |
|
|
AnonJr
Moderator
United States
5768 Posts |
Posted - 17 February 2009 : 16:53:35
|
I use it, I just don't obsess over SEO. But that's not a discussion I want to get into - my opinions are considered anathema by many.  < |
|
|
| |
Topic |
|
Topic Locked
