| Author |
 Topic  |
|
|
philwhite
Starting Member
Germany
47 Posts |
 Posted - 04 January 2005 : 15:23:17
|
I just know this is going to sound very foolish indeed, but here goes:
We've just put up a Snitz forum to replace a creaky old proprietary one and imported all the old threads with a script. Not too pretty, but it worked by dummying up a lot of the missing information.
Just exactly how will Google burrow down through 5000+ old topics (yes, we want them all on Google) when very few of them have even been viewed yet in the new forum? How, indeed, does Google actually index dynamic pages when the actual content is not physically present on the site?
Above all, do I have to do anything server-side or with respect to the Snitz code to make sure everything is found when the bots get round to re-indexing?
Sorry for my ignorance, but it comes naturally to me...
www.wordwizard.com if you're interested in language and don't mind a pretty ugly forum. The colours were not my idea! |
Phil White |
Edited by - philwhite on 04 January 2005 17:06:16 |
|
|
muzishun
Senior Member
United States
1079 Posts |
Posted - 04 January 2005 : 17:28:47
|
I'm not sure that Google will crawl every topic on your site. I've got a stat monitor running on my domain, and here's the breakdown of hits on the site:
6 different robots Hits
Inktomi Slurp 232
Googlebot (Google) 37
GigaBot 24
Alexa (IA Archiver) 13
SurveyBot 1
We've got a few hundred pages, counting the topics. I don't know too much about how the different spiders index sites, but I don't think they index each and every page. |
Bill Parrott
Senior Web Programmer, University of Kansas
Co-Owner and Code Monkey, Eternal Second Designs (www.eternalsecond.com)
Personal Website (www.chimericdream.com) |
 |
|
|
dayve
Forum Moderator
USA
5820 Posts |
Posted - 04 January 2005 : 19:56:06
|
I have been using a script for the past 2-3 years called Topic Maker which creates static pages of my forum topics which make it easer for a crawler/robot to index. These static pages forward to the actual forum pages. Since starting this long ago, Google has indeed indexed a lot of my forum discussions. The addition of having an RSS Feed I think also helps from some articles that I've come across in the past, but I'm not for sure if it's had a real impact.
Here is an example of a Google indexed page.
http://www.google.com/search?hl=en&lr=&safe=on&q=trent+reznor+selling+home
You should see a topic from Burning Souls Forum using these keywords.
I think you can find Topic Maker at http://www.snitzbitz.com although I think Google can handle dynamic pages on its own now. |
 |
Edited by - dayve on 04 January 2005 20:15:32 |
|
|
|
redbrad0
Advanced Member
USA
3725 Posts |
|
|
muzishun
Senior Member
United States
1079 Posts |
Posted - 04 January 2005 : 23:42:34
|
| It is being indexed, but I think what Phil was looking for is something to index each individual topic. I've made sure that the site itself is indexed on Google, and I think the Topic Maker that dayve suggested is probably the best way to index all topics if you want to do that. I've got it on my forum as well, and it generates HTML pages for every topic. I agree that spiders can index asp, but I'm not sure about their ability to crawl through a database. |
Bill Parrott
Senior Web Programmer, University of Kansas
Co-Owner and Code Monkey, Eternal Second Designs (www.eternalsecond.com)
Personal Website (www.chimericdream.com) |
|
|
|
johnny bravo
New Member
78 Posts |
Posted - 05 January 2005 : 03:06:40
|
I use Google Adds on my site and they come up on every page. By doing this I am forcing Google to index ever topic as it changes and as new ones are created. Plus I earn cash from doing it.
I don't really know if this works or not but how could the Google adds show relevant adds if was not having to read the page first.
On a new topic it takes about 15mins for relevant ads to appear.
www.elvethamheathforum.info
|
|
|
|
philwhite
Starting Member
Germany
47 Posts |
Posted - 05 January 2005 : 05:54:01
|
Brad,
Google appears not to automatically index topics as you can see from this Snitz forum itself.
Try this:
http://www.google.com/search?hl=en&q=%22I+have+a+pretty+annoying+problem+with+my+subscriptions%22
and it doesn't find a topic from 18 April last year, namely
http://forum.snitz.com/forum/topic.asp?TOPIC_ID=52573
The degree to which this Snitz forum has been indexed appears to be very low.
If you put the following into Google:
Snitz site:forum.snitz.com
it returns only 180 hits. I would expect that the word "Snitz" comes up on more than 180 topics in this forum! I guess that Google only finds the pages that people are actually viewing when it is crawling the site.
Dave,
Rather than keeping a static copy of all 5000+ threads, I was playing with the idea of automatically generating a link to each (non-hidden) topic on some sort of a links page, but Topic Maker may do the job (I'm a little concerned about web space, though!)
Johnny
That, of course, is ideal. The ad solution would not work for us though, as we are opposed to ads on our site!
It seems to me that nobody really understands the mechanisms involved here. |
Phil White |
|
|
|
johnny bravo
New Member
78 Posts |
Posted - 08 January 2005 : 10:52:29
|
Even thought my site is hit by Google bots evey day it still does not appear in any google searches? Does anybody know how look it takes for this to start happening? My site is about 1 month old.
www.elvethamheathforum.info
|
|
|
|
philwhite
Starting Member
Germany
47 Posts |
Posted - 13 January 2005 : 12:23:33
|
Sorry to bang on about this, but I'm still none the wiser.
As an example, put "The color article is very interesting" (with quotes) into Google, and it delivers two results from our site, both of which show the post.asp screen. It appears that Google has found the new site (only a few days after it went up!) and is slowly gathering information, but it seems utterly random. Older posts made since the site was put up on January 1, but earlier than the one quoted, have not been found. The imported posts haven't been found at all, and anything that is found appears to be a link to post.asp. It seems that it only finds posts which are being edited at the moment googlebot visits the site.
In contrast, I can't seem to get a single hit on any phrase I've tried in Johnny Bravo's forum, even though the forum's "much" older than ours.
As soon as the provider makes the relevant directory writable, I'll get Topic Maker installed, which should solve the issue for us, but I'd love to understand the logic and mechanics behind the phenomenon above. |
Phil White |
Edited by - philwhite on 13 January 2005 15:50:14 |
|
|
|
Scottie
Starting Member
24 Posts |
Posted - 22 February 2005 : 17:44:27
|
It seems as though the answer to this would be an RSS or Atom feed generator asp script. Does anyone have one already written?
|
|
|
|
Podge
Support Moderator
Ireland
3776 Posts |
|
| |
Topic |
|
Topic Locked
