| Author |
 Topic  |
|
|
jfitz
Junior Member
USA
345 Posts |
 Posted - 23 June 2003 : 15:44:03
|
Is there a simple way to allow Google to crawl my registration-required forums without opening up a big security hole? I was thinking maybe one could add something to inc_header.asp to recognize a Google crawler ip and set m_lev so that it would appear that the robot was logged on.
I'm looking for suggestions, comments, or warnings about how or why not to attempt this.
Thanks. |
--Jördan
It's a wasted day if you don't spend at least part of it flying upside down. |
|
|
VodkaFish
Average Member
USA
654 Posts |
Posted - 23 June 2003 : 16:25:17
|
| Since people would be able to access the cached page of your registration-required site - doesn't that defeat the purpose? |
v ø d k â f ï § h |
 |
|
|
lofty
Junior Member
USA
158 Posts |
Posted - 23 June 2003 : 16:32:51
|
quote:
Since people would be able to access the cached page of your registration-required site - doesn't that defeat the purpose?
But you could always use Google's no-cache meta tag to avoid having it cache your pages. I would check both the IP and the HTTP_USER_AGENT.
You might be careful though, because I'm not sure if Google would consider this to be cloaking. |
|
|
|
jfitz
Junior Member
USA
345 Posts |
Posted - 23 June 2003 : 16:53:28
|
| The reason for the question is that Google has started a program called AdSense where you show their ads on your site and earn on a per-click basis, but for the ads to work, they need to crawl your pages so the ads can be context sensitive. So I was trying to figure how to let Google crawl a registration-required forum without being itself registered. As far as the cached pages do, I don't think that's a big deal, but that is something I had not considered. |
--Jördan
It's a wasted day if you don't spend at least part of it flying upside down. |
|
|
|
jfitz
Junior Member
USA
345 Posts |
Posted - 23 June 2003 : 17:01:47
|
quote:
Originally posted by lofty
But you could always use Google's no-cache meta tag to avoid having it cache your pages. I would check both the IP and the HTTP_USER_AGENT.
You might be careful though, because I'm not sure if Google would consider this to be cloaking.
Googles Webmaster FAQ explains the no-cache tag, so I would presume that's considered fair play by them. |
--Jördan
It's a wasted day if you don't spend at least part of it flying upside down. |
|
|
|
lofty
Junior Member
USA
158 Posts |
Posted - 23 June 2003 : 17:18:47
|
Well, yes, the no-cache tag is fair play, but that's not what I was thinking of. Giving Google a different page than what the average user sees could be considered cloaking, which they frown upon very heavily. I would send them a friendly email asking if it's ok and see if they respond.
You're probably in the clear though because you are serving the exact same content that your paid users are seeing. |
|
|
|
jfitz
Junior Member
USA
345 Posts |
Posted - 23 June 2003 : 17:21:24
|
| Thanks for the comment, lofty. Yes, it would be the exact page. I just want to avoid making the robot have to log in. Google suggests triggering on the USER_AGENT, so I'm going to give it a try and see what my web site logs show. |
--Jördan
It's a wasted day if you don't spend at least part of it flying upside down. |
|
|
|
VodkaFish
Average Member
USA
654 Posts |
Posted - 23 June 2003 : 17:48:46
|
jfitz - regarding their Adsense program, most people say they receive prompt, polite emails from them. So I would do exactly what lofty suggested - write them  If they think your idea is ok, then you'll be fine implementing it.
And yes, you can mark their IP range and set the mlev so it can browse. |
v ø d k â f ï § h |
|
|
|
jfitz
Junior Member
USA
345 Posts |
Posted - 23 June 2003 : 17:53:33
|
I've been using AdWords for several months, and the've been very helpful and cooperative. I have a message in already with the AdSense tech folks about using their service on SSL; I'll add a second inquiry about what lofty suggested.
Thanks for the suggestions. |
--Jördan
It's a wasted day if you don't spend at least part of it flying upside down. |
|
|
|
jfitz
Junior Member
USA
345 Posts |
Posted - 27 June 2003 : 23:17:59
|
I got the reply from AdSense, and this is what they said:
We can crawl the password-protected pages if you check the HTTP_USER_AGENT for our content targeted crawler which is
'Mediapartners-Google*'
I've pretty much figured out what additions / changes to make to inc_header.asp, but need to do some more testing. |
--Jördan
It's a wasted day if you don't spend at least part of it flying upside down. |
|
|
| |
Topic |
|
Topic Locked
