| Author |
 Topic  |
|
endomorph
Junior Member
United Kingdom
128 Posts |
 Posted - 17 February 2006 : 17:17:34
|
Can anyone help me please and sorry if I have posted in the wrong forum.
I am sure that in the past there was a MOD to allow SE spiders into private forums so they could still be spidered.
It was either a code modification or instructions on how to set up a membership for each spider.
Can anyone enlighten me as I want to lock some of my forums down but I still want them spidered.
Cheers all.
< |
Need help with your Snitz ? Most Snitz & ASP custom coding undertaken. Email for info | Search Engine Optimisation |
Edited by - OneWayMule on 21 February 2006 04:45:29 |
|
|
Davio
Development Team Member
Jamaica
12217 Posts |
Posted - 17 February 2006 : 17:21:30
|
| If you are going to make parts of your forum private, why would you still want it spidered? If it is spidered the information contained in the private forums won't be private.< |
Support Snitz Forums
|
 |
|
|
Podge
Support Moderator
Ireland
3775 Posts |
|
|
endomorph
Junior Member
United Kingdom
128 Posts |
Posted - 17 February 2006 : 18:07:17
|
Davio,
I thought that question may come up. I want to make the forums private so only paid members can go in. I still want the content spidered as it gives me a lot of pages in the search engines and I'm not too bothered if people make the effort to view through a cache.
Podge,
I thought it may be in there or in topic.asp, but could you expand on how I could validate a spider as my asp is not that up to scratch.
Many thanks.< |
Need help with your Snitz ? Most Snitz & ASP custom coding undertaken. Email for info | Search Engine Optimisation |
|
|
|
Davio
Development Team Member
Jamaica
12217 Posts |
Posted - 17 February 2006 : 18:08:02
|
quote:
There are legimitate reasons for allowing robots into secure areas of the forum.
Care to share some of those legitimate reasons with me Sir Podge? 
endomorph, if I could use the search engine to read the content that are in the private forums, why would I need to pay to access it?
I'm not trying to persuade you from doing what you want to do. Just trying to find the logic behind it. If there is no logic behind it, I'll just leave it alone.  < |
Support Snitz Forums
|
|
|
|
Podge
Support Moderator
Ireland
3775 Posts |
|
|
Davio
Development Team Member
Jamaica
12217 Posts |
Posted - 17 February 2006 : 18:25:00
|
quote:
Someone used my correct title at last!
LoL! Don't let it get to your head now. 
quote:
Davio, I'd love to tell you but I cannot. Sorry.
Can you give me any legitimate reasons as to why not? < |
Support Snitz Forums
|
|
|
|
endomorph
Junior Member
United Kingdom
128 Posts |
Posted - 17 February 2006 : 18:25:15
|
If I stop the search engines from spidering the forums, I would loose around 2000 pages in the serps.
I am using the logic of "People are nosey and want to know what is behind a closed door" to help with donations on my site. The information is interesting, not confidential/secure. Besides, most of my users are not savvy enought to realise they they view a cache of the page.< |
Need help with your Snitz ? Most Snitz & ASP custom coding undertaken. Email for info | Search Engine Optimisation |
|
|
|
Davio
Development Team Member
Jamaica
12217 Posts |
Posted - 17 February 2006 : 18:30:15
|
Sounds like it is more ideal to have a mod that pulls a sentence or 2 randomly from topics in your private forums and display them prominently somewhere public.
Or to pull the title of the topics and display them publicly, so you could entice them.
Isn't there a mod that displays part of the title of the last topic in a forum on the default.asp page? Mosuing over it would display the full title in a caption.
That could be used to entice persons to join up. Just some ideas anyway. < |
Support Snitz Forums
|
|
|
|
endomorph
Junior Member
United Kingdom
128 Posts |
Posted - 17 February 2006 : 18:32:28
|
It's the loss of pages in the SERPS that would be a bigger killer for me. I also need Adsense Mediabot to still go in to make sure the ads remain on target.
Anyway chaps, back to the coding. Could I use this as a basis to start with -
userAgent = Request.ServerVariables("HTTP_USER_AGENT")
'############### Spider
If inStr(userAgent,"bot") > 0 or _
inStr(userAgent,"perl") > 0 or _
inStr(userAgent,"java") > 0 or _
inStr(userAgent,"libw") > 0 or _
inStr(userAgent,"crawl") > 0 or _
inStr(userAgent,"DA ") > 0 or _
inStr(userAgent,"msn") > 0 or _
inStr(userAgent,"google") > 0 or _
inStr(userAgent,"download") > 0 then
< |
Need help with your Snitz ? Most Snitz & ASP custom coding undertaken. Email for info | Search Engine Optimisation |
Edited by - endomorph on 17 February 2006 18:44:32 |
|
|
|
Podge
Support Moderator
Ireland
3775 Posts |
Posted - 17 February 2006 : 18:50:43
|
This is untested but it should give you a starting point. Don't hold me responsible if its not secure or if someone finds a way around it.
Add the following after the line in red on inc_func_secure.asp
function chkForumAccess(fForum, UserNum, Display)
if (left(Request.ServerVariables("HTTP_USER_AGENT"), 6) = "Google") and & _
(Request.ServerVariables("REMOTE_ADDR") > Dot2LongIP("127.0.64.1") AND & _
Request.ServerVariables("REMOTE_ADDR") < Dot2LongIP("127.0.79.254") then
chkForumAccess = true
exit function
end if
You can alter the user agent bit above to suit your needs and you'll need to find out the i.p. addresses of the bots you want to allow.
Put the following function somwhere on inc_func_secure.asp
Function Dot2LongIP (ByVal DottedIP)
Dim i, pos
Dim PrevPos, num
If DottedIP = "" Then
Dot2LongIP = 0
Else
For i = 1 To 4
pos = InStr(PrevPos + 1, DottedIP, ".", 1)
If i = 4 Then
pos = Len(DottedIP) + 1
End If
num = Int(Mid(DottedIP, PrevPos + 1, pos - PrevPos - 1))
PrevPos = pos
Dot2LongIP = ((num Mod 256) * (256 ^ (4 - i))) + Dot2LongIP
Next
End If
End Function< |
Podge.
The Hunger Site - Click to donate free food | My Blog | Snitz 3.4.05 AutoInstall (Beta!)
My Mods: CAPTCHA Mod | GateKeeper Mod
Tutorial: Enable subscriptions on your board
Warning: The post above or below may contain nuts. |
|
|
|
endomorph
Junior Member
United Kingdom
128 Posts |
Posted - 17 February 2006 : 19:09:06
|
Podge,
OK I will give yours a try, but I was thinking maybe somethine like this taken from topic.asp -
userAgent = Request.ServerVariables("HTTP_USER_AGENT")
If inStr(userAgent,"bot") > 0 then
chkForumAccess = true
elseif mLev = 4 then
AdminAllowed = 1
ForumChkSkipAllowed = 1
elseif mLev = 3 then
if chkForumModerator(Forum_ID, chkString(strDBNTUserName,"decode")) = "1" then
AdminAllowed = 1
ForumChkSkipAllowed = 1
else
if lcase(strNoCookies) = "1" then
AdminAllowed = 1
ForumChkSkipAllowed = 0
else
AdminAllowed = 0
ForumChkSkipAllowed = 0
end if
end if
elseif lcase(strNoCookies) = "1" then
AdminAllowed = 1
ForumChkSkipAllowed = 0
else
AdminAllowed = 0
ForumChkSkipAllowed = 0
end if
end if
Red code is original, green code is new. Would this work ? Maybe not as secure as yours.< |
Need help with your Snitz ? Most Snitz & ASP custom coding undertaken. Email for info | Search Engine Optimisation |
Edited by - endomorph on 17 February 2006 19:17:29 |
|
|
|
Podge
Support Moderator
Ireland
3775 Posts |
|
|
Davio
Development Team Member
Jamaica
12217 Posts |
|
|
Podge
Support Moderator
Ireland
3775 Posts |
|
|
MarkJH
Senior Member
United Kingdom
1722 Posts |
|
|
Topic |
|
Allowing spiders into private forums
