Allowing spiders into private forums

Snitz™ Forums 2000
https://forum.snitz.com/forumTopic/Posts/61070?pagenum=1
05 November 2025, 12:58

Topic

endomorph
Allowing spiders into private forums
17 February 2006, 17:17


Can anyone help me please and sorry if I have posted in the wrong forum.

I am sure that in the past there was a MOD to allow SE spiders into private forums so they could still be spidered.
It was either a code modification or instructions on how to set up a membership for each spider.
Can anyone enlighten me as I want to lock some of my forums down but I still want them spidered.
Cheers all. <

 

Replies ...

Davio
17 February 2006, 17:21


If you are going to make parts of your forum private, why would you still want it spidered? If it is spidered the information contained in the private forums won't be private.<
Podge
17 February 2006, 17:32


There are legimitate reasons for allowing robots into secure areas of the forum.
The safest way is to amend the chkForumAccess function in inc_func_secure.asp to allow robots based on their user agent and i.p. address. Both user agent and i.p. address can be spoofed and you will be compromising the security of the data in those forums. Weigh up the consequences before you proceed.<
endomorph
17 February 2006, 18:07


Davio,

I thought that question may come up. I want to make the forums private so only paid members can go in. I still want the content spidered as it gives me a lot of pages in the search engines and I'm not too bothered if people make the effort to view through a cache.
Podge,

I thought it may be in there or in topic.asp, but could you expand on how I could validate a spider as my asp is not that up to scratch.
Many thanks.<
Davio
17 February 2006, 18:08


There are legimitate reasons for allowing robots into secure areas of the forum.
Care to share some of those legitimate reasons with me Sir Podge? wink
endomorph, if I could use the search engine to read the content that are in the private forums, why would I need to pay to access it?
I'm not trying to persuade you from doing what you want to do. Just trying to find the logic behind it. If there is no logic behind it, I'll just leave it alone. bigsmile<
Podge
17 February 2006, 18:21


Someone used my correct title at last! Davio, I'd love to tell you but I cannot. Sorry.
endomorph, you do realise that all of the content will be available in Google's cache? Would you not be better off to post excerpts of the content in a public place where prospective customers can read it?<
Davio
17 February 2006, 18:25


Davio, I'd love to tell you but I cannot. Sorry.
Can you give me any legitimate reasons as to why not? wink<
endomorph
17 February 2006, 18:25


If I stop the search engines from spidering the forums, I would loose around 2000 pages in the serps.
I am using the logic of "People are nosey and want to know what is behind a closed door" to help with donations on my site. The information is interesting, not confidential/secure. Besides, most of my users are not savvy enought to realise they they view a cache of the page.<
Davio
17 February 2006, 18:30


Sounds like it is more ideal to have a mod that pulls a sentence or 2 randomly from topics in your private forums and display them prominently somewhere public.
Or to pull the title of the topics and display them publicly, so you could entice them. Isn't there a mod that displays part of the title of the last topic in a forum on the default.asp page? Mosuing over it would display the full title in a caption.
That could be used to entice persons to join up. Just some ideas anyway. smile<
endomorph
17 February 2006, 18:32


It's the loss of pages in the SERPS that would be a bigger killer for me. I also need Adsense Mediabot to still go in to make sure the ads remain on target.
Anyway chaps, back to the coding. Could I use this as a basis to start with -

userAgent = Request.ServerVariables("HTTP_USER_AGENT")

'############### Spider
If inStr(userAgent,"bot") > 0 or _
inStr(userAgent,"perl") > 0 or _
inStr(userAgent,"java") > 0 or _
inStr(userAgent,"libw") > 0 or _
inStr(userAgent,"crawl") > 0 or _
inStr(userAgent,"DA ") > 0 or _
inStr(userAgent,"msn") > 0 or _
inStr(userAgent,"google") > 0 or _
inStr(userAgent,"download") > 0 then

<
Podge
17 February 2006, 18:50


This is untested but it should give you a starting point. Don't hold me responsible if its not secure or if someone finds a way around it.

Add the following after the line in red on inc_func_secure.asp
Code:

function chkForumAccess(fForum, UserNum, Display)

	if (left(Request.ServerVariables("HTTP_USER_AGENT"), 6) = "Google") and & _
		(Request.ServerVariables("REMOTE_ADDR") > Dot2LongIP("127.0.64.1") AND & _
		Request.ServerVariables("REMOTE_ADDR") < Dot2LongIP("127.0.79.254") then
	chkForumAccess = true
	exit function
	end if

You can alter the user agent bit above to suit your needs and you'll need to find out the i.p. addresses of the bots you want to allow.
Put the following function somwhere on inc_func_secure.asp
Code:

Function Dot2LongIP (ByVal DottedIP)
Dim i, pos
Dim PrevPos, num
If DottedIP = "" Then
    Dot2LongIP = 0
Else
    For i = 1 To 4
        pos = InStr(PrevPos + 1, DottedIP, ".", 1)
        If i = 4 Then 
            pos = Len(DottedIP) + 1
        End If
        num = Int(Mid(DottedIP, PrevPos + 1, pos - PrevPos - 1))
        PrevPos = pos
        Dot2LongIP = ((num Mod 256) * (256 ^ (4 - i))) + Dot2LongIP
    Next
End If
End Function
<
endomorph
17 February 2006, 19:09


Podge,

OK I will give yours a try, but I was thinking maybe somethine like this taken from topic.asp -

Code:
userAgent = Request.ServerVariables("HTTP_USER_AGENT")

If inStr(userAgent,"bot") > 0 then 
        chkForumAccess = true

elseif mLev = 4 then
	AdminAllowed = 1
	ForumChkSkipAllowed = 1
elseif mLev = 3 then
	if chkForumModerator(Forum_ID, chkString(strDBNTUserName,"decode")) = "1" then
		AdminAllowed = 1
		ForumChkSkipAllowed = 1
	else
		if lcase(strNoCookies) = "1" then
			AdminAllowed = 1
			ForumChkSkipAllowed = 0
		else
			AdminAllowed = 0
			ForumChkSkipAllowed = 0
		end if
	end if
elseif lcase(strNoCookies) = "1" then
 	AdminAllowed = 1
	ForumChkSkipAllowed = 0
else   
 	AdminAllowed = 0
	ForumChkSkipAllowed = 0
end if
end if

Red code is original, green code is new. Would this work ? Maybe not as secure as yours.<
Podge
17 February 2006, 21:30


The only way to find out is to try. You can get an extension for Firefocks which will allow you to change the user agent. Its called User Agent Switcher (funnily enough).<
Davio
18 February 2006, 03:22


You mean, Firefox Podge, don't you? wink<
Podge
18 February 2006, 04:59


Yes.<
MarkJH
18 February 2006, 09:45


Care to share some of those legitimate reasons with me Sir Podge?
Google Adsense?<
Podge
18 February 2006, 10:30


I'm not sayin' nothin.<
MarkJH
18 February 2006, 12:07


Lord Mark at your service. tongue<
Podge
18 February 2006, 12:13


Reveal hidden contentIntellitext, Yahoo Ads, Clicksor, etc.
Mum's the word.<
MarkJH
18 February 2006, 14:23


Hehe. <
endomorph
20 February 2006, 08:28


Thanks Podge worked a treat<
endomorph
20 February 2006, 08:38


Thanks Podge your the don ! OK now if I may, a little extension to your code.
On my site I have removed the IQC field from everywhere and am using it to mark members as "Paid" if they have donated. Now I want to allow those members into certain forums that I lock off. To save adding each paid member to each locked off forum's "Allowed Members" list, I want to use the ICQ field.
How would you do this in inc_func_secure ?
If viewer is not signed in, give them error page
If viewer is signed in and has 'Paid' in the ICQ field, then chkForumAccess = true
end if
end if

Cheers<
© 2000-2021 Snitz™ Communications