Allowing spiders into private forums

The Forum has been Updated

The code has been upgraded to the latest .NET core version. Please check instructions in the Community Announcements about migrating your account.

Allowing spiders into private forums - Postet den 17 Feb 2006 (1113 Views)

endomorph

Junior Member

Innlegg: 128

128

Can anyone help me please and sorry if I have posted in the wrong forum.

I am sure that in the past there was a MOD to allow SE spiders into private forums so they could still be spidered.
It was either a code modification or instructions on how to set up a membership for each spider.
Can anyone enlighten me as I want to lock some of my forums down but I still want them spidered.
Cheers all. <

Need help with your Snitz ? Most Snitz & ASP custom coding undertaken. Email for info | Search Engine Optimisation

Sist redigert av

Postet den 17 Feb 2006

Davio

Development Team Member

Innlegg: 12217

12217

If you are going to make parts of your forum private, why would you still want it spidered? If it is spidered the information contained in the private forums won't be private.<

Support Snitz Forums

Postet den 17 Feb 2006

Podge

Support Moderator

Innlegg: 3776

3776

There are legimitate reasons for allowing robots into secure areas of the forum.
The safest way is to amend the chkForumAccess function in inc_func_secure.asp to allow robots based on their user agent and i.p. address. Both user agent and i.p. address can be spoofed and you will be compromising the security of the data in those forums. Weigh up the consequences before you proceed.<

Podge. The Hunger Site - Click to donate free food | My Blog | Snitz 3.4.05 AutoInstall (Beta!)
My Mods: CAPTCHA Mod | GateKeeper Mod Tutorial: Enable subscriptions on your board
Warning: The post above or below may contain nuts.

Postet den 17 Feb 2006

endomorph

Junior Member

Innlegg: 128

128

Davio,

I thought that question may come up. I want to make the forums private so only paid members can go in. I still want the content spidered as it gives me a lot of pages in the search engines and I'm not too bothered if people make the effort to view through a cache.
Podge,

I thought it may be in there or in topic.asp, but could you expand on how I could validate a spider as my asp is not that up to scratch.
Many thanks.<

Need help with your Snitz ? Most Snitz & ASP custom coding undertaken. Email for info | Search Engine Optimisation

Postet den 17 Feb 2006

Davio

Development Team Member

Innlegg: 12217

12217

There are legimitate reasons for allowing robots into secure areas of the forum.

Care to share some of those legitimate reasons with me Sir Podge?

endomorph, if I could use the search engine to read the content that are in the private forums, why would I need to pay to access it?
I'm not trying to persuade you from doing what you want to do. Just trying to find the logic behind it. If there is no logic behind it, I'll just leave it alone.

Support Snitz Forums

Postet den 17 Feb 2006

Podge

Support Moderator

Innlegg: 3776

3776

Someone used my correct title at last! Davio, I'd love to tell you but I cannot. Sorry.
endomorph, you do realise that all of the content will be available in Google's cache? Would you not be better off to post excerpts of the content in a public place where prospective customers can read it?<

Postet den 17 Feb 2006

Podge

Support Moderator

Innlegg: 3776

3776

This is untested but it should give you a starting point. Don't hold me responsible if its not secure or if someone finds a way around it.

Add the following after the line in red on inc_func_secure.asp

Code:


function chkForumAccess(fForum, UserNum, Display)

	if (left(Request.ServerVariables("HTTP_USER_AGENT"), 6) = "Google") and & _
		(Request.ServerVariables("REMOTE_ADDR") > Dot2LongIP("127.0.64.1") AND & _
		Request.ServerVariables("REMOTE_ADDR") < Dot2LongIP("127.0.79.254") then
	chkForumAccess = true
	exit function
	end if

You can alter the user agent bit above to suit your needs and you'll need to find out the i.p. addresses of the bots you want to allow.
Put the following function somwhere on inc_func_secure.asp

Code:


Function Dot2LongIP (ByVal DottedIP)
Dim i, pos
Dim PrevPos, num
If DottedIP = "" Then
    Dot2LongIP = 0
Else
    For i = 1 To 4
        pos = InStr(PrevPos + 1, DottedIP, ".", 1)
        If i = 4 Then 
            pos = Len(DottedIP) + 1
        End If
        num = Int(Mid(DottedIP, PrevPos + 1, pos - PrevPos - 1))
        PrevPos = pos
        Dot2LongIP = ((num Mod 256) * (256 ^ (4 - i))) + Dot2LongIP
    Next
End If
End Function