Allowing spiders into private forums - Posted (1112 Views)
endomorph
Junior Member
endomorph
Posts: 128
128
Can anyone help me please and sorry if I have posted in the wrong forum.

I am sure that in the past there was a MOD to allow SE spiders into private forums so they could still be spidered.
It was either a code modification or instructions on how to set up a membership for each spider.
Can anyone enlighten me as I want to lock some of my forums down but I still want them spidered.
Cheers all. <
Need help with your Snitz ? Most Snitz & ASP custom coding undertaken. Email for info | Search Engine Optimisation
Last edited by OneWayMule on 21 February 2006, 04:45
 Sort direction, for dates DESC means newest first  
 Page size 
Posted
Davio
Development Team Member
Davio
Posts: 12217
12217
If you are going to make parts of your forum private, why would you still want it spidered? If it is spidered the information contained in the private forums won't be private.<
Support Snitz Forums
Posted
Podge
Support Moderator
Podge
Posts: 3776
3776
There are legimitate reasons for allowing robots into secure areas of the forum.
The safest way is to amend the chkForumAccess function in inc_func_secure.asp to allow robots based on their user agent and i.p. address. Both user agent and i.p. address can be spoofed and you will be compromising the security of the data in those forums. Weigh up the consequences before you proceed.<
Podge. The Hunger Site - Click to donate free food | My Blog | Snitz 3.4.05 AutoInstall (Beta!)
My Mods: CAPTCHA Mod | GateKeeper Mod Tutorial: Enable subscriptions on your board
Warning: The post above or below may contain nuts.
Posted
endomorph
Junior Member
endomorph
Posts: 128
128
Davio,

I thought that question may come up. I want to make the forums private so only paid members can go in. I still want the content spidered as it gives me a lot of pages in the search engines and I'm not too bothered if people make the effort to view through a cache.
Podge,

I thought it may be in there or in topic.asp, but could you expand on how I could validate a spider as my asp is not that up to scratch.
Many thanks.<
Need help with your Snitz ? Most Snitz & ASP custom coding undertaken. Email for info | Search Engine Optimisation
Posted
Davio
Development Team Member
Davio
Posts: 12217
12217
There are legimitate reasons for allowing robots into secure areas of the forum.
Care to share some of those legitimate reasons with me Sir Podge? wink
endomorph, if I could use the search engine to read the content that are in the private forums, why would I need to pay to access it?
I'm not trying to persuade you from doing what you want to do. Just trying to find the logic behind it. If there is no logic behind it, I'll just leave it alone. bigsmile<
Support Snitz Forums
Posted
Podge
Support Moderator
Podge
Posts: 3776
3776
Someone used my correct title at last! Davio, I'd love to tell you but I cannot. Sorry.
endomorph, you do realise that all of the content will be available in Google's cache? Would you not be better off to post excerpts of the content in a public place where prospective customers can read it?<
Podge. The Hunger Site - Click to donate free food | My Blog | Snitz 3.4.05 AutoInstall (Beta!)
My Mods: CAPTCHA Mod | GateKeeper Mod Tutorial: Enable subscriptions on your board
Warning: The post above or below may contain nuts.
Posted
Davio
Development Team Member
Davio
Posts: 12217
12217
Davio, I'd love to tell you but I cannot. Sorry.
Can you give me any legitimate reasons as to why not? wink<
Support Snitz Forums
Posted
endomorph
Junior Member
endomorph
Posts: 128
128
If I stop the search engines from spidering the forums, I would loose around 2000 pages in the serps.
I am using the logic of "People are nosey and want to know what is behind a closed door" to help with donations on my site. The information is interesting, not confidential/secure. Besides, most of my users are not savvy enought to realise they they view a cache of the page.<
Need help with your Snitz ? Most Snitz & ASP custom coding undertaken. Email for info | Search Engine Optimisation
Posted
Davio
Development Team Member
Davio
Posts: 12217
12217
Sounds like it is more ideal to have a mod that pulls a sentence or 2 randomly from topics in your private forums and display them prominently somewhere public.
Or to pull the title of the topics and display them publicly, so you could entice them. Isn't there a mod that displays part of the title of the last topic in a forum on the default.asp page? Mosuing over it would display the full title in a caption.
That could be used to entice persons to join up. Just some ideas anyway. smile<
Support Snitz Forums
Posted
endomorph
Junior Member
endomorph
Posts: 128
128
It's the loss of pages in the SERPS that would be a bigger killer for me. I also need Adsense Mediabot to still go in to make sure the ads remain on target.
Anyway chaps, back to the coding. Could I use this as a basis to start with -

userAgent = Request.ServerVariables("HTTP_USER_AGENT")

'############### Spider
If inStr(userAgent,"bot") > 0 or _
inStr(userAgent,"perl") > 0 or _
inStr(userAgent,"java") > 0 or _
inStr(userAgent,"libw") > 0 or _
inStr(userAgent,"crawl") > 0 or _
inStr(userAgent,"DA ") > 0 or _
inStr(userAgent,"msn") > 0 or _
inStr(userAgent,"google") > 0 or _
inStr(userAgent,"download") > 0 then

<
Need help with your Snitz ? Most Snitz & ASP custom coding undertaken. Email for info | Search Engine Optimisation
Last edited by endomorph on 17 February 2006, 18:44
Posted
Podge
Support Moderator
Podge
Posts: 3776
3776
This is untested but it should give you a starting point. Don't hold me responsible if its not secure or if someone finds a way around it.

Add the following after the line in red on inc_func_secure.asp
Code:

function chkForumAccess(fForum, UserNum, Display)

	if (left(Request.ServerVariables("HTTP_USER_AGENT"), 6) = "Google") and & _
		(Request.ServerVariables("REMOTE_ADDR") > Dot2LongIP("127.0.64.1") AND & _
		Request.ServerVariables("REMOTE_ADDR") < Dot2LongIP("127.0.79.254") then
	chkForumAccess = true
	exit function
	end if

You can alter the user agent bit above to suit your needs and you'll need to find out the i.p. addresses of the bots you want to allow.
Put the following function somwhere on inc_func_secure.asp
Code:

Function Dot2LongIP (ByVal DottedIP)
Dim i, pos
Dim PrevPos, num
If DottedIP = "" Then
    Dot2LongIP = 0
Else
    For i = 1 To 4
        pos = InStr(PrevPos + 1, DottedIP, ".", 1)
        If i = 4 Then 
            pos = Len(DottedIP) + 1
        End If
        num = Int(Mid(DottedIP, PrevPos + 1, pos - PrevPos - 1))
        PrevPos = pos
        Dot2LongIP = ((num Mod 256) * (256 ^ (4 - i))) + Dot2LongIP
    Next
End If
End Function
<
Podge. The Hunger Site - Click to donate free food | My Blog | Snitz 3.4.05 AutoInstall (Beta!)
My Mods: CAPTCHA Mod | GateKeeper Mod Tutorial: Enable subscriptions on your board
Warning: The post above or below may contain nuts.
You Must enter a message