Snitz Forums 2000
Snitz Forums 2000
Home | Profile | Register | Active Topics | Members | Search | FAQ
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 Snitz Forums 2000 MOD-Group
 MOD Add-On Forum (W/Code)
 Auto Update Spider ID MOD : Active Users
 New Topic  Topic Locked
 Printer Friendly
Previous Page | Next Page
Author Previous Topic Topic Next Topic
Page: of 12

aspwiz
Junior Member

250 Posts

Posted - 08 April 2004 :  07:21:06  Show Profile  Visit aspwiz's Homepage
Hi.... Javascript (especially related to navigation, and frames) often poses a problem for search engine spiders.

I am happy to offer any SEO (Search Engine Optimisation) advice to members of this forum. Just pop along to my site and post in the SEO forum.

Sorry I've been absent from this mod for a while... I've just been very hecticcally busy!
Go to Top of Page

nickw
Junior Member

Ireland
193 Posts

Posted - 03 June 2004 :  17:42:16  Show Profile
So, here I am with this MOD, and I am getting notification on a new BOT.

Agent="msnbot/0.11 (+http://search.msn.com/msnbot.htm)"

How do I add this to my list?

Thanks,

Nick
Go to Top of Page

masterao
Senior Member

Sweden
1678 Posts

Posted - 03 June 2004 :  19:26:38  Show Profile  Visit masterao's Homepage
Add it to the current list of agents so that it looks like this (addition marked red):

"ZyBorg/||ZYBORG||DEAD LINK CHECKER**" & _
"Mediapartners-Google||GOOGLE ADSENSE||CONTEXTUAL ADVERT BOT" & _
"msnbot||MSNBot||Prototype Web-crawling robot"


Thanks for the tip, nickw.

Jan
===========
FR Portal Forums | Active Users 4.0.20 Mod
Go to Top of Page

nickw
Junior Member

Ireland
193 Posts

Posted - 04 June 2004 :  12:12:12  Show Profile
Excellent Jan ;-) Now we can build this list up :-)

Now it is: Edited 9th June 2004
AgentCsvStr = 	"Mediapartners-Google||GOOGLE ADSENSE||ADSENSE CRAWLER**" & _
			"Googlebot||GOOGLE||SEARCH ENGINE SPIDER**" & _
			"NPBot||Name Protect||Domain Research Bot**" & _
			"Scooter||ALTAVISTA||SEARCH ENGINE SPIDER**" & _
			"ia_archiver||ALEXA||SEARCH ENGINE SPIDER**" & _
			"FAST-WebCrawler||ALL THE WEB||SEARCH ENGINE SPIDER**" & _
			"Speedy Spider||ENTIRE WEB||SEARCH ENGINE SPIDER**" & _
			"ArchitextSpider||EXCITE||SEARCH ENGINE SPIDER**" & _
			"ArchitectSpider||EXCITE||SEARCH ENGINE SPIDER**" & _
			"Ask Jeeves/Teoma||ASK JEEVES / TEOMA||SEARCH ENGINE SPIDER**" & _
			"Slurp/||INKTOMI||SEARCH ENGINE SPIDER**" & _
			"Slurp.so/1.0||INKTOMI||SEARCH ENGINE SPIDER**" & _
			"UltraSeek||INFOSEEK||SEARCH ENGINE SPIDER**" & _
			"InfoSeek Sidewinder||INFOSEEK||SEARCH ENGINE SPIDER**" & _
			"MantraAgent||LOOKSMART||SEARCH ENGINE SPIDER**" & _
			"Lycos_Spider_(T-Rex)||LYCOS||SEARCH ENGINE SPIDER**" & _
			"HenryTheMiragoRobot||MIRAGO||SEARCH ENGINE SPIDER**" & _
			"MSNBOT/0.1||MSN SEARCH||SEARCH ENGINE SPIDER**" & _
			"Gulliver||NORTHERN LIGHT||SEARCH ENGINE SPIDER**" & _
			"Scrubby||SCRUB THE WEB||SEARCH ENGINE SPIDER**" & _
			"teoma_agent1||TEOMA||SEARCH ENGINE SPIDER**" & _
			"marvin/infoseek||WEBSEEK||SEARCH ENGINE SPIDER**" & _
			"SlySearch/1.3||SLYSEARCH||SEARCH ENGINE SPIDER**" & _
			"Szukacz||SZUKACZ.PL||SEARCH ENGINE SPIDER**" & _
			"IE 5.5 Compatible Browser||IE 5.5 Compatible Browser||Unknown**" & _
			"almaden||IBM||RESEARCH BOT**" & _
			"Google CHTML Proxy/1.0||GOOGLE PROXY SERVER||PROXY SERVER**" & _
			"http://grub.org||GRUB||RESEARCH BOT**" & _
			"NutchOrg||NUTCH||OPEN SOURCE SPIDER**" & _
			"InternetSeer.com||INTERNET SEER||WEBSITE MONITORING SERVICE**" & _
			"Baiduspider+||BAIDU||SEARCH ENGINE SPIDER**" & _
			"Xenu Link Sleuth||XENU LINK CHECKER||DEAD LINK CHECKER**" & _
			"Mozilla/5.||Mozilla 5.x||**" & _
			"Mozilla/4.||Mozilla 4.x||**" & _
			"Mozilla/3.||Mozilla 3.x||**" & _
			"NetMonitor/||NetMonitor||WEBSITE MONITORING SERVICE**" & _
			"TurnitinBot/1.5||TurnItInBot||PLAGIARISM RESEARCH BOT**" & _
			"dloader(NaverRobot)||NAVER ROBOT||KOREA TELECOM**" & _
			"ZyBorg/||ZYBORG||DEAD LINK CHECKER**" & _
			"QuepasaCreep||QUEPASA.COM||SEARCH ENGINE SPIDER**" & _
			"Microsoft URL Control||POSSIBLE EMAIL COLLECTOR||POSSIBLE VUNERABILITY SCANNER**" & _
			"Avant Browser||Avant Browser||Avant Browser**" & _
			"msnbot/0.11||MSNBot||Prototype Web-crawling robot**" & _
			"Openbot/3.0||Openfind.com||Prototype Web-crawling robot**"


Nick

Edited by - nickw on 09 June 2004 14:29:54
Go to Top of Page

nickw
Junior Member

Ireland
193 Posts

Posted - 04 June 2004 :  16:58:59  Show Profile
Mmmm... Hi Again Jan...

So today, I get another email... even thought the above is added... ideas?

Email:
Agent="msnbot/0.11 (+http://search.msn.com/msnbot.htm)"
IP: 213.94.253.31


Nick
Go to Top of Page

OneWayMule
Dev. Team Member & Support Moderator

Austria
4969 Posts

Posted - 05 June 2004 :  04:53:46  Show Profile  Visit OneWayMule's Homepage  Send OneWayMule an ICQ Message
You probably have to change this line:
			"msnbot||MSNBot||Prototype Web-crawling robot"

to
			"msnbot/0.11||MSNBot||Prototype Web-crawling robot"

My MODs:
Birthdays - Custom Policy - F.A.Q. Administration - Forum Rules - Guestbook
Links Manager - MyOwnGoogle - Profile Views - Search Log - WebSearch

Useful stuff:
Forum and MOD Installation - MOD Installation Guide - Snitz v3.4.05 Readme - Free ASP Hosts - Support Snitz
Go to Top of Page

masterao
Senior Member

Sweden
1678 Posts

Posted - 05 June 2004 :  07:18:16  Show Profile  Visit masterao's Homepage
I second OWMs suggestion.

Jan
===========
FR Portal Forums | Active Users 4.0.20 Mod
Go to Top of Page

nickw
Junior Member

Ireland
193 Posts

Posted - 05 June 2004 :  09:27:30  Show Profile
Already tried that guys... same problem :(


Nick
Go to Top of Page

masterao
Senior Member

Sweden
1678 Posts

Posted - 05 June 2004 :  16:10:47  Show Profile  Visit masterao's Homepage
I checked the code a little and I missed something. You need to add ** after the entry. ** is used to split the crawler-list into the array.

It should look like this:

"msnbot||MSNBot||Prototype Web-crawling robot**"

Jan
===========
FR Portal Forums | Active Users 4.0.20 Mod
Go to Top of Page

nickw
Junior Member

Ireland
193 Posts

Posted - 06 June 2004 :  09:39:07  Show Profile
Thanks Jan, I'll give that a go, and edit the list above

Nick
Go to Top of Page

MarkJH
Senior Member

United Kingdom
1722 Posts

Posted - 06 June 2004 :  12:54:08  Show Profile  Visit MarkJH's Homepage
Nick, you have the Mediapartners-Google spider listed twice.

Bandlink.net - http://www.bandlink.net/
Bandlink Music Forums - http://www.bandlink.net/forum/
Go to Top of Page

DavidRhodes
Senior Member

United Kingdom
1222 Posts

Posted - 06 June 2004 :  14:07:11  Show Profile
Great mod, just a couple of things:

Why not use the email facilities built into Snitz rather than depending on CDONTS?
Can the code be changed to show "Search Engine" rather than "Unknown" for the browser when displayed on the page?

The UK MkIVs Forum

Edited by - DavidRhodes on 06 June 2004 14:09:07
Go to Top of Page

masterao
Senior Member

Sweden
1678 Posts

Posted - 06 June 2004 :  19:03:32  Show Profile  Visit masterao's Homepage
David, it should display CRAWLER instead of unknown if you have added all the changes for this mod. This is included in the modified active_users.asp available on the first page in this topic.

Jan
===========
FR Portal Forums | Active Users 4.0.20 Mod
Go to Top of Page

nickw
Junior Member

Ireland
193 Posts

Posted - 09 June 2004 :  14:30:35  Show Profile
quote:
Originally posted by MarkJH

Nick, you have the Mediapartners-Google spider listed twice.



Thanks Mark, I edited that out now. Well spotted

Nick
Go to Top of Page

DavidRhodes
Senior Member

United Kingdom
1222 Posts

Posted - 09 June 2004 :  15:44:57  Show Profile
quote:
Originally posted by masterao

David, it should display CRAWLER instead of unknown if you have added all the changes for this mod. This is included in the modified active_users.asp available on the first page in this topic.



I see. CRAWLER should be displayed instead of Guest #, but should it still show "unknown" above the crawler name in the next column

I've now added the following lines

AGENTSTR = fUserAgent(rsAM("AU_USER_AGENT"))


AGENTSTR = fUserAgent(rsAG("AU_USER_AGENT"))


and edited to display

"                <td bgcolor=""" & CColor & """ valign=middle nowrap><font face=""" & strDefaultFontFace & """ size=""" & strDefaultFontSize & """ color=""" & strDefaultFontColor & """>" 
	If instr(AGENTSTR,"SEARCH ENGINE SPIDER") then
		response.write "CRAWLER"
		c=c-1
	else
		response.write fLang(strLangMOD_Ls3kAU_01130) & c 
	end if
	response.write "</font></td>" & VBNewLine


It still shows as Guest #, anything else i've missed?

The UK MkIVs Forum
Go to Top of Page

DavidRhodes
Senior Member

United Kingdom
1222 Posts

Posted - 09 June 2004 :  15:51:21  Show Profile
also just added
elseif instr(ua, "inktomi") then
      strBrowser = "INKTOMI"
      strSystem = "SEARCH ENGINE SPIDER"
to the fUserAgent function

here's a text file if someone could help me out
http://www.uk-mkivs.net/forum/active_users.txt

The UK MkIVs Forum

Edited by - DavidRhodes on 09 June 2004 15:54:26
Go to Top of Page

masterao
Senior Member

Sweden
1678 Posts

Posted - 09 June 2004 :  18:49:37  Show Profile  Visit masterao's Homepage
That is odd. Which search engine is it that shows up as a guest? Is it all or just one specific?

Jan
===========
FR Portal Forums | Active Users 4.0.20 Mod
Go to Top of Page

DavidRhodes
Senior Member

United Kingdom
1222 Posts

Posted - 10 June 2004 :  07:22:41  Show Profile
I've only seen one online. This was shown as Unknown *linebreak* MSNBot in the os/browser column.

The UK MkIVs Forum
Go to Top of Page

masterao
Senior Member

Sweden
1678 Posts

Posted - 10 June 2004 :  16:47:08  Show Profile  Visit masterao's Homepage
Ah, then it isn't a bug. It will only display CRAWLER for those search engines who has the following profile (keypart marked red):

"FAST-WebCrawler||ALL THE WEB||SEARCH ENGINE SPIDER**"

Msn has "msnbot/0.11||MSNBot||Prototype Web-crawling robot**" and will not match the if-statement. If you want it to do so, change Prototype Web-crawling robot with SEARCH ENGINE SPIDER, and it will show up as a crawler and not a guest.

Jan
===========
FR Portal Forums | Active Users 4.0.20 Mod
Go to Top of Page

DavidRhodes
Senior Member

United Kingdom
1222 Posts

Posted - 10 June 2004 :  17:33:10  Show Profile
I've just tried that change and it didn't work, it still shows MSNBot as a guest.

If it only recognises crawler with "SEARCH ENGINE SPIDER", what is the point of listing those which don't have this in the AgentCsvStr string?

The UK MkIVs Forum

Edited by - DavidRhodes on 10 June 2004 17:33:38
Go to Top of Page

masterao
Senior Member

Sweden
1678 Posts

Posted - 11 June 2004 :  16:01:02  Show Profile  Visit masterao's Homepage
I don't know what aspwiz's intent was, so I cannot answer about the reason. I agree with you that it seems odd to not list all as search engine spiders.

Jan
===========
FR Portal Forums | Active Users 4.0.20 Mod
Go to Top of Page

nickw
Junior Member

Ireland
193 Posts

Posted - 11 June 2004 :  17:19:05  Show Profile
Well, I just changed all mine, now they show as crawlers (except one, which I was not sure about).

AgentCsvStr = 		"Googlebot||GOOGLE||SEARCH ENGINE SPIDER**" & _
			"NPBot||Name Protect||SEARCH ENGINE SPIDER**" & _
			"Scooter||ALTAVISTA||SEARCH ENGINE SPIDER**" & _
			"ia_archiver||ALEXA||SEARCH ENGINE SPIDER**" & _
			"FAST-WebCrawler||ALL THE WEB||SEARCH ENGINE SPIDER**" & _
			"Speedy Spider||ENTIRE WEB||SEARCH ENGINE SPIDER**" & _
			"ArchitextSpider||EXCITE||SEARCH ENGINE SPIDER**" & _
			"ArchitectSpider||EXCITE||SEARCH ENGINE SPIDER**" & _
			"Ask Jeeves/Teoma||ASK JEEVES / TEOMA||SEARCH ENGINE SPIDER**" & _
			"Slurp/||INKTOMI||SEARCH ENGINE SPIDER**" & _
			"Slurp.so/1.0||INKTOMI||SEARCH ENGINE SPIDER**" & _
			"UltraSeek||INFOSEEK||SEARCH ENGINE SPIDER**" & _
			"InfoSeek Sidewinder||INFOSEEK||SEARCH ENGINE SPIDER**" & _
			"MantraAgent||LOOKSMART||SEARCH ENGINE SPIDER**" & _
			"Lycos_Spider_(T-Rex)||LYCOS||SEARCH ENGINE SPIDER**" & _
			"HenryTheMiragoRobot||MIRAGO||SEARCH ENGINE SPIDER**" & _
			"MSNBOT/0.1||MSN SEARCH||SEARCH ENGINE SPIDER**" & _
			"Gulliver||NORTHERN LIGHT||SEARCH ENGINE SPIDER**" & _
			"Scrubby||SCRUB THE WEB||SEARCH ENGINE SPIDER**" & _
			"teoma_agent1||TEOMA||SEARCH ENGINE SPIDER**" & _
			"marvin/infoseek||WEBSEEK||SEARCH ENGINE SPIDER**" & _
			"SlySearch/1.3||SLYSEARCH||SEARCH ENGINE SPIDER**" & _
			"Szukacz||SZUKACZ.PL||SEARCH ENGINE SPIDER**" & _
			"IE 5.5 Compatible Browser||IE 5.5 Compatible Browser||Unknown**" & _
			"almaden||IBM||SEARCH ENGINE SPIDER**" & _
			"Google CHTML Proxy/1.0||GOOGLE PROXY SERVER||SEARCH ENGINE SPIDER**" & _
			"http://grub.org||GRUB||SEARCH ENGINE SPIDER**" & _
			"NutchOrg||NUTCH||SEARCH ENGINE SPIDER**" & _
			"InternetSeer.com||INTERNET SEER||SEARCH ENGINE SPIDER**" & _
			"Baiduspider+||BAIDU||SEARCH ENGINE SPIDER*" & _
			"Xenu Link Sleuth||XENU LINK CHECKER||SEARCH ENGINE SPIDER**" & _
			"Mozilla/5.||Mozilla 5.x||SEARCH ENGINE SPIDER**" & _
			"Mozilla/4.||Mozilla 4.x||SEARCH ENGINE SPIDER**" & _
			"Mozilla/3.||Mozilla 3.x||SEARCH ENGINE SPIDER**" & _
			"NetMonitor/||NetMonitor||SEARCH ENGINE SPIDER**" & _
			"TurnitinBot/1.5||TurnItInBot||SEARCH ENGINE SPIDER**" & _
			"dloader(NaverRobot)||NAVER ROBOT||SEARCH ENGINE SPIDER**" & _
			"ZyBorg/||ZYBORG||SEARCH ENGINE SPIDER**" & _
			"Mediapartners-Google||Google Adsense||SEARCH ENGINE SPIDER**" & _
			"QuepasaCreep||QUEPASA.COM||SEARCH ENGINE SPIDER**" & _
			"Microsoft URL Control||POSSIBLE EMAIL COLLECTOR||SEARCH ENGINE SPIDER**" & _
			"Avant Browser||Avant Browser||SEARCH ENGINE SPIDER**" & _
			"msnbot/0.11||MSNBot||SEARCH ENGINE SPIDER**" & _
			"SKTGbot/1.0||SKTGBot||SEARCH ENGINE SPIDER**" & _
			"Openbot/3.0||Openfind.com||SEARCH ENGINE SPIDER**"

Nick
Go to Top of Page

DavidRhodes
Senior Member

United Kingdom
1222 Posts

Posted - 11 June 2004 :  18:28:51  Show Profile
quote:
Originally posted by DavidRhodes
here's a text file if someone could help me out
http://www.uk-mkivs.net/forum/active_users.txt



Can anyone see anything i've missed? It's still showing MSNBot as a guest

The UK MkIVs Forum
Go to Top of Page

nickw
Junior Member

Ireland
193 Posts

Posted - 19 June 2004 :  18:39:31  Show Profile
Hi David,

When I changed all of them to "SEARCH ENGINE SPIDER" as above, they stopped showing as guest. I am no coder... so I do not know why.

Nick
Go to Top of Page

masterao
Senior Member

Sweden
1678 Posts

Posted - 19 June 2004 :  18:50:22  Show Profile  Visit masterao's Homepage
David, have you changed the second listing for msnbot to search engine spider? It is listed twice, as MSNBOT/0.1 and msnbot/0.11? Change the newer version and it should work for you

Nick, I think you can remove the older version of msnbot from the list.

Jan
===========
FR Portal Forums | Active Users 4.0.20 Mod
Go to Top of Page
Page: of 12 Previous Topic Topic Next Topic  
Previous Page | Next Page
 New Topic  Topic Locked
 Printer Friendly
Jump To:
Snitz Forums 2000 © 2000-2016 Snitz™ Communications Go To Top Of Page
This page was generated in 0.21 seconds. Powered By: Snitz Forums 2000 Version 3.4.07