Snitz Forums 2000
Snitz Forums 2000
Home | Profile | Register | Active Topics | Members | Search | FAQ
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 Snitz Forums 2000 MOD-Group
 MOD Add-On Forum (W/Code)
 Auto Update Spider ID MOD : Active Users
 New Topic  Topic Locked
 Printer Friendly
Previous Page
Author Previous Topic Topic Next Topic
Page: of 19

dabugster
Junior Member

USA
168 Posts

Posted - 08 May 2005 :  22:15:04  Show Profile  Visit dabugster's Homepage  Send dabugster an AOL message  Send dabugster a Yahoo! Message
Cool ... i got it.
I will try to put it up on my board later to see if it works better.

Thanks a lot!
Go to Top of Page

samyot
Junior Member

Canada
242 Posts

Posted - 09 May 2005 :  13:15:38  Show Profile  Visit samyot's Homepage  Send samyot a Yahoo! Message
Does anybody have an updated list of user agents?

Sylvain Amyot
Forum | Home

MY OTHER SITES:
http://www.mypcsecurity.ca
http://www.mynature.ca
http://www.myworkshop.ca
http://www.toptechsites.com
http://www.topsportsites.com
http://www.mysportsforums.net
Go to Top of Page

hellind
Junior Member

Singapore
123 Posts

Posted - 10 May 2005 :  12:56:30  Show Profile
I emailed AspWiz and no reply.

He had asked Chuck to take over, why dont' Chuck take the honours and email AspWiz? :)
Go to Top of Page

dabugster
Junior Member

USA
168 Posts

Posted - 18 May 2005 :  10:33:10  Show Profile  Visit dabugster's Homepage  Send dabugster an AOL message  Send dabugster a Yahoo! Message
Has anyone else used this:
ASP Stats Generator

It is a free stats generator, and while it does not tell you if the visitor is a spider or not, it does tell you the browser and OS. Even some really new ones.

Any one know how to implement this system in to this spider mod?
It uses some XML pages but i fugure someone with more ASP expierience that i might be able to adapt it.
I have been trying to play around with it, to no avail.

Mine is getting to where there are too many 'Unknown System'and 'Unknown User Agent'
And everyone is a visitor, even when it says, 'Inktomi.com'

It has been driving me crazy lately and no matter what i try it seem like it never works right.
Go to Top of Page

hellind
Junior Member

Singapore
123 Posts

Posted - 27 May 2005 :  08:27:45  Show Profile
Basically we have to use the built-in list.

So delete

AgentCSVStr = Application("AUCACHE")
If Application("AUDataLastCached") <> "" or request.querystring("mode") = "reset" Then
    If CDate(Application("AUDataLastCached")) < now()-2 or request.querystring("mode") = "reset" Then 'Update Cache every 2 days
        DoCache()
    else
        AgentCSVStr = Application("AUCACHE")
    end if
else
    DoCache()
end if


And whenever you get the email, you update the list yourself.
Go to Top of Page

dabugster
Junior Member

USA
168 Posts

Posted - 27 May 2005 :  12:12:03  Show Profile  Visit dabugster's Homepage  Send dabugster an AOL message  Send dabugster a Yahoo! Message
Cool!!!!!
That worked!
Thank you hellhind, i was getting so sick of everyone being a guest all the time and all that unknow system and browser.
It works perfect now.
I have been getting like 25 emails a day and i try to add them where i think i should, but just to be sure, when i get these emails:

New User Agent Reported by: Ratdaddyvince.com Forum ( http://www.ratdaddyvince.com )
User Agent String = "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
User Agent IP: 68.142.250.11
Script date: 21.10.2004
User Agent Update Location: http://www.dotmagic.co.uk/activeuseragents.htm
User Agent Update Info: :: <span style="background-color:red; color:white; padding:2px;">ASP Script Update is available. <a href="">Click Here.</a> </span>


What part of that do i add and where?

Here is a link to my active_users.asp page:
http://www.ratdaddyvince.com/forum/active_users.txt
Go to Top of Page

hellind
Junior Member

Singapore
123 Posts

Posted - 27 May 2005 :  13:52:04  Show Profile
From your email, the useragent is
"Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

And the format of AgentCsvStr has three parts seperated by ||.
Useragent-to-match || Site || System

Therefore you should another line to AgentCsvStr, if it doesn't exist already:
"Yahoo! Slurp||YAHOO||SEARCH ENGINE SPIDER" & vbcrlf & _

PS: "Yahoo! Slurp" matches the UA string, so it is a hit.
YAHOO || SEARCH ENGINE SPIDER can be anything you want it displayed.

Go to Top of Page

hellind
Junior Member

Singapore
123 Posts

Posted - 27 May 2005 :  13:54:30  Show Profile
I see you have added a lot of other lines in your active_users.txt

Delete
	"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)" & vbcrlf & _
	"Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" & vbcrlf & _
	"Mediapartners-Google/2.1" & vbcrlf & _
	"msnbot/1.0 (+http://search.msn.com/msnbot.htm)" & vbcrlf & _
	"Mediapartners-Google/2.1" & vbcrlf & _
	"Mozilla/4.0 (compatible; grub-client-2.6.0)" & vbcrlf & _
	"Googlebot/2.1 (+http://www.google.com/bot.html)" & vbcrlf & _
	"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)" & vbcrlf & _
	"Mozilla/4.0 (compatible; grub-client-2.3)" & vbcrlf & _
	"ConveraCrawler/0.5 (+http://www" & vbcrlf & _
	"DoCoMo/2.0 P900i(c100;TB;W24H11)" & vbcrlf & _
	"afifefhnaponugwFmunoifaimi rg" & vbcrlf & _
	"cbpoexwkdvv6Dopnfpyjj" & vbcrlf & _
	"gsldmdc y hfnkqoqvwAanwsdfeaemgimu" & vbcrlf & _
	"dkqxydvrwatmwdhsfehhwjjksmeyq" & vbcrlf & _
	"HTTP agent" & vbcrlf & _
	"Port Huron Labs" & vbcrlf & _
	"dwmmrpiaihuilxxgbbbly" & vbcrlf & _
	"NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)" & vbcrlf & _
	"WebRingChecker/1.71 LWP/5.65" & vbcrlf & _
	"Mediapartners-Google/2.1" & vbcrlf & _
	"Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" & vbcrlf & _
Check the original but older text.

http://www.zestkith.com/Upload/active_users.txt


Edited by - hellind on 27 May 2005 13:55:55
Go to Top of Page

dabugster
Junior Member

USA
168 Posts

Posted - 27 May 2005 :  14:30:22  Show Profile  Visit dabugster's Homepage  Send dabugster an AOL message  Send dabugster a Yahoo! Message
I added those like that cause i didn't know what i was supposed to add.
Now that you explained it to me, i changed the list to the one in that link you posted.
Now will it still email me when there is a crawler on my site that isn't in the list?
Go to Top of Page

hellind
Junior Member

Singapore
123 Posts

Posted - 28 May 2005 :  01:41:32  Show Profile
It is emailing you because you need to use the builtin list, and not the list by ASPWIZ.
You delete this as was instructed before.


AgentCSVStr = Application("AUCACHE")
If Application("AUDataLastCached") <> "" or request.querystring("mode") = "reset" Then
    If CDate(Application("AUDataLastCached")) < now()-2 or request.querystring("mode") = "reset" Then 'Update Cache every 2 days
        DoCache()
    else
        AgentCSVStr = Application("AUCACHE")
    end if
else
    DoCache()
end if


Go to Top of Page

dabugster
Junior Member

USA
168 Posts

Posted - 28 May 2005 :  02:02:31  Show Profile  Visit dabugster's Homepage  Send dabugster an AOL message  Send dabugster a Yahoo! Message
Well it is working now so that is all that matters to me i guess.
Thanks Bro

BUG
Go to Top of Page

dabugster
Junior Member

USA
168 Posts

Posted - 31 May 2005 :  10:38:27  Show Profile  Visit dabugster's Homepage  Send dabugster an AOL message  Send dabugster a Yahoo! Message
I found a new spider.
Become.com's BecomeBot

Here is the code i added for it:

	"Anonymization.Net||Anonymous Web Surfing||WEB ANONYMYZER" & vbcrlf & _
	"BecomeBot/2.||Become.com||SEARCH ENGINE SPIDER" & vbcrlf & _
	"lmspider||lmspider lmspider@scansoft.com||Speech Recognition Research Bot"


then

  if instr(ua, "WebCrawler") then strBrowser = "Webcrawler.com"
  if instr(ua, "wombat") then strBrowser = "Web Wombat"
  if instr(ua, "BecomeBot/2.") then strBrowser = "BecomeBot.com"


then

  elseif instr(ua, "FreeBSD") then
    strSystem = "FreeBSD"
  elseif instr(ua, "BecomeBot/2.") then
    strSystem = "SEARCH ENGINE SPIDER"
  else
    strSystem = "<span title=""" & ua & """>Unknown System</span>"
  end if

Go to Top of Page

dabugster
Junior Member

USA
168 Posts

Posted - 18 June 2005 :  14:55:43  Show Profile  Visit dabugster's Homepage  Send dabugster an AOL message  Send dabugster a Yahoo! Message
Found some more spiders:

"msnbot/0.11||http://search.msn.com/msnbot.htm||SEARCH ENGINE SPIDER" & vbcrlf & _
"OmniExplorer_Bot/1.07||http://www.omni-explorer.com||Internet Categorizer" & vbcrlf & _
"ConveraCrawler/0.8||http://www.authoritativeweb.com/crawl||SEARCH ENGINE SPIDER" & vbcrlf & _
"noxtrumbot/1.0||crawler@noxtrum.com||SEARCH ENGINE SPIDER" & vbcrlf & _


Edited by - dabugster on 18 June 2005 22:57:15
Go to Top of Page
Page: of 19 Previous Topic Topic Next Topic  
Previous Page
 New Topic  Topic Locked
 Printer Friendly
Jump To:
Snitz Forums 2000 © 2000-2021 Snitz™ Communications Go To Top Of Page
This page was generated in 0.25 seconds. Powered By: Snitz Forums 2000 Version 3.4.07