Snitz Forums 2000
Snitz Forums 2000
Home | Profile | Register | Active Topics | Members | Search | FAQ
Username:
 Password:
Save Password
Forgot your Password?

 All Forums
 Help Groups for Snitz Forums 2000 Users
 Help: General / Classic ASP versions(v3.4.XX)
 Crawlers		  New Topic  Topic Locked
 Printer Friendly
Author Previous Topic Topic Next Topic  

l2k man
New Member

United Kingdom
74 Posts
Posted - 23 April 2010 :  17:59:45  Show Profile
We seem to be having problems with the server being too busy and therefore users not being able to access the site. I may be wrong but it seems that the problem arises soon after a robot has accessed the site. I have robots.txt to disallow access to some files, but I noticed I had a googlebot as an unknown active user for over 20 hours!! It was going through our calendar day by day and had got to 2014 !!!! I shut the site for a while so it gave up.
Do these robots increase the site traffic to such an extent that the server can't cope. Our database is MS Access, which I know has problems.
"MS Access has a limit of 255 connections per server, across all sites. Scripts that open many connections, or do not close connections after use, can quickly exceed this limit. Leaving a connection open means that a script has to time out. This can take several minutes - during which time the connection is unavailable to the same, and other, scripts. "
Am I right in thinking these bots are the source of my problem?

AnonJr
Moderator

United States
5768 Posts
Posted - 23 April 2010 :  21:39:44  Show Profile  Visit AnonJr's Homepage
They may or may not be the source, but its likely to be a contributing factor.

What does your robots.txt look like?
Go to Top of Page

l2k man
New Member

United Kingdom
74 Posts
Posted - 24 April 2010 :  04:22:05  Show Profile
It's my first venture into robots.txt so please don't be toooo critical!!!

User-agent: *
Disallow: /chat/
Disallow: /sharedfiles/
Disallow: /pics/
Disallow: /wpics/

I have another bot, from the same source, crawling the events calendar now. Been on there for 4 hours now. (66.249.65.56)

fyi our website is laser2000.org.uk
Go to Top of Page

Shaggy
Support Moderator

Ireland
6780 Posts
Posted - 29 April 2010 :  06:27:38  Show Profile
Disallow all forum files bar topic.asp; that's the only one with content that you want the bots to be indexing.
Search is your friend
“I was having a mildly paranoid day, mostly due to the
fact that the mad priest lady from over the river had
taken to nailing weasels to my front door again.”
Go to Top of Page

bobby131313
Senior Member

USA
1163 Posts
Posted - 29 April 2010 :  10:33:20  Show Profile  Visit bobby131313's Homepage
quote:
Disallow all forum files bar topic.asp; that's the only one with content that you want the bots to be indexing.

Ooohhh I disagree Shaggy. Where is the SE gonna find the topics without the others?

Here's what I disallow...

Disallow: /forum/active.asp
Disallow: /forum/faq.asp
Disallow: /forum/password.asp
Disallow: /forum/pop_
Disallow: /forum/inc_
Disallow: /forum/admin_
Disallow: /forum/post
Disallow: /forum/policy.asp
Disallow: /forum/register.asp
Disallow: /forum/search.asp
Disallow: /forum/split.asp
Disallow: /forum/unanswered.asp
Disallow: /forum/cal
Switch the order of your title tags
Edited by - bobby131313 on 29 April 2010 10:34:38
Go to Top of Page
  Previous Topic Topic Next Topic  
 New Topic  Topic Locked
 Printer Friendly
Jump To:
Snitz Forums 2000 © 2000-2021 Snitz™ Communications Go To Top Of Page
This page was generated in 0.66 seconds. Powered By: Snitz Forums 2000 Version 3.4.07