I was checking out what robots visit one of my sites, and I found a list. I need help in knowing which to allow and which to not allow.
Googlebot
Unknown robot (identified by 'crawl')
MJ12bot
Yandex bot
Phantom
Unknown robot (identified by 'bot/' or 'bot-')
Unknown robot (identified by hit on 'robots.txt')
Unknown robot (identified by 'robot')
Unknown robot (identified by 'spider')
Yahoo Slurp
Exabot
SurveyBot
Java (Often spam bot)
Jakarta commons-httpclient
Netcraft

The largest consumers are:

Googlebot (44.50 MB)
Unknown robot (identified by 'crawl') (73.97 MB)
MJ12bot (192.09 MB)
Yandex bot (10.90 MB)
Phantom (1.11 MB)