I never really checked before, but I have a lot of evil spiders crawling
around my server. Some of them respect my robots.txt file and others do not.
Some of the ones that do are still *very* pushy. So I decided to shut that
bastards off. Here's what I added to my httpd.conf:
RewriteLog logs/rewrite_log
RewriteLogLevel 1
RewriteCond %{HTTP_USER_AGENT} ^Baiduspider.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^msnbot.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^NaverBot.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Sogou-Test-Spider.*
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4.0.*
RewriteCond %{HTTP_USER_AGENT} ^T-Mobile Dash.*
RewriteRule .* - [F,L]
and inside each of the virtual domains, I added:
RewriteEngine On
RewriteOptions Inherit
Here's the problem. What I want to see is the rewrite_log telling me what it
has redirected or failed. Instead, I'm getting a line telling me every link
that it does NOT rewrite. For example:
72.30.65.61 - - [10/Oct/2009:14:28:24 --0400] \
[vdom.syslang.net/sid#b7298ed0][rid#b6b488e8/initial] (1) pass through /d1/fn
I have googled my brains out and it seems like others have had the same
questions. I see no answers. If anyone has any idea I love to hear it.
I understand that nod_rewrite is complicated, but what I'd like to end up with
a log of all the spiders that got rejected by my rules. Current;y, the
access_log tells me where the attempt is, the error_log tell me nothing and
the rewrite_log is telling me more than I want with none of what I need.
The goal is to see the spiders bouncing off.
Anyone?
--
Time flies like the wind. Fruit flies like a banana. Stranger things have .0.
happened but none stranger than this. Does your driver's license say Organ ..0
Donor?Black holes are where God divided by zero. Listen to me! We are all- 000
individuals! What if this weren't a hypothetical question?
steveo at syslang.net
Attachment:
signature.asc
Description: OpenPGP digital signature
-- fedora-list mailing list fedora-list@xxxxxxxxxx To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list Guidelines: http://fedoraproject.org/wiki/Communicate/MailingListGuidelines
