How to configure Mod Security and CSF Firewall to allow Googlebots

steitieh

Well-known member
Joined
Apr 23, 2014
Messages
113
Points
18
In the case Google bots increased a large number of connections to the website, if on your VPS installed Mod Security and CSF Firewall then it can be misunderstood as being offensive websites and bots can be blocked. Consequently, google bot will not be able to visit your site again. To fix this, you need config Mod Firewall Security Rules and CSF as following ways:

1. For Mod Security Rules

Code:
#Allow googlebots
SecRule REMOTE_HOST googlebot.com$ allow,pass
 
#Add this to your config. The right way to identify Google bot is it's User-Agent.
SecRule REQUEST_HEADERS:User-Agent "Googlebot" phase:1,nolog,allow,ctl:ruleEngine=off
 
# GoogleBot by user-agent…
SecRule HTTP_USER_AGENT Google” nolog,allow
SecRule HTTP_USER_AGENT Googlebot” nolog,allow
SecRule HTTP_USER_AGENT GoogleBot” nolog,allow
SecRule HTTP_USER_AGENT googlebot” nolog,allow
SecRule HTTP_USER_AGENT Googlebot-Image” nolog,allow
##
SecRule HTTP_USER_AGENT AdsBot-Google” nolog,allow
SecRule HTTP_USER_AGENT Googlebot-Image/1.0″ nolog,allow
SecRule HTTP_USER_AGENT Googlebot/2.1″ nolog,allow
SecRule HTTP_USER_AGENT Googlebot/Test” nolog,allow
SecRule HTTP_USER_AGENT Mediapartners-Google/2.1″ nolog,allow
SecRule HTTP_USER_AGENT Mediapartners-Google*” nolog,allow
SecRule HTTP_USER_AGENT msnbot” nolog,allow
2. For CSF Firewall

We add into the file /etc/csf/csf.rignore with following codes

Code:
.googlebot.com
With Yahoo, Bing

Code:
.crawl.yahoo.net 
.search.msn.com
Hope it helps!
 

Latest Hosting OffersNew Reviews

Sponsors

Tag Cloud

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top