steitieh
Well-known member
- Joined
- Apr 23, 2014
- Messages
- 113
- Points
- 18
In the case Google bots increased a large number of connections to the website, if on your VPS installed Mod Security and CSF Firewall then it can be misunderstood as being offensive websites and bots can be blocked. Consequently, google bot will not be able to visit your site again. To fix this, you need config Mod Firewall Security Rules and CSF as following ways:
1. For Mod Security Rules
2. For CSF Firewall
We add into the file /etc/csf/csf.rignore with following codes
With Yahoo, Bing
Hope it helps!
1. For Mod Security Rules
Code:
#Allow googlebots
SecRule REMOTE_HOST googlebot.com$ allow,pass
#Add this to your config. The right way to identify Google bot is it's User-Agent.
SecRule REQUEST_HEADERS:User-Agent "Googlebot" phase:1,nolog,allow,ctl:ruleEngine=off
# GoogleBot by user-agent…
SecRule HTTP_USER_AGENT Google†nolog,allow
SecRule HTTP_USER_AGENT Googlebot†nolog,allow
SecRule HTTP_USER_AGENT GoogleBot†nolog,allow
SecRule HTTP_USER_AGENT googlebot†nolog,allow
SecRule HTTP_USER_AGENT Googlebot-Image†nolog,allow
##
SecRule HTTP_USER_AGENT AdsBot-Google†nolog,allow
SecRule HTTP_USER_AGENT Googlebot-Image/1.0″ nolog,allow
SecRule HTTP_USER_AGENT Googlebot/2.1″ nolog,allow
SecRule HTTP_USER_AGENT Googlebot/Test†nolog,allow
SecRule HTTP_USER_AGENT Mediapartners-Google/2.1″ nolog,allow
SecRule HTTP_USER_AGENT Mediapartners-Google*†nolog,allow
SecRule HTTP_USER_AGENT msnbot†nolog,allowWe add into the file /etc/csf/csf.rignore with following codes
Code:
.googlebot.com
Code:
.crawl.yahoo.net
.search.msn.com
