How to set up a robots.txt to control Google spider crawl my site?

johnyplex · Jan 27, 2015

Hi everyone,

I have just established a new site and I don't want google spider crawl some parts (folder, files) on my site
Are threre any ways to set up a robots.txt to control Google spider crawl my site?

for example:
I want google spider crawl all my site but not for /payment/ folder and /private_data/ folder
I don't want google spider crawl a image file like my_image_file.jpg
I don't want google spider crawl all file start with mydata

Any suggestions?

Nytshade · Jan 27, 2015

You can do that in the Google Webmaster Tools, if you're using wordpress there are free plugins you can use.

Hope this video will help:

[video=youtube;Hi41dBV3BxM]https://www.youtube.com/watch?v=Hi41dBV3BxM[/video]

Hassan · Jan 27, 2015

Code:

User-agent: *
Disallow: /payment/
Disallow: /private_data/
Disallow: /mydata

User-agent: Googlebot-Image
Disallow: /images/my_image_file.jpg

How to set up a robots.txt to control Google spider crawl my site?

johnyplex

Member

Nytshade

Well-known member

Hassan

Well-known member

Latest posts New threads

Latest posts

New threads

Latest Hosting Offers New Reviews

Latest hosting offers

Popular Contributors - Past 30 Days

Sponsors

Tag Cloud