How to set up a robots.txt to control Google spider crawl my site?

johnyplex

Member
Registered
Joined
Jan 23, 2015
Messages
28
Points
3
Hi everyone,

I have just established a new site and I don't want google spider crawl some parts (folder, files) on my site
Are threre any ways to set up a robots.txt to control Google spider crawl my site?

for example:
I want google spider crawl all my site but not for /payment/ folder and /private_data/ folder
I don't want google spider crawl a image file like my_image_file.jpg
I don't want google spider crawl all file start with mydata

Any suggestions?
 

Hassan

Well-known member
Registered
Joined
Nov 11, 2014
Messages
433
Points
0
Code:
User-agent: *
Disallow: /payment/
Disallow: /private_data/
Disallow: /mydata

User-agent: Googlebot-Image
Disallow: /images/my_image_file.jpg
 
Older Threads
Replies
6
Views
3,189
Replies
3
Views
3,092
Replies
5
Views
3,546
Latest Threads
Replies
0
Views
29
Replies
0
Views
31
Replies
0
Views
24
Replies
0
Views
25
Recommended Threads
Replies
1
Views
2,149
Replies
31
Views
7,708
Replies
6
Views
2,059
Replies
0
Views
54

Latest postsNew threads

Latest Hosting OffersNew Reviews

Sponsors

Tag Cloud

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top