Crawl delay in robots.txt?

Kaz Wolfe

Well-known member
Registered
Joined
Jul 7, 2016
Messages
604
Points
28
I checked robots.txt of a website on the internet and I saw they putted these line in their robots.txt.

Code:
User-agent: Googlebot
Crawl-delay: 10
What is mean of Crawl-delay: 10 in robots.txt? does anyone explain me?

Thanks in advance
 

hostens

Well-known member
Hosting Provider
Registered
Joined
Jan 18, 2017
Messages
215
Points
28
The "crawl-delay" just slows down the bots, that are scanning your website. In this case, it is for Googlebot. It instructs the bot to wait 10 seconds after a crawl action. This means that it will access your website after next 10 seconds. It may take some more time for the bots to index your site, but you should not worry if the website does not have more than 8000 pages (if it had more, it may take more than a day to index your website).
Usually, this crawl delay is set for the bot to get into your website fewer times compared with organic hits.
 

Kaz Wolfe

Well-known member
Registered
Joined
Jul 7, 2016
Messages
604
Points
28
Kaz Wolfe
Thanks for explaining me, I should add Crawl-delay to robots.txt for my website or let Google decide that? I also checked other websites and some have set 200 for Crawl-delay. I don't know I should give this with a proper number.
 

royceD

Member
Registered
Joined
Nov 8, 2018
Messages
18
Points
3
I checked robots.txt of a website on the internet and I saw they putted these line in their robots.txt.

Code:
User-agent: Googlebot
Crawl-delay: 10
What is mean of Crawl-delay: 10 in robots.txt? does anyone explain me?

Thanks in advance
Crawl-delay is a robots.txt file directive that allows you to set up a delay in scanning pages to reduce the incoming peak load on the server when the site is crawled by the search robot.
 
Newer Threads
Replies
1
Views
2,174
Replies
6
Views
2,368
Replies
1
Views
1,657
Replies
7
Views
1,781
Latest Threads
Replies
0
Views
12
Replies
1
Views
16
Replies
2
Views
36

Latest postsNew threads

Latest Hosting OffersNew Reviews

Sponsors

Tag Cloud

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top