Crawl delay in robots.txt?

Kaz Wolfe · Jun 30, 2018

I checked robots.txt of a website on the internet and I saw they putted these line in their robots.txt.

Code:

User-agent: Googlebot
Crawl-delay: 10

What is mean of Crawl-delay: 10 in robots.txt? does anyone explain me?

Thanks in advance

hostens · Jun 30, 2018

The "crawl-delay" just slows down the bots, that are scanning your website. In this case, it is for Googlebot. It instructs the bot to wait 10 seconds after a crawl action. This means that it will access your website after next 10 seconds. It may take some more time for the bots to index your site, but you should not worry if the website does not have more than 8000 pages (if it had more, it may take more than a day to index your website).
Usually, this crawl delay is set for the bot to get into your website fewer times compared with organic hits.

royceD · Nov 13, 2018

Kaz Wolfe said:
I checked robots.txt of a website on the internet and I saw they putted these line in their robots.txt.

Code:

User-agent: Googlebot Crawl-delay: 10

What is mean of Crawl-delay: 10 in robots.txt? does anyone explain me?

Thanks in advance

Crawl-delay is a robots.txt file directive that allows you to set up a delay in scanning pages to reduce the incoming peak load on the server when the site is crawled by the search robot.

Crawl delay in robots.txt?

Kaz Wolfe

Well-known member

hostens

Well-known member

Kaz Wolfe

Well-known member

royceD

Member

Maxoq

Well-known member

royceD

Member

Latest posts New threads

Latest posts

New threads

Latest Hosting Offers New Reviews

Latest hosting offers

Popular Contributors - Past 30 Days

Sponsors

Tag Cloud