Crawl Rate & Indexing Optimisation

Ensure your key landing pages are regularly crawled and indexed

Someone Using A Laptop And Tablet At The Same Time

What is Crawl Rate Optimisation?

Crawl rate optimisation relates to the crawl rate, demand and in turn budget that search engines apply to your site in terms of the number of URLs they are able to, and in turn, will crawl.

You want search engines to be able to easily crawl and download key, primary versions of each page on your site and to prevent them from crawling unnecessary, duplicate, thin or low quality content that may dilute your overall site quality.

What is Crawl Rate?

This is the number of URLs per second at which search engines will crawl on your site.

What is Crawl Demand?

This is the demand for new and existing content on your site to be crawled. This is determined by the domain authority (the number of links pointing at the pages of your site) and the regularity of new content being added. Strong sites with regularly added content will be crawled more often than weaker sites that rarely update.

What is Crawl Budget?

A combination of crawl rate and demand to determine how many pages search engines can and will crawl on your site per visit.

What impacts Crawl Budget?

  • Faceted navigation (aka search filters) – these can generate 00s if not 000s of URLs on your site, in particular ecommerce sites, none of which are unique or add value to the site
  • Duplicate content – don’t waste your crawl budget or dilute your site quality by allowing them to visit identical content on different URLs
  • Broken pages and redirects – every internal link to a URL on your site represents a page that can be crawled, these should be live pages that are linked to directly – avoid broken pages and redirect chains as these consume budget
  • XML sitemaps – these tell search engines which pages on your site to visit, their importance and how often they change – make sure the pages here are primary, unique pages you want people to find, and leave out the ones you don’t
  • Internal links – make sure that both users and search engines can find your key pages quickly and easily when navigating your site
  • Site & server Speed - the faster a page can be downloaded, the most pages can be crawled per visit

What are the benefits of Crawl Rate Optimisation?

Whilst Google states that Crawl Rate itself is no a ranking factor, we have found that by optimising a website to ensure that search engines only visit key, primary pages on a regular basis, and eliminating issues and content that waste crawl budget, the organic performance of key landing pages on your site will improve.

For example, if you have 10 key ecommerce category pages, but search filters mean that 500 variations are crawlable and indexed, by blocking the unnecessary 490 the organic performance of the remaining 10 will improve.

What is Indexing Optimisation?

This ensures that search engines only indexes the primary versions of pages on your site and avoids indexing duplicate content which then effectively means your site quality is diluted as your pages complete with each other for relevant terms.

With inefficient crawl rate optimisation in place you can end up with several, hundreds and in some cases thousands of versions of what are essentially the same page.

How Can Organic Digital Help?

The audit process ensures your site adheres to best practice by assessing the configuration and/or performance of:

  • Your existing level of indexed content
  • XML Sitemap Configuration
  • robots.txt Coniguration
  • Canonical Tag Configuration
  • Internal links to redirects, redirect chains, broken pages and server errors
  • The volume and inclusion of internal links to key pages

Useful Crawl Rate and Indexing Optimisation Resources

If you want to find out more about crawl optimisation, the following resource is probably the best article you will come across, alternatively, if you want any assistance implementing crawl optimsiation on your site, fill in the contact form below:

Get In Touch

Is your site in need of crawl rate or indexing optimsiation? Fill in the form below to find out how we can help