Crawl Rate & Indexing Optimisation

Ensure your key landing pages are regularly crawled and indexed

Someone Using A Laptop And Tablet At The Same Time

What is Crawl Rate Optimisation?

Crawl rate optimisation relates to the crawl rate, demand and in turn budget that search engines apply to your site in terms of the number of URLs they are able to, and in turn, will crawl.

You want search engines to be able to easily crawl and download key, primary versions of each page on your site and to prevent them from crawling unnecessary, duplicate, thin or low quality content that may dilute your overall site quality.

What is Crawl Rate?

This is the number of URLs per second at which search engines will crawl on your site.

What is Crawl Demand?

This is the demand for new and existing content on your site to be crawled. This is determined by the domain authority (the number of links pointing at the pages of your site) and the regularity of new content being added. Strong sites with regularly added content will be crawled more often than weaker sites that rarely update.

What is Crawl Budget?

A combination of crawl rate and demand to determine how many pages search engines can and will crawl on your site per visit.

What impacts Crawl Budget?

  • Faceted navigation (aka search filters) – these can generate 00s if not 000s of URLs on your site, in particular ecommerce sites, none of which are unique or add value to the site
  • Duplicate content – don’t waste your crawl budget or dilute your site quality by allowing them to visit identical content on different URLs
  • Broken pages and redirects – every internal link to a URL on your site represents a page that can be crawled, these should be live pages that are linked to directly – avoid broken pages and redirect chains as these consume budget
  • XML sitemaps – these tell search engines which pages on your site to visit, their importance and how often they change – make sure the pages here are primary, unique pages you want people to find, and leave out the ones you don’t
  • Internal links – make sure that both users and search engines can find your key pages quickly and easily when navigating your site
  • Site & server Speed - the faster a page can be downloaded, the most pages can be crawled per visit

What are the benefits of Crawl Rate Optimisation?

Whilst Google states that Crawl Rate itself is no a ranking factor, we have found that by optimising a website to ensure that search engines only visit key, primary pages on a regular basis, and eliminating issues and content that waste crawl budget, the organic performance of key landing pages on your site will improve.

For example, if you have 10 key ecommerce category pages, but search filters mean that 500 variations are crawlable and indexed, by blocking the unnecessary 490 the organic performance of the remaining 10 will improve.

What is Indexing Optimisation?

This ensures that search engines only indexes the primary versions of pages on your site and avoids indexing duplicate content which then effectively means your site quality is diluted as your pages complete with each other for relevant terms.

With inefficient crawl rate optimisation in place you can end up with several, hundreds and in some cases thousands of versions of what are essentially the same page.

How Can Organic Digital Help?

The audit process ensures your site adheres to best practice by assessing the configuration and/or performance of:

  • Your existing level of indexed content
  • XML Sitemap Configuration
  • robots.txt Coniguration
  • Canonical Tag Configuration
  • Internal links to redirects, redirect chains, broken pages and server errors
  • The volume and inclusion of internal links to key pages

Useful Crawl Rate and Indexing Optimisation Resources

If you want to find out more about crawl optimisation, the following resource is probably the best article you will come across, alternatively, if you want any assistance implementing crawl optimsiation on your site, fill in the contact form below:

Latest Testimonials

The following are genuine testimonials left by actual real life people on Organic Digital's Google Business Listing. To read more, visit the testimonials page.

"Dave is super easy to work with - he's fast to respond to any query I might have, knows his stuff, is thorough and proactive. It's a pleasure working with him."

Liz Walmsley Liz Walmsley

"Dave helped us get our website into shape and SEO friendly. His technical knowledge is to a very high level which is essential to achieving success in the fast changing world of Search Marketing."

Imran Farooq Imran Farooq

"We have been working with Organic Digital for over a year now and have nothing but high praise for the quality and detail of the SEO technical audits.

We specifically enjoy the rationale included in the reports on the 'why' each audit area is important, as well as easy-to-follow recommendations that can be passed directly onto the digital and or development team to implement.

Since implementing Organic Digital’s SEO updates we have also seen a lift in SEO performance aligned to delivering objectives.

We highly recommend Organic Digital"

Jon Rayner Jon Rayner

Get In Touch

Is your site in need of crawl rate or indexing optimsiation? Fill in the form below to find out how we can help