What is Crawl Rate Optimisation?
Crawl rate optimisation relates to the crawl rate, demand and in turn budget that search engines apply to your site in terms of the number of URLs they are able to, and in turn, will crawl.
You want search engines to be able to easily crawl and download key, primary versions of each page on your site and to prevent them from crawling unnecessary, duplicate, thin or low quality content that may dilute your overall site quality.
What is Crawl Rate?
This is the number of URLs per second at which search engines will crawl on your site.
What is Crawl Demand?
This is the demand for new and existing content on your site to be crawled. This is determined by the domain authority (the number of links pointing at the pages of your site) and the regularity of new content being added. Strong sites with regularly added content will be crawled more often than weaker sites that rarely update.
What is Crawl Budget?
A combination of crawl rate and demand to determine how many pages search engines can and will crawl on your site per visit.
What impacts Crawl Budget?
- Faceted navigation (aka search filters) – these can generate 00s if not 000s of URLs on your site, in particular ecommerce sites, none of which are unique or add value to the site
- Duplicate content – don’t waste your crawl budget or dilute your site quality by allowing them to visit identical content on different URLs
- Broken pages and redirects – every internal link to a URL on your site represents a page that can be crawled, these should be live pages that are linked to directly – avoid broken pages and redirect chains as these consume budget
- XML sitemaps – these tell search engines which pages on your site to visit, their importance and how often they change – make sure the pages here are primary, unique pages you want people to find, and leave out the ones you don’t
- Internal links – make sure that both users and search engines can find your key pages quickly and easily when navigating your site
- Site & server Speed - the faster a page can be downloaded, the most pages can be crawled per visit
What are the benefits of Crawl Rate Optimisation?
Whilst Google states that Crawl Rate itself is no a ranking factor, we have found that by optimising a website to ensure that search engines only visit key, primary pages on a regular basis, and eliminating issues and content that waste crawl budget, the organic performance of key landing pages on your site will improve.
For example, if you have 10 key ecommerce category pages, but search filters mean that 500 variations are crawlable and indexed, by blocking the unnecessary 490 the organic performance of the remaining 10 will improve.
What is Indexing Optimisation?
This ensures that search engines only indexes the primary versions of pages on your site and avoids indexing duplicate content which then effectively means your site quality is diluted as your pages complete with each other for relevant terms.
With inefficient crawl rate optimisation in place you can end up with several, hundreds and in some cases thousands of versions of what are essentially the same page.
How Can Organic Digital Help?
The audit process ensures your site adheres to best practice by assessing the configuration and/or performance of:
- Your existing level of indexed content
- XML Sitemap Configuration
- robots.txt Coniguration
- Canonical Tag Configuration
- Internal links to redirects, redirect chains, broken pages and server errors
- The volume and inclusion of internal links to key pages
Useful Crawl Rate and Indexing Optimisation Resources
If you want to find out more about crawl optimisation, the following resource is probably the best article you will come across, alternatively, if you want any assistance implementing crawl optimsiation on your site, fill in the contact form below:
Latest Testimonials
The following are genuine testimonials left by actual real life people on Organic Digital's Google Business Listing. To read more, visit the testimonials page.
"Dave recently helped us with SEO crawl issues on our site and was of great assistance."
Jack Neville
yundies.co.uk
"Dave is a well seasoned SEO professional with great experience in his craft. He's always on hand when you have questions or require his advice and is willing to integrate into the business like another member of the team to ensure SEO can play a central role in the business strategy."
Jay Lau
lullabellz.com
"If you're looking for someone to help get your Analytical reporting in order, on time, to budget with a common sense approach - look no further. Dave is excellent at what he does, patient and efficient. We'll be working with him for many years to come."
Roly Peto
fullingmill.co.uk
"I've worked with Dave on a GA4 project with numerous potential hurdles and he's been nothing but brilliant. Dave provides well thought-out solutions to challenges and is a fantastic communicator, sharing clear and concise instructions."
Nick Le Chat
riseatseven.com
"We've worked alongside Dave at Organic Digital to deploy GA4 tracking on our custom Shopify Plus store for Tatti Lashes. Dave is very knowledgeable in all areas of Google Analytics and has been a pleasure to work with."
Dave Speake
madebyfield.co.uk
