If you are going to invest time and money into SEO, you need to make sure that your site is being indexed correctly. Although it is a pretty minor part of SEO, you’d be amazed at the amount of indexation issues that can arise, particularly with large site. The good news is that in most cases these issues are easily fixed. If you’re having indexation issues, follow this guide or get in touch for custom advice.

Why doesn’t Google crawl every page on my sitemap?

Google will rarely crawl every single page on your sitemap. The first thing to be aware of is that a sitemap is inclusive rather than exclusive. This means that Google will crawl some pages that aren’t on your sitemap as well as not crawl some pages that are. There are a couple of reasons why Google may not crawl certain pages on your sitemap in particular. Almost always, these are due to duplicate content or thin content issues. If you have a doorway page, for instance, that has very little unique content on it, Google just won’t bother.

But shouldn’t Google index every page on my site?

No, Google shouldn’t index every page on your site. In fact, Google doesn’t index every page on any site. Not even the biggest sites in the world. Especially the biggest sites in the world, in fact. Some pages just aren’t relevant enough to be included in the search results so Google and Bing don’t bother. Stop worrying about getting every single page of your site into the index and start worrying about making your content as good as it can be.

Here’s what you can do to make sure every page on your sitemap gets crawled

All this being said, there are several things that you can do (/not do) to make sure that every page on your sitemap does get crawled by Google. We recommend the following points.

Make sure that your links are canonical and that they don’t redirect

If Google is to crawl your sitemap properly, it want to see the final destination URLS, not redirects that take them around your site. This means that for landing pages in particular you should be adding the final destination to your sitemap.

Stop creating new sitemap in order to get crawled

If you realize that some of your pages aren’t getting crawled, it is normal to think that it is an issue with your sitemaps. But it almost certainly isn’t. Rather, it is a problem with your pages. So fix them first.

Don’t create thin content pages

Every page on your site should add value to the user. And Google won’t index pages that it thinks add little value. This is especially important if you have doorway pages or thin landing pages like local service pages. If these are regurgitated content they probably won’t get indexed.

Have a proper 404 page

One way to stop the creation of thin doorway pages is to have a proper 404 page in place. This way, instead of a thin content page getting created when the content can’t be found, the page will redirect to your 404 page.

Get help from Google’s Indexation Report

Google is on your side when it comes to indexing and they want to make it as easy as possible for you. To this end, they have created a new indexation report which can be found in the Search Console and helps you to identify what indexation issues your site is facing and how to overcome them.

If you’re still having indexation issues after reading this, reach out to a member of our team for a consultation on how we can help.

This is how to make sure Google crawls your site properly