If the Search Engine doesn’t index your website on Google, you might as well not exist. You won’t come up in any search results, and you won’t get any traffic from search engines.
Since you’re here, I’m going to guess that you already know this. So let’s get right to the point.
This article tells you how to fix any of these issues:
- Not all of your website is indexed.
- Some of your pages are indexed, but others are not.
- Your website’s new pages aren’t being indexed quickly enough.
But first, let’s make sure we’re all on the same page and fully understand this indexing nonsense with the help of this ultimate guide provided by SEO Agency.
What does it mean to crawl and index your website on Google?
Google crawls the web to find new web pages, which they then add to their index. They do this with a tool called Googlebot, which is a web spider.
Confused? Let’s talk about a few important words.
- Crawling: is the process of following links to find new content on the web.
- Indexing: is the process of putting every page on the Internet into a vast database.
- A web spider: is a piece of software that crawls the web on a large scale.
- Googlebot: Google’s web spider.
When you Google something, you are asking Google to give you all of the relevant pages from their index. Google’s ranking algorithm does its best to sort the pages so that you see the best and most relevant ones first. This is because there are often millions of pages that fit the bill.
The most important thing I want to say here is that indexing and ranking are different.
Index your website on Google is like showing up for a race, but the ranking is like winning.
You can’t win the race if you don’t even show up for it.
How to know if you’re indexed in Google
Go to Google and type site:yourwebsite.com into the search bar.
This number gives you a rough idea of how many of your pages Google has looked at.
Use the same site:yourwebsite.com/web-page-slug operator to find out if a certain URL is indexed.
Google’s search page for sites
If the page isn’t indexed, there won’t be any results.
Now, it’s important to note that if you use Google Search Console, you can use the Coverage report to get a better idea of how your website is being indexed. Just visit:
Google Search Console > Coverage > Index
Check how many valid pages there are (with and without warnings).
If the sum of these two numbers is anything other than zero, Google has indexed at least some of the pages on your site. If you don’t, you have a big problem because none of your web pages will be found by search engines.
SIDENOTE. Don’t you use Google Search Console? Sign up. It’s at no cost. Google Search Console should be used by anyone who runs a website and wants Google to send people to it. That’s how important it is.
You can also use Search Console to see if a particular page has been indexed. Paste the URL into the URL Inspection tool to do this.
If Google has found that page, it will say “URL is on Google.” So index your website on Google is done!
If the page isn’t indexed, “URL is not on Google” will show up.
Index your website on Google, How to Get That?
Noticed that Google doesn’t list your site or page? Here’s what:
- Visit Google Search Console.
- Go to the tool for checking URLs.
- In the search bar, paste the URL you want to index your website on Google.
- Let Google check the URL first.
- Click the button that says “Ask to be indexed.”
- When you publish a new post or page, you should follow these steps. You’re telling Google that you’ve added something new to your site and that they should check it out.
But asking for indexing isn’t likely to fix the problems that keep Google from indexing old pages. If that’s the case, use the list below to figure out what’s wrong and how to fix it.
If you’ve already tried some of these, here are quick links to each one:
1) Remove crawl blocks from your robots.txt file
Index your website on Google hasn’t done yet? It might be because there is a crawl block in a file called robots.txt.
Go to yourdomain.com/robots.txt to see if this is the case.
Look for one of the following two pieces of code:
Both of these tell Googlebot that it can’t crawl any of your site’s pages. To fix the problem, get rid of them. It’s really that easy.
If Google isn’t indexing any web pages, the problem could also be a crawl block in robots.txt. Paste the URL into the URL inspection tool in Google Search Console to see if this is true. Click on the Coverage block to see more information, then look for “Crawl allowed?” No: error “blocked by robots.txt.”
This means that robots.txt has blocked the page.
If that’s the case, check your robots.txt file again to see if there are any “do not allow” rules for that page or section.
Remove where necessary.
2) Remove rogue noindex tags
If you tell Google not to index a page, it won’t. This is a good way to keep some web pages secret. You can do it two ways:
Method 1: meta tag
Index your website on Google won’t be done, especially pages that have either of these meta tags in their <head> section:
This is a meta robots tag, which tells search engines if they can index the page or not.
Run a crawl with Ahrefs Site Audit to find all of the pages on your site that have a noindex meta tag. Go to the Report on Indexability. Look for pages that say “Noindex page.”
Click through to see all the pages that are affected. Remove the noindex meta tag from pages where it doesn’t belong.
Method 2: X-Robots-Tag
The X-Robots-Tag HTTP response header is also taken into account by crawlers. You can do this by using a server-side scripting language like PHP, putting the code in your.htaccess file, or changing the way your server is set up.
You can find out if Google can’t crawl a page because of this header by using the URL inspection tool in Search Console. Just type in your URL and look for the “Indexing allowed?” message. No: ‘noindex’ found in http header ‘X-Robots-Tag'”
Run a crawl of your site with Ahrefs’ Site Audit tool and then use the “Robots information in HTTP header” filter in the Page Explorer to check for this problem:
Tell your developer to keep this header from coming back from pages you want to be indexed.
3) Add the page to the sitemap
A sitemap tells Google which pages on your site are valuable and which ones aren’t. It might also tell you how often you should crawl them again.
Google should be able to find pages on your website even if they’re not in your sitemap, but it’s still a good idea to include them. After all, there’s no reason to make things hard for Google.
Use the URL inspection tool in Search Console to see if a page is in your sitemap. If you see “URL is not on Google” and “Sitemap: N/A,” it’s not in your sitemap and hasn’t been indexed.
Don’t you use Search Console? Head to your sitemap URL—usually, yourdomain.com/sitemap.xml—and search for the page.
Or, run a crawl in Ahrefs’ Site Audit to find all the crawlable and indexable pages that aren’t in your sitemap. Use the following filters in Page Explorer:
Add these pages to your sitemap, because they should be there. Once you’re done, ping this URL to let Google know that you’ve changed your sitemap:
http://www.google.com/ping?sitemap=http://yourwebsite.com/sitemap_url.xml
Change the last part to the URL of your sitemap. Then you should see something similar to this:
That should make it faster to index the page or index your website on Google.
4) Remove rogue canonical tags
A canonical tag tells Google which version of a page is the best one to use. It kind of looks like this:
<link rel=”canonical” href=”/page.html/”>
Most pages either don’t have a canonical tag or have one that points back to itself. This tells Google that the page itself is probably the best and only version. In other words, you want this page to show up in search results.
But if your page has a rogue canonical tag, it could be telling Google about a preferred version of this page that doesn’t exist. Then your page won’t be indexed.
Use Google’s URL inspection tool to see if there is a canonical. If the canonical tag points to another page, you will see a message that says “Alternate page with canonical tag.”
If this shouldn’t be there and you want the page to be indexed, remove the canonical tag.
Note: Not all the time, canonical tags are bad. Most pages that use these tags do so for a good reason. Check the canonical page if you see that your page has a canonical set. If this really is the best version of the page and there’s no reason to also index the other page, the canonical tag should stay.
Run a crawl in Ahrefs’ Site Audit tool to quickly find rogue canonical tags on your whole site. Go to the page navigator. Make these changes:
This looks for pages in your sitemap that have canonical tags that don’t link back to themselves. Since you probably would like to index the pages in your sitemap, you should find out if this filter gives you any results.
It’s likely that either these pages have a broken canonical or they shouldn’t be in your sitemap at all.
5) Make sure the page isn’t alone
Pages that don’t have any internal links to them are called “orphan pages.”
Google finds new content by crawling the web, which means that they can’t find orphaned pages that way. Even people who visit your website won’t be able to find them.
Crawl your site with Ahrefs’ Site Audit to find orphan pages. Next, look in the Links report for errors that say “Orphan page (has no incoming internal links)”:
This shows all the pages on your site that can be indexed and are in your sitemap but don’t have any internal links to them.
Note: This method only works if both of the following are true:
- Your sitemaps have all the pages you want Google to find.
- You checked the box to have the crawl start with the pages in your sitemaps.
Not sure if your sitemap has all the pages you want to be indexed? Here’s what:
Get a list of all your website’s pages on your site (via your CMS)
- Crawl your website
- Compare the URLs on the two lists.
- Orphan pages are URLs that were not found during the crawl.
There are two ways to fix pages that don’t have a parent:
- If the page doesn’t matter, you can delete it and take it off your sitemap.
- If the page is important, you should link to it from other pages on your website.
6) Fix Nofollow internal links
Links with a rel=”nofollow” tag are “nofollow” links. They stop PageRank from being sent to the destination URL. Google also doesn’t follow links that tell it not to.
What Google says about it is as follows:
When we use nofollow, the target links are taken out of our overall graph of the web. But the target pages may still show up in our index if other sites link to them without using nofollow or if the URLs are sent to Google in a Sitemap.
In short, you should follow all internal links that lead to pages that can be indexed.
To do this, crawl your site with the Site Audit tool from Ahrefs. Check the Links report for “Page has nofollow incoming internal links only” errors on pages that can be indexed:
If you want Google to list the page, remove the “nofollow tag” from these internal links. If not, delete the page or tell Google not to index it.
7) Add “powerful” links to your own site
Google finds new content on your website by “crawling” it. If you don’t link to the page you want them to see, they might not be able to find it.
Adding some internal links to the page is a simple way to solve this problem. You can do that from any other website that Google can crawl and index. But if you want Google to index the page ASAP, you should do it from one of your “stronger” pages.
Why? Because Google is likely to re-crawl these pages more quickly than pages that aren’t as important.
To do this, go to Ahrefs’ Site Explorer, type in your domain, and then go to the Best by links report.
This shows all of your website’s pages in order of their URL rating (UR). In other words, the most important pages come up first.
Look through this list and find relevant pages that you can link to the page in question.
The next time Google crawls the page, it will see that link and follow it.
PRO TIP: Paste the page where the internal link was added into Google’s URL inspection tool. Click the “Request indexing” button to tell Google that something on the page has changed and that they should recrawl it as soon as possible. This could make it easier for them to find the internal link and, in turn, the page you want them to index.
8) Make sure the page is unique & valuable
Google probably won’t index low-quality pages because they don’t offer anything useful to its users. Here is what John Mueller of Google had to say about indexing in 2018:
He makes it sound like your website or web page needs to be “awesome and inspiring” for Google to list it.
If you’ve ruled out technical issues as the reason for the lack of indexing, it could be because there isn’t enough value. Because of this, you should look at the page with fresh eyes and ask yourself, “Is this page really helpful?”Would a user find this website useful if they clicked on it from the search results?
If you answered no to either of these questions, you need to work on your content.
With Ahrefs’ Site Audit tool and URL Profiler, you can find more low-quality pages that aren’t indexed. To do this, go to Site Audit’s Page Explorer and use these settings:
This will bring back “thin” pages that can be indexed but don’t get any organic traffic at the moment. That is, there is a good chance that they aren’t indexed.
Export the report, then copy and paste all the URLs into URL Profiler and run a Google Indexation check.
Note: If you do this for a lot of pages, it’s best to use proxies (i.e., over 100). If you don’t, you risk having your IP banned by Google. If you can’t do that, you can look for a “free bulk Google indexation checker” on Google. There are a few of these tools, but most of them can only handle about 25 pages at once.
Check the quality of any pages that aren’t indexed. If you need to, make changes, and then ask Google Search Console to reindex your site.
You should also try to fix problems with content that is already out there. Google probably won’t index pages that are the same or almost the same. Use the Site Audit Duplicate content report to look for these problems.
9) Get rid of low-quality pages to get the most out of your “crawl budget.”
If your website has too many low-quality pages, you are just wasting the crawl budget.
This is what Google says about it:
If server resources are wasted on [low-value-add pages], crawl activity will be taken away from pages that do have value, which could make it take a long time to find great content on a site.
Imagine that a teacher is grading essays and one of them is yours. If they have ten essays to grade, it won’t take long for them to get to yours. If they have 100, it will take a little longer. If they have 1000s, they have too much work, and it’s possible they’ll never get to your essay.
Google does say that “crawl budget is not something most publishers need to worry about” and that “if a site has fewer than a few thousand URLs, it will be crawled efficiently most of the time.”
Still, it’s never a bad idea to get rid of pages on your website that aren’t very good. It can only help crawl the budget in some way.
You can use our content audit template to find pages that might not be very good or aren’t relevant that you can get rid of.
10) Make sure your backlinks are good
Backlinks tell Google that a website page is important. Since someone is linking to it, it must be worth something. These are pages that Google wants to add to its index.
Google doesn’t just index web pages with backlinks so that everything is clear. There are a lot (billions) of pages that have been indexed but have no backlinks. But because Google thinks pages with high-quality links are more important, they are likely to crawl and re-crawl those pages faster than those without them. This makes the indexing go faster.
On the blog, we have a lot of information about how to build good backlinks.
Indexing ≠ ranking
Google’s indexing of your website or web page doesn’t mean that it will rank higher or get more traffic.
They are not the same thing.
When Google indexes your site, it means that Google knows about it. It doesn’t mean that they’re going to put it at the top of the list for any good queries.
This is where SEO comes in. SEO is the art of making sure your web pages rank well for certain searches.
SEO, in short, means:
- Finding out what your customers are looking for.
- Making content around those topics.
- Optimizing those pages for your target keywords.
- Building backlinks.
- Republishing content often helps it stay “evergreen.”
You must be logged in to post a comment.