How Do You Check if a Page is Indexable by Google?

To check if a page is indexable by Google, test four gates in order: the robots.txt rule, the meta robots tag, the canonical tag, and the HTTP status code. A page is indexable only when all four allow indexing. One blocked gate removes the page from search results.

Each gate fails in a different way, and each failure produces a different symptom in Google Search Console. Checking them in order saves time, because an earlier gate hides the ones behind it. A page blocked in robots.txt never reveals its meta robots tag, so testing the tag first tells you nothing.

What does indexable mean?

Indexable means Google is permitted to store a page in its index and show it in search results.

Indexability differs from crawlability. Crawlability decides whether a crawler may fetch a URL. Indexability decides whether the fetched page may be stored. A page can be crawlable and not indexable, and a page can be indexed without ever being crawled.

The distinction matters because the two are controlled by different mechanisms. Crawling is controlled by robots.txt. Indexing is controlled by the robots meta tag and the X-Robots-Tag header.

How do you check the robots.txt rule?

Check the robots.txt rule by opening yourdomain.com/robots.txt and matching the URL path against every Disallow line under a user-agent that applies to Googlebot.

A matching Disallow line stops the crawl. A stopped crawl prevents Google from reading anything on the page, including a noindex instruction. This is the most common cause of a page that refuses to leave the index no matter how many times noindex is added.

Three details decide whether a rule matches:

• Longest match wins - Google applies the most specific rule, not the first one. An Allow line longer than a competing Disallow line takes priority.
• Paths are case sensitive - /Blog/ and /blog/ are different rules. Case mismatches silently fail to block anything.
• A trailing slash narrows the rule - Disallow: /blog blocks /blog-archive too. Disallow: /blog/ does not.

Build or repair the file with our Robots.txt Generator before uploading it, because a stray wildcard can deindex an entire directory.

How do you check the meta robots tag?

Check the meta robots tag by viewing the page source and searching the <head> for <meta name="robots">.

A value of noindex removes the page from search results. A value of index, follow permits indexing. A missing tag also permits indexing, because indexing is the default behaviour.

The same instruction can arrive as an HTTP response header named X-Robots-Tag. The header is invisible in the page source, which makes it a frequent blind spot on PDFs, images and pages served through a CDN rule. Any conflict resolves to the most restrictive value: one noindex anywhere wins.

Our Meta Robots Tag Checker reads both the tag and the header in a single request, so a header-level block cannot hide from the audit.

How do you check the canonical tag?

Check the canonical tag by finding <link rel="canonical"> in the <head> and confirming it points at the page’s own URL.

A canonical pointing elsewhere tells Google to index the other URL instead. The page then disappears from results without any error appearing anywhere. Search Console reports it as “Alternate page with proper canonical tag”, which reads like a success message and is routinely ignored.

Four canonical faults account for most cases:

• A relative path that resolves wrongly - Absolute URLs remove the ambiguity entirely.
• A protocol or subdomain mismatch - An http:// canonical on an https:// page splits the signal.
• A template default left in place - Every page canonicalising to the homepage is a template bug, not a strategy.
• A canonical chain - Page A points to B, B points to C. Google follows one hop reliably and no further.

Confirm the value resolves correctly with our Canonical URL Checker.

How do you check the HTTP status code?

Check the HTTP status code by requesting the URL and reading the response line. Only a 200 OK response is indexable in its own right.

Every other class of response changes the outcome:

• 301 and 308 - The signal transfers permanently to the target URL. The source URL leaves the index.
• 302 and 307 - The redirect is temporary, so Google usually keeps indexing the source URL.
• 404 and 410 - The page is dropped. A 410 is dropped faster because it asserts permanence.
• 500 and 503 - Google retries. Sustained server errors eventually remove the page.

Redirect chains deserve particular attention. Each extra hop adds latency and dilutes the signal reaching the destination. Trace the full path with our Redirect Checker and flatten every chain to a single hop.

Why is a page crawled but not indexed?

A page is crawled but not indexed when Google fetched it successfully and then chose not to store it. The status “Crawled – currently not indexed” is a quality judgement, not a technical block.

All four gates pass in this state. Nothing is misconfigured. Google has simply decided the page does not add enough to the index to justify a place in it.

Three causes explain most of these pages. Thin content offers less than the pages already ranking. Duplicate content repeats what another URL on the same site says. Low internal link depth signals that the site itself does not treat the page as important.

The fix is editorial rather than technical. Strengthen the page, or link to it from pages that already perform, or consolidate it into a stronger URL with a redirect.

How do you check indexability in bulk?

Check indexability in bulk by running a list of URLs through a status checker first, then auditing the tags on everything that returns 200.

The order matters. Status codes eliminate the majority of problem URLs in one pass, which leaves a much smaller set worth inspecting tag by tag.

Run the list through our Bulk HTTP Status Checker to isolate the redirects and errors. Then pull titles, descriptions and robots directives for the survivors with our Bulk Meta Tags Extractor. Two passes over a few hundred URLs takes minutes and surfaces every gate failure on the site.

Final Thoughts

Indexability is a sequence of four permissions, and diagnosis means testing them in the order Google applies them. Robots.txt decides the fetch. The robots tag decides the storage. The canonical decides which URL gets credit. The status code decides whether there is a page to store at all.

Work through them in that order whenever a page goes missing from search results, and start with our Meta Robots Tag Checker for a single URL.