Overview
Introduction
Whether a page ends up indexed depends on several independent signals working together: robots.txt has to allow crawling it, no noindex directive can be present in the HTML or response headers, and its canonical tag needs to point search engines at the right URL rather than confusing which version is authoritative.
Checking each of those separately means juggling three or four different tools and manually reconciling what they say. This tool runs all of them together and reduces the result to one clear verdict: can this page be indexed, and if not, exactly which signal is responsible.
What Is Crawlability Checker?
A combined indexability checker that cross-references a robots.txt disallow test, meta robots/googlebot and X-Robots-Tag noindex and nofollow directives, and canonical tag status, then rolls all of it into a single indexable/blocked verdict.
It works two ways for each input: fetching a URL directly from your browser when the target allows cross-origin requests, or reading content you paste in yourself, which works regardless of CORS.
How Crawlability Checker Works
robots.txt content (if provided) is parsed into User-agent groups and tested against the page's path using the standard longest-match precedence rules; the HTML is parsed with the browser's DOMParser to read the meta robots tag and canonical link; pasted response headers are checked for an X-Robots-Tag line.
Each signal produces its own item in the breakdown with a good/warning/missing status and explanation. The final verdict is "blocked" if either a noindex directive or a robots.txt disallow rule was found; a nofollow directive and canonical issues are called out in the verdict note without flipping the indexable/blocked result on their own, since neither one alone prevents indexing.
When To Use Crawlability Checker
As a first stop when a page unexpectedly isn't appearing in search results, to see every contributing signal at once instead of checking them one by one.
Before launching a new page or section, to confirm nothing (a leftover staging noindex tag, an overly broad robots.txt rule, a missing canonical) is quietly working against it.
During a site migration or replatform, since CMS and CDN changes commonly touch robots.txt, response headers, and canonical tags all at once.
Often used alongside Robots.txt Tester, Canonical URL Checker and Meta Tag Checker.
Features
Advantages
- Combines robots.txt, meta robots/googlebot, X-Robots-Tag, and canonical checks into one verdict instead of requiring several separate tools.
- Explains the real difference between a robots.txt disallow (blocks crawling) and a noindex directive (blocks indexing) rather than treating them as identical.
- Reuses this category's existing canonical tag logic exactly, so its results are consistent with running the Canonical URL Checker on the same page directly.
Limitations
- A live fetch only works when the target site sends permissive CORS headers, which most ordinary websites don't; use the paste fallback for the HTML, headers, and robots.txt inputs instead.
- The verdict is based on the signals provided - skipping the headers or robots.txt inputs means those checks show as "missing," not as confirmed clear, so a partial input set can understate what's actually blocking a page.
- This tool evaluates what the page's own signals say; it can't tell you whether Google has actually crawled or indexed the page yet, which depends on crawl scheduling this tool has no visibility into.
Examples
Best Practices & Notes
Best Practices
- Provide all three inputs (HTML, headers, and robots.txt) when diagnosing a specific indexing problem, since a partial check can miss the actual cause and give false confidence.
- Treat a "blocked" verdict caused only by robots.txt differently from one caused by noindex - fixing the former means updating the disallow rule, while the latter means removing the meta tag or header entirely.
- Re-run this check after any CDN, reverse-proxy, or CMS change, since those layers can quietly alter response headers or robots.txt without touching the page's own HTML.
Developer Notes
The lib deliberately duplicates a small amount of noindex/nofollow token-parsing logic that also exists in noindex-checker.ts and nofollow-checker.ts, rather than extracting a shared abstraction across three otherwise-independent single-purpose checker tools - the duplicated logic is a few lines of regex/token matching, and a premature shared module would couple three pages together for negligible reuse benefit.
Crawlability Checker Use Cases
- Running a single combined check when a page isn't appearing in search results and the cause isn't obvious
- Pre-launch verification that a new page has no lingering noindex tag, overly broad robots.txt rule, or canonical issue
- Post-migration auditing across robots.txt, headers, and canonical tags at once after a CMS or CDN change
Common Mistakes
- Fixing a robots.txt disallow rule and assuming that alone restores indexing, without checking whether a noindex tag is also present and independently blocking the page.
- Treating a canonical tag issue as the reason a page isn't indexed at all, when in practice it usually just risks the wrong URL being the one that gets indexed, not exclusion outright.
Tips
- Test the exact path you're investigating against robots.txt, not just the domain's root - disallow rules are often scoped to specific subdirectories and won't show up if you test "/" instead.