What is AI crawlability?
AI crawlability is whether the crawlers behind AI search engines and assistants are allowed and able to fetch a page. A page that ChatGPT, Claude, Perplexity or Google AI Overviews cannot fetch cannot be shown or quoted by them, however good the content is.
AI Crawlability Check answers three questions about any public URL, in the order a crawler meets them:
- Can AI crawlers reach it? robots.txt rules for 29 crawlers, firewall refusals, noindex and nosnippet directives.
- Can it be indexed? Status code, canonical, copy present without JavaScript, sitemap coverage, speed and structured data.
- Is it quotable? Answer-first structure, question headings, statistics with sources, named entities, author and freshness.
It runs 70 measured checks across four tools and returns a 0 to 100 score, the measured value behind every check and a fix for each failure.
How do you connect AI Crawlability Check?
Add one URL to your AI client as a remote Model Context Protocol server. The server uses the Streamable HTTP transport and needs no sign-in.
https://mcp.milan-novotny.com/mcp
- Claude (web and desktop). Open Settings, go to Connectors, choose "Add custom connector" and paste the URL.
- Claude Code. Run this command in a terminal:
claude mcp add --transport http ai-crawlability https://mcp.milan-novotny.com/mcp
- Cursor and other clients with a JSON config. Add this entry to the MCP servers file:
{ "mcpServers": { "ai-crawlability": { "url": "https://mcp.milan-novotny.com/mcp" } } }
Then ask in plain language. Three prompts that work:
- "Are we blocking GPTBot, ClaudeBot or PerplexityBot on https://example.com/pricing?"
- "Check whether this page can be indexed by AI search engines, and list the fixes."
- "Run all the crawlability checks on our five most important URLs and give me one table."
What can AI Crawlability Check do?
The server has five read-only tools. Four fetch a live URL and measure it. One returns a review rubric for the judgment a script cannot make.
| Tool | Question it answers | What it returns |
|---|---|---|
check_ai_crawler_access | Which AI crawlers may fetch this URL? | Allowed or blocked for 29 crawlers with the deciding robots.txt rule, plus 9 scored checks: robots.txt health, catch-all rules, server refusals, noindex, snippets, sitemaps and llms.txt. |
check_ai_indexability | Can this page be indexed once a crawler reaches it? | 15 scored checks: status and redirects, noindex, canonical, copy in raw HTML, JavaScript dependence, sitemap coverage, response time, HTML size, structured data and dates. |
check_ai_citability | Is the content easy for an AI engine to quote? | 22 scored checks on structure, evidence, entities and freshness. |
check_seo_basics | Are the on-page SEO basics in place? | 24 scored checks: title, meta description, H1, heading order, keyword use, slug, canonical, links, alt text and schema. |
get_review_rubric | How should a person or model judge the copy? | The AI citability rubric or the pre-publish SEO rubric as text. |
This is an MCP server and not a prompt because a language model cannot reliably fetch robots.txt, match crawler rules or count the words in raw HTML on its own. The server measures and the model explains.
Which AI crawlers does it check?
The crawler access check covers 29 crawlers from 14 organisations: 19 that build AI search indexes or fetch pages when a person asks a question, and 10 that collect or control training data. It follows RFC 9309, the robots.txt standard: the most specific user-agent group wins, the longest matching rule wins, and Allow beats Disallow on a tie.
Which crawlers decide whether a page appears in AI answers?
These 19 search and answer-time crawlers decide visibility. Blocking one removes the page from that product's answers. The last column is each vendor's stated position on robots.txt.
| Crawler | Owner | What it is for | robots.txt |
|---|---|---|---|
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Honors |
ChatGPT-User | OpenAI | Live fetch when a ChatGPT user or custom GPT asks | May not apply (user-triggered) |
Claude-SearchBot | Anthropic | Crawls to improve search results in Claude | Honors |
Claude-User | Anthropic | Live fetch when a Claude user asks | Honors |
PerplexityBot | Perplexity | Perplexity search index; not used for model training | Honors |
Perplexity-User | Perplexity | Live fetch when a Perplexity user asks | Generally ignores (user-triggered) |
Googlebot | Google Search, including AI Overviews and AI Mode | Honors | |
Bingbot | Microsoft | Bing index, which also grounds Copilot | Honors |
Applebot | Apple | Search across Siri, Spotlight and Safari | Honors |
DuckAssistBot | DuckDuckGo | Real-time fetch for DuckDuckGo AI answers; no training | Honors |
meta-webindexer | Meta | Meta AI search results and citations | Not stated |
meta-externalfetcher | Meta | User-requested link fetches for Meta AI | May bypass (user-triggered) |
Amzn-SearchBot | Amazon | Search in Amazon products such as Alexa; no training | Honors |
Amzn-User | Amazon | Live fetch for Amazon user requests; no training | May not follow all rules (user-triggered) |
MistralAI-Index | Mistral | Mistral search index; no training | Not stated |
MistralAI-User | Mistral | Live fetch when a Le Chat user asks | Honors |
YouBot | You.com | You.com search index | Honors |
Google-Agent | Google-hosted agents browsing for a user | Generally ignores (user-triggered) | |
Claude-Agent | Anthropic | Hosted Claude browser acting for a user | Not stated |
Which crawlers collect training data?
These 10 crawlers and control tokens govern model training. Blocking them is a legitimate choice and does not remove a page from AI search, which is why it costs 8 of 100 points here while blocking answer-time crawlers costs 30.
| Crawler | Owner | What it is for | robots.txt |
|---|---|---|---|
GPTBot | OpenAI | OpenAI model training | Honors |
ClaudeBot | Anthropic | Anthropic model training | Honors |
Google-Extended | Control token for Gemini training and grounding; no effect on Search | Honors | |
Applebot-Extended | Apple | Control token for Apple AI training on Applebot data | Honors |
meta-externalagent | Meta | Training Meta AI models and improving products | Not stated |
Amazonbot | Amazon | Improves Amazon products; data may train Amazon AI models | Honors |
MistralAI-Training | Mistral | Mistral training datasets | Honors |
CCBot | Common Crawl | Open web archive, widely reused as AI training data | Honors |
AI2Bot | Allen Institute for AI | Training open language models | Not stated |
Bytespider | ByteDance | ByteDance crawler, reported as model training | No vendor documentation found |
Names and purposes verified against each vendor's own crawler documentation on 19 November 2026: OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, Mistral, DuckDuckGo, You.com, Common Crawl and the Allen Institute for AI. Bytespider has no vendor documentation and is included because it appears in most block lists.
robots.txt is a request, not a lock. Several vendors document that their user-triggered fetchers may not follow it, because a person asked for that page. The check reports what your robots.txt says and what each vendor says it does with it.
The check also reads Content-Signal lines, the robots.txt extension Cloudflare introduced for stating search, ai-input and ai-train preferences, and reports them as written.
What does the indexability check measure?
The indexability check measures 15 things that decide whether a page can enter a search or AI index after a crawler fetches it. The three that fail most often on modern sites are copy that only exists after JavaScript runs, a canonical tag that points at another URL, and a page missing from the XML sitemap.
| Area | Checks |
|---|---|
| Response | HTTP status, redirects, response time, HTML size. |
| Directives | noindex in meta tags and headers, robots.txt rules for answer-time bots, canonical target, snippet permissions. |
| Rendering | Words present in the raw HTML, empty app-shell containers and script count, title and H1 in the initial response. |
| Discovery and meaning | XML sitemap coverage, valid structured data, a machine-readable date, a declared language. |
The page is read as raw HTML without running JavaScript, because that is how most AI crawlers read it. A client-rendered app that shows 2,000 words in a browser can show 40 words to a crawler.
What does the citability check measure?
The citability check measures 22 signals in four groups and weights them to a score out of 100. A score of 85 or more is Strong, 70 to 84 is Good, 50 to 69 is Leaking and under 50 is Weak.
| Group | Points | Checks |
|---|---|---|
| Access | 30 | Indexable, answer-time bots allowed, training bots allowed, copy in the HTML without JavaScript, snippets allowed. |
| Structure | 30 | One H1, an opening paragraph that answers first, question headings, lists and tables, an FAQ block, paragraph length, sentence length. |
| Evidence | 25 | Quantified facts per 1,000 words, statistics with a year or a named source, links to outside sources, named entities, hedging and hype phrases. |
| Entity and freshness | 15 | Schema markup, a named author, a machine-readable date, Person or Organization schema with sameAs, llms.txt. |
The citability score is tuned for articles and guides, so a product page or a home page loses evidence points by design. On 19 November 2026 the generative engine optimization statistics page on this site scored 98 and the about page scored 90.
What are the limits of AI Crawlability Check?
AI Crawlability Check measures whether a page can be crawled, indexed and quoted. It does not measure whether any AI engine does so today.
- It queries no index. It cannot confirm that a page is in ChatGPT's, Perplexity's or Google's index.
- Bot-specific firewall rules are invisible. It sees robots.txt, meta tags and headers. It reports when its own request is refused, and it never pretends to be another company's crawler.
- No JavaScript. Pages are read as raw HTML on purpose.
- Sitemaps are sampled. It reads at most 5 sitemap files, and reports "not checked" on larger sites instead of a false failure.
- One request, one location. Response time is a single measurement from a Cloudflare data centre.
- Public pages only. Only public http and https addresses are accepted, and pages behind a login cannot be checked.
The server keeps no state and stores no input. It identifies itself as milan-novotny-mcp/1.0. The full privacy policy and terms are one short page.
FAQ
Is the AI Crawlability Check MCP free?
Yes. The AI Crawlability Check MCP is free, needs no account and no API key. You only need an AI client that supports remote MCP servers, such as Claude or Claude Code.
How do I check if my site blocks GPTBot or ClaudeBot?
Connect the server and ask your assistant to check AI crawler access for a URL. The check_ai_crawler_access tool reads the site's robots.txt and reports allowed or blocked for 29 AI crawlers, with the exact rule that decided each one.
Which AI crawlers does it check?
It checks 29 crawlers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, DuckAssistBot, meta-webindexer, meta-externalfetcher, Amzn-SearchBot, Amzn-User, MistralAI-Index, MistralAI-User, YouBot, Google-Agent, Claude-Agent, GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, MistralAI-Training, CCBot, AI2Bot and Bytespider. Every name except Bytespider was verified against the vendor's own crawler documentation on 19 November 2026.
Does a high score mean ChatGPT or Perplexity will cite my page?
No. The scores measure whether a page can be crawled, indexed and quoted. They do not query any AI engine, so they cannot tell you whether a page is cited today. That also depends on the query, the competition and each engine's index.
Can it see a firewall that blocks AI bots?
Only partly. The server requests the page once with its own user agent and reports if that request is refused, which is a sign that a firewall or bot manager is filtering automated clients. It never impersonates another company's crawler, so it cannot test a rule aimed at one specific bot.
Does the server store the URLs I check?
No. The server keeps no state and has no database. It fetches the page you name plus that site's robots.txt, llms.txt and sitemaps, returns the result and keeps nothing.