Free MCP server · No account, no key · Updated 19 Nov 2026

AI Crawlability Check

AI Crawlability Check is a free Model Context Protocol (MCP) server that tells your AI assistant whether 29 AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, can reach a web page, whether the page can be indexed, and whether its content is quotable.

What is AI crawlability?

AI crawlability is whether the crawlers behind AI search engines and assistants are allowed and able to fetch a page. A page that ChatGPT, Claude, Perplexity or Google AI Overviews cannot fetch cannot be shown or quoted by them, however good the content is.

AI Crawlability Check answers three questions about any public URL, in the order a crawler meets them:

  1. Can AI crawlers reach it? robots.txt rules for 29 crawlers, firewall refusals, noindex and nosnippet directives.
  2. Can it be indexed? Status code, canonical, copy present without JavaScript, sitemap coverage, speed and structured data.
  3. Is it quotable? Answer-first structure, question headings, statistics with sources, named entities, author and freshness.

It runs 70 measured checks across four tools and returns a 0 to 100 score, the measured value behind every check and a fix for each failure.

How do you connect AI Crawlability Check?

Add one URL to your AI client as a remote Model Context Protocol server. The server uses the Streamable HTTP transport and needs no sign-in.

https://mcp.milan-novotny.com/mcp
  1. Claude (web and desktop). Open Settings, go to Connectors, choose "Add custom connector" and paste the URL.
  2. Claude Code. Run this command in a terminal:
    claude mcp add --transport http ai-crawlability https://mcp.milan-novotny.com/mcp
  3. Cursor and other clients with a JSON config. Add this entry to the MCP servers file:
    {
      "mcpServers": {
        "ai-crawlability": { "url": "https://mcp.milan-novotny.com/mcp" }
      }
    }

Then ask in plain language. Three prompts that work:

  • "Are we blocking GPTBot, ClaudeBot or PerplexityBot on https://example.com/pricing?"
  • "Check whether this page can be indexed by AI search engines, and list the fixes."
  • "Run all the crawlability checks on our five most important URLs and give me one table."

What can AI Crawlability Check do?

The server has five read-only tools. Four fetch a live URL and measure it. One returns a review rubric for the judgment a script cannot make.

ToolQuestion it answersWhat it returns
check_ai_crawler_accessWhich AI crawlers may fetch this URL?Allowed or blocked for 29 crawlers with the deciding robots.txt rule, plus 9 scored checks: robots.txt health, catch-all rules, server refusals, noindex, snippets, sitemaps and llms.txt.
check_ai_indexabilityCan this page be indexed once a crawler reaches it?15 scored checks: status and redirects, noindex, canonical, copy in raw HTML, JavaScript dependence, sitemap coverage, response time, HTML size, structured data and dates.
check_ai_citabilityIs the content easy for an AI engine to quote?22 scored checks on structure, evidence, entities and freshness.
check_seo_basicsAre the on-page SEO basics in place?24 scored checks: title, meta description, H1, heading order, keyword use, slug, canonical, links, alt text and schema.
get_review_rubricHow should a person or model judge the copy?The AI citability rubric or the pre-publish SEO rubric as text.
Analyst note

This is an MCP server and not a prompt because a language model cannot reliably fetch robots.txt, match crawler rules or count the words in raw HTML on its own. The server measures and the model explains.

Which AI crawlers does it check?

The crawler access check covers 29 crawlers from 14 organisations: 19 that build AI search indexes or fetch pages when a person asks a question, and 10 that collect or control training data. It follows RFC 9309, the robots.txt standard: the most specific user-agent group wins, the longest matching rule wins, and Allow beats Disallow on a tie.

Which crawlers decide whether a page appears in AI answers?

These 19 search and answer-time crawlers decide visibility. Blocking one removes the page from that product's answers. The last column is each vendor's stated position on robots.txt.

CrawlerOwnerWhat it is forrobots.txt
OAI-SearchBotOpenAISurfaces sites in ChatGPT searchHonors
ChatGPT-UserOpenAILive fetch when a ChatGPT user or custom GPT asksMay not apply (user-triggered)
Claude-SearchBotAnthropicCrawls to improve search results in ClaudeHonors
Claude-UserAnthropicLive fetch when a Claude user asksHonors
PerplexityBotPerplexityPerplexity search index; not used for model trainingHonors
Perplexity-UserPerplexityLive fetch when a Perplexity user asksGenerally ignores (user-triggered)
GooglebotGoogleGoogle Search, including AI Overviews and AI ModeHonors
BingbotMicrosoftBing index, which also grounds CopilotHonors
ApplebotAppleSearch across Siri, Spotlight and SafariHonors
DuckAssistBotDuckDuckGoReal-time fetch for DuckDuckGo AI answers; no trainingHonors
meta-webindexerMetaMeta AI search results and citationsNot stated
meta-externalfetcherMetaUser-requested link fetches for Meta AIMay bypass (user-triggered)
Amzn-SearchBotAmazonSearch in Amazon products such as Alexa; no trainingHonors
Amzn-UserAmazonLive fetch for Amazon user requests; no trainingMay not follow all rules (user-triggered)
MistralAI-IndexMistralMistral search index; no trainingNot stated
MistralAI-UserMistralLive fetch when a Le Chat user asksHonors
YouBotYou.comYou.com search indexHonors
Google-AgentGoogleGoogle-hosted agents browsing for a userGenerally ignores (user-triggered)
Claude-AgentAnthropicHosted Claude browser acting for a userNot stated

Which crawlers collect training data?

These 10 crawlers and control tokens govern model training. Blocking them is a legitimate choice and does not remove a page from AI search, which is why it costs 8 of 100 points here while blocking answer-time crawlers costs 30.

CrawlerOwnerWhat it is forrobots.txt
GPTBotOpenAIOpenAI model trainingHonors
ClaudeBotAnthropicAnthropic model trainingHonors
Google-ExtendedGoogleControl token for Gemini training and grounding; no effect on SearchHonors
Applebot-ExtendedAppleControl token for Apple AI training on Applebot dataHonors
meta-externalagentMetaTraining Meta AI models and improving productsNot stated
AmazonbotAmazonImproves Amazon products; data may train Amazon AI modelsHonors
MistralAI-TrainingMistralMistral training datasetsHonors
CCBotCommon CrawlOpen web archive, widely reused as AI training dataHonors
AI2BotAllen Institute for AITraining open language modelsNot stated
BytespiderByteDanceByteDance crawler, reported as model trainingNo vendor documentation found

Names and purposes verified against each vendor's own crawler documentation on 19 November 2026: OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, Mistral, DuckDuckGo, You.com, Common Crawl and the Allen Institute for AI. Bytespider has no vendor documentation and is included because it appears in most block lists.

Analyst note

robots.txt is a request, not a lock. Several vendors document that their user-triggered fetchers may not follow it, because a person asked for that page. The check reports what your robots.txt says and what each vendor says it does with it.

The check also reads Content-Signal lines, the robots.txt extension Cloudflare introduced for stating search, ai-input and ai-train preferences, and reports them as written.

What does the indexability check measure?

The indexability check measures 15 things that decide whether a page can enter a search or AI index after a crawler fetches it. The three that fail most often on modern sites are copy that only exists after JavaScript runs, a canonical tag that points at another URL, and a page missing from the XML sitemap.

AreaChecks
ResponseHTTP status, redirects, response time, HTML size.
Directivesnoindex in meta tags and headers, robots.txt rules for answer-time bots, canonical target, snippet permissions.
RenderingWords present in the raw HTML, empty app-shell containers and script count, title and H1 in the initial response.
Discovery and meaningXML sitemap coverage, valid structured data, a machine-readable date, a declared language.

The page is read as raw HTML without running JavaScript, because that is how most AI crawlers read it. A client-rendered app that shows 2,000 words in a browser can show 40 words to a crawler.

What does the citability check measure?

The citability check measures 22 signals in four groups and weights them to a score out of 100. A score of 85 or more is Strong, 70 to 84 is Good, 50 to 69 is Leaking and under 50 is Weak.

GroupPointsChecks
Access30Indexable, answer-time bots allowed, training bots allowed, copy in the HTML without JavaScript, snippets allowed.
Structure30One H1, an opening paragraph that answers first, question headings, lists and tables, an FAQ block, paragraph length, sentence length.
Evidence25Quantified facts per 1,000 words, statistics with a year or a named source, links to outside sources, named entities, hedging and hype phrases.
Entity and freshness15Schema markup, a named author, a machine-readable date, Person or Organization schema with sameAs, llms.txt.

The citability score is tuned for articles and guides, so a product page or a home page loses evidence points by design. On 19 November 2026 the generative engine optimization statistics page on this site scored 98 and the about page scored 90.

What are the limits of AI Crawlability Check?

AI Crawlability Check measures whether a page can be crawled, indexed and quoted. It does not measure whether any AI engine does so today.

  • It queries no index. It cannot confirm that a page is in ChatGPT's, Perplexity's or Google's index.
  • Bot-specific firewall rules are invisible. It sees robots.txt, meta tags and headers. It reports when its own request is refused, and it never pretends to be another company's crawler.
  • No JavaScript. Pages are read as raw HTML on purpose.
  • Sitemaps are sampled. It reads at most 5 sitemap files, and reports "not checked" on larger sites instead of a false failure.
  • One request, one location. Response time is a single measurement from a Cloudflare data centre.
  • Public pages only. Only public http and https addresses are accepted, and pages behind a login cannot be checked.

The server keeps no state and stores no input. It identifies itself as milan-novotny-mcp/1.0. The full privacy policy and terms are one short page.

FAQ

Is the AI Crawlability Check MCP free?

Yes. The AI Crawlability Check MCP is free, needs no account and no API key. You only need an AI client that supports remote MCP servers, such as Claude or Claude Code.

How do I check if my site blocks GPTBot or ClaudeBot?

Connect the server and ask your assistant to check AI crawler access for a URL. The check_ai_crawler_access tool reads the site's robots.txt and reports allowed or blocked for 29 AI crawlers, with the exact rule that decided each one.

Which AI crawlers does it check?

It checks 29 crawlers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, DuckAssistBot, meta-webindexer, meta-externalfetcher, Amzn-SearchBot, Amzn-User, MistralAI-Index, MistralAI-User, YouBot, Google-Agent, Claude-Agent, GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, MistralAI-Training, CCBot, AI2Bot and Bytespider. Every name except Bytespider was verified against the vendor's own crawler documentation on 19 November 2026.

Does a high score mean ChatGPT or Perplexity will cite my page?

No. The scores measure whether a page can be crawled, indexed and quoted. They do not query any AI engine, so they cannot tell you whether a page is cited today. That also depends on the query, the competition and each engine's index.

Can it see a firewall that blocks AI bots?

Only partly. The server requests the page once with its own user agent and reports if that request is refused, which is a sign that a firewall or bot manager is filtering automated clients. It never impersonates another company's crawler, so it cannot test a rule aimed at one specific bot.

Does the server store the URLs I check?

No. The server keeps no state and has no database. It fetches the page you name plus that site's robots.txt, llms.txt and sitemaps, returns the result and keeps nothing.

About the author

Milan Novotný is the Senior Director of Growth Marketing at CloudTalk, a B2B SaaS company. The organic programme he leads was a 2026 Search Engine Land Awards finalist for Best Use of AI Technology in Search Marketing, and won Silver at the European Search Awards and Gold at Digital PIE. AI Crawlability Check is a personal project. Full track record.