Skip to content

AI Crawlability Check MCP server

io.github.Milan344s/ai-crawlability-checkMilan344sstreamable-httpv2.0.0Scanned

Check if AI crawlers like GPTBot and ClaudeBot can reach, index and quote any web page.

5 tools: 2 change things and 3 only read.

Tools: 0 delete, 0 irreversible, 2 write, 3 read.
  • Reads private data · 0
  • Reads text others write · 5
  • Can send data out · 4
  • No findings

Tools listed without sign-in. Remote posture

With Anlyon

2 tools here change things. Govern them.

  • Anlyon lists the tools and adopts none.
  • Each tool you adopt gets approvals and receipts.
  • Your agent never holds the key.
Govern this server →

A tool your agent calls directly is not routed through Anlyon and is not governed by it. Anlyon Radar only lists and labels, and these labels are the same either way.

What the AI Crawlability Check MCP server can do

The AI Crawlability Check MCP server is published by the GitHub account Milan344s. It has 5 tools. It can create or change data (2 tools: check_ai_crawler_access and check_ai_indexability), but cannot delete data. 3 tools only read.

5 tools bring in text other people can write (check_ai_crawler_access, check_ai_indexability, check_ai_citability and 2 more); 4 tools can send data out of your system (check_ai_crawler_access, check_ai_indexability, check_ai_citability and 1 more).

None of the fixed checks matched anything in its tool definitions. That is not a safety guarantee.

Its remote endpoint lists its tools without sign-in.

Last scanned Nov 19, 2026, version 2.0.0.

Tools

Grouped by what each tool does, most consequential first. Open one for its reasons and full description; descriptions are the publisher's own words.

5 tools

Changes things 2Claude Code: ask
  • check_ai_crawler_accessReports which AI crawlers may fetch a public URL.Reads text others writeCan send data outNo findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    What it can reach

    • untrusted-inputThe server marks it openWorldHint: true: it reaches things outside the server's own data.
    • outboundThe input takes a URL (input.url): a request to a URL can carry data to whoever runs it.
    Description, as the server wrote it
    Reports which AI crawlers may fetch a public URL. Reads the site's robots.txt and returns allowed or blocked, with the deciding rule, for each known AI crawler from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon and others, split into search and answer-time bots versus training bots. Also checks whether the server answers an automated request, noindex and nosnippet directives, declared sitemaps, Content-Signal lines and llms.txt. Use for "are we blocking GPTBot or ClaudeBot?", "can ChatGPT read this page?" or an AI crawler audit of robots.txt.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: true
  • check_ai_indexabilityChecks whether a public page can be indexed by search and AI engines once a crawler reaches it.Reads text others writeCan send data outNo findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    What it can reach

    • untrusted-inputThe server marks it openWorldHint: true: it reaches things outside the server's own data.
    • outboundThe input takes a URL (input.url): a request to a URL can carry data to whoever runs it.
    Description, as the server wrote it
    Checks whether a public page can be indexed by search and AI engines once a crawler reaches it. Measures HTTP status and redirects, noindex, canonical target, how much copy exists in the raw HTML without JavaScript, client-side rendering signals, title and H1, XML sitemap coverage, response time, HTML size, structured data validity, dates and language. Use for "why is this page not showing up in ChatGPT or Perplexity?", "is this page indexable?" or to check a JavaScript-rendered page. It does not query any index.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: true
Only reads 3Show 3 read-only tools
  • check_ai_citabilityFetches a public page and scores how quotable its content is for AI engines once it is crawled and indexed.Reads text others writeCan send data outNo findings

    Why it is in this group

    • The description's first verb is "fetch": a word for reading.

    What it can reach

    • untrusted-inputThe server marks it openWorldHint: true: it reaches things outside the server's own data.
    • untrusted-inputThe input takes a URL (input.url), and the tool reads it: it reads web pages, which anyone can write.
    • outboundThe input takes a URL (input.url): a request to a URL can carry data to whoever runs it.

    Claude Code: ask, because it is tagged outbound.

    Description, as the server wrote it
    Fetches a public page and scores how quotable its content is for AI engines once it is crawled and indexed. Measures answer-first structure, question headings, lists and tables, FAQ, paragraph and sentence length, statistic and source density, named entities, schema, author, freshness and llms.txt, plus a short crawler-access summary. Use for "will AI cite this page?", "is this article LLM-friendly?" or a GEO check. Tuned for articles and guides.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: true
  • check_seo_basicsFetches a public page and runs 24 measurable on-page SEO checks: title and meta description length, H1, heading hierarchy, keyword placement and density, slug, canonical, internal …Reads text others writeCan send data outNo findings

    Why it is in this group

    • The description's first verb is "fetch": a word for reading.

    What it can reach

    • untrusted-inputThe server marks it openWorldHint: true: it reaches things outside the server's own data.
    • untrusted-inputThe input takes a URL (input.url), and the tool reads it: it reads web pages, which anyone can write.
    • outboundThe input takes a URL (input.url): a request to a URL can carry data to whoever runs it.

    Claude Code: ask, because it is tagged outbound.

    Description, as the server wrote it
    Fetches a public page and runs 24 measurable on-page SEO checks: title and meta description length, H1, heading hierarchy, keyword placement and density, slug, canonical, internal and external links, anchor text, image alt text, schema and social tags. Pass the target keyword to enable the keyword checks. Does not cover rankings or backlinks.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: true
  • get_review_rubricReturns a scoring rubric as text for the judgment items the URL tools cannot measure: geo (would an AI engine quote this article?) or seo (pre-publish SEO review).Reads text others writeNo findings

    Why it is in this group

    • The name "get_review_rubric" says "get": a word for reading.

    What it can reach

    • untrusted-inputThe name "get_review_rubric" says "get" and "review": it reads comments, which other people can write.
    • untrusted-inputThe description's first sentence says "return" and "url": it reads web pages, which anyone can write.
    Description, as the server wrote it
    Returns a scoring rubric as text for the judgment items the URL tools cannot measure: geo (would an AI engine quote this article?) or seo (pre-publish SEO review). Fetches nothing.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false

This is not a safety rating, and no finding is not a safety guarantee. The groups and findings come from fixed checks of the tool definitions the server returned on the date shown, not from its code or what it does when called. How the checks work.

Exfil surface

Which tools read private data, which read text someone else can write, and which can send data out. One of each is a path a prompt injection can use.

No private-data, untrusted-input, outbound path in this server alone: no tool is tagged private-data.

Reads private data0
Reads files, mail, messages, databases, secrets, calendars or repositories.
Reads text others write5
Brings in text someone else can write: web pages, inbound email, messages, issues and comments.
Can send data out4
Can carry data to people or systems: it sends, posts or publishes, or takes a recipient, channel or URL.

Other servers in the same agent add their own tools to these paths. Audit your whole setup to see them together.

Remote posture

What anyone on the internet sees at the remote endpoint: an anonymous initialize and tools/list, and the public OAuth metadata.

Tools listed without sign-in

Looked at Nov 19, 2026, 17:19 UTC, unchanged since Nov 19, 2026. No credentials were sent and no tool was called. What is looked at.

Reachable
yes
HTTP status of an anonymous initialize
200
tools/list served without credentials
yes
Tools that change things, listed without credentials
2
Protocol version
2025-06-18
WWW-Authenticate on 401
not observed
resource_metadata in WWW-Authenticate
not observed
RFC 9728 protected resource metadata
not observed
Authorization servers
not observed
Legacy HTTP+SSE transport
no
Dynamic client registration
not observed
Client ID metadata documents
not observed
  • WarnThe whole server

    The endpoint served tools/list without credentials, and the list has 2 tools that change things (2 write). Whether a call also works without credentials was not tested: tools are never called.

    Matched text

    https://mcp.milan-novotny.com/mcp: tools/list returned 5 tools to an anonymous client, including check_ai_crawler_access (write), check_ai_indexability (write)

    What this check looks for

Findings

Matches of fixed checks, with the text they matched.

No check matched anything in these tool definitions. That is not a safety guarantee.

History

Changes to its tool definitions and to its remote endpoint, newest first.

No change seen yet. Its tools are rescanned every day, and a remote endpoint is looked at once a week.

Install

Pinned to version 2.0.0. Values in angle brackets are yours to fill in.

claude mcp add --transport http ai-crawlability-check https://mcp.milan-novotny.com/mcp
claude mcp add --transport http ai-crawlability-check https://mcp.milan-novotny.com/mcp

Run in a terminal.

Claude Code permissions

Allow the reads, ask before a change or a way out, deny the deletes. Paste into .claude/settings.json.

.claude/settings.json
{
  "permissions": {
    "allow": [
      "mcp__ai-crawlability-check__get_review_rubric"
    ],
    "ask": [
      "mcp__ai-crawlability-check__check_ai_crawler_access",
      "mcp__ai-crawlability-check__check_ai_indexability",
      "mcp__ai-crawlability-check__check_ai_citability",
      "mcp__ai-crawlability-check__check_seo_basics"
    ],
    "deny": []
  }
}

Tools that only read are allowed, tools that change things ask first, tools that delete or cancel are denied. A read that can send data out (it takes a URL, say) asks first too. It assumes the server is installed as ai-crawlability-check. Review it before you use it.

Capability tags behind 5 rules
  • mcp__ai-crawlability-check__check_ai_crawler_access untrusted-input, outbound
  • mcp__ai-crawlability-check__check_ai_indexability untrusted-input, outbound
  • mcp__ai-crawlability-check__check_ai_citability untrusted-input, outbound
  • mcp__ai-crawlability-check__check_seo_basics untrusted-input, outbound
  • mcp__ai-crawlability-check__get_review_rubric untrusted-input

Pin

Keep the decision: fail CI when the tools change, follow every change, or show a badge.

Pin in CI

Fail a build when the tools you reviewed are not the tools the server serves today.

curl -s "https://radar.anlyon.com/api/v1/servers/io.github.Milan344s/ai-crawlability-check/verify?hash=22ca20bfe0670b4f9ab8051f17d5d51e52a73c79e8bfe33113471b7f6b451a20" | jq -e '.changed == false'

Pinned hash 22ca20bfe067, from Nov 19, 2026, 14:18 UTC.

Watch

Get told when its tools change: a new tool, a new description, a new class.

https://radar.anlyon.com/servers/io.github.Milan344s/ai-crawlability-check/changes.xml

An Atom feed. Works in any feed reader or Slack's RSS app.

Badge

Anlyon Radar badge for io.github.Milan344s/ai-crawlability-check
[![Anlyon Radar](https://radar.anlyon.com/badge/io.github.Milan344s/ai-crawlability-check.svg)](https://radar.anlyon.com/servers/io.github.Milan344s/ai-crawlability-check)

Source: MCP Registry · repository. Synced Nov 19, 2026, 14:17 UTC.

AI Crawlability Check MCP server: questions and answers

Answered from the scan above, not from the publisher's description.

Is the AI Crawlability Check MCP server safe?

No scan can promise that. What its definitions show: 5 tools, 2 of which can change, send or delete something, and 0 findings from fixed checks. Allow the reads, ask before each change, and deny the deletes; the permissions block on this page does that.

Can the AI Crawlability Check MCP server delete data?

No tool is classed as destructive: going by their definitions, none of them deletes, cancels or revokes anything.

Can the AI Crawlability Check MCP server do things that cannot be undone?

No tool is classed as irreversible: none of them sends, pays, publishes or merges, going by their definitions.

What tools does the AI Crawlability Check MCP server have?

5 tools: 2 write, 3 read. check_ai_crawler_access, check_ai_indexability, check_ai_citability, check_seo_basics and get_review_rubric.

Does the AI Crawlability Check MCP server need sign-in or OAuth?

Its remote endpoint listed its tools to an anonymous client, without sign-in.

Does the AI Crawlability Check MCP server send data to outside services?

4 tools can carry data out (it sends, posts or publishes, or takes a recipient, channel or URL): check_ai_crawler_access, check_ai_indexability, check_ai_citability and check_seo_basics.

Has the AI Crawlability Check MCP server changed its tools?

Not since Anlyon Radar started tracking it. Last scanned Nov 19, 2026.