Crawl Census MCP server
io.github.taylorsmithgg/crawl-censustaylorsmithggstreamable-httpv1.0.0Scanned
Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?
7 tools: 6 change things and 1 only read.
- Reads private data · 0
- Reads text others write · 2
- Can send data out · 0
- No findings
Tools listed without sign-in. Remote posture
With Anlyon
6 tools here change things. Govern them.
- Anlyon lists the tools and adopts none.
- Each tool you adopt gets approvals and receipts.
- Your agent never holds the key.
A tool your agent calls directly is not routed through Anlyon and is not governed by it. Anlyon Radar only lists and labels, and these labels are the same either way.
What the Crawl Census MCP server can do
The Crawl Census MCP server is published by the GitHub account taylorsmithgg. It has 7 tools. It can create or change data (6 tools: scan_site, census_stats, crawl_preflight, agent_profile and 2 more), but cannot delete data. 1 tool only reads.
2 tools bring in text other people can write (crawl_preflight and submit_domains).
None of the fixed checks matched anything in its tool definitions. That is not a safety guarantee.
Its remote endpoint lists its tools without sign-in.
Last scanned Nov 20, 2026, version 1.0.0.
Tools
Grouped by what each tool does, most consequential first. Open one for its reasons and full description; descriptions are the publisher's own words.
7 tools
Changes things 6Claude Code: ask
agent_profileWhat this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network e…No findings
Why it is in this group
- Nothing in its name or description says what it does, so it is counted as a write.
Description, as the server wrote it What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.
census_factsEvery headline finding from the census as discrete, dated records rather than prose.No findings
Why it is in this group
- Nothing in its name or description says what it does, so it is counted as a write.
Description, as the server wrote it Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.
census_statsCorpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.No findings
Why it is in this group
- Nothing in its name or description says what it does, so it is counted as a write.
Description, as the server wrote it Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
crawl_preflightDecide whether a crawler may fetch a list of domains before spending requests on them.Reads text others writeNo findings
Why it is in this group
- Nothing in its name or description says what it does, so it is counted as a write.
What it can reach
- untrusted-inputThe name "crawl_preflight" says "crawl": it reads web pages, which anyone can write.
Description, as the server wrote it Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.
scan_siteRun a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt.No findings
Why it is in this group
- The description's first verb is "run": a word for a change.
Description, as the server wrote it Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.
submit_domainsQueue domains the census has not measured yet so a later crawl_preflight can answer them.Reads text others writeNo findings
Why it is in this group
- The name "submit_domains" says "submit": a word for a change.
What it can reach
- untrusted-inputThe description's first sentence says "crawl": it reads web pages, which anyone can write.
Description, as the server wrote it Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.
Only reads 1Show 1 read-only tool
site_reportReturn the most recent stored audit for a domain without triggering a new scan.No findings
Why it is in this group
- The description's first verb is "return": a word for reading.
Description, as the server wrote it Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
This is not a safety rating, and no finding is not a safety guarantee. The groups and findings come from fixed checks of the tool definitions the server returned on the date shown, not from its code or what it does when called. How the checks work.
Exfil surface
Which tools read private data, which read text someone else can write, and which can send data out. One of each is a path a prompt injection can use.
No private-data, untrusted-input, outbound path in this server alone: no tool is tagged private-data or outbound.
- Reads private data0
- Reads files, mail, messages, databases, secrets, calendars or repositories.
- Reads text others write2
- Brings in text someone else can write: web pages, inbound email, messages, issues and comments.
- Can send data out0
- Can carry data to people or systems: it sends, posts or publishes, or takes a recipient, channel or URL.
Other servers in the same agent add their own tools to these paths. Audit your whole setup to see them together.
Remote posture
What anyone on the internet sees at the remote endpoint: an anonymous initialize and tools/list, and the public OAuth metadata.
Looked at Nov 20, 2026, 19:11 UTC, unchanged since Oct 22, 2026. No credentials were sent and no tool was called. What is looked at.
- Reachable
- yes
- HTTP status of an anonymous initialize
- 200
- tools/list served without credentials
- yes
- Tools that change things, listed without credentials
- 6
- Protocol version
- 2025-06-18
- WWW-Authenticate on 401
- not observed
- resource_metadata in WWW-Authenticate
- not observed
- RFC 9728 protected resource metadata
- not observed
- Authorization servers
- not observed
- Legacy HTTP+SSE transport
- no
- Dynamic client registration
- not observed
- Client ID metadata documents
- not observed
- WarnThe whole server
The endpoint served tools/list without credentials, and the list has 6 tools that change things (6 write). Whether a call also works without credentials was not tested: tools are never called.
What this check looks forMatched text
https://crawlcensus.com/mcp: tools/list returned 7 tools to an anonymous client, including scan_site (write), census_stats (write), crawl_preflight (write), agent_profile (write), census_facts (write) and 1 more
Findings
Matches of fixed checks, with the text they matched.
No check matched anything in these tool definitions. That is not a safety guarantee.
History
Changes to its tool definitions and to its remote endpoint, newest first.
No change seen yet. Its tools are rescanned every day, and a remote endpoint is looked at once a week.
Install
Pinned to version 1.0.0. Values in angle brackets are yours to fill in.
claude mcp add --transport http crawl-census https://crawlcensus.com/mcpclaude mcp add --transport http crawl-census https://crawlcensus.com/mcpRun in a terminal.
Claude Code permissions
Allow the reads, ask before a change or a way out, deny the deletes. Paste into .claude/settings.json.
{
"permissions": {
"allow": [
"mcp__crawl-census__site_report"
],
"ask": [
"mcp__crawl-census__scan_site",
"mcp__crawl-census__census_stats",
"mcp__crawl-census__crawl_preflight",
"mcp__crawl-census__agent_profile",
"mcp__crawl-census__census_facts",
"mcp__crawl-census__submit_domains"
],
"deny": []
}
}Tools that only read are allowed, tools that change things ask first, tools that delete or cancel are denied. A read that can send data out (it takes a URL, say) asks first too. It assumes the server is installed as crawl-census. Review it before you use it.
Capability tags behind 2 rules
- mcp__crawl-census__crawl_preflight untrusted-input
- mcp__crawl-census__submit_domains untrusted-input
Pin
Keep the decision: fail CI when the tools change, follow every change, or show a badge.
Pin in CI
Fail a build when the tools you reviewed are not the tools the server serves today.
curl -s "https://radar.anlyon.com/api/v1/servers/io.github.taylorsmithgg/crawl-census/verify?hash=5d0afbdea63de91efc329b016f575846c686929e4418a1ab97b7f703955ac1e5" | jq -e '.changed == false'Pinned hash 5d0afbdea63d, from Nov 20, 2026, 15:17 UTC.
Watch
Get told when its tools change: a new tool, a new description, a new class.
https://radar.anlyon.com/servers/io.github.taylorsmithgg/crawl-census/changes.xmlAn Atom feed. Works in any feed reader or Slack's RSS app.
Badge
[](https://radar.anlyon.com/servers/io.github.taylorsmithgg/crawl-census)Source: MCP Registry · repository. Synced Oct 21, 2026, 18:50 UTC.
Crawl Census MCP server: questions and answers
Answered from the scan above, not from the publisher's description.
Is the Crawl Census MCP server safe?
- No scan can promise that. What its definitions show: 7 tools, 6 of which can change, send or delete something, and 0 findings from fixed checks. Allow the reads, ask before each change, and deny the deletes; the permissions block on this page does that.
Can the Crawl Census MCP server delete data?
- No tool is classed as destructive: going by their definitions, none of them deletes, cancels or revokes anything.
Can the Crawl Census MCP server do things that cannot be undone?
- No tool is classed as irreversible: none of them sends, pays, publishes or merges, going by their definitions.
What tools does the Crawl Census MCP server have?
- 7 tools: 6 write, 1 read. scan_site, census_stats, crawl_preflight, agent_profile, census_facts, submit_domains and site_report.
Does the Crawl Census MCP server need sign-in or OAuth?
- Its remote endpoint listed its tools to an anonymous client, without sign-in.
Does the Crawl Census MCP server send data to outside services?
- None of its tools sends, posts or publishes, or takes a recipient, channel or URL.
Has the Crawl Census MCP server changed its tools?
- Not since Anlyon Radar started tracking it. Last scanned Nov 20, 2026.