Crawl Census
Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?
| Endpoint | https://crawlcensus.com/mcp (streamable-http) |
|---|---|
| Registry version | 1.0.0 |
| Now | 7 tools · grade A (96) · pin ts_1f02b08b64268fffc69e874cc26ef9e4 · protocol 2025-06-18 |
| First / last listed | 2026-10-30 01:19 UTC / 2026-10-30 01:19 UTC |
| Grade over time | 2026-10-30: A (96), 7 tools |
Is this your server? Own a server? ToolFit hosted watch re-checks it every 24 h ($0.25 per 30 days) or 6 h ($0.60) and sends you an HMAC-signed webhook when its tools or grade change: POST https://api.youraitools.co/toolfit/v1/watch. To opt out, see About & opt-out.
Tools: scan_site site_report census_stats crawl_preflight agent_profile census_facts submit_domains
scan_site | Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail. params: domain* |
site_report | Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site. params: domain* |
census_stats | Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption. params: none |
crawl_preflight | Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it) params: agent*, domains* |
agent_profile | What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the corr params: agent* |
census_facts | Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date params: none |
submit_domains | Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a params: domains* |
Entries describe changes in tool definitions. They are not statements about a server, its quality or its author. The grade measures how the tool definitions fit model providers (token cost, schema compatibility, confusable names) as scored by the ToolFit engine at the time of the listing. JSON: append .json. Opt-out: how or hello@youraitools.co.