RobotsGate
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
| Endpoint | https://robotsgate.mike-tusa.workers.dev/mcp (streamable-http) |
|---|---|
| Registry version | 0.3.1 |
| Now | 4 tools · grade A (98) · pin ts_e80e76741677c039975d9f3e81643ad5 · protocol 2025-06-18 |
| First / last listed | 2026-11-06 02:00 UTC / 2026-11-06 02:00 UTC |
| Grade over time | 2026-11-06: A (98), 4 tools |
Is this your server? Own a server? ToolFit hosted watch re-checks it every 24 h ($0.25 per 30 days) or 6 h ($0.60) and sends you an HMAC-signed webhook when its tools or grade change: POST https://api.youraitools.co/toolfit/v1/watch. To opt out, see About & opt-out.
Tools: list_crawlers generate_robots validate_robots check_site
list_crawlers | Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers. params: category |
generate_robots | Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate. params: categories, agents, default, disallow_paths, allow_paths, sitemaps, sitemap, custom_rules, date |
validate_robots | Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site. params: robots_txt*, path |
check_site | Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check. params: url*, path |
Entries describe changes in tool definitions. They are not statements about a server, its quality or its author. The grade measures how the tool definitions fit model providers (token cost, schema compatibility, confusable names) as scored by the ToolFit engine at the time of the listing. JSON: append .json. Opt-out: how or hello@youraitools.co.