Octocrawl
Pages, crawls and URL batches as clean Markdown and JSON for AI agents, with an Evidence Record.
| Endpoint | https://mcp.octocrawl.dev/mcp (streamable-http) |
|---|---|
| Registry version | 0.8.13 |
| Now | 2 tools · grade D (61) · pin ts_4700c0159a3fba7124330a0da96bdd1d · protocol 2025-06-18 |
| First / last listed | 2026-11-14 01:36 UTC / 2026-11-14 01:36 UTC |
| Grade over time | 2026-11-14: D (61), 2 tools |
Is this your server? Own a server? ToolFit hosted watch re-checks it every 24 h ($0.25 per 30 days) or 6 h ($0.60) and sends you an HMAC-signed webhook when its tools or grade change: POST https://api.youraitools.co/toolfit/v1/watch. To opt out, see About & opt-out.
Tools: scrape map
scrape | Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name what the content cannot vouch for (client_rendered_suspected, robots_overridden), and agentHints the next step. params: url*, mode, allowlistedDomains, formats, includeLinks, debug, markdownOffset, onlyMainContent, timeout, maxFileBytes, includeTags, excludeTags, headers, mobile, fastMode, blockAds, removeBase64Images, verify, integration |
map | List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the de params: url*, search, sitemap, includeSubdomains, ignoreQueryParameters, limit, timeout, includePaths, excludePaths, regexOnFullURL, crawlEntireDomain, deduplicateSimilarURLs, mode, debug, integration |
Entries describe changes in tool definitions. They are not statements about a server, its quality or its author. The grade measures how the tool definitions fit model providers (token cost, schema compatibility, confusable names) as scored by the ToolFit engine at the time of the listing. JSON: append .json. Opt-out: how or hello@youraitools.co.