Web Scraping & Parsing
Robots Sitemap Inspector
Parse robots.txt and linked sitemaps for any URL - SEO and AI-crawler access debugging from Cloudflare's edge
Fetch and parse robots.txt plus linked sitemaps for SEO and AI-crawler debugging from Cloudflare's edge.
Features
- Parses
User-agent,Allow,Disallow, andSitemapdirectives - Inspects up to 5 sitemap URLs from robots.txt
- Distinguishes
urlsetvssitemapindexand returns sample URLs - Stateless; no bindings required
API Reference
GET /inspect
Prop
Type
Example Request
curl "https://your-worker.workers.dev/inspect?url=https://www.cloudflare.com"Success Response
url string
Input URL that was inspected
robots object
Parsed robots.txt: present, url, groups, sitemaps, optional rawPreview / error
sitemaps array
Per-sitemap results with ok, type, urlCount, sampleUrls or childSitemaps
{
"url": "https://www.cloudflare.com/",
"robots": {
"present": true,
"url": "https://www.cloudflare.com/robots.txt",
"groups": [{ "userAgent": "*", "allow": [], "disallow": ["/cdn-cgi/"] }],
"sitemaps": ["https://www.cloudflare.com/sitemap.xml"]
},
"sitemaps": [
{
"url": "https://www.cloudflare.com/sitemap.xml",
"ok": true,
"type": "sitemapindex",
"urlCount": 3,
"childSitemaps": ["https://www.cloudflare.com/sitemap-0.xml"]
}
]
}Error Codes
400- Missing or invalidurl(INVALID_URL)502- Unexpected fetch failure (FETCH_ERROR)
Missing robots.txt (404) still returns 200 with robots.present: false.
Use Cases
- Debug crawl rules before launching content
- Verify sitemap discovery from robots.txt
- Complement AI bot visibility checks with full robots/sitemap context
- Audit SEO crawl configuration from the edge
Limitations
- Fetches at most 5 sitemaps and samples a small set of URLs
- Does not recursively expand every child sitemap in an index
- Robots matching semantics are summarized, not a full crawler simulator
Use in your project
Copy these files into an existing Worker. Prefer Deployment to try the full experiment first. Source: apps/experiments/robots-sitemap-inspector.
DependenciesNoneBindingsNonePlatformFetch API
import { FETCH_TIMEOUT_MS, MAX_SITEMAPS, USER_AGENT } from "../constants/defaults";import type { InspectResponse } from "../types/inspect";import { parseRobotsTxt } from "./robots";import { inspectSitemap } from "./sitemap";import { originFromUrl } from "./url";async function fetchRobots( robotsUrl: string): Promise<{ ok: true; text: string } | { ok: false; status?: number; error: string }> { const controller = new AbortController(); const timeoutId = setTimeout(() => controller.abort(), FETCH_TIMEOUT_MS); try { const res = await fetch(robotsUrl, { method: "GET", redirect: "follow", signal: controller.signal, headers: { "User-Agent": USER_AGENT, Accept: "text/plain,*/*" }, }); clearTimeout(timeoutId); if (!res.ok) { return { ok: false, status: res.status, error: `HTTP ${res.status}` }; } return { ok: true, text: await res.text() }; } catch (e) { clearTimeout(timeoutId); return { ok: false, error: e instanceof Error ? e.message : "Unknown error" }; }}/** * Inspect robots.txt and linked sitemaps for a URL's origin. */export async function inspectRobotsAndSitemaps(url: string): Promise<InspectResponse> { const origin = originFromUrl(url); const robotsUrl = `${origin}/robots.txt`; const robotsFetch = await fetchRobots(robotsUrl); if (!robotsFetch.ok) { return { url, robots: { present: false, url: robotsUrl, groups: [], sitemaps: [], error: robotsFetch.error, }, sitemaps: [], }; } const parsed = parseRobotsTxt(robotsFetch.text); const sitemapUrls = parsed.sitemaps.slice(0, MAX_SITEMAPS); const sitemaps = await Promise.all(sitemapUrls.map((s) => inspectSitemap(s))); return { url, robots: { present: true, url: robotsUrl, groups: parsed.groups, sitemaps: parsed.sitemaps, rawPreview: robotsFetch.text.slice(0, 500), }, sitemaps, };}Deployment
Deploy
Follow the deployment wizard. No bindings or secrets required.
Test your deployment
curl "https://your-worker.workers.dev/inspect?url=https://example.com"Local Development
cd apps/experiments/robots-sitemap-inspector
npm install
npm run devcurl "http://localhost:8787/inspect?url=https://www.cloudflare.com"