Robots Sitemap Inspector

Fetch and parse robots.txt plus linked sitemaps for SEO and AI-crawler debugging from Cloudflare's edge.

Features

• Parses User-agent, Allow, Disallow, and Sitemap directives
• Inspects up to 5 sitemap URLs from robots.txt
• Distinguishes urlset vs sitemapindex and returns sample URLs
• Stateless; no bindings required

API Reference

GET /inspect

Example Request

curl "https://your-worker.workers.dev/inspect?url=https://www.cloudflare.com"

Success Response

url string

Input URL that was inspected

robots object

Parsed robots.txt: present, url, groups, sitemaps, optional rawPreview / error

sitemaps array

Per-sitemap results with ok, type, urlCount, sampleUrls or childSitemaps

{
  "url": "https://www.cloudflare.com/",
  "robots": {
    "present": true,
    "url": "https://www.cloudflare.com/robots.txt",
    "groups": [{ "userAgent": "*", "allow": [], "disallow": ["/cdn-cgi/"] }],
    "sitemaps": ["https://www.cloudflare.com/sitemap.xml"]
  },
  "sitemaps": [
    {
      "url": "https://www.cloudflare.com/sitemap.xml",
      "ok": true,
      "type": "sitemapindex",
      "urlCount": 3,
      "childSitemaps": ["https://www.cloudflare.com/sitemap-0.xml"]
    }
  ]
}

Error Codes

• 400 - Missing or invalid url (INVALID_URL)
• 502 - Unexpected fetch failure (FETCH_ERROR)

Missing robots.txt (404) still returns 200 with robots.present: false.

Use Cases

• Debug crawl rules before launching content
• Verify sitemap discovery from robots.txt
• Complement AI bot visibility checks with full robots/sitemap context
• Audit SEO crawl configuration from the edge

Limitations

• Fetches at most 5 sitemaps and samples a small set of URLs
• Does not recursively expand every child sitemap in an index
• Robots matching semantics are summarized, not a full crawler simulator

Deployment

curl "https://your-worker.workers.dev/inspect?url=https://example.com"

Local Development

cd apps/experiments/robots-sitemap-inspector
npm install
npm run dev

curl "http://localhost:8787/inspect?url=https://www.cloudflare.com"

Cloudflare Features Used

• Workers - Edge compute runtime
• Fetch API - Fetch robots.txt and sitemaps

Next Steps