Make your site visible to AI crawlers and agents

AI assistants and crawlers increasingly decide whether your docs are usable from machine-readable surfaces: robots.txt, sitemaps, llms.txt, OpenAPI, MCP, and clean metadata. If those are missing or contradictory, agents skip you - even when the HTML is excellent.

This catalog dogfoods those surfaces (/llms.txt, /openapi.json, /developers, MCP). You can audit and generate them with focused experiments running on Cloudflare Workers.

Cloudflare products that help (directly or as the runtime)Copy Anchor Link

Also relevant: Cloudflare’s own guidance around AI crawlers and content signals evolves quickly - start from AI Crawl Control and your zone’s bot settings.

The agent-friendly checklistCopy Anchor Link

1. robots.txt - allow the AI bots you want; do not accidentally blanket-deny
2. sitemap.xml - list canonical URLs you want indexed
3. llms.txt - short index of when-to-use + key links for LLM agents
4. Markdown alternatives - Accept: text/markdown or per-page markdown routes
5. OpenAPI / API catalog - for callable APIs (OpenAPI on Workers)
6. MCP - if you expose tools (Model Context Protocol)
7. Accurate metadata - title, description, canonical, OG tags
8. Security headers - trustworthy defaults without breaking embeds you need
9. Content-Signal / usage preferences - declare search vs training preferences where you publish them

Experiments that helpCopy Anchor Link

Practical robots.txt tips for AICopy Anchor Link

• Be explicit per user-agent when policies differ (retrieval vs training)
• Keep Sitemap: absolute
• Do not Disallow: / for * if you want search/AI input - prefer path-level rules
• Re-test after CDN or framework changes; generators often overwrite robots
• Align robots with zone-level AI Crawl Control so edge policy and file policy do not contradict

AI Bot Visibility is the fastest way to see how popular crawlers interpret your file.

llms.txt that agents actually useCopy Anchor Link

A useful llms.txt is not a dump of every URL. Include:

• One-paragraph product summary
• When to use / not to use
• Links to OpenAPI, MCP, full markdown dump
• A curated experiment or doc list

This site’s /llms.txt follows that shape. Generate starting material for arbitrary sites with Website to llms.txt, then edit for accuracy.

Hosting tip: serve llms.txt from a Worker or static assets with long cache and easy redeploys (Workers static assets when applicable).

OpenAPI + MCP for callable surfacesCopy Anchor Link

If agents should call you, not just read you:

• Publish OpenAPI 3.x at a stable URL
• Keep operationIds unique and descriptions concrete
• Return JSON errors with error + code (+ hints)
• Offer MCP when tool-calling clients are a target audience

Workers are a natural host for these JSON surfaces (Workers routing).

Metadata and trust pagesCopy Anchor Link

Agents and humans both use About / Contact / Privacy as trust anchors. Ensure:

• Titles/descriptions via Metadata Extractor patterns
• Security headers via Security Headers Grader
• Canonical URLs that match the sitemap

Ship the auditsCopy Anchor Link

1. Run AI Bot Visibility against your production origin
2. Inspect robots/sitemap with Robots Sitemap Inspector
3. Review zone settings in AI Crawl Control
4. Publish an llms.txt at the site root
5. Ensure titles/descriptions via Metadata Extractor
6. Recheck after each major docs deploy

Agent discoverability is SEO for machines. These experiments turn it into a checklist you can automate - each one deploys as a small Worker on Cloudflare, and the products above are the levers you actually tune in the dashboard and docs.

Production checklistCopy Anchor Link

1. Align robots.txt with AI Crawl Control zone settings.
2. Publish sitemap.xml and llms.txt.
3. Offer Markdown / OpenAPI / MCP where APIs exist.
4. Keep About / Contact / Privacy findable.
5. Re-audit after framework or CDN changes with AI Bot Visibility.
6. Cache machine-readable docs aggressively via Cache.

FAQCopy Anchor Link

Allow all AI bots?Copy Anchor Link

Only if that matches your content policy. Many teams allow retrieval bots and restrict training bots - be explicit per user-agent.