AI Gateway Dashboard
Route Workers AI through Cloudflare AI Gateway and surface cache hits, latency, and gateway metadata for observability demos
Call Workers AI through AI Gateway instead of direct inference. Returns generated text plus gateway metadata (cache status, latency). Optionally compare cached vs fresh requests for the same prompt.
Features
- POST /generate - text generation via AI Gateway
- Returns latency and cache hit/miss metadata
- Optional compareCache mode runs cached vs skipCache requests
API Reference
POST /generate
prompt string (required) - Text prompt for the model.
compareCache boolean (optional) - Run cached and fresh requests and compare latency.
Default model: @cf/meta/llama-3.1-8b-instruct-fast.
Example Request
curl -X POST "https://your-worker.workers.dev/generate" \
-H "Content-Type: application/json" \
-d '{"prompt":"Explain Workers AI in one sentence","compareCache":true}'Error Codes
400-INVALID_BODY,MISSING_PROMPT502-AI_ERROR
Use Cases
- Learn AI Gateway cache behavior vs direct Workers AI calls
- Compare latency for repeated prompts in prototyping
- Reference pattern for gateway options in production Workers
Limitations
- Requires Workers AI and AI Gateway enabled on your account
- Cache metadata depends on gateway configuration
- Text generation only; fixed default model
Use in your project
Copy these files into an existing Worker. Prefer Deployment to try the full experiment first. Source: apps/experiments/ai-gateway-dashboard.
import { DEFAULT_MODEL, GATEWAY_ID } from "../constants/defaults";import type { Env } from "../types/env";import type { CacheComparison, GatewayMetadata, GenerateResponse } from "../types/generate";type AiGatewayRunOptions = { gateway: { id: string; skipCache?: boolean; };};type AiRunResult = { response?: string; result?: string; "cf-aig-cache-status"?: string;};function extractResponseText(out: unknown): string { const result = out as AiRunResult; return result.response ?? result.result ?? "";}function extractCacheStatus(out: unknown, skipCache: boolean): GatewayMetadata["cacheStatus"] { if (skipCache) return "BYPASS"; const headerStatus = (out as AiRunResult)["cf-aig-cache-status"]; if (headerStatus === "HIT" || headerStatus === "MISS") { return headerStatus; } return "UNKNOWN";}async function runThroughGateway( env: Env, prompt: string, skipCache: boolean): Promise<{ text: string; gateway: GatewayMetadata }> { const startedAt = performance.now(); const options: AiGatewayRunOptions = { gateway: { id: GATEWAY_ID, skipCache, }, }; const out = await env.AI.run(DEFAULT_MODEL, { prompt }, options); const latencyMs = Math.round(performance.now() - startedAt); return { text: extractResponseText(out), gateway: { id: GATEWAY_ID, model: DEFAULT_MODEL, latencyMs, cacheStatus: extractCacheStatus(out, skipCache), skipCache, }, };}export async function generateWithGateway( env: Env, prompt: string, compareCache: boolean): Promise<GenerateResponse> { if (!compareCache) { const result = await runThroughGateway(env, prompt, false); return { response: result.text, gateway: result.gateway, }; } const cached = await runThroughGateway(env, prompt, false); const uncached = await runThroughGateway(env, prompt, true); const cacheComparison: CacheComparison = { cached: { latencyMs: cached.gateway.latencyMs, cacheStatus: cached.gateway.cacheStatus, skipCache: false, }, skipCache: { latencyMs: uncached.gateway.latencyMs, cacheStatus: "BYPASS", skipCache: true, }, latencyDeltaMs: uncached.gateway.latencyMs - cached.gateway.latencyMs, }; return { response: cached.text, gateway: cached.gateway, cacheComparison, };}Deployment
Configure bindings
AI binding in wrangler.json. Use gateway id default or your own gateway ID in code.
Test your deployment
See the experiment README for curl examples.
Local Development
cd apps/experiments/ai-gateway-dashboard
npm install
npm run devConfiguration
AI binding in wrangler.json. Use gateway id default or your own gateway ID in code.