Blog20 min read
Background jobs at the edge with Cloudflare Queues
Deep Cloudflare Queues guide - producers, consumers, retries, idempotency, vs Workflows, batch AI, and the Task Queue experiment.
Screenshot pipelines, bulk AI calls, and fan-out notifications should not block the user’s request. Cloudflare Queues let a Worker publish messages and a consumer process them asynchronously with retries - keeping the HTTP path under the catalog’s ~60 second guideline.
Learn the wiring with Task Queue, then specialize.
Cloudflare products in this pattern
| Product | Role | Docs |
|---|---|---|
| Queues | Durable message buffer between Workers | Product home |
| Queue producers | env.QUEUE.send | Config |
| Queue consumers | Batch handler Worker | Consumers |
| Workers | Produce + consume entrypoints | Runtime |
| Workflows | Multi-step durable orchestration | Workflows |
| R2 | Store large payloads by reference | Objects |
| Durable Objects | Job dashboards / coordination | State |
| Workers AI | Async inference consumers | AI |
Queues vs doing it inline vs Workflows
| Work | Prefer | Docs |
|---|---|---|
| Validate input + enqueue | Request Worker (producer) | Queues |
| Slow AI / scrape / email | Queue consumer | Consumers |
| Multi-step durable waits | Workflows | Workflows |
| Sticky per-entity coordination | Durable Objects | DOs |
If the user must wait for the result, stream or poll - do not hold the original request open for batch jobs. See also Queues vs other products in Cloudflare’s docs for positioning.
Producer / consumer shape
HTTP POST /jobs → producer Worker → queue.send(message)
↓
consumer Worker batch handler
↓
R2 / D1 / webhook side effectsOfficial guides:
Message schema tips:
- Include a
jobIdandcreatedAt - Version the payload (
v: 1) - Keep messages small; store blobs in R2 and pass keys
- Design for at-least-once delivery (delivery guarantees)
Experiments that teach the ecosystem
| Goal | Experiment | Product |
|---|---|---|
| Basic producer/consumer | Task Queue | Queues |
| Inspect job flow with a DO view | Queue Job Visualizer | Queues + Durable Objects |
| Async embeddings | AI Batch Infer | Queues + Workers AI |
| Broader event fan-in | Event Pipeline | Pipelines / R2 |
| Multi-step orchestration | Workflows Pipeline Demo | Workflows |
Reliability checklist
- Make consumers idempotent - duplicates happen (delivery guarantees)
- Configure retries / dead-letter intentionally (retries)
- Log
jobIdend-to-end - Separate “accept job” (
202) from “job finished” - Monitor lag; a silent consumer is worse than a failing HTTP route
- Bound batch sizes and concurrency to stay within Worker limits
Deploy Task Queue
- Read Queues get started
- Open Task Queue
- Deploy producer + consumer bindings
- Enqueue a few jobs from the documented route
- Watch the consumer drain them
- Map your payload onto the same wiring
- Consider Workflows when steps need durable waits
You get a working async spine for edge apps - the difference between a demo API and something you can put behind a button that says “processing,” built on Cloudflare Queues rather than a self-hosted broker.
Production checklist
- Version message schemas.
- Keep payloads small; put blobs in R2.
- Make consumers idempotent (delivery guarantees).
- Configure batch size, concurrency, and retries (batching & retries).
- Separate accept (
202) from completion. - Prefer Workflows when steps need durable waits and branching.
FAQ
Queue vs Workflow vs Cron?
- Queue: many independent messages
- Workflow: multi-step durable process
- Cron: time-based tick (Cron Triggers)
Related experiments
These deployable references implement the patterns in this article. Open the docs for API details, then Deploy or clone the source.
- Task Queue - Cloudflare Queues reference: enqueue tasks over HTTP and process them asynchronously with a consumer Worker
- Queue Job Visualizer - Cloudflare Queues producer/consumer with KV-backed job status and simulated retries - visual job queue reference
- AI Batch Infer - Workers AI async batch embeddings with queueRequest and polling - process large embedding jobs without blocking the request path
- Event Pipeline - Ingest events into Cloudflare Pipelines streaming to R2/Iceberg - event ingestion and analytics lake pattern
- Workflows Pipeline Demo - Durable fetch → AI summarize → R2 pipeline with Cloudflare Workflows - durable multi-step orchestration reference