This site is not affiliated with or endorsed by Cloudflare, Inc. It showcases deployable experiments across most Cloudflare products.

Blog20 min read

Background jobs at the edge with Cloudflare Queues

Deep Cloudflare Queues guide - producers, consumers, retries, idempotency, vs Workflows, batch AI, and the Task Queue experiment.

Warehouse shelving packed with stored inventory boxes
Photo by CHUTTERSNAP on Unsplash

Screenshot pipelines, bulk AI calls, and fan-out notifications should not block the user’s request. Cloudflare Queues let a Worker publish messages and a consumer process them asynchronously with retries - keeping the HTTP path under the catalog’s ~60 second guideline.

Learn the wiring with Task Queue, then specialize.

Cloudflare products in this pattern

ProductRoleDocs
QueuesDurable message buffer between WorkersProduct home
Queue producersenv.QUEUE.sendConfig
Queue consumersBatch handler WorkerConsumers
WorkersProduce + consume entrypointsRuntime
WorkflowsMulti-step durable orchestrationWorkflows
R2Store large payloads by referenceObjects
Durable ObjectsJob dashboards / coordinationState
Workers AIAsync inference consumersAI

Queues vs doing it inline vs Workflows

WorkPreferDocs
Validate input + enqueueRequest Worker (producer)Queues
Slow AI / scrape / emailQueue consumerConsumers
Multi-step durable waitsWorkflowsWorkflows
Sticky per-entity coordinationDurable ObjectsDOs

If the user must wait for the result, stream or poll - do not hold the original request open for batch jobs. See also Queues vs other products in Cloudflare’s docs for positioning.

Producer / consumer shape

HTTP POST /jobs → producer Worker → queue.send(message)
                                      ↓
                               consumer Worker batch handler
                                      ↓
                               R2 / D1 / webhook side effects

Official guides:

Message schema tips:

  • Include a jobId and createdAt
  • Version the payload (v: 1)
  • Keep messages small; store blobs in R2 and pass keys
  • Design for at-least-once delivery (delivery guarantees)

Experiments that teach the ecosystem

GoalExperimentProduct
Basic producer/consumerTask QueueQueues
Inspect job flow with a DO viewQueue Job VisualizerQueues + Durable Objects
Async embeddingsAI Batch InferQueues + Workers AI
Broader event fan-inEvent PipelinePipelines / R2
Multi-step orchestrationWorkflows Pipeline DemoWorkflows

Reliability checklist

  1. Make consumers idempotent - duplicates happen (delivery guarantees)
  2. Configure retries / dead-letter intentionally (retries)
  3. Log jobId end-to-end
  4. Separate “accept job” (202) from “job finished”
  5. Monitor lag; a silent consumer is worse than a failing HTTP route
  6. Bound batch sizes and concurrency to stay within Worker limits

Deploy Task Queue

  1. Read Queues get started
  2. Open Task Queue
  3. Deploy producer + consumer bindings
  4. Enqueue a few jobs from the documented route
  5. Watch the consumer drain them
  6. Map your payload onto the same wiring
  7. Consider Workflows when steps need durable waits

You get a working async spine for edge apps - the difference between a demo API and something you can put behind a button that says “processing,” built on Cloudflare Queues rather than a self-hosted broker.

Production checklist

  1. Version message schemas.
  2. Keep payloads small; put blobs in R2.
  3. Make consumers idempotent (delivery guarantees).
  4. Configure batch size, concurrency, and retries (batching & retries).
  5. Separate accept (202) from completion.
  6. Prefer Workflows when steps need durable waits and branching.

FAQ

Queue vs Workflow vs Cron?

  • Queue: many independent messages
  • Workflow: multi-step durable process
  • Cron: time-based tick (Cron Triggers)

Related experiments

These deployable references implement the patterns in this article. Open the docs for API details, then Deploy or clone the source.

  • Task Queue - Cloudflare Queues reference: enqueue tasks over HTTP and process them asynchronously with a consumer Worker
  • Queue Job Visualizer - Cloudflare Queues producer/consumer with KV-backed job status and simulated retries - visual job queue reference
  • AI Batch Infer - Workers AI async batch embeddings with queueRequest and polling - process large embedding jobs without blocking the request path
  • Event Pipeline - Ingest events into Cloudflare Pipelines streaming to R2/Iceberg - event ingestion and analytics lake pattern
  • Workflows Pipeline Demo - Durable fetch → AI summarize → R2 pipeline with Cloudflare Workflows - durable multi-step orchestration reference

Cloudflare Experiments is an independent open-source catalog - not affiliated with Cloudflare, Inc.