A worker sends a receipt, then crashes before acknowledging the job. The receipt arrived. The queue still sees unfinished work. Another worker picks it up and sends it again.
The queue cannot resolve that uncertainty by itself. It knows whether the job was acknowledged; it does not know what happened inside the email service.
That gap is where most of the design of Backstage comes from. It is a worker orchestration SDK for TypeScript/Bun and Go, with providers for Redis Streams, RabbitMQ, and Kafka. It handles task registration, chaining, retry decisions, and settlement through a common worker interface. The provider handles delivery.
Redis Streams is the default. It is also where the project started. But describing Backstage as a Redis queue now leaves out the architectural decision that matters: the worker should express what needs to happen to a job without implementing each broker’s acknowledgement protocol.
A delivery has three ways to settle
The worker receives a task name, payload, delivery count, and execution metadata. Alongside those fields are three operations:
This is the TypeScript delivery interface. The Go SDK has the corresponding contract, with context-aware methods.
A handler that finishes successfully is acknowledged. A failure goes through the retry policy. Once the delivery count exceeds the configured limit, the worker requests dead-lettering. The error travels with that decision, rather than existing only in the worker’s logs.
Those operations mean different things underneath. Redis can leave an entry pending for a reclaimer. RabbitMQ can publish a retry copy and acknowledge the original delivery. Kafka can schedule another delivery and advance the consumer’s committed position. The worker calls the same methods, but the providers retain their own recovery mechanisms.
That distinction matters when operating the system. A shared API gives application code a consistent shape; it does not make broker behaviour identical.
Jobs, topics, delays, and deduplication are separate capabilities
A provider must supply jobs. It can also supply topics, delays, and deduplication. Those optional capabilities can be supplied separately when the chosen provider does not have them.
Jobs distribute work between consumers. Topics deliver application events to subscribers. Delays hold work until a future time. Deduplication claims a key for a bounded period before publication.
I separated these because choosing a broker should not require pretending it implements every feature in the same way.
| Provider | Jobs and topics | Delays | Deduplication |
| --- | --- | --- | --- |
| Redis Streams | Included | Redis sorted sets and a promotion loop | Included |
| RabbitMQ | Included | Delayed-message plugin, or a supplied delay capability | Supply separately |
| Kafka | Included | Supply separately | Supply separately |
The worker checks its required capabilities at startup. Kafka jobs require delays because retries need somewhere to wait. RabbitMQ discovers whether its delayed-message exchange is available during initialization; without it, a delay capability must be supplied. Calling a feature that needs an unavailable capability fails explicitly.
For example, a TypeScript worker can use Kafka for delivery and a Redis-backed delay store:
This is the provider configuration; handlers and worker startup follow it. The Kafka adapter also needs the optional kafkajs dependency. A single-broker development cluster needs partitions: 1 and replicationFactor: 1, overriding the adapter’s defaults of three each.
When the worker starts, it binds the delay store to the active jobs provider and starts its promotion loop. Due work is published into Kafka. It is not left on a Redis work queue merely because Redis held the timer. The separately created delay store also needs to be closed when the application shuts down.
Publishing across those two systems is not atomic. The delay store retains claimed entries until publication succeeds and retries abandoned claims. A crash after publication but before removing the claim can still produce a duplicate.
Task chaining keeps the handoff explicit
A handler can return a named next task, its payload, and an optional delay. Returning nothing finishes the chain.
Here, processOrder and sendReceipt are application functions. The example uses the default Redis provider.
The worker enqueues or schedules the next task before acknowledging the current delivery. If that handoff fails, the current task goes through failure handling instead of being silently marked complete.
There is still a gap between publishing the next task and acknowledging this one. A crash in that gap can cause the current handler to run again and publish the next task twice. Each step needs an idempotency strategy, including steps that mostly dispatch more work.
This is task chaining, not durable replay of arbitrary application code. Backstage does not memoize every completed operation inside a handler or provide a transaction across the chain.
Redis recovers pending work
With the Redis Streams provider, workers read through consumer groups. Delivery puts an entry into the group’s pending-entry list. Successful acknowledgement removes it from that list; deleting the underlying stream entry is a separate configuration choice.
The provider limits new reads according to its available in-flight allowance. It also uses a dedicated connection for blocking reads, so an idle XREADGROUP ... BLOCK does not hold up publishing or acknowledgements on the shared connection.
When a handler fails, the Redis delivery remains pending. The provider records the error, and a reclaimer looks for entries old enough to retry. It uses XPENDING to inspect idle deliveries and XCLAIM to transfer them to a consumer. Retry timing depends on the idle threshold and the reclaim loop, with backoff metadata checked during reclaim.
This is why the common retry({ delayMs }) method should not be read as a promise of identical timing on every provider. The Redis implementation does not create a new timer for every failed delivery. RabbitMQ and Kafka take different paths.
A pending entry also does not prove that its worker has died. A healthy handler may simply be taking longer than expected. Reclaiming it can produce concurrent execution. Timeout settings need to reflect the work, and external side effects still need protection against duplicates.
Successful Redis acknowledgements are batched to reduce round trips. That leaves another small replay window: a handler may finish before its acknowledgement reaches Redis. The job can be delivered again even though the application function returned successfully.
RabbitMQ and Kafka recover differently
The RabbitMQ adapter publishes persistent messages into durable queues and waits for publisher confirms. Consumers use manual acknowledgements and a prefetch limit.
On a handled failure, a retry is published or scheduled before the original delivery is acknowledged. Dead-lettering similarly waits for the failed job to be published to the dead-letter queue before acknowledging the source delivery. A worker losing its connection leaves unacknowledged work for broker redelivery.
The Kafka adapter uses explicit offset commits. Its settlement tracking advances the committed position only through settled messages within a partition; finishing later work must not commit past earlier unfinished work. A handled failure schedules a retry before settling the original record. Dead-lettering publishes to the dead-letter topic before settlement.
The TypeScript Kafka producer is configured for idempotent production and all-replica acknowledgements. That does not make the application handler exactly-once. The adapter does not use Kafka transactions to combine a handler’s side effects, retry publication, and offset commit.
These details belong in the provider. Putting them there keeps broker-specific recovery code out of every application handler without concealing the differences from someone operating the system.
Topics need their own delivery contract
A job queue and a broadcast serve different purposes. Several workers sharing a job consumer group should divide the work. Subscribers to an application event may each need their own copy.
Backstage exposes topics separately from jobs. The Redis provider uses a stream per topic with subscriber groups. RabbitMQ uses a topic exchange and subscriber queues. Kafka uses topics and consumer groups.
Subscription identity and start position still matter. Redis and Kafka can use retained history according to their subscription settings. RabbitMQ subscriptions without a named group use temporary queues, and creating a new queue does not replay messages that were published before it existed. “Supports topics” is not a promise of the same retention and replay behaviour across all three.
The boundary is still the side effect
Backstage’s delivery model is at-least-once. Its usefulness depends on taking that literally.
An idempotency key at the email or payment boundary can prevent a repeated delivery from repeating the effect. A database transaction can keep related application changes together. A submission deduplication key can suppress repeated enqueue requests for a while. These solve different problems: deduplicating submission does not prevent a worker from executing an accepted job twice.
Shutdown has the same boundary. The worker stops subscriptions and gives active handlers a finite grace period before closing its provider. If work remains unfinished, the broker’s recovery path still matters. Graceful shutdown reduces interruptions; it does not replace redelivery handling.
The provider interface is deliberately small, but the responsibility it expresses is specific: publish work, deliver it, and settle it according to an explicit policy. Redis, RabbitMQ, and Kafka implement that responsibility differently. Application code should know which parts Backstage coordinates and which guarantees still depend on the broker, its configuration, and the system receiving the side effect.
That is the answer to the worker dying. The process can disappear. The work needs a recovery path, and that path must be allowed to run the handler again.