Queue vs Stream vs Pub/Sub: Three Questions That Decide How a Message Behaves
Queue, pub/sub and stream answer three separate questions about a message: does a read remove it, who gets a copy, who tracks progress. With a runnable demo.
Queue, pub/sub and stream are often compared as three versions of the same thing. That comparison makes the choice unclear, because the three words answer different questions about a message. A queue and a stream differ in whether a read removes the message and in who remembers progress. Pub/sub only says who gets a copy. Its durable forms are built on queues or on a log, and a plain push subscription keeps a message only while its delivery retries last.
For a first system, one pairing covers most needs. Work that one worker out of a pool should handle goes on a queue. Facts that several services react to go on pub/sub, where every subscriber owns a durable queue. Both can deliver a message more than once, so handlers have to tolerate repeats. A log is the next step when a consumer must read the past, or when ordered traffic outgrows what an ordered queue can carry.
The same three answers explain why an unfamiliar broker behaves differently from the one you already know. The questions pick up where Thinking in Events stops: that guide separates commands from events and leaves the delivery details open.
Three Questions Behind the Names#
Every channel has to answer three questions about a message, and each answer can be chosen separately.
- Does a read remove it? A queue reads destructively: a delivered message is held for one consumer, and it is gone once that consumer acknowledges it. A log reads non-destructively: the record stays until its retention period ends, and it can be read again.
- Who gets a copy? Either one consumer out of a pool receives the message, or every subscriber receives its own copy.
- Who remembers progress? Either the broker tracks the state of every message, or each consumer tracks a single position in an ordered log.
A queue is one fixed set of answers: the read is destructive, one consumer receives the message, and the broker tracks it. A stream is another set: records stay, and every reader keeps its own position. Pub/sub answers the second question only. It says nothing about deletion or progress, so those two answers come from whatever backs each subscription.
The table shows the answers for each shape. Two terms in the last column are explained further down. A partition is one ordered slice of a log, and a consumer group is a set of readers that divide the partitions among themselves and share their progress.
| Question | Queue | Pub/sub | Stream (log) |
|---|---|---|---|
| Does the message exist after a read? | No, it is deleted on acknowledgement | Depends on what backs each subscription | Yes, until retention expires |
| Who receives a given message? | One consumer from the pool | Every subscription gets a copy; consumers inside one subscription share | Every consumer group gets every record from its starting offset; inside a group, one member per partition |
| Who tracks progress? | Broker, per message | Depends on what backs each subscription | Each consumer group, one offset per partition |
| Can a late joiner read the past? | No | No, unless the subscription existed earlier or retention and seek are configured | Yes, by starting or seeking at any retained offset |
| Ordering | Lost once consumers compete; FIFO variants keep it per message group | No guarantee by default; ordering keys can add it per key | Per partition only |
| How do consumers scale? | Add consumers freely | Add consumers inside a subscription | Up to the partition count per classic consumer group |
The diagram shows the two durable ways to build pub/sub. In the first chain, a topic copies each message into one queue per subscriber. In the second chain, both services read the same log, and each one keeps its own offset.
A Runnable Demo of Each Shape#
The three shapes fit in one file with no infrastructure. WorkQueue keeps the state of every message and deletes a message on ack. Topic holds one WorkQueue per subscriber and copies each published message into all of them. Log keeps every record and stores one committed offset per reader.
type Msg = { id: string; body: string };
// Queue: the broker tracks every message. ack deletes it, nack puts it back at the end.
class WorkQueue {
private ready: Msg[] = [];
private inFlight = new Map<string, Msg>();
send(m: Msg): void {
this.ready.push(m);
}
receive(): Msg | undefined {
const m = this.ready.shift();
if (m) this.inFlight.set(m.id, m);
return m;
}
ack(id: string): void {
this.inFlight.delete(id);
}
nack(id: string): void {
const m = this.inFlight.get(id);
if (!m) return;
this.inFlight.delete(id);
this.ready.push(m);
}
get size(): number {
return this.ready.length + this.inFlight.size;
}
}
// Pub/sub: one input, one queue per subscriber. A late subscriber starts empty.
class Topic {
private subs = new Map<string, WorkQueue>();
subscribe(name: string): WorkQueue {
const q = new WorkQueue();
this.subs.set(name, q);
return q;
}
publish(m: Msg): void {
for (const q of this.subs.values()) q.send({ ...m });
}
}
// Log: records stay. The log stores one committed offset per reader. seek is replay.
class Log {
private records: Msg[] = [];
private committed = new Map<string, number>();
append(m: Msg): void {
this.records.push(m);
}
// Reading moves nothing. It returns records from the reader's committed offset.
read(reader: string, max: number): Msg[] {
const from = this.committed.get(reader) ?? 0;
return this.records.slice(from, from + max);
}
// Progress moves only when the reader commits what it has processed.
commit(reader: string, count: number): void {
this.committed.set(reader, (this.committed.get(reader) ?? 0) + count);
}
seek(reader: string, offset: number): void {
this.committed.set(reader, offset);
}
get size(): number {
return this.records.length;
}
}
The second half of the file sends the same four orders through each structure.
const orders: Msg[] = ['o1', 'o2', 'o3', 'o4'].map((id) => ({ id, body: `order ${id}` }));
const ids = (batch: Msg[]): string => batch.map((m) => m.id).join(', ') || '(nothing)';
function drain(q: WorkQueue): Msg[] {
const seen: Msg[] = [];
for (let m = q.receive(); m; m = q.receive()) {
seen.push(m);
q.ack(m.id);
}
return seen;
}
console.log('== Queue: two workers compete ==');
const queue = new WorkQueue();
for (const o of orders) queue.send(o);
const workers = ['worker-A', 'worker-B'];
let turn = 0;
let failedOnce = false;
for (let m = queue.receive(); m; m = queue.receive()) {
const worker = workers[turn++ % workers.length];
if (m.id === 'o2' && !failedOnce) {
failedOnce = true;
queue.nack(m.id);
console.log(`${worker} received ${m.id} -> nack (handler failed)`);
} else {
queue.ack(m.id);
console.log(`${worker} received ${m.id} -> ack`);
}
}
console.log(`messages left in the queue: ${queue.size}`);
console.log('\n== Pub/sub: every subscriber gets a copy ==');
const topic = new Topic();
const billing = topic.subscribe('billing');
const email = topic.subscribe('email');
for (const o of orders.slice(0, 3)) topic.publish(o);
const analytics = topic.subscribe('analytics');
topic.publish(orders[3]);
console.log(`billing received: ${ids(drain(billing))}`);
console.log(`email received: ${ids(drain(email))}`);
console.log(`analytics (subscribed after o3) received: ${ids(drain(analytics))}`);
console.log('\n== Log: each reader commits its own offset ==');
const log = new Log();
for (const o of orders) log.append(o);
function consume(reader: string, max: number): string {
const batch = log.read(reader, max);
log.commit(reader, batch.length);
return ids(batch);
}
console.log(`fast reader: ${consume('fast', 4)}`);
console.log(`slow reader: ${consume('slow', 1)}`);
console.log(`slow reader reads ${ids(log.read('slow', 1))} and stops before commit`);
console.log(`slow reader after restart: ${consume('slow', 1)}`);
console.log(`audit (joined late): ${consume('audit', 4)}`);
console.log(`fast reader again: ${consume('fast', 4)}`);
log.seek('fast', 1);
console.log(`fast reader after seek(1): ${consume('fast', 4)}`);
console.log(`records still in the log: ${log.size}`);
Save both parts, in this order, as channels.ts. Node runs the file directly where type stripping is on by default. That means 22.18 or later on the 22 line, and every release from 23.6 on, including Node 24.
node channels.ts
This is the output of a run on Node 22:
== Queue: two workers compete ==
worker-A received o1 -> ack
worker-B received o2 -> nack (handler failed)
worker-A received o3 -> ack
worker-B received o4 -> ack
worker-A received o2 -> ack
messages left in the queue: 0
== Pub/sub: every subscriber gets a copy ==
billing received: o1, o2, o3, o4
email received: o1, o2, o3, o4
analytics (subscribed after o3) received: o4
== Log: each reader commits its own offset ==
fast reader: o1, o2, o3, o4
slow reader: o1
slow reader reads o2 and stops before commit
slow reader after restart: o2
audit (joined late): o1, o2, o3, o4
fast reader again: (nothing)
fast reader after seek(1): o2, o3, o4
records still in the log: 4
In the queue block, every delivery goes to exactly one worker. worker-B fails on o2 and returns it with nack. o2 comes back only after o4, and worker-A handles it the second time. The send order was o1 to o4, but the completion order is o1, o3, o4, o2. Once every message is acknowledged, the queue holds nothing.
The pub/sub block gives billing and email all four orders each, because the topic copies every message into both queues. analytics subscribes after o3 is published and receives only o4. The three earlier orders were never copied into its queue, so no retry can bring them back.
The log block is the only one where reading leaves the data in place. The fast reader takes all four records and the slow reader takes one, and neither read affects the other. Reading alone moves nothing. The slow reader reads o2, stops before its commit, and gets o2 again after the restart. audit joins last and still starts at o1. A second read by the fast reader returns nothing, because its committed offset is already at the end. After seek(1) it reads o2, o3 and o4 again, and the log still holds four records.
The answer to the progress question is visible in the fields of the classes. WorkQueue needs the inFlight map because it tracks every message. Log only stores one number per reader.
The demo models the semantics and leaves out everything else a broker does. There is no network, no visibility timeout, no partition and no consumer group. Putting a nacked message at the back of the queue is also a simplification, because brokers differ in where a redelivered message lands. Starting every new reader at the first record is a simplification too, and the stream section covers what a broker does by default. The same section covers the in-memory position that a Kafka consumer keeps ahead of its committed offset, which the demo leaves out. The next sections add those details from the documentation of the brokers themselves.
Queues and Competing Consumers#
A queue works like the ticket rail in a restaurant kitchen: one cook takes a ticket, and no other cook prepares that order. Enterprise Integration Patterns calls this a Point-to-Point Channel (opens in new tab), which “ensures that only one receiver will receive a particular message”. Adding receivers does not change that guarantee. The Competing Consumers (opens in new tab) pattern uses it to scale: “Create multiple Competing Consumers on a single channel so that the consumers can process multiple messages concurrently.”
The broker makes this work by tracking every message. Amazon SQS is a clear example. According to its visibility timeout documentation (opens in new tab), a received message “remains in the queue but becomes temporarily invisible to other consumers”. The consumer has to delete the message after processing it. If it does not do so before the timeout expires, “the message becomes visible again in the queue and can be retrieved by another consumer.” The default timeout is 30 seconds. In the demo, the inFlight map does this tracking, without a timeout.
Three consequences follow from that mechanism. First, a message can be delivered more than once, so handlers must be safe to run twice. The idempotency guide covers how to do that. Second, a message that always fails keeps coming back until something removes it. That is the job of a dead-letter queue.
Third, the order of messages is not preserved. Microsoft’s Competing Consumers pattern (opens in new tab) warns that the order in which consumers receive messages “isn’t guaranteed”. Redelivery moves messages as well. The SQS metrics page (opens in new tab) describes one case for standard queues. A message that is received three or more times without being deleted moves to the back of the queue. Ordered queue variants restore order per key. In an SQS FIFO queue, messages in the same message group (opens in new tab) “are always processed one at a time, in strict order”, while messages of different groups can be in flight together. The Kafka migration post covers the throughput limits of that mechanism.
Pub/Sub and the Durability of a Subscription#
Pub/sub answers the question of who gets a copy. A Publish-Subscribe Channel (opens in new tab) “delivers a copy of a particular event to each receiver”. The same page describes the structure: “It has one input channel that splits into multiple output channels, one for each subscriber.” A mailing list behaves the same way: every member receives each message, and a new member does not receive earlier messages.
Microsoft’s Competing Consumers pattern (opens in new tab) states the contrast with a queue in two sentences: “In the Competing Consumers pattern, one consumer receives each message for processing. In the Publisher-Subscriber pattern, all consumers receive every message.”
The pattern exists because fan-out through plain queues puts the work on the sender. Microsoft’s Publisher-Subscriber pattern (opens in new tab) describes that situation: “To support multiple consumers, the sender must create a separate queue for each consumer”. Pub/sub moves the copying into the broker. The sender publishes once and does not know who subscribed.
What happens while a subscriber is offline is a separate matter, and sources describe it differently. The same Microsoft page lists as a benefit that “the broker retains messages even if one or more receivers are offline.” Enterprise Integration Patterns treats that behaviour as a pattern of its own. The solution of the Durable Subscriber (opens in new tab) pattern is to “make the messaging system save messages published while the subscriber is disconnected.” Amazon SNS shows what happens without durable storage for the subscriber. Its delivery retry documentation (opens in new tab) says that once the retry policy is exhausted, SNS “stops retrying the delivery and discards the message”. The exception is a subscription with a dead-letter queue attached.
The two descriptions fit together once durability is seen as a property of each subscription. A subscription backed by a queue keeps messages for an absent consumer, and a plain push subscription only retries for a limited time. That is why the common AWS fan-out puts an SQS queue behind every SNS subscription. According to the SNS fan-out documentation (opens in new tab), messages then “persist in an Amazon SQS queue for other applications to process later.” Cross-account SNS to SQS fan-out covers that setup in detail. Fan-out to isolated consumer accounts also gives every consumer its own queue, with EventBridge doing the fan-out.
Durability does not give a new subscriber the past. Google Pub/Sub’s subscription overview (opens in new tab) states: “Only messages published to the topic after the subscription is created are available to subscriber clients.” This is the analytics line in the demo output. The same page notes that messages carry no ordering guarantee by default. With message ordering enabled on the subscription, messages that share an ordering key arrive in the order Pub/Sub received them.
Pub/sub also combines with competing consumers. Google Pub/Sub’s documentation (opens in new tab) describes both levels at once. Across subscriptions, “Each subscription receives a copy”. Inside one subscription with several subscriber applications, “Each of the subscriber applications gets a subset of the published messages”. The demo’s Topic has the same structure, because each subscriber’s output is a WorkQueue that several workers could share.
Streams as Append-Only Logs#
The word “stream” names a storage shape here: an append-only log. RabbitMQ’s streams documentation (opens in new tab) gives a compact definition. Streams “model an append-only log of messages that can be repeatedly read until they expire”. A shared logbook is a fair picture: entries are only added at the end, and every reader keeps a personal bookmark.
Reading does not remove anything. Kafka’s introduction (opens in new tab) says that events in a topic “can be read as often as needed” and that “events are not deleted after consumption”. How long they stay is “a per-topic configuration setting”. In the demo, the last output line shows four records still in the log after all the reads.
A log is usually split into partitions, and order holds inside one partition only. The Kafka introduction puts it this way: “any consumer of a given topic-partition will always read that partition’s events in exactly the same order as they were written.” Events with the same key go to the same partition, so order is kept per key. There is no order across partitions.
Progress is a position that the reader moves. Confluent’s consumer design page (opens in new tab) describes it: “As the consumer makes progress, it commits the offsets of messages it has successfully processed”. Kafka stores each committed offset on the broker, per group and partition, and the consumer decides when it advances. According to the same page, a consumer can also “specify an offset to reconsume data if needed.” That second ability is replay, and it is what seek(1) does in the demo.
Reading and committing are separate steps. Kafka’s consumer documentation (opens in new tab) names two positions. The consumer’s own position “automatically advances every time the consumer receives messages”. The committed position “is the last offset that has been stored securely”, and after a failure “this is the offset that the consumer will recover to”. A consumer that commits after processing and stops in between reads the uncommitted records again. The slow reader in the demo shows this with o2, so handlers on a log need the same idempotency as handlers on a queue.
A reader that joins late does not get the past automatically. The same page says that when a group is first initialized, its consumers “typically begin reading from either the earliest or latest offset in each partition”. In Kafka the auto.offset.reset (opens in new tab) setting makes that choice, and its default is latest, the end of the log. To read history, a new group has to set it to earliest or seek back.
Readers that share committed offsets form a consumer group, and the scaling limit comes from that group. The same page states: “Each partition is consumed by exactly one consumer within each consumer group at any given time.” A topic with six partitions therefore gives a group at most six working consumers, and a seventh adds no throughput.
RabbitMQ shows both shapes inside one broker. According to the same streams page (opens in new tab), streams have “non-destructive consumer semantics” and let consumers “attach at any point in the log”. Its queue types “have destructive consume behaviour”, so “it is not possible to re-read messages that have been consumed”.
A stream is often equated with real-time processing, which mixes two things. Kafka’s introduction defines event streaming to include both “storing these event streams durably for later retrieval” and “processing, and reacting to the event streams in real-time as well as retrospectively.” The log is the storage part. Stream processing is an optional layer on top of it.
Shard and partition mechanics are covered in the Kinesis deep dive.
Hybrids That Brokers Ship#
Since the three answers are independent, brokers combine them in more than three ways. Four combinations come up often.
| Combination | Does a read remove it? | Who gets a copy? | Who remembers progress? |
|---|---|---|---|
| SNS topic with an SQS queue per subscriber | Yes, each queue deletes on acknowledgement | Every queue; workers on one queue compete | Broker, per message |
| Google Pub/Sub subscription | Yes by default; retention is opt-in | Every subscription; clients inside one share | Broker, per message in each subscription |
| Kafka consumer group | No | Every group; one member per partition inside a group | Each consumer group, one committed offset per partition, stored by the broker |
| Kafka share group | No | Members share records without exclusive partition ownership | Broker, per record |
Kafka describes consumer groups as exactly this kind of combination. The introduction in its 2.4 documentation (opens in new tab) says that messaging “traditionally has two models: queuing and publish-subscribe”. The page names the weakness of each. Queues “aren’t multi-subscriber”, and pub/sub “has no way of scaling processing since every message goes to every subscriber”. Kafka’s answer follows: “The consumer group concept in Kafka generalizes these two concepts”. Between groups Kafka behaves like pub/sub, and inside a group the members divide the partitions. The same text adds the limit: “there cannot be more consumer instances in a consumer group than partitions.”
Kafka share groups put queue behaviour on top of the same log. The design proposal, KIP-932 (opens in new tab), says that “The number of consumers in a share group can exceed the number of partitions in a topic”. It also says that “Records are acknowledged on an individual basis”. The proposal adds that it “does not add the concept of a ‘queue’ to Kafka per se”. That fits the model here: the records stay in the log, while delivery and progress tracking move to the broker, record by record.
The Kafka 4.2 upgrade notes (opens in new tab) give the status of the feature: “Queues for Kafka (KIP-932) is production-ready in Apache Kafka 4.2”. The price is order. Confluent’s general availability announcement (opens in new tab) says that “Share groups sacrifice ordering for elastic scaling and per-message control”. At the time of that announcement, only Java clients were supported and exactly-once semantics were a roadmap item. Check the current state before depending on a non-Java client or on exactly-once semantics.
Google Pub/Sub is pub/sub with queue-like subscriptions, and replay is opt-in. According to its replay documentation (opens in new tab), replaying acknowledged messages requires retention to be configured on the topic or on the subscription. After that, a seek to a past time delivers those messages again.
SNS in front of SQS is pub/sub whose outputs are queues. Each subscriber gets a durable backlog, and a message is gone from that queue once the subscriber’s consumer deletes it. The AWS messaging comparison covers when to choose it over the other AWS services.
Product categories do not follow the three shapes closely. Microsoft’s Publisher-Subscriber page lists Azure Event Hubs among pub/sub technologies and adds that it “uses a log-based streaming model rather than traditional pub/sub messaging”. Asking the three questions about a product is more reliable than reading its category.
Choosing a Shape for a New Channel#
The kind of message suggests the first answer. A command asks one receiver to do something, so it fits a queue owned by the receiver. An event states that something happened, so it fits pub/sub or a log owned by the publisher. Thinking in Events explains the distinction, and Commands vs Events Across Bounded Contexts covers who owns which channel.
The flowchart starts at the default from the opening and names the cases that override it.
The trade-offs of the default suit a first system. Each subscriber scales by adding workers to its own queue, and a slow subscriber only fills its own queue. Messages wait there while that subscriber is down. In exchange, every subscriber needs its own queue and alarms, and nobody can read the past unless retention or an archive was switched on beforehand. The AWS messaging comparison recommends the same pairing in AWS terms, as SNS in front of SQS.
Order per key alone does not call for a log. A FIFO queue keeps order inside a message group, and an ordering key does the same for a pub/sub subscription. The price is head-of-line blocking. According to the SQS metrics page (opens in new tab), “A failed message blocks its message group until it’s deleted or expires.”
A log is worth its extra work when one of these needs is present:
- A new consumer has to read events from before it existed.
- A fixed bug has to be replayed over records that were already processed.
- Ordered traffic per key grows beyond what an ordered queue can carry.
- Many consumers read the same high-volume flow at different speeds.
The Kafka migration post goes through these signals in detail. When one of them is a known requirement from the start, begin with the log and skip the default. In the other direction, non-durable push is enough when a missed message is harmless, as with live UI updates or cache invalidation.
What Each Shape Costs#
Each shape gives something different and asks for something different in return.
| Shape | What it gives | What it costs | Typical failure |
|---|---|---|---|
| Queue | Simple to reason about; consumers scale freely | No history after acknowledgement; order is lost under competing consumers; duplicate deliveries are possible | A message that always fails keeps cycling until a DLQ catches it |
| Pub/sub with a queue per subscriber | Producers do not know consumers; a slow consumer only delays itself | One queue and its alarms per subscriber; no replay without retention or an archive; new subscribers start from now | A subscription created after the events it needed |
| Non-durable pub/sub | The fewest parts to operate | Messages are lost once delivery retries run out while a subscriber is down | Unnoticed gaps after an outage |
| Log | Replay, late joiners, order per partition, many readers of one copy | Storage for retention; the partition count caps the parallelism of a classic group; consumers manage offsets | A stuck record blocks its partition for that group |
Assumptions That Fail After Launch#
Most surprises with a broker come from expecting one shape’s answer from another shape. Each bold line below is an assumption that turns out to be wrong, followed by what happens instead.
A second service on the same queue receives every message too. The two services compete, so each one gets part of the messages. In the demo’s queue block, worker-A never sees o4. When both services need every message, each one needs its own subscription.
A queue can replay after a bug fix. Acknowledged messages are deleted, so there is nothing left to run again. Decide early whether history matters. If it does, choose a log or add an archive on purpose.
A new subscriber starts with the full history. It starts from the moment its subscription exists, as analytics did. Create subscriptions before producers start. When backfill is a requirement, choose a log and start the new group at the earliest offset.
More consumers always drain a backlog faster. A classic consumer group stops gaining throughput at the partition count. Size the partition count for the parallelism you expect, or look at share groups when order does not matter.
The queue keeps messages in order. Competing workers and redelivery both change the order, and o2 showed the redelivery case. Write handlers that tolerate reordering, or use a group key where order per entity matters.
A log is the safe choice for every workload. A background job queue usually needs no history, no order and no second reader. On a log it still inherits partition planning, offset management and consumer lag. A queue is the simpler tool for that job.
Backlog Signals for Each Shape#
A queue falls behind when messages wait too long. Its two signals are therefore the age of the oldest message and the number of waiting messages. SQS publishes them as ApproximateAgeOfOldestMessage and ApproximateNumberOfMessagesVisible, and counts in-flight messages as ApproximateNumberOfMessagesNotVisible (SQS metrics (opens in new tab)). According to the same page, messages that are received repeatedly but never deleted are excluded from the age metric. DLQ arrivals therefore need their own alarm. Pub/sub needs the same two signals per subscription, because a topic-level number can look healthy while one subscriber is far behind. With a queue per subscriber, the queue metrics already give that view.
A log read by a classic consumer group has no per-message state to count. Its signal is consumer lag: how far a group’s offset is behind the newest record in each partition. Kinesis reports the same idea as a time, the iterator age, and the Kinesis deep dive shows it climbing when consumers fall behind. Whatever the shape, it is worth counting redeliveries, arrivals in the DLQ, and duplicates caught by the idempotency check.
When the Default Holds#
The default is a queue for work and fan-out into a queue per subscriber. It holds as long as consumers only need messages from now on. Where order per key matters, use the ordered variant of the same shape. Move a flow to a log when a consumer has to read history or replay after a fix, or when ordered traffic outgrows an ordered queue. Use non-durable push only where a gap in the data is harmless. For the next channel you add, write down the broker’s three answers first: whether a read removes the message, who gets a copy, and who remembers progress.
References#
- Point-to-Point Channel - Enterprise Integration Patterns (opens in new tab) - Defines the channel on which only one receiver gets each message.
- Competing Consumers - Enterprise Integration Patterns (opens in new tab) - Several consumers on one channel processing messages concurrently; the pattern behind scaling queue workers.
- Publish-Subscribe Channel - Enterprise Integration Patterns (opens in new tab) - One input channel split into an output channel per subscriber, each receiving its own copy.
- Durable Subscriber - Enterprise Integration Patterns (opens in new tab) - Saving messages for a disconnected subscriber, described as a pattern of its own.
- Competing Consumers pattern - Microsoft Learn (opens in new tab) - Contrasts one consumer per message with pub/sub and discusses ordering and idempotent processing.
- Publisher-Subscriber pattern - Microsoft Learn (opens in new tab) - The broker-side copy per consumer, its benefits, and a note on the log-based model of Event Hubs.
- Introduction - Apache Kafka (opens in new tab) - Events stay after consumption, retention is set per topic, and order holds per partition.
- Introduction (Kafka 2.4) - Apache Kafka (opens in new tab) - Kafka’s description of queuing and publish-subscribe, and how consumer groups generalise both.
- Kafka Consumer Design: Consumers, Consumer Groups, and Offsets - Confluent (opens in new tab) - One consumer per partition per group, committed offsets, and rewinding to consume again.
- Consumer and Share Consumer Configs (Kafka 4.4) - Apache Kafka (opens in new tab) - The
auto.offset.resetsetting, which decides where a group without a committed offset starts and defaults tolatest. - KafkaConsumer (kafka 4.4.0 API) - Apache Kafka (opens in new tab) - The consumer’s position versus its committed position, and which one a restarted consumer recovers to.
- KIP-932: Queues for Kafka - Apache Kafka wiki (opens in new tab) - The design proposal for share groups: more consumers than partitions and per-record acknowledgement.
- Upgrading (Kafka 4.2) - Apache Kafka (opens in new tab) - Upgrade notes that declare Queues for Kafka production-ready in Apache Kafka 4.2.
- Kafka Queue Semantics Now GA with Share Consumer API - Confluent (opens in new tab) - How share groups trade ordering for elastic scaling, with the limitations at general availability.
- Streams and Superstreams (Partitioned Streams) - RabbitMQ (opens in new tab) - Non-destructive log reads next to RabbitMQ’s destructive queues, inside one broker.
- Amazon SQS visibility timeout - AWS (opens in new tab) - How a received message is hidden, deleted on success, or made visible again for another consumer.
- Using the message group ID with Amazon SQS FIFO Queues - AWS (opens in new tab) - Strict ordering inside a message group of a FIFO queue.
- Fanout Amazon SNS notifications to Amazon SQS queues for asynchronous processing - AWS (opens in new tab) - Pub/sub whose outputs are queues, so each subscriber’s messages persist until processed.
- Amazon SNS message delivery retries - AWS (opens in new tab) - Push delivery retries and what happens to a message once they are exhausted.
- Overview of the Pub/Sub service - Google Cloud (opens in new tab) - Fan-out across subscriptions and load sharing inside one subscription.
- Subscription overview - Google Cloud Pub/Sub (opens in new tab) - New subscriptions receive only messages published after they exist, with no ordering by default and opt-in ordering per key.
- Replay and purge messages with seek - Google Cloud Pub/Sub (opens in new tab) - Opt-in replay through retained acknowledged messages and seeking to a past time.
- Available CloudWatch metrics for Amazon SQS - AWS (opens in new tab) - Backlog and age metrics, how a repeatedly received message moves to the back of a standard queue, and how a failed message blocks its FIFO message group.
Related posts
Decide which message may cross a bounded context: receivers own command schemas, publishers own event schemas. Default to integration events and name every command.
domain-driven-design · event-driven · messaging +4
Learn the mental shift behind event-driven systems: announce facts instead of issuing commands. Covers naming, decoupling, eventual consistency, and idempotency.
event-driven · messaging · idempotency +2
An exploration thesis: vendor lock-in in event-driven systems lives in the bus topology, not the runtime, and wasmCloud plus NATS make the bus portable.
webassembly · messaging · event-driven +3
A pragmatic guide for designers working with async backends: three interaction patterns, when to use each, and four anti-patterns to push back against.
event-driven · state-management · design-patterns +2
Learn how the Transactional Outbox Pattern solves the dual-write problem in distributed systems, with practical implementations using PostgreSQL, DynamoDB, and CDC tools.
distributed-systems · microservices · event-driven +5