Event and Queue Architecture

Why the Request Stopped Being Enough

Sync HTTP cannot be the only bus

A checkout that calls email, ledger, inventory, and analytics inline will fail in the last hop and leave the first three committed. Queues separate "we accepted the fact" from "every downstream has reacted." The user-facing request writes a fact. Workers fan out.

That split is the whole subject. Everything else (partitions, competing consumers, dead letters) exists to keep that split correct under retry, crash, and backlog.

schema · two planes

REQUEST PLANE                         ASYNC PLANE
POST /orders                          queue / log / bus
  validate                              │
  begin txn                             ├── email worker
  insert orders                         ├── ledger worker
  insert outbox  ─────────────────────► ├── inventory worker
  commit                                └── search indexer
  201 + order id

  The 201 does not mean email sent.
  It means the fact is durable and will be published.

Pick a vocabulary and keep it

Command: do this (SendInvoice). Event: this happened (InvoicePaid). Queues carry both. Do not name a command like an event. Consumers that think they are reacting to history will start issuing new side effects from a replay.

Delivery Guarantees

Guarantee	What it means	What you still build
At-most-once	A crash can drop the message	Accept loss, or do not use this for money
At-least-once	A crash can duplicate the message	Idempotent consumers, dedupe store
Effectively-once	Broker + app cooperate so duplicates do not double-apply	Transactional produce/consume, idempotency keys

Exactly-once is a marketing phrase

Brokers that advertise exactly-once usually mean: produce and consume can be tied to an internal transaction so a message is not lost or double-appended inside that log. Your email provider is still at-least-once. Your HTTP webhook is still at-least-once. Design every consumer as if the message will arrive twice, possibly hours apart.

Queue (competing consumers)

Each message goes to one worker in a group. Good for commands. AMQP-style brokers and SQS live here.

Log (pub/sub + offset)

Each consumer group reads the same stream. Good for events many teams must see. Kafka-style logs live here.

Inbox / mailbox

Per-aggregate serial mailbox. Good when order inside one entity matters and global order does not.

Delayed / scheduled

Visible later. Timeouts, retries with jitter, and "remind me" jobs. Not a cron replacement for fleet-wide schedules unless you mean it.

The Transactional Outbox

A transaction crystal splitting into a data record and an event capsule — Write the row and the outbox row in one database transaction. A publisher drains the outbox. Dual-write to the bus inside the request is how events vanish.

Dual-write is the bug

If you COMMIT the order and then PUBLISH, a crash between those lines loses the event. If you publish first and then commit, you emit a fact that rolled back. The outbox puts both writes in one ACID transaction. A separate process (poller, logical replication, trigger) publishes committed outbox rows and marks them sent.

SQL · outbox in the same transaction

BEGIN;
INSERT INTO orders (id, status, total_cents)
VALUES ($1, 'placed', $2);

INSERT INTO outbox (id, topic, payload, created_at)
VALUES ($3, 'order.placed', $4::jsonb, now());
COMMIT;

-- publisher (another process)
SELECT id, topic, payload FROM outbox
 WHERE published_at IS NULL
 ORDER BY created_at
 LIMIT 100
 FOR UPDATE SKIP LOCKED;

mermaid · flowchart

flowchart TD
  A[HTTP handler] --> B[BEGIN]
  B --> C[Write aggregate]
  C --> D[Write outbox row]
  D --> E[COMMIT]
  E --> F[201 to client]
  D -.-> G[Publisher]
  G --> H{Broker ack?}
  H -->|yes| I[Mark published_at]
  H -->|no| G

CDC is an outbox you did not design

Change data capture from the WAL can replace a handwritten outbox table. You still need a stable event schema, a way to ignore uninteresting tables, and a rule for what a tombstone means. CDC that dumps row images is not a public event contract.

Consumer Idempotency

Remember the work, not the delivery

Store event_id (or a hash of aggregate id + version) in a processed table with a unique constraint. Apply side effects in the same transaction as the insert when the side effect is local. For an external side effect (send email), record "we decided to send" before the provider call, and make the provider call itself idempotent with its own key.

Receive

Pull or push. Hold a visibility timeout or uncommitted offset until you finish.

Dedupe

Insert event_id. Unique violation means "already applied." Ack and stop.

Apply

Mutate local state or call an external API with the same idempotency key.

Ack

Only after commit. Ack-before-commit is at-most-once dressed as a queue.

TypeScript · consume once

async function consume(event: { id: string; type: string; data: unknown }) {
  const inserted = await db.query(
    "INSERT INTO processed_events (id) VALUES ($1) ON CONFLICT DO NOTHING",
    [event.id]
  );
  if (inserted.rowCount === 0) return; // duplicate delivery
  await apply(event);
}

Poison Messages and Dead Letters

A magenta poisoned capsule diverted from a cyan queue river into a dead-letter basin — A message that fails validation forever must leave the hot path. Retries without a ceiling are a self-inflicted outage.

Three failure classes

Transient. 503, lock timeout, network blip. Retry with jitter. Keep the message in the main queue.
Deterministic poison. Schema fail, missing required field, business rule that will never pass. After N attempts, dead-letter. Page a human or a replay tool.
Slow poison. Handler that livelocks. Visibility timeout expires, another worker takes it, now two workers fight. Cap concurrency per aggregate key.

Dead letters need an owner

A DLQ nobody reads is disk usage. Give it a dashboard, a replay command, and a time-to-live. Replaying without fixing the handler puts the poison right back on the river.

Event Shape

CloudEvents as the envelope

CloudEvents (CNCF) standardizes id, source, type, specversion, time, and a data payload. Use it when events cross team or vendor boundaries. Inside one service you can be leaner. Do not invent a fourth envelope if you already have one.

JSON · CloudEvents

{
  "specversion": "1.0",
  "id": "01J8Z3K4N2M",
  "source": "https://orders.example.com/app",
  "type": "com.example.order.placed.v1",
  "time": "2026-08-26T12:00:00Z",
  "datacontenttype": "application/json",
  "data": {
    "orderId": "ord_01J",
    "totalCents": 4200,
    "currency": "KES"
  }
}

Version the type name

order.placed.v1 can add optional fields. v2 is for incompatible changes. Consumers subscribe to types they understand. A single "payload version inside data" field is how you get half-upgraded workers.

Backpressure and Ordering

Lag is a first-class SLO

Watch consumer lag (offset delay or queue depth), not only error rate. A silent backlog is a delayed outage. Shed load on the producer (429 the HTTP plane) before the queue grows without a drain plan.

Global order is expensive. Order per partition key (customer id, order id) is usually enough. If two keys must be strictly ordered, they are one key. If they are not, do not share a partition just to feel safe.

Need	Mechanism	Cost
One worker per aggregate	Keyed mailbox / partition key	Hot keys stall that key only
Fan-out to many teams	Log + consumer groups	Each group pays storage/read
Burst absorption	Queue depth + autoscale workers	Lag grows; set a max depth alarm

Key Takeaways

Principles

Assume at-least-once. Every consumer is idempotent or it is wrong.
Never dual-write. Outbox or CDC, then publish.
Ack after commit. Ack-before-write drops work on a crash.
Separate commands from events. Replay of history must not re-issue commands.
Dead-letter with an owner and a replay path. Infinite retry is an outage.
Envelope the payload (CloudEvents or equivalent). Version the type name.
Order per key, not globally, unless you have a single-thread reason.
Alarm on lag. A quiet, growing queue is already an incident.