Skip to content

ADR 0014 — Ordered processing and convoys

Status: Proposed (design)
Date: 2026-08-14
Related: ADR 0006, ADR 0007, ADR 0013, RabbitMQ, issue #135


Context

Operators (and BizTalk migrants) need in-order processing for a business key (order id, IBAN, device id) and a story for convoys: related messages that must not overtake each other. A later message in the same convoy must not start while an earlier one is still running, retrying, or sitting on the DLQ.

Today MeshFlows does not guarantee order:

  • AMQP trigger dispatch (POST /invoke/amqp/dispatch) can start several workflows for one publish (1→N).
  • Orchestrator queue workers use prefetch > 1 (ORCH_WORKER_PREFETCH_SMALL default 20) and multiple replicas.
  • Subscriber ACK is after a synchronous Orchestrator invoke (ADR 0013).
  • Retry / Dashboard DLQ replay can re-offer a message after a newer message for the same key has already started.

correlation_id exists on the envelope but is used as a trace/run id, not as a FIFO session.

BizTalk-style parallel convoys (wait until all members of a correlation set arrive) need dehydrate/resume (ADR 0007) and are out of scope for the first implementation slice.


Decision

1) Ordering is opt-in, per key — never global FIFO

Default behaviour stays parallel (throughput). Ordered processing is declared on the AMQP trigger (and forwarded onto the run envelope):

invocation:
  type: amqp
  ordering:
    key: header:OrderId          # or context:amqp.headers.order_id / CEL later
    on_dlq: block                # block | skip (see §4)

Messages without a resolvable key keep today’s parallel path.

There is no total order across unrelated keys, workflows, or the whole broker.

2) Sequential convoy = single-flight per order_key

For a given (workflow_name, order_key):

  • At most one run is in-flight (accepted / queued / running / interrupted).
  • Later messages for the same key are parked in arrival order until the in-flight run reaches a terminal status (completed, failed after final retry policy, cancelled).
  • Arrival order = broker delivery order for that consumer path, recorded when the message is accepted (monotonic seq per key).

This is the MeshFlows mapping of a BizTalk sequential convoy.

3) Hold the convoy in Postgres, not a second broker

Stay on RabbitMQ (ADR 0006). Do not introduce Kafka partitions or a new broker for FIFO.

v1 store (next to run_state_store):

Table (logical) Role
message_order_lock one row per (workflow, order_key) with current run_id / status
message_order_hold parked envelopes FIFO (seq, payload/ref, headers)

Worker loop for an ordered message:

  1. Resolve order_key. If empty → existing parallel execute.
  2. Try to acquire the lock. If free → start run, ACK/complete as today.
  3. If held → insert into message_order_hold, ACK the RabbitMQ delivery (so the broker is not a second timeout clock), do not start the workflow.
  4. When a run for that key becomes terminal → dequeue the next hold row and enqueue it onto orchestrator.run.{small|large} as today.

Parking after ACK is intentional: RabbitMQ consumer timeouts must not release a convoy member out of order. The hold table is the source of truth.

4) Redelivery inside a convoy

Event Behaviour
In-flight run retries (workflow retry / queue retry) Key stays locked. Held messages wait.
In-flight run goes to DLQ on_dlq: block (default): lock remains; hold queue does not advance until an operator retries or skips. on_dlq: skip: mark that seq abandoned, start next hold.
Operator retries from DLQ (#110 Dashboard) Re-enter at the front of that key’s FIFO (original seq), never appended as a new last message.
Duplicate / idempotent re-offer of seq N while N is in-flight No second run; existing idempotency key wins.
Duplicate of seq N after N completed No-op (idempotency), do not unlock out of order.

Do not start message N+1 while N is on the DLQ unless on_dlq: skip.

5) What this is not

  • Not a peek-lock complete step for RabbitMQ (ADR 0013 stays).
  • Not parallel convoys / correlation-set dehydrate (follow ADR 0007).
  • Native Service Bus topic publish + subscription peek-lock/complete is a connector (#127); triggers are ADR 0015 option C.
  • Not prefetch=1 on the shared work queue as the product solution (that only serializes everything on one replica and still races with DLQ replay).

First implementation slice (not this PR)

  1. Trigger YAML ordering.key + envelope header X-MeshFlows-Order-Key.
  2. Postgres lock + hold + release-on-terminal.
  3. Tests: two messages same key with prefetch > 1 complete in publish order; DLQ of N blocks N+1 until retry; different keys still overlap.

Dashboard UX for “blocked keys” can wait; DLQ retry already exists.


Consequences

  • Throughput for unordered traffic is unchanged.
  • Ordered workflows trade latency/throughput for correctness per key.
  • Operators must pick a stable key; a missing key silently falls back to parallel (log a warning, do not fail the trigger).
  • Parallel convoys remain a documented gap until checkpoint/resume exists.

See also