Skip to content

ADR 0006 — RabbitMQ persistence with two-track message policy (CEL, 512KB)

Status: Proposed (target architecture)
Date: 2026-06-02
Related: ADR 0005, Architecture, Orchestrator

Context

The orchestrator is being shaped more strictly as a service-provider-driven runtime with:

  • minimal memory footprint in the runner;
  • durable message persistence;
  • open-source components that run locally on Kubernetes;
  • explicit separation between small and large messages.

CEL was chosen for routing expressions.
RabbitMQ (not Kafka) was chosen for queueing/persistence because the primary use case is command/work queue processing, not large-scale event log replay.

Decision

1) Messaging stack (OSS, local K8s)

The runtime uses:

  • RabbitMQ as the broker for command/work queues;
  • PostgreSQL for run/step/state/idempotency persistence;
  • MinIO (S3-compatible) for payload offload of large messages.

Everything runs as containers on local Kubernetes (kind/k3d/minikube) and is cloud-neutral.

2) Two-track message policy

The orchestrator enforces a hard split on payload size:

  • Track small: payload <= 512KB
  • Track large: payload > 512KB

Track small (<= 512KB)

  • Inline payload in message envelope allowed.
  • In-flow transformations allowed (within budget/limits).
  • Content-based routing allowed with CEL on headers + payload fields.

Track large (> 512KB)

  • Payload is handled as a reference (payload_ref), not inline.
  • Orchestrator processes envelope + metadata only.
  • Transformations happen only via explicit service_call to provider services.
  • No inline content transformations in the runner.

3) Envelope contract

Every queue message contains at minimum:

  • message_id (UUID)
  • correlation_id
  • workflow_name
  • track (small | large)
  • headers (technical + business)
  • idempotency_key
  • exactly one of:
  • payload_inline (small)
  • payload_ref (bucket, key, version, sha256) (large)

4) RabbitMQ topology

Minimal topology:

  • Exchange orchestrator.commands (topic/direct)
  • Queue orchestrator.run.small
  • Queue orchestrator.run.large
  • Queue orchestrator.retry.small
  • Queue orchestrator.retry.large
  • Queue orchestrator.dlq

Rules:

  • retry via delayed backoff pattern (TTL + dead-letter routing);
  • poison messages to orchestrator.dlq;
  • worker acks only after durable state write.

5) State & idempotency

PostgreSQL manages:

  • run lifecycle (accepted, queued, running, succeeded, failed, dead_lettered);
  • step checkpoints;
  • idempotency receipts;
  • audit/events.

Processing is at-least-once with idempotent handling via idempotency_key.

6) Memory footprint rules

Runner rules:

  • keep only envelope + necessary step context in memory;
  • do not dump large payloads into trace by default;
  • bounded concurrency + prefetch limits;
  • fetch/stream large payloads only when a service call needs them.

Consequences

Positive

  • clear, enforceable separation between light and heavy payload processing;
  • lower memory pressure on orchestrator workers;
  • better reliability through durable queue + persisted state;
  • easier to run locally than a Kafka-first setup.

Negative / trade-offs

  • extra components (RabbitMQ, Postgres, MinIO) increase operational footprint;
  • large track requires reference management and hash validation;
  • CEL governance needed (limits on expression complexity and evaluation time).

Implementation guardrails

  • No backward compatibility for legacy inline-invocation paths.
  • New workflow functionality must be provider-driven (service_call first).
  • Inline transformations only for small track and within strict limits.
  • Large track policy enforced at API ingress.

Follow-up work items

  1. ~~Publish contracts: queue envelope JSON schema + DB schema.~~ (done)
  2. ~~Adapt orchestrator API: size-based track selection and payload offload.~~ (done)
  3. ~~Implement worker split: small/large queue consumers with different budgets.~~ (done)
  4. Add CEL routing engine for small track.
  5. Add retry/DLQ policies and operational dashboards.
  6. ~~PostgreSQL run-state writes in enqueue/worker flow.~~ (done)
  7. Per-step workflow_steps persistence from runner trace (not yet wired).