Skip to content

ADR 0007 — Run checkpointing & crash recovery

Status: Proposed (design)
Date: 2026-06-19
Related: ADR 0006, Orchestrator messaging blueprint, ADR 0002, ADR 0004

Context

MeshFlows executes workflows in-process in the orchestrator (workflow_runner.run_workflow). During a run the engine keeps:

  • step outputs (outputs[step_id])
  • workflow context (string map, including scoped variables and context_set values)
  • trace (filesystem under TRACES_DIR)

Today:

Mechanism What it persists Used for crash recovery?
trace_store Step I/O files during run; trace.json at end No — observability only
RunStateStore (Postgres) Run lifecycle for queue path (accepted → running → …) No — no step checkpoints
workflow_steps table (migration) Schema exists Not written by code
Step retry_policy Retries within same process Transient errors only
Idempotency Duplicate successful enqueue Prevents double success, not resume
Variables provider Stateless operations only External compute; orchestrator context holds values

After orchestrator/worker crash or pod kill, a run may leave Postgres on running, partial trace files on disk, and no way to continue from step N+1.

ADR 0006 already names step checkpoints and durable queue semantics; this ADR defines what recovery means and a phased path to get there without blocking current delivery (variables provider, external triggers, dashboard runs).

Goals

  1. Durable run ledger — operators can see whether a run is in progress, finished, or stale (worker died).
  2. Step-level progress — which steps completed, failed, or were skipped for a given run_id.
  3. Defined recovery semantics — explicit behaviour on retry (full re-run vs resume), not implicit hope.
  4. Same mental model for context — workflow context remains the execution source of truth; providers hold optional external state (ADR 0002).
  5. Queue-first — async path (/enqueue + workers) gets durability first; sync paths (HTTP, dashboard, scheduler) can opt in later.

Non-goals (initial phases)

  • Exactly-once side effects inside arbitrary service_call steps (callers must be idempotent).
  • Automatic resume inside parallel lanes without explicit scope policy (Phase 3+).
  • Cross-region active/active orchestrator failover.
  • Replacing trace_store; traces remain the debugging/UX surface; Postgres is the operational ledger.

Terminology

Term Meaning
Run ledger Postgres rows: workflow_runs, workflow_steps, workflow_events
Checkpoint Persisted record that step S finished (or failed) with enough data to resume or audit
Resume Start execution at step S+1 using restored context + outputs, not re-running S
Replay Full workflow execution again (new or same run_id policy — see below)
Stale run workflow_runs.status = running with no heartbeat beyond timeout

Decision

1. Two layers: ledger + checkpoint payload

Layer A — Run ledger (minimal, always cheap)

Extend RunStateStore to write:

  • workflow_steps on step start and finish (status, timestamps, error).
  • workflow_events for fine-grained audit (step.started, step.succeeded, step.failed, step.skipped).
  • workflow_runs.heartbeat_at (new column) updated periodically during long runs.

Layer B — Checkpoint payload (resume data)

New table workflow_run_checkpoints (or JSON blob column on latest step row — prefer separate table for large context):

CREATE TABLE workflow_run_checkpoints (
    run_id          TEXT NOT NULL REFERENCES workflow_runs(run_id) ON DELETE CASCADE,
    step_id         TEXT NOT NULL,
    attempt         INTEGER NOT NULL DEFAULT 1,
    scope_path      TEXT NOT NULL DEFAULT '',  -- e.g. '' | 'fanout.lane_fast' | 'repeat_gate.iter_2'
    context_json    JSONB NOT NULL,            -- masked context snapshot
    outputs_json    JSONB NOT NULL,            -- outputs map (step_id -> body/preview ref)
    previous_body_ref TEXT,                    -- optional ref to storage for large "previous" chain
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (run_id, scope_path, step_id, attempt)
);
  • context_json: workflow context after the step (secrets redacted, same rules as trace).
  • outputs_json: serialisable outputs dict; large bodies → storage ref + sha256 (same pattern as large track payloads).
  • scope_path: encodes nested execution (parallel lane, repeat_until iteration). Root main line uses ''.

Checkpoint after each successful step (and on controlled failure before compensation). Skipped steps (when, run_after) recorded in ledger only — no checkpoint advance.

2. Recovery modes (explicit, not magic)

Product defines three modes per workflow (YAML hint or runtime default):

Mode On worker retry / operator action Guarantee
replay (default) Full workflow from step 1 At-least-once attempt; idempotency may return prior success
resume Continue after last committed checkpoint Best-effort resume if checkpoint exists; else fail with 409 no_checkpoint
manual No automatic retry; operator uses dashboard retrigger Human decides payload

Default = replay until resume is proven for a workflow’s step types.

Resume preconditions:

  1. Same workflow_name and compatible workflow definition hash (store workflow_revision on workflow_runs — content hash of steps YAML at start).
  2. Last checkpoint’s scope_path is unambiguous (no in-flight parallel lane without per-lane checkpoint — Phase 3).
  3. Downstream service_call steps are idempotent or use explicit idempotency keys.

3. Integration with queue worker (ADR 0006)

Align worker ACK with ledger:

dequeue → running (+ heartbeat loop)
  → execute with checkpoint writer
  → succeeded → receipt + ACK
  → failed (retryable) → NACK/retry exchange (when topology exists)
  → failed (terminal) / max retries → failed + DLQ
  → crash mid-run → message redelivered OR stale-run sweeper marks `interrupted`

Crash during running:

  • Message: at-least-once redelivery (fix requeue=False → retry topology per blueprint).
  • Ledger: stale-run job marks runs with heartbeat_at < now - threshold as interrupted.
  • Recovery: new delivery uses recovery_mode on envelope (replay | resume) — default replay; optional header X-MeshFlows-Recovery: resume.

Idempotency interaction:

  • message_receipts still means “this idempotency key already succeeded” → return cached outcome, do not resume a different partial run.
  • Separate idempotency key required if operator intentionally wants a second full run.

4. Sync runs (HTTP, dashboard, scheduler)

Phase 1: no change to execution path; optional read-only mirror of step events to Postgres when DSN set (same code path as queue, trigger_type recorded on workflow_runs).

Phase 2: dashboard shows ledger + “stale / interrupted” badge; retrigger uses replay.

Phase 3 (optional): dashboard “Resume” button when recovery_mode=resume and checkpoint exists.

Rationale: sync runs are short-lived requests; durability investment belongs on the queue track first.

5. Variables provider & context

Checkpoint stores orchestrator context (including values set via context_set and merged scoped_context init).

  • The variables provider is stateless — it is not a resume source of truth (ADR 0008).
  • On resume, restore context from checkpoint; no provider resync.

This matches the rule: provider computes; context holds values.

6. Control-flow scope rules (phased)

Construct Phase 1 ledger Phase 2 checkpoint Phase 3 resume
Linear steps ✓ ✓ root scope ✓
if / when / run_after / try ✓ ✓ ✓
repeat_until ✓ ✓ per iteration in scope_path ✓
for_each ✓ ✓ scope_path = {step_id}.item_{i} ✓
parallel ✓ per lane ✓ per lane Lane-level resume only
workflow_call ✓ child run_id link Child run own ledger Child resume independent

Parallel: resume enters only lanes whose checkpoint is succeeded; incomplete lanes re-run from lane start (not mid-lane unless lane has single linear chain).

7. Heartbeat & stale-run sweeper

  • workflow_runs.heartbeat_at updated every N seconds from run_workflow loop (config ORCH_RUN_HEARTBEAT_SECONDS, default 15).
  • Background task (orchestrator API or worker sidecar): UPDATE workflow_runs SET status = 'interrupted' WHERE status = 'running' AND heartbeat_at < ….
  • Dashboard: filter runs interrupted / running (stale).

8. API surface (incremental)

Endpoint Purpose
GET /api/runs/{run_id}/ledger Run + steps + events (dashboard proxy)
POST /api/runs/{run_id}/resume Operator resume (resume mode, authz)
POST /run/{workflow}/enqueue Add optional recovery_mode in body

Internal: RunStateStore.upsert_step(...), write_checkpoint(...), touch_heartbeat(...).

Workstreams & sequencing

Phase R0 — Honest baseline (1–2 days)

  • Document current behaviour in orchestrator wiki (no resume).
  • Fix queue worker requeue=False vs ADR 0006 blueprint (retry/DLQ) — separate task, tracked as dependency for R2.

Phase R1 — Run ledger for all executions (MVP)

ID Deliverable Acceptance
R1.1 RunStateStore step APIs + migration heartbeat_at, trigger_type, workflow_revision Unit tests
R1.2 Hook workflow_runner: on step start/finish → ledger (when DSN set) Integration test: linear workflow writes 3 step rows
R1.3 Create workflow_runs row for sync _execute (not only enqueue) Dashboard can list run status for dashboard trigger
R1.4 Stale-run sweeper + interrupted status Run left running → marked interrupted after timeout

Outcome: Operators see progress and stale runs; no resume yet.

Phase R2 — Checkpoint payload + replay discipline

ID Deliverable Acceptance
R2.1 workflow_run_checkpoints migration + write after each succeeded step Checkpoint count = succeeded steps
R2.2 Queue redelivery → replay by default; idempotency unchanged Duplicate enqueue still returns prior success
R2.3 Store workflow_revision hash; refuse resume on mismatch Test: edit YAML → resume blocked

Phase R3 — Resume (opt-in per workflow)

ID Deliverable Acceptance
R3.1 recovery_mode: replay \| resume on workflow doc (default replay) Validation in /admin/validate
R3.2 run_workflow entry: restore context/outputs from last checkpoint Test: fail after step 2 → resume completes step 3 only
R3.3 repeat_until + for_each scope_path Fixture workflow green
R3.4 Dashboard “Resume” for interrupted + checkpoint Admin only

Phase R4 — Parallel & child runs

  • Per-lane checkpoints and partial parallel resume.
  • workflow_call child resume linked via parent_run_id (already in trace meta).

Consequences

Positive

  • Clear operator story: running / succeeded / failed / interrupted.
  • Foundation for true resume without overloading trace files.
  • Aligns code with ADR 0006 schema already in migrations.
  • Context-first model preserved for variables (ADR 0002).

Risks / trade-offs

  • Postgres write amplification (every step); mitigate with async batch or “checkpoint every N steps” flag later.
  • Resume with non-idempotent service_call can double side effects — workflows must opt in to resume.
  • Parallel resume is inherently harder; default stays replay.
  • Larger context snapshots → storage offload (same as large track).

Open questions

  1. Should trace.json be written incrementally (crash partial trace) in parallel with ledger? (Recommended: yes, low effort — RunTrace.flush_partial().)
  2. Retention: checkpoint TTL vs trace retention (TRACES_MAX_RUNS).
  3. Is interrupted a terminal status or can it transition to running on resume?

See also