Skip to content

Architecture Overview

MeshFlows Engine is a service-oriented runtime for workflow orchestration.

Core Components

  • Gateway: external API and request routing layer.
  • Orchestrator: workflow execution runtime and policy enforcement.
  • Capability registry: provider register / heartbeat / discovery (CAPABILITY_REGISTRY_URL).
  • Transformers, SES, Variables, egress providers: capability providers called via service_call.
  • Storage: workflow/config/secrets persistence and runtime documents.
  • Identity: token issuance and identity functions.
  • Scheduler: timed workflow invocation.

Logical Request Path

  1. Client calls Gateway (/v1/api/*, /v1/run/*, or /v1/transform).
  2. Gateway resolves edge route mapping for /v1/api/* using exact method + path matching.
  3. Gateway forwards to Orchestrator and preserves correlation headers.
  4. Orchestrator executes workflow steps (built-in control/context in-process).
  5. Integration/transform/script steps are service_call: orchestrator looks up the capability in the capability registry, then POSTs to the provider URL (capability_name + path).
  6. Trace, run ledger, and trigger data are stored and exposed via API.

Data and Control Planes

  • Control plane: workflow definitions, connections, policies, schedules, runtime config.
  • Data plane: workflow payloads and step outputs.
  • Ops plane: health, readiness, traces, and admin endpoints.

Deployment Model

  • Containerized microservices, typically deployed on Kubernetes.
  • Service discovery through cluster DNS and service names.
  • Configuration via environment variables and mounted files/secrets.
  • Persistent data for Storage and message broker state.

Runtime Characteristics

  • Stateless API services where possible.
  • Explicit request correlation using X-Request-ID and correlation headers.
  • Async HTTP communication between services.
  • Retry/idempotency/throttling controls in orchestration runtime.

Messaging and Persistence Direction

  • Broker: RabbitMQ (durable work queues, retry, DLQ).
  • State: PostgreSQL for workflow run/step lifecycle and idempotency receipts.
  • Payload offload: MinIO for large message bodies by reference.
  • Two-track policy:
  • small track (<= 512KB): inline payload, optional in-flow transforms, CEL content routing.
  • large track (> 512KB): payload-by-reference, transforms only via provider services.

See ADR 0006 for normative details.

  • Services: 02-services.md
  • Orchestrator API: 04-orchestrator-api.md
  • CI/CD architecture: 11-cicd-pipeline.md
  • ADR: ../adr/0006-rabbitmq-persistence-two-track-routing.md