Orchestrator¶
Responsibility¶
Loads runtime definitions and executes workflow steps via workflow_runner.py.
Integrations, transforms, and scripts run as type: service_call after discovery through the capability registry (CAPABILITY_REGISTRY_URL). Built-in control/context steps stay in-process.
For asynchronous execution and durability, orchestrator uses:
- RabbitMQ command queues (small/large tracks + retry + DLQ)
- PostgreSQL run/step state persistence
- MinIO / storage payload references for large messages
Inbound Authentication¶
- HTTP invocation auth (OAuth2 scopes and/or bearer token fallback)
- scheduled / dashboard / AMQP invocation auth
- admin reload token (
X-Reload-Token)
Outbound Authentication¶
- Calls capability providers over the internal network (URLs from registry, not
EGRESS_*env) - Uses storage bearer token for runtime reads/writes
- Optional SES / registry bearer tokens as configured
Key Config¶
CAPABILITY_REGISTRY_URL(required forservice_call)CAPABILITY_REGISTRY_TOKEN,CAPABILITY_REGISTRY_CACHE_TTL_SECONDSORCH_RUNTIME_STOREWORKFLOWS_DIR,CONNECTIONS_DIR,POLICIES_DIRORCH_RELOAD_TOKENORCH_ASYNC_QUEUE_ENABLEDORCH_MESSAGE_RABBITMQ_URL,ORCH_MESSAGE_WORK_EXCHANGEORCH_MESSAGE_TRACK_THRESHOLD_BYTES(default target:524288)ORCH_MESSAGE_STORAGE_UPLOAD_TIMEOUT_SECONDSORCH_MESSAGE_DATABASE_DSN/POSTGRES_DSNas deployed- MinIO / storage credentials for large-track offload
Do not configure orchestrator with EGRESS_*_URL — discovery is only CAPABILITY_REGISTRY_URL. Gateway/dashboard may still probe egress health with those env vars.
Cluster Service port: orchestrator:8080.
Health¶
GET /healthzGET /readyz
Entry points¶
POST /run/{workflow_name}POST /run/{workflow_name}/enqueue(returns202)POST /invoke/scheduledPOST /invoke/amqpPOST /invoke/amqp/dispatchPOST /invoke/dashboardGET /api/runs*, traces, trigger-eventsPOST /admin/validate?strict=true
See Orchestrator API.
Async Queue Entry¶
POST /run/{workflow_name}/enqueue(returns202)- Requires async queue mode enabled (
ORCH_ASYNC_QUEUE_ENABLED=true) - Small track (
<= 512KB): queues payload inline - Large track (
> 512KB): uploads payload as artifact and queuespayload_ref
Queue Workers¶
python -m app.queue_worker(no HTTP server)ORCH_WORKER_TRACK=small|largeselects queue whenORCH_MESSAGE_CONSUMER_QUEUEis unset- Kubernetes deployments:
orchestrator-worker-smallorchestrator-worker-large- Uses the same orchestrator image/runtime config as the API pod (workflows, connections, traces, registry discovery)
Run state persistence¶
- Driver:
psycopg(async pool) - Env:
ORCH_MESSAGE_DATABASE_DSN(Postgres run ledger; required on the API orchestrator and queue workers for Operations / checkpoints) - Schema:
engine/services/orchestrator/db/migrations/0001_workflow_messaging.sql - Tables:
workflow_runs,workflow_steps,message_receipts,workflow_events - When DSN is unset, run-state writes are skipped (queue flow still works)
Workflow Validation Contract¶
Orchestrator validates workflow YAML through POST /admin/validate using a layered model:
- contract validation (always)
- provider semantic validation (optional when provider capability supports it)
Use ?strict=true so missing registry capabilities fail CI/publish.
Recommended response shape (aligned with API reference):
{
"valid": true,
"contract_status": "passed",
"provider_status": "passed",
"steps": [
{
"step_id": "publish_order",
"contract": {
"status": "passed",
"diagnostics": []
},
"provider": {
"status": "passed",
"diagnostics": []
}
}
]
}
Diagnostic entry fields:
severity:error|warning|infocode: stable machine-readable codemessage: human-readable explanationpath: optional field path in step/request payload
Blocking policy recommendation:
- contract errors always block publish/reload
- provider errors block only in strict mode
- provider unavailable yields
provider_status=skippedin non-strict mode
Two-Track Message Policy¶
- Small track (
<= 512KB) - Inline payload in queue message.
- In-flow transformations allowed (bounded by execution limits).
-
Content-based routing using CEL expressions.
-
Large track (
> 512KB) - Payload stored in object storage; queue carries
payload_ref. - No inline transforms in runner.
- Transformations only through provider
service_callsteps.
See ADR 0006 for the normative contract and queue topology.