Engine Internals: How Workflows Start¶
This page explains the main workflow start paths:
- HTTP sync (
/run/{workflow_name}) - HTTP async enqueue (
/run/{workflow_name}/enqueue) - Scheduler (
/invoke/scheduled) - AMQP / queue (
/invoke/amqp,/invoke/amqp/dispatch, workers) - Dashboard (
/invoke/dashboard)
1. Scheduler → Orchestrator Start Path¶
- Scheduler stores and evaluates cron jobs.
- When a job fires, Scheduler calls Orchestrator scheduled-invocation endpoint.
- Orchestrator validates scheduled auth (
SCHEDULE_INVOCATION_TOKENand optional OAuth mode). - Orchestrator resolves workflow, trigger context, runtime config, and cursor state.
run_workflow()executes step-by-step inworkflow_runner.py.- Trace and trigger-event records are persisted.
- Result is returned to Scheduler caller.
Key endpoint:
POST /invoke/scheduled
2. HTTP → Orchestrator Start Path¶
- Client calls Gateway route for a workflow.
- Gateway enforces edge auth/policy and proxies to Orchestrator.
- Orchestrator resolves workflow by URL path (
/run/{workflow_name}). - Invocation auth is validated (
HTTP_INVOCATION_TOKEN, per-workflow token, or OAuth scopes). - Input payload and request context are normalized.
run_workflow()executes all steps and applies retry/throttle/idempotency rules.- Orchestrator returns final response and
X-Request-IDfor correlation.
Key endpoints:
POST /run/{workflow_name}(sync)POST /run/{workflow_name}/enqueue(async202; requires messaging env)
Body-selected
POST /runis deprecated / unsupported as the public contract — always use path-based/run/{name}(or gateway aliases).
3. AMQP / Queue Start Path¶
Two related patterns:
Trigger subscriber → Orchestrator¶
rabbitmq-trigger-subscriber(egress-rabbitmq image role) consumes broker messages.- Subscriber validates the event and calls
POST /invoke/amqp(or the dispatch path as configured). - Orchestrator authorizes the AMQP path, loads the target workflow, and runs (or enqueues) it.
Orchestrator enqueue → workers¶
- Client (or gateway) calls
POST /run/{name}/enqueue. - API pod builds a command envelope; large payloads offload to storage (
payload_ref). - Message is published to
ORCH_MESSAGE_WORK_EXCHANGE(run.small/run.large). orchestrator-worker-small/orchestrator-worker-largeconsume and execute via the samerun_workflow()path.- Run ledger updates go to Postgres when
ORCH_MESSAGE_DATABASE_DSNis set.
Key env: ORCH_MESSAGE_RABBITMQ_URL, ORCH_MESSAGE_*, ORCH_ASYNC_QUEUE_ENABLED.
4. Dashboard Start Path¶
- Dashboard IAM checks user scope.
- Dashboard calls
POST /invoke/dashboard. - Orchestrator validates dashboard token / OAuth and executes the workflow.
Why Step Classes Live in Orchestrator¶
The step classes in workflow_runner.py are orchestrator execution contracts, not microservice internals.
- Orchestrator owns workflow schema validation and step dispatch for built-in types (
context_*,if,parallel, …) plus the genericservice_call. - Capability providers own protocol-specific I/O (HTTP, Gmail, S3, transforms, SES, …).
- A step class defines what workflow YAML is valid and how orchestration chooses the next action.
For Google Workspace specifically:
- There are no first-class
gmail_*/calendar_*/drive_*step types. - Workflows use
type: service_callwithcapability_name: egress.google_workspace.*and the provider path (e.g./gmail/list). - Discovery is via the capability registry — not an orchestrator
EGRESS_GOOGLE_WORKSPACE_URLcode path.
This split keeps orchestration deterministic while isolating external protocol complexity.
Reload and Runtime Data Flow¶
- Flows/connections/policies are uploaded to Storage.
POST /admin/reloadin Orchestrator re-pulls runtime data from the configured runtime store.- In-memory workflow/connection/policy snapshots are atomically swapped under reload lock.
If reload fails, check:
- connection schema compatibility in runtime store
- unresolved secret refs
- storage API health and payload validity
- capability registry reachability for
service_callvalidation (?strict=true)
Orchestrator vs RabbitMQ Trigger Subscriber¶
MeshFlows splits execution from trigger ingestion:
orchestrator: API-facing execution runtime for HTTP, schedule, dashboard, and AMQP invoke endpoints.rabbitmq-trigger-subscriber: RabbitMQ ingress adapter (egress-rabbitmq image) that consumes queue/topic messages and calls Orchestrator.
This keeps Orchestrator focused on workflow execution and isolates broker-consumer behavior.
Why the Logs Look the Same¶
- Both are Python/FastAPI services and can emit similar infra logs (
httpx, probes, OTEL). - The queue subscriber no longer hosts Orchestrator workflow runtime internals.
So seeing similar lines in both pods is expected behavior, not necessarily a duplicate execution bug.
What Actually Differs¶
The main difference is role configuration (environment variables), not code identity.
- API orchestrator focuses on endpoint-driven execution (
/run/{name},/invoke/*). - Queue subscriber focuses on RabbitMQ consume/ack/nack and forwards valid events to Orchestrator.
Queue subscriber deployment reference:
engine/deploy/k8s/rabbitmq-trigger-subscriber-deployment.yaml
Practical Log Differentiation¶
To reduce operator confusion:
- Filter by pod/deployment name (
orchestrator-*vsrabbitmq-trigger-subscriber-*). - Filter by Kubernetes component labels.
- Use different log levels per role (for example INFO on queue subscriber, DEBUG only when debugging).
- Optionally add an explicit role field in log context (
apivsqueue-subscriber).