Skip to content

Engine Internals: How Workflows Start

This page explains the two main workflow start paths:

  • scheduler-triggered start (/invoke/scheduled)
  • HTTP-triggered start (/run/{workflow_name} or /run)

1. Scheduler -> Orchestrator Start Path

  1. Scheduler stores and evaluates cron jobs.
  2. When a job fires, Scheduler calls Orchestrator scheduled-invocation endpoint.
  3. Orchestrator validates scheduled auth (SCHEDULE_INVOCATION_TOKEN and optional OAuth mode).
  4. Orchestrator resolves workflow, trigger context, runtime config, and cursor state.
  5. run_workflow() executes step-by-step in workflow_runner.py.
  6. Trace and trigger-event records are persisted.
  7. Result is returned to Scheduler caller.

Key endpoint:

  • POST /invoke/scheduled

2. HTTP -> Orchestrator Start Path

  1. Client calls Gateway route for a workflow.
  2. Gateway enforces edge auth/policy and proxies to Orchestrator.
  3. Orchestrator endpoint resolves workflow by path/body.
  4. Invocation auth is validated (HTTP_INVOCATION_TOKEN, per-workflow token, or OAuth scopes).
  5. Input payload and request context are normalized.
  6. run_workflow() executes all steps and applies retry/throttle/idempotency rules.
  7. Orchestrator returns final response and X-Request-ID for correlation.

Key endpoints:

  • POST /run/{workflow_name}
  • POST /run

Why Step Classes Live in Orchestrator

The many step classes in workflow_runner.py are orchestrator execution contracts, not microservice internals.

  • Orchestrator owns workflow schema validation and step dispatch.
  • Egress microservices own protocol-specific I/O details.
  • A step class defines what workflow YAML is valid and how orchestration chooses the next action.

For Google Workspace specifically:

  • Step models (gmail_*, calendar_*, drive_*) stay in orchestrator.
  • API calls and token exchange run in egress-google-workspace.

This split keeps orchestration deterministic while isolating external protocol complexity.

Reload and Runtime Data Flow

  1. Flows/connections/policies are uploaded to Storage.
  2. POST /admin/reload in Orchestrator re-pulls runtime data from the configured runtime store.
  3. In-memory workflow/connection/policy snapshots are atomically swapped under reload lock.

If reload fails, check:

  • connection schema compatibility in runtime store
  • unresolved secret refs
  • storage API health and payload validity

Orchestrator vs RabbitMQ Trigger Subscriber

MeshFlows splits execution from trigger ingestion:

  • orchestrator: API-facing execution runtime for HTTP and scheduled invocations.
  • rabbitmq-trigger-subscriber: RabbitMQ ingress adapter (egress-rabbitmq image) that consumes queue/topic messages and calls Orchestrator.

This keeps Orchestrator focused on workflow execution and isolates broker-consumer behavior.

Why the Logs Look the Same

  • Both are Python/FastAPI services and can emit similar infra logs (httpx, probes, OTEL).
  • The queue subscriber no longer hosts Orchestrator workflow runtime internals.

So seeing similar lines in both pods is expected behavior, not necessarily a duplicate execution bug.

What Actually Differs

The main difference is role configuration (environment variables), not code identity.

  • API orchestrator focuses on endpoint-driven execution (/run, /invoke/scheduled).
  • Queue subscriber focuses on RabbitMQ consume/ack/nack and forwards valid events to Orchestrator.

Queue subscriber deployment reference:

  • engine/deploy/k8s/rabbitmq-trigger-subscriber-deployment.yaml

Practical Log Differentiation

To reduce operator confusion:

  1. Filter by pod/deployment name (orchestrator-* vs rabbitmq-trigger-subscriber-*).
  2. Filter by Kubernetes component labels.
  3. Use different log levels per role (for example INFO on queue subscriber, DEBUG only when debugging).
  4. Optionally add an explicit role field in log context (api vs queue-subscriber).