Backend (Engine) — Microservices Infrastructure¶
The Engine is the core infrastructure and microservices layer of MeshFlows.
For how this differs from UI-first automation tools or embedded in-app workflow engines, see Why MeshFlows.
What is the Engine?¶
The Engine provides all the operational infrastructure needed to: - Execute workflows — YAML-driven workflow orchestration - Handle requests — Gateway for public API access and alias route mapping - Transform data — XSLT, XML↔JSON, CSV, Liquid templates - Store state — Runtime key-value store & document management - Authenticate users — JWT tokens, OAuth 2.0, identity management - Schedule tasks — CronJob-based workflow triggers - Manage integrations — HTTP, FTP, SSH, RabbitMQ connections - Monitor everything — Observability stack (Tempo, Loki, Prometheus)
Services at a Glance¶
| Service | Purpose |
|---|---|
| Gateway | Public API entry point |
| Orchestrator | Workflow execution (orchestrator:8080); discovers providers via registry |
| Capability Registry | Register / heartbeat / /v1/registry/capabilities |
| Transformers | transformers.* via service_call |
| SES / Variables | script.execute, variables.* |
| Storage | Artifacts, secrets, runtime documents |
| Identity | User management & JWT |
| Scheduler | Cron / named schedule triggers |
| Dashboard | Web UI for workflows |
| Capability providers | HTTP, FTP, SSH/SFTP, SMTP/IMAP, RabbitMQ, S3, Postgres, Weatherstack, Positionstack, Joke API, Google Workspace, … |
| RabbitMQ | Message broker |
Orchestrator does not use EGRESS_*_URL for discovery — only CAPABILITY_REGISTRY_URL. See Services Overview.
Start Here¶
New Platform Setup¶
- Prerequisites
- Kubernetes Deployment — or AKS Deployment
- Secrets Management — platform / CI; for
$secret:see Runtime Secrets - Operations
Local Development¶
- Local Development
- Develop and Test
- Longhorn Storage Setup
- Flows Overview (Forgejo mirror: Flows Overview)
API and Runtime Internals¶
- Architecture
- Services Overview
- Gateway API
- Edge routes (implementation)
- Trigger policy-chain redesign plan
- Orchestrator API
- Orchestrator Messaging Blueprint
I’m an Operator — What Do I Need to Know?¶
First Time Setup?¶
- Prerequisites — Get your environment ready
- Kubernetes Deployment — Deploy to your cluster
- Operations & Monitoring — Run and manage the system
- Forgejo Container Package Cleanup — Registry retention (keep
vX.Y.Z)
Local Development?¶
Going Deeper?¶
- Architecture Overview — System design
- Scheduler Operations — Cron triggers, monitoring, troubleshooting
- Security & Hardening — Production hardening
- Observability Setup — Monitoring & tracing
Deployment Options¶
Docker Compose (Development/Testing)¶
cd engine
docker-compose up -d
Kubernetes (Production)¶
cd engine/deploy
kubectl apply -k overlays/linux-cluster/
Azure Kubernetes Service (AKS)¶
cd engine/deploy
kubectl apply -k overlays/aks/ --load-restrictor=LoadRestrictionsNone
Minikube (Learning)¶
# From repo root — preferred
./deploy-minikube.ps1
# Or: kubectl apply -k engine/deploy/overlays/minikube/
engine/deploy/k8s/overlays/minikube/ is a deprecation stub — use apply-via-canonical.sh or the canonical overlay directly.
Key Features¶
- YAML Workflows — Declarative workflow definitions
- Parallel Execution — Concurrent step execution
- Retry & Compensation — Automatic failure recovery
- Data Transformations —
transformers.*capability providers - Integrations — registry-driven
service_call(HTTP, FTP, SSH/SFTP, RabbitMQ, S3, …) - Edge Route Aliases — Public endpoint-to-workflow mapping in Gateway
- OAuth 2.0 — Inbound and outbound authentication
- Observability — OpenTelemetry + Tempo + Loki
- Storage ACLs — Role-based access control
- Scheduler — CronJob or HTTP-driven triggers
- Dashboard — Visual workflow editor
Technology Stack¶
- Language: Python 3.11+ (FastAPI)
- Container: Docker
- Orchestration: Kubernetes (Kustomize)
- Storage: Longhorn persistent volumes
- Message Broker: RabbitMQ
- Observability: OpenTelemetry + Tempo + Loki + Prometheus
- Testing: pytest
- CI/CD: Forgejo Actions + container image build pipeline
Common Tasks¶
Deploy to Production¶
→ Kubernetes Deployment How-to
Configure Secrets¶
→ Deploy Runtime Secrets — $secret: / Dashboard / Storage
→ Secrets Management (platform / CI) — Forgejo → Kubernetes
Setup Longhorn Storage¶
Monitor with Tempo/Loki¶
Secure Your Deployment¶
→ Security & Hardening Reference
Understand Our CI/CD Pipeline¶
→ CI/CD Pipeline Architecture — How builds, tests, and deployments work
Clean up Forgejo container packages¶
→ Forgejo Container Package Cleanup — what to keep (vX.Y.Z, recent build tags) and how to delete -pr/-dev without breaking deploy
Capability providers¶
→ Integration end-to-end — new provider + connection + workflow + trigger
→ Add a Capability Provider — Python, Go, and C# starters
→ Capability provider contract — registration, designer metadata, UI ownership
→ Deploy integration — images, kustomize, CI checklist
Internal docs (repo only, not in wiki)¶
| Topic | Location | Why not published |
|---|---|---|
| Edge control backlog / tickets | internal/edge-control-layer-*.md (except sprint1 guide, now reference/20) |
Planning/backlog |
| Script Execution Service design | internal/script-execution-service-*.md |
Superseded by Script Steps + engine/services/script-execution-service/README.md |
| Platform proposals | internal/proposals/ |
Pre-implementation proposals |
| AI scratch prompts | internal/meshflows-from-scratch-prompt* |
Not operator/developer documentation |
| Connector issue backlog | internal/codeberg-issue-requests-connectors.md |
Internal tracker |
Develop and Test Locally¶
→ Develop and Test How-to — Local setup through CI promotion
Understand OAuth¶
Next Steps¶
- Operators/DevOps: Start with Prerequisites
- Developers: Check out Flows (Workflows) documentation (Forgejo mirror: Flows (Workflows))
- Learning: Try Local Development