Skip to content

Backend (Engine) — Microservices Infrastructure

The Engine is the core infrastructure and microservices layer of MeshFlows.

What is the Engine?

The Engine provides all the operational infrastructure needed to: - Execute workflows — YAML-driven workflow orchestration - Handle requests — Gateway for public API access and alias route mapping - Transform data — XSLT, XML↔JSON, Liquid templates - Store state — Runtime key-value store & document management - Authenticate users — JWT tokens, OAuth 2.0, identity management - Schedule tasks — CronJob-based workflow triggers - Manage integrations — HTTP, FTP, SSH, RabbitMQ connections - Monitor everything — Observability stack (Tempo, Loki, Tempo)

Services at a Glance

Service Purpose
Gateway Public API entry point (port 8080)
Orchestrator Workflow execution engine (port 8083)
Transformers Data transformations (port 8081)
Storage Key-value store with ACLs (port 8086)
Identity User management & JWT (port 8088)
Scheduler CronJob triggers (port 8089)
Dashboard Web UI for workflows (port 8090)
Egress Services HTTP, FTP, SSH, RabbitMQ (ports 8082-8087)
RabbitMQ Message broker (ports 5672, 15672)

Start Here

New Platform Setup

  1. Prerequisites
  2. Kubernetes Deployment
  3. Secrets Management
  4. Operations

Local Development

  1. Local Development
  2. Longhorn Storage Setup
  3. Flows Overview (Forgejo mirror: Flows Overview)

API and Runtime Internals

  1. Architecture
  2. Services Overview
  3. Gateway API
  4. Orchestrator API
  5. Orchestrator Messaging Blueprint

I’m an Operator — What Do I Need to Know?

First Time Setup?

  1. Prerequisites — Get your environment ready
  2. Kubernetes Deployment — Deploy to your cluster
  3. Operations & Monitoring — Run and manage the system

Local Development?

  1. Local Development (Docker Compose)
  2. Secrets Management

Going Deeper?

Deployment Options

Docker Compose (Development/Testing)

cd engine
docker-compose up -d
All services + RabbitMQ + observability stack in one command.

Kubernetes (Production)

cd engine/deploy
kubectl apply -k overlays/linux-cluster/
Full production-ready deployment with persistence, networking, and monitoring.

Minikube (Learning)

cd engine/deploy
kubectl apply -k overlays/minikube/
Simplified deployment for learning and testing on a local cluster.

Key Features

  • YAML Workflows — Declarative workflow definitions
  • Parallel Execution — Concurrent step execution
  • Retry & Compensation — Automatic failure recovery
  • Data Transformations — XSLT 1.0, XML↔JSON, Liquid
  • Integrations — HTTP, FTP, SSH, RabbitMQ
  • Edge Route Aliases — Public endpoint-to-workflow mapping in Gateway
  • OAuth 2.0 — Inbound and outbound authentication
  • Observability — OpenTelemetry + Tempo + Loki
  • Storage ACLs — Role-based access control
  • Scheduler — CronJob or HTTP-driven triggers
  • Dashboard — Visual workflow editor

Technology Stack

  • Language: Python 3.11+ (FastAPI)
  • Container: Docker
  • Orchestration: Kubernetes (Kustomize)
  • Storage: Longhorn persistent volumes
  • Message Broker: RabbitMQ
  • Observability: OpenTelemetry + Tempo + Loki + Tempo
  • Testing: pytest
  • CI/CD: Forgejo Actions + container image build pipeline

Common Tasks

Deploy to Production

→ Kubernetes Deployment How-to

Configure Secrets

→ Secrets Management How-to

Setup Longhorn Storage

→ Longhorn Storage How-to

Monitor with Tempo/Loki

→ Observability Reference

Secure Your Deployment

→ Security & Hardening Reference

Understand Our CI/CD Pipeline

→ CI/CD Pipeline Architecture — How builds, tests, and deployments work

Understand OAuth

→ Authentication Reference

Next Steps

  1. Operators/DevOps: Start with Prerequisites
  2. Developers: Check out Flows (Workflows) documentation (Forgejo mirror: Flows (Workflows))
  3. Learning: Try Local Development