Skip to content

CI/CD Pipeline Architecture

The MeshFlows CI/CD pipeline automates building, testing, and deploying engine components with intelligent change detection.

Complete Pipeline Flow

graph TD
    A[Push to main/develop] --> B[Detect Changes]

    B --> C{Any Changes?}
    C -->|Engine code| D["📦 Build Engine Services<br/>Smart - only changed"]
    C -->|Wiki docs| E["📚 Build Wiki<br/>MkDocs → Docker"]
    C -->|Flows| F["🔍 Validate Workflows<br/>YAML syntax check"]

    D --> G{All Builds<br/>Success?}
    E --> G

    F --> H["Validate Structure<br/>Preflight checks"]
    H --> I{Structure<br/>OK?}

    G -->|Build fails| X["❌ STOP<br/>Fix & Retry"]
    I -->|Validation fails| X

    G -->|All OK| J[Test Engine Code]
    I -->|All OK| Z["Run Tests"]

    F --> K{Flows<br/>Changed?}
    K -->|Yes| L["🚀 Test Deployment<br/>Deploy to Lab"]
    K -->|No| M["⏭️ Skip Workflow Tests"]

    J --> N{Tests<br/>Pass?}
    N -->|Fail| X
    N -->|Pass| O["🎯 Verification Gate<br/>Both build + test?"]

    O --> P{Both<br/>Success?}
    P -->|No| X
    P -->|Yes| Q["✅ DEPLOY<br/>to Lab Cluster"]

    Q --> R["Deploy Engine"]
    Q --> S["Deploy Wiki"]
    R --> T["Deploy Workflows"]

    T --> U["🚀 LIVE on Lab"]

Detailed Job Dependency Chain

graph LR
    DC["detect-changes"]
    DB["detect-test-changes"]

    subgraph Build["🔨 BUILD PHASE"]
        BES["build-engine-services<br/>Matrix: only changed services"]
        BW["build-wiki<br/>If wiki/ changed"]
    end

    subgraph Test["✅ TEST PHASE"]
        TE["test-engine<br/>If code changed"]
        VW["validate-workflows<br/>If flows/ changed"]
        VS["validate-structure<br/>Always runs"]
        VK["validate-k8s-manifests"]
    end

    subgraph Verify["🎯 VERIFY PHASE"]
        VBT["verify-build-and-tests<br/>Gate: both passed?"]
    end

    subgraph Deploy["🚀 DEPLOY PHASE"]
        DE["deploy-engine"]
        DW["deploy-wiki"]
        DF["deploy-workflows"]
    end

    subgraph Rollback["⏮️ ROLLBACK"]
        RB["rollback<br/>Manual trigger"]
    end

    DC --> BES
    DC --> BW
    DB --> TE
    DB --> VW

    BES --> BW
    BW --> VBT

    VS --> TE
    TE --> VW
    VW --> VK
    VK --> VBT

    VBT --> DE
    VBT --> DW

    DE --> DF

    style DC fill:#FFE6B3
    style DB fill:#FFE6B3
    style VBT fill:#FFB3B3,stroke:#FF0000,stroke-width:3px
    style DF fill:#B3D9FF
    style RB fill:#FFB3B3

Smart Change Detection

Build-Time Detection (detect-changes)

graph BT
    A["Git diff<br/>HEAD~1 → HEAD"]

    A --> B{Check paths}

    B -->|engine/services/gateway/| C["gateway: true"]
    B -->|engine/services/orchestrator/| D["orchestrator: true"]
    B -->|engine/services/storage/| E["storage: true"]
    B -->|engine/services/transformers/| F["transformers: true"]
    B -->|engine/services/identity/| G["identity: true"]
    B -->|engine/services/scheduler/| H["scheduler: true"]
    B -->|engine/services/dashboard/| I["dashboard: true"]
    B -->|engine/services/| J["egress-*: true"]
    B -->|wiki/| K["wiki: true"]
    B -->|No engine changes| L["any-engine: false<br/>Skip all builds"]

    C --> M["build-engine-services<br/>runs with matrix filter"]
    D --> M
    E --> M
    F --> M
    G --> M
    H --> M
    I --> M
    J --> M
    K --> N["build-wiki<br/>runs only if true"]

    style A fill:#FFF9E6
    style M fill:#B3E5FC
    style N fill:#B3E5FC

Test-Time Detection (detect-test-changes)

graph BT
    A["Git diff<br/>HEAD~1 → HEAD"]

    A --> B{Check paths}

    B -->|engine/services/services/| C["engine-code: true"]
    B -->|engine/tests/tests/| C
    B -->|engine/config/| C
    B -->|flows/| D["flows: true"]
    B -->|No changes| E["any-changes: false"]

    C --> F["test-engine<br/>runs pytest"]
    D --> G["validate-workflows<br/>runs YAML check"]

    style A fill:#FFF9E6
    style F fill:#B3E5FC
    style G fill:#B3E5FC

Real-World Scenarios

Scenario 1: Fix Gateway Bug

Changes:
  - engine/services/gateway/main.py  # ← Only this changed

graph LR
    A["Push"] --> B["detect-changes"]
    B -->|gateway=true| C["build-gateway"]
    B -->|orchestrator=false| D["⏭️ skip orchestrator"]
    B -->|storage=false| E["⏭️ skip storage"]

    C --> F["deploy-engine<br/>updates gateway pod"]

    G["test-engine"] --> F

    F --> H["🟢 LIVE<br/>gateway running new code"]

    style C fill:#90EE90
    style H fill:#90EE90
    style D fill:#FFFFCC
    style E fill:#FFFFCC

Result: Build time ~2-3 min (vs. ~15 min for all services)

Scenario 2: Add New Workflow

Changes:
  - flows/workflows/new-workflow.yaml

graph LR
    A["Push"] --> B["detect-changes"]
    B -->|no engine changes| C["⏭️ skip all builds"]

    D["detect-test-changes"]
    D -->|flows=true| E["validate-workflows"]
    D -->|engine-code=false| F["⏭️ skip engine tests"]

    E --> G["Run workflow validation"]
    G --> H["YAML syntax check"]

    H --> I["Deploy workflows"]
    I --> J["🟢 New workflow available"]

    style C fill:#FFFFCC
    style F fill:#FFFFCC
    style E fill:#90EE90
    style J fill:#90EE90

Result: Test time ~30 sec (engine tests skipped)

Scenario 3: Update Documentation

Changes:
  - wiki/docs/index.md
  - wiki/mkdocs.yml

graph LR
    A["Push"] --> B["detect-changes"]
    B -->|no engine changes| C["⏭️ skip engine builds"]
    B -->|wiki=true| D["build-wiki"]

    E["detect-test-changes"]
    E -->|no code changes| F["⏭️ skip all tests"]

    D --> G["Build MkDocs container"]
    G --> H["deploy-wiki"]

    H --> I["🟢 Documentation updated"]

    style C fill:#FFFFCC
    style F fill:#FFFFCC
    style D fill:#90EE90
    style H fill:#90EE90
    style I fill:#90EE90

Result: Build + Deploy time ~2-3 min

Deployment Decision Tree

graph TD
    A["Build & Test<br/>Pipeline Starts"]

    A --> B["(1) detect-changes<br/>Analyze git diff"]
    B --> C["(2) Build Phase<br/>docker build push"]

    C --> D{All builds<br/>successful?}
    D -->|No| E["❌ FAILED<br/>Fix and retry"]
    D -->|Yes| F["(3) Test Phase<br/>pytest + validation"]

    F --> G{All tests<br/>pass?}
    G -->|No| E
    G -->|Yes| H["(4) Verification Gate<br/>verify-build-and-tests"]

    H --> I{Confirm both<br/>success?}
    I -->|No| E
    I -->|Yes| J["(5) Deploy Approved<br/>deploy-engine"]

    J --> K["(6) Deploy Wiki<br/>deploy-wiki<br/>parallel with engine"]

    K --> L["(7) Wait for Engine<br/>Ready?"]
    L -->|No| M["❌ Engine failed<br/>Investigate"]
    L -->|Yes| N["(8) Deploy Workflows<br/>deploy-workflows"]

    N --> O["(9) Health Checks<br/>Verify pods running"]

    O --> P{All pods<br/>healthy?}
    P -->|No| Q["⚠️ Warning<br/>Check logs"]
    P -->|Yes| R["🟢 LIVE<br/>All services ready"]

    style E fill:#FFB3B3
    style H fill:#FFE6B3,stroke:#FF8C00,stroke-width:2px
    style R fill:#90EE90

Performance Impact

Before Change Detection (Old)

pie title Build Time Distribution (All Services)
    "Gateway Build" : 2
    "Orchestrator Build" : 2
    "Transformers Build" : 2
    "Storage Build" : 2
    "Identity Build" : 2
    "Scheduler Build" : 2
    "Dashboard Build" : 2
    "Egress Services Build" : 2
    "Total overhead" : 1

Total: ~15-20 minutes for any change

After Change Detection (New)

pie title Build Time - Typical Gateway Fix Only
    "Gateway Build" : 2
    "Tests Run" : 1
    "Deploy" : 1
    "Skipped Services" : 16

Total: ~3-5 minutes (75% faster)

Pipeline Guardrails

No Direct Push-to-Deploy

graph LR
    A["❌ Old: Push"] -->|Direct| B["Deploy"]

    C["✅ New: Push"] --> D["Build"]
    D --> E["Test"]
    E --> F{Both OK?}

    F -->|No| G["❌ BLOCKED"]
    F -->|Yes| H["Deploy"]

    style B fill:#FFB3B3,stroke:#FF0000
    style G fill:#FFB3B3
    style H fill:#90EE90

Verification Gates

graph TD
    A["verify-build-and-tests"]
    A -->|Checks| B["Build Status: success?"]
    A -->|Checks| C["Test Status: success?"]
    A -->|Checks| D["No conflicts in staging?"]

    B --> E{All checks<br/>pass?}
    C --> E
    D --> E

    E -->|No| F["🛑 Deployment Blocked"]
    E -->|Yes| G["✅ Proceed to Deploy"]

    style A fill:#FFE6B3,stroke:#FF8C00,stroke-width:3px
    style F fill:#FFB3B3
    style G fill:#90EE90

Rollback Strategy

Manual rollback available via workflow_dispatch:

graph LR
    A["Manual Trigger<br/>workflow_dispatch"] --> B["Rollback Job"]

    B --> C["kubectl rollout undo<br/>all deployments"]

    C --> D["Revert to Previous<br/>Container Images"]

    D --> E["health-checks"]

    E --> F{Services<br/>healthy?}

    F -->|No| G["⚠️ Check logs"]
    F -->|Yes| H["✅ Rollback Complete<br/>Previous version live"]

    style A fill:#FFE6B3
    style H fill:#90EE90

Monitoring & Alerts

Post-deployment verification step checks:

graph LR
    A["After Deploy"] --> B["Get Pod Status"]

    B --> C{Ready<br/>Replicas?}

    C -->|Expected| D["✅ Healthy"]
    C -->|Fewer| E["⚠️ Rollout Issue"]

    D --> F["Verify Image Version<br/>commit SHA"]
    E --> G["Inspect pod events<br/>Check logs"]

    F --> H{Running<br/>new image?}
    H -->|Yes| I["🟢 Deployment Confirmed"]
    H -->|No| J["❌ Image Pull Issue"]

    style I fill:#90EE90
    style J fill:#FFB3B3

Summary

Feature Benefit
Smart Builds Skip unchanged services → 3-5x faster
Smart Tests Run only relevant validation → 50% faster
Verification Gate Prevent broken deploys → production stability
Change Detection Zero manual configuration → always accurate
Manual Rollback Quick recovery option → business continuity
Health Checks Post-deploy verification → confidence

→ Result: Fast, safe, automated deployment pipeline

Current Deployment Notes (2026)

  • Engine deploy now applies latest image tags for all runtime services during cluster rollout.
  • Deploy workflow ensures required auth secrets exist before rollout:
    • meshflows-identity-auth
    • meshflows-orchestrator-auth
  • Deploy workflow also removes stale ReplicaSets that still reference unresolved placeholder images (for example IMAGE_ORCHESTRATOR) to prevent stuck rollouts.
  • Wiki deploy runs as non-root on container port 8080 and the service still exposes port 80 via targetPort mapping.