Skip to content

Tuning, Limits, and Load Testing

This page provides practical guidance for performance tuning and resource limits in MeshFlows Engine, with a focus on keeping an existing (possibly small) cluster usable.

Goals

  • Keep the current cluster stable.
  • Prevent noisy-neighbor behavior with sensible limits.
  • Avoid over-reserving CPU/memory via aggressive requests.
  • Build an iterative path toward realistic load testing.

Core Principle: Requests vs Limits

  • requests decide scheduling. If requests are too high, pods may not schedule.
  • limits cap burst usage. If limits are too low, pods are throttled or OOM-killed.
  • For small clusters: keep requests conservative, set limits to protect the node.

Safe Starting Strategy for Small Clusters

  1. Keep existing requests low unless pods are consistently CPU-starved.
  2. Add or tune CPU limits to avoid unbounded contention.
  3. Observe throttling and latency before increasing requests.
  4. Increase requests only for components that are continuously busy.

Suggested Baseline (Small Cluster)

Use this as a starting point, then tune from metrics:

  • Gateway: request 50m, limit 200m
  • Orchestrator: request 50m, limit 1000m
  • Orchestrator queue subscriber: request 50m, limit 1000m
  • Transformers: request 50m, limit 500m
  • Scheduler: request 100m, limit 500m
  • Storage: request 50m, limit 250m

Notes:

  • The orchestrator can be bursty; a higher CPU limit can reduce latency spikes.
  • If the cluster is very constrained, reduce orchestrator limits first before touching requests.

Configuration Advice

Kubernetes

  • Keep resource settings in overlays per environment profile.
  • Prefer a dedicated profile (for example: small/dev/tst/prd) over editing base manifests.
  • Apply and verify:
kubectl apply -k deploy/k8s/overlays/<profile>
kubectl get pods -n <namespace>

Compose (Local)

  • Keep default behavior predictable via environment variables.
  • For JavaScript steps, use SES execution and keep inline disabled.

Recommended JS-related settings:

  • ORCH_SES_ENABLED=true
  • ORCH_SES_URL=http://ses:8080
  • ORCH_SES_FALLBACK_TO_INLINE_JS=false
  • ORCH_ENABLE_INLINE_JS=false

What to Monitor During Tuning

  • Pod restarts and OOM kills.
  • CPU throttling (container_cpu_cfs_throttled_seconds_total).
  • P95/P99 API latency on gateway and orchestrator.
  • Queue depth and processing lag.
  • Error rate (4xx/5xx) and timeout rate.

Quick checks:

kubectl top pods -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl describe pod <pod-name> -n <namespace>
  1. Capture baseline for 24-48 hours.
  2. Adjust one variable at a time (for example: orchestrator CPU limit).
  3. Re-run representative traffic.
  4. Compare latency, throttling, and error rates.
  5. Keep the change only if metrics improve without regressions.

Load Testing Plan (Phased)

Phase 1: Smoke Load

  • Goal: verify basic stability under light concurrent load.
  • Duration: 5-10 minutes.
  • Traffic: one representative workflow, low concurrency.

Phase 2: Sustained Load

  • Goal: understand steady-state behavior and throttling.
  • Duration: 30-60 minutes.
  • Traffic: realistic mix (HTTP and scheduled triggers).

Phase 3: Stress and Breakpoint

  • Goal: find saturation points and failure modes.
  • Increase concurrency gradually until SLO breach.
  • Record breakpoint, recovery time, and bottlenecks.

Minimal k6 Example

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 10,
  duration: '5m',
};

export default function () {
  const body = JSON.stringify({ xml: '<root/>' });
  const res = http.post('http://gateway:8080/v1/run/minimal', body, {
    headers: { 'Content-Type': 'application/json' },
  });
  check(res, {
    'status is 200': (r) => r.status === 200,
  });
  sleep(1);
}

Rollback Guidance

  • Keep previous manifests and image tags available.
  • If regressions appear, rollback immediately:
kubectl rollout undo deployment/orchestrator -n <namespace>
kubectl rollout undo deployment/gateway -n <namespace>
  • Revert only the last tuning step, then re-test.

Next Steps

  • Create a dedicated load-test workflow set in flows/workflows/.
  • Add automated k6 runs in CI for smoke and sustained profiles.
  • Track SLOs (latency, error rate, throughput) per release.

Repository References

  • k6 profiles: engine/tests/loadtest/profiles/
  • Loadtest guide: engine/tests/loadtest/README.md
  • Small cluster checklist: Small Cluster Tuning Checklist