Skip to content

Small Cluster Tuning Checklist

Use this checklist when running MeshFlows on a constrained cluster and you want stability first.

Scope

  • Keep the current cluster usable.
  • Avoid unschedulable pods.
  • Limit noisy-neighbor effects.
  • Tune gradually based on observed metrics.

1. Baseline (Before Any Change)

Run and record:

kubectl get pods -n <namespace> -o wide
kubectl top pods -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Capture at least:

  • Pod restarts (per deployment).
  • CPU and memory usage per pod.
  • Any OOMKill or failed scheduling events.

2. Resource Rules

  • Keep requests conservative (scheduling safety).
  • Use limits to cap bursts.
  • Change one service at a time.
  • Observe for at least 30-60 minutes after each change.

3. Decision Thresholds (Practical)

Use these as default guardrails:

  • CPU throttling:
  • If throttling is consistently high for orchestrator or gateway, increase CPU limit first.
  • Error rate:
  • If 5xx or timeout rate exceeds 2% under normal traffic, pause rollout and inspect bottlenecks.
  • Latency:
  • If p95 latency degrades more than 30% versus baseline, rollback the last tuning change.
  • Restarts:
  • Any repeated OOM restarts in a 30-minute window means memory limits are too tight.
  • Scheduling:
  • If pods become Pending due to insufficient CPU/memory, reduce requests or add cluster capacity.

4. Suggested Order of Tuning

  1. Orchestrator CPU limit.
  2. Gateway CPU limit.
  3. Transformers CPU/memory.
  4. Storage and RabbitMQ after app-layer tuning.
  5. Only then increase requests where needed.

5. Rollback Triggers

Rollback immediately when one of these occurs:

  • New Pending pods after change.
  • Sustained timeout spikes.
  • Error rate > 5% for more than 10 minutes.
  • Repeated OOM kills.

Rollback commands:

kubectl rollout undo deployment/orchestrator -n <namespace>
kubectl rollout undo deployment/gateway -n <namespace>

6. Weekly Hygiene

  • Re-check kubectl top trends.
  • Revisit limits after feature changes.
  • Validate JS inline allowlist is still minimal.
  • Keep loadtest smoke profile in CI or release checks.