Observability

MeshFlows observability combines service health, request traces, trigger events, and platform telemetry.

Observability Signals

• Health and readiness endpoints per service (/healthz, /readyz).
• Workflow execution traces (/api/traces*).
• Trigger audit events (/api/trigger-events*).
• OpenTelemetry traces exported to collector/backends.
• Service logs from runtime containers.

Trace Correlation Model

• X-Request-ID is used for end-to-end request correlation.
• Gateway forwards correlation context to Orchestrator.
• Orchestrator emits trace documents per run and per-step artifacts.

Suggested Stack

• OpenTelemetry Collector for ingestion and routing.
• Tempo/Tempo for distributed tracing.
• Loki (or equivalent) for log aggregation.
• Optional dashboards/alerts in your preferred monitoring tool.

Common Diagnostic Workflow

1. Confirm service readiness with /readyz.
2. Locate failing run via /api/traces?status=failed.
3. Inspect step-level data through trace step endpoints.
4. Check trigger event stream for dropped/failed trigger attempts.
5. Correlate with infrastructure logs and broker metrics.

Best Practices

• Capture enough context to reproduce failures without logging secrets.
• Retain traces long enough for incident review windows.
• Alert on repeated trigger failures and rising latency percentiles.
• Use consistent request identifiers across all integrations.