Observability¶
MeshFlows observability combines service health, request traces, trigger events, and platform telemetry.
Observability Signals¶
- Health and readiness endpoints per service (
/healthz,/readyz). - Workflow execution traces (
/api/traces*). - Trigger audit events (
/api/trigger-events*). - OpenTelemetry traces exported to collector/backends.
- Service logs from runtime containers.
Trace Correlation Model¶
X-Request-IDis used for end-to-end request correlation.- Gateway forwards correlation context to Orchestrator.
- Orchestrator emits trace documents per run and per-step artifacts.
Suggested Stack¶
- OpenTelemetry Collector for ingestion and routing.
- Tempo for distributed tracing.
- Loki (or equivalent) for log aggregation.
- Optional dashboards/alerts in your preferred monitoring tool.
Common Diagnostic Workflow¶
- Confirm service readiness with
/readyz. - Locate failing run via
/api/traces?status=failed. - Inspect step-level data through trace step endpoints.
- Check trigger event stream for dropped/failed trigger attempts.
- Correlate with infrastructure logs and broker metrics.
Best Practices¶
- Capture enough context to reproduce failures without logging secrets.
- Retain traces long enough for incident review windows.
- Alert on repeated trigger failures and rising latency percentiles.
- Use consistent request identifiers across all integrations.
Business events (thin convention)¶
Technical traces do not replace business milestones (order placed, invoice
posted). MeshFlows does not ship BAM / Invictus. Workflows emit a greppable
line from the existing type: log step.
- Convention + fieldset: Business events
- Demo:
business_event_demo(POST /v1/api/examples/business-event-demo) - Loki:
{component="orchestrator"} |= "meshflows.business_event" - Metrics: LogQL
count_over_time(... |= "meshflows.business_event" [5m])— not a Prometheus counter or Grafana pack
Operations milestones (type: milestone / context.milestone) are a
different surface: they roll up onto the runtime-root row in the Operations
list, they are not Loki events.