Skip to content

MeshFlows Capability Registry v0.1.x: Implementation Plan and Feasibility Study

Document Version: 1.0
Last Updated: April 2026
Status: Ready for Engineering Review
Target Wiki Path: wiki/docs/engine/reference/16-capability-registry-implementation-plan-and-feasibility.md

Executive Summary

This document provides a comprehensive roadmap for implementing the MeshFlows Capability Registry v0.1.x EPIC—a modular, self-healing service discovery and capability management system. The implementation spans 15 weeks across five phases (v0.1.1 through v0.1.5), with clear architectural decisions, phased deliverables, dashboard integration points, and operational runbooks.

Key Outcomes: - Unified capability discovery for Gateway and Orchestrator - Automatic service re-registration with backoff and heartbeat logic - Step-level versioning and provider validation in workflows - Dashboard console for capability validation, route mapping, and incident tracking - Operator runbooks for common failure scenarios

Part 1: Architecture Decision Records (ADRs)

These eight ADRs establish the technical foundation and must be reviewed and approved before v0.1.1 begins. Each ADR captures decision rationale, alternatives considered, and implementation implications.

ADR-001: Registry Storage Model (In-Memory vs. Persistent Snapshot)

Decision: Adopt in-memory storage with optional snapshot persistence for MVP.

Rationale: - Service instances are ephemeral by design (tied to pod lifecycle). - Re-registration on registry restart is self-healing; no data loss risk. - In-memory avoids database dependencies and operational complexity for v0.1.x. - Optional snapshot support (daily or on-demand) enables disaster recovery without blocking MVP.

Alternatives Considered: 1. Full persistent database (PostgreSQL, etcd): adds operational overhead, not required for v0.1.x. 2. Pure in-memory without snapshot: acceptable but limits observability of registry state during incidents.

Implementation Details: - Snapshot format: JSON file with timestamp, service instance count, capability count. - Snapshot location: optional PVC mount at /registry-snapshots/ (e.g., orchestrator-flows-data). - Snapshot triggers: cron job (daily) or manual operator command via Kubernetes Job. - No restoration on restart; snapshot is observational only in MVP.

Go/No-Go Criteria: - Registry handles ≥500 service instances without memory concerns. - Instance eviction is deterministic and observed in tests. - Snapshot restore time (if implemented in v0.1.2) is <5 seconds.

ADR-002: Re-Registration Algorithm (Backoff, Heartbeat Thresholds, Generation Semantics)

Decision: Use exponential backoff with jitter for re-registration, generation tracking for lease validation, and configurable heartbeat thresholds.

Rationale: - Exponential backoff prevents thundering herd on registry restart. - Jitter ensures spread across time, reducing spike load. - Generation tags prevent stale re-registration cycles in eventual consistency scenarios. - Clear heartbeat thresholds (e.g., 3 consecutive failures) provide deterministic reconnect triggers.

Recommended Settings (Configurable): - Initial backoff: 100 ms - Max backoff: 5 minutes - Backoff multiplier: 2 - Jitter: random 0–100 ms - Heartbeat interval: 30 seconds (provider-side) - Lease TTL: 90 seconds (registry-side) - Heartbeat failure threshold: 3 consecutive failures triggers re-register - Generation assigned by registry on successful register; provider echoes it in heartbeat.

Implementation Details: - Provider SDK encapsulates backoff logic; accessible via register_with_backoff() helper. - Registry compares heartbeat generation against stored generation; mismatch triggers 410 Gone response. - Provider re-register on: 404, 410, generation mismatch, or N consecutive heartbeat failures. - Backoff state stored per service instance in provider; reset on successful heartbeat.

Go/No-Go Criteria: - 100 simulated providers re-registering simultaneously produce <50 QPS spike. - Median re-registration latency: <100 ms. - Generation mismatches logged and monitored; 0 spurious re-registers in 1-hour smoke test.

ADR-003: Discovery Client Caching (TTL, Last-Known-Good, Invalidation Triggers)

Decision: Implement discovery client with configurable short-lived cache (default 30s), persistent last-known-good snapshot, and event-driven invalidation.

Rationale: - Short TTL (30s) reduces stale route serving while limiting registry query load. - Last-known-good snapshot ensures consumers continue operating if registry is temporarily unavailable. - Event-driven invalidation (via webhook or polling for changes) enables faster updates without full refresh. - Separate cache and snapshot prevents losing good routes if cache is cleared.

Cache Behavior: - Cache TTL: 30 seconds (default, configurable per consumer: Gateway, Orchestrator, Dashboard). - Last-known-good: persisted in-memory; updated on successful cache refresh. - Invalidation triggers: - Manual GET /v1/discovery/invalidate call (admin/operator action). - Polling-based change detection (hash of discovery response). - Optional webhook from registry (future v0.1.2+ enhancement).

Implementation Details: - Consumer makes GET /v1/discovery/capabilities?min_version=X with caching layer. - On cache miss or expiry, fetch fresh response from registry; compare hash against stored. - If hash matches, skip last-known-good update; reuse in-memory reference. - If hash differs, update last-known-good and cache. - If registry unreachable, serve last-known-good with X-Cache-Source: snapshot header.

Go/No-Go Criteria: - Cache hit rate ≥95% in steady state with 30s TTL. - Last-known-good enables 100% uptime on simulated registry outage (10 min) for Gateway/Orchestrator. - Invalidation via manual call completes within 100 ms.

ADR-004: Layered Validation Policy (Provider Calls, Timeouts, Strict vs. Non-Strict Modes)

Decision: Three-layer validation model (contract, provider, runtime); strict mode blocks on contract/provider failure; non-strict mode allows with warnings.

Rationale: - Contract layer provides deterministic, fast feedback without external calls. - Provider layer adds semantic validation for protocol-specific constraints. - Runtime layer is the final safety net. - Strict/non-strict modes allow gradual migration of workflows without full rewrite.

Layer Definitions:

  1. Contract Layer (Orchestrator, no external calls):
  2. Step shape and required fields
  3. Capability ID, provider, and version syntax

  4. Cross-field constraints in workflow YAML

  5. Provider Layer (Optional, calls provider service):

  6. Only if provider metadata indicates validate_supported: true
  7. Timeout: 5 seconds (default, configurable)
  8. Endpoint: POST /v1/capabilities/{capability_id}/validate

  9. Provider returns {valid: bool, diagnostics: []}

  10. Runtime Layer (Always enforced):

  11. Destination/resource existence
  12. Payload constraints
  13. Auth and connection validation at execution time

Strict vs. Non-Strict: - Strict mode (default for new workflows): contract failure → block; provider timeout/failure → block - Non-Strict mode (legacy compatibility): contract failure → block; provider timeout → warning only - Bypass flag: WORKFLOW_VALIDATE_PROVIDER_STRICT=false (environment or per-workflow setting)

Implementation Details: - Validation triggered on: workflow upload, publish, dashboard reload, Orchestrator startup. - Validation result persisted in workflow metadata: {contract_status, provider_status, diagnostics[]} - Dashboard renders inline diagnostics per field.

Go/No-Go Criteria: - Contract validation latency: <10 ms per step. - Provider validation latency (with timeout): <5 seconds per capability. - Strict mode blocks 100% of invalid workflows; non-strict mode allows with warning.

ADR-005: Step Versioning & Capability Version Constraints

Decision: Use semantic versioning for step contracts and capabilities; support min/max constraints in workflow YAML.

Rationale: - Semantic versioning provides clear, understood compatibility signals. - Constraint ranges (e.g., >=1.0 <2.0) allow workflows to express intent without hardcoding exact versions. - Step version (e.g., "1.1") tracks workflow's step contract expectation separately from provider capability version. - Enables gradual provider evolution without breaking existing workflows.

Version Fields in Workflow YAML: - step_version: version of the generic service-backed step contract (e.g., "1.1") - capability.version (or capability_version): constraint on provider capability version (e.g., ">=1.0 <2.0")

Compatibility Rules: - If step_version omitted: default to "1.0" - If capability_version omitted: default to >=0.0.0 (any version acceptable) - Orchestrator performs version compatibility check at workflow load time. - Failed compatibility check: validation error with clear message (e.g., "Workflow requires JMS messaging.publish >=1.0 <2.0; registry has 2.1.0 only").

Implementation Details: - Version parsing via semver library (Python: packaging.version; Go: semver package). - Version matching logic: check available capabilities against constraint. - If multiple providers satisfy constraint, selection is deterministic (e.g., by registration order or priority hint).

Go/No-Go Criteria: - Version parsing handles all semver edge cases (pre-release, build metadata). - Constraint matching latency: <5 ms per step. - Backward compatibility: workflows without versions default safely to v1.0.

ADR-006: Service Identity & Registry API Authentication (mTLS vs. JWT)

Decision: Use mTLS for registry API authentication in v0.1.1; JWT support as optional v0.1.2 feature.

Rationale: - mTLS is cluster-standard (already deployed for inter-service communication). - Simple certificate-based identity avoids issuer/key management complexity. - Registry can verify client certificate CN against known service names. - JWT adds flexibility for future cross-cluster or external provider scenarios (v0.1.2+).

Implementation Details: - Registry APIs (/v1/services/register, /v1/services/{id}/heartbeat) require TLS client certificate. - Certificate CN must match service identity (e.g., cn=egress-jms). - Registry maintains whitelist of allowed service names. - Discovery APIs (/v1/discovery/) allow authenticated service consumers (Gateway, Orchestrator) via certificate or JWT bearer token.

Kubernetes Deployment Implications: - Provider pods mount secrets: service-cert and service-key. - Orchestrator/Gateway pods mount registry-ca.crt for server verification. - Registry pod mounts: registry-cert, registry-key, registry-ca.crt, and client-ca.crt for client verification. - Network policies restrict registry to orchestrator, gateway, and provider namespaces.

Go/No-Go Criteria: - mTLS handshake latency: <50 ms. - Certificate validation correct for all service identities. - Unauthenticated requests to protected endpoints are rejected.

ADR-007: RBAC Model for Capability Registration

Decision: Role-based access control via service identity; no additional ACL database in v0.1.1.

Rationale: - Service identity (mTLS CN) serves as role identity. - Registry maintains static whitelist of allowed registrations per service role. - Simplifies v0.1.1; ACL database can be added in v0.1.2 if needed.

Role Model: - messaging-provider: can register messaging. capabilities - transform-provider: can register transform. capabilities - webhook-provider: can register webhook.* capabilities - admin: can register any capability, manage registry

Authorization Logic: - On POST /v1/services/register, registry extracts client CN from mTLS certificate. - Registry checks CN against whitelist: {service_role: [allowed_capability_ids]} - If match, register accepted; otherwise, 403 Forbidden with clear error.

Implementation Details: - Whitelist stored in registry service config: config/registry-rbac.yaml - RBAC rules are static in v0.1.1; operator updates config and restarts registry (or implements hot-reload in v0.1.2). - Audit trail: log all register/heartbeat/discovery requests with service identity.

Go/No-Go Criteria: - Unauthorized registration attempts are rejected with 403. - Authorized registrations succeed without latency penalty. - Audit logs capture all requests with timestamps and identities.

ADR-008: Dashboard Validation Console UI & Capabilities Catalog

Decision: Implement two primary dashboard views: (1) Capabilities Catalog (read-only discovery), (2) Validation Console (workflow validation results & re-validation trigger).

Rationale: - Catalog provides operator visibility into registered capabilities and providers. - Validation Console helps workflow authors understand validation results and iterate on step definitions. - Separates discovery/monitoring concerns (Catalog) from authoring/validation concerns (Console).

Capabilities Catalog View: - Lists all active service instances and their capabilities. - Displays: service name, version, instance ID, endpoint, capabilities, lease expiry. - Filter by: provider, capability ID, service name. - Action: click capability to view full metadata and contract schema.

Validation Console View: - Displays validation results for uploaded/published workflows. - Shows: contract status (pass/fail), provider status (pass/warning/fail), per-step diagnostics. - Inline field-level errors/warnings linked to workflow YAML line numbers. - Action: trigger re-validation manually (e.g., after provider restart).

Interactive Route Map (v0.1.4+): - For ≤50 routes: interactive graph showing route → capability → provider → service instance relationships. - For >50 routes: searchable list fallback with filter options. - Click node to see details (capability metadata, provider health, trigger count).

Go/No-Go Criteria: - Catalog loads ≤1 second with 1000+ capabilities. - Validation Console displays results accurately matching Orchestrator validation logic. - Route Map (v0.1.4) renders graph in ≤2 seconds for 50 routes.

Part 2: Phased Implementation Roadmap (v0.1.1 through v0.1.5)

Phase Overview

Phase Version Duration Focus Key Deliverables Dashboard Impact
1 v0.1.1 Weeks 1–3 Registry foundation, Orchestrator discovery Registry service, provider re-registration, Orchestrator client Workflows tab: provider health
2 v0.1.2 Weeks 4–6 Gateway integration, route merging Gateway discovery client, merged route serving, last-known-good cache Routes tab: discovered routes vs. static routes
3 v0.1.3 Weeks 7–9 Dashboard UX for validation & catalog Capabilities Catalog view, Validation Console, per-step diagnostics New tabs: Capabilities, Validation, Diagnostics
4 v0.1.4 Weeks 10–12 Operational UX & incident tracking Route Map, Trigger Health, Incident Actions, health timeline New tabs: Route Map, Incidents; enhanced health view
5 v0.1.5 Weeks 13–15 Observability & runbooks Grafana dashboards, alert rules, runbooks for common incidents Metrics view, logs view, incident playbooks

Phase v0.1.1: Registry Foundation & Orchestrator Discovery (Weeks 1–3)

Objectives: - Deploy Registry service to Kubernetes. - Implement provider re-registration logic in engine services. - Build Orchestrator discovery client and generic service_call step support. - Establish baseline observability (metrics, logs).

Key Deliverables:

  1. Registry Service
  2. RESTful API: POST /v1/services/register, PUT /v1/services/{id}/heartbeat, DELETE /v1/services/{id}
  3. Discovery API: GET /v1/discovery/capabilities, GET /v1/discovery/health
  4. Instance storage: in-memory with optional snapshot support
  5. Lease management: evict stale instances after TTL expiry

  6. mTLS authentication for all endpoints

  7. Provider Re-Registration SDK

  8. Python/Go helper library for service registration
  9. Automatic backoff + jitter for failed registration
  10. Heartbeat loop with failure detection

  11. Generation tracking and re-registration triggers

  12. Orchestrator Discovery Client

  13. Lightweight cache (30s TTL) wrapping registry discovery API
  14. Last-known-good snapshot persisted in memory
  15. Capability resolution logic for generic steps

  16. Fallback to static configuration if registry unavailable

  17. Workflow YAML Enhancements

  18. Support step_version and capability.capability_version fields
  19. Backward compatibility: omitted versions default safely to v1.0

  20. Validation error messages for unsupported step versions

  21. Observability

  22. Registry metrics: active instances, capabilities, register/heartbeat success/failure counts
  23. Provider metrics: registry re-registration attempts/successes, heartbeat latency
  24. Structured logs: all register/heartbeat events with service identity and timestamp

Exit Criteria: - Registry service passes integration tests (10+ providers registering/re-registering). - Orchestrator resolves generic service_call steps to registered capabilities. - Provider re-registration triggers correctly on simulated registry unavailability (5+ minute outage). - Metrics and logs are properly exported to observability backend. - Zero regressions in existing workflow execution paths.

Owner Assignment: Platform Engineering (Registry), Core Engine Team (Orchestrator Integration)

Dashboard Impact: Workflows tab displays provider health status (connected/disconnected) and last heartbeat timestamp.

Go/No-Go Gate: - Registry handles ≥100 concurrent service registrations. - P99 registration latency <100 ms. - Orchestrator fallback to static config works correctly with registry unavailable.

Phase v0.1.2: Gateway Integration & Route Discovery (Weeks 4–6)

Objectives: - Implement Gateway discovery client with static/dynamic route merging. - Build route conflict resolution algorithm. - Establish last-known-good route cache for availability. - Enable public route aliases for new service variants.

Key Deliverables:

  1. Gateway Discovery Client
  2. Fetch discovered routes from registry: GET /v1/discovery/routes
  3. Parse route metadata: path, methods, service endpoints

  4. 30s TTL cache with last-known-good snapshot

  5. Route Merge Algorithm

  6. Deterministic priority: static routes > dynamic with priority > other dynamic
  7. Conflict resolution: explicit static route wins; if multiple dynamic, use priority hint or registration order

  8. Route validation: ensure no circular references or invalid configurations

  9. Last-Known-Good Route Cache

  10. Persist successful route set in memory
  11. On registry unavailability, serve last-known-good with X-Cache-Source: snapshot header

  12. Operator can manually invalidate cache via DELETE /v1/discovery/invalidate endpoint

  13. Dynamic Route Aliases

  14. Registry returns optional public_paths per capability
  15. Example: JMS message publish capability exposes alias route /api/messaging/publish
  16. Routes generated dynamically; tied to capability lifecycle

Exit Criteria: - Gateway correctly merges static and dynamic routes without conflicts. - Route serving continues uninterrupted during simulated registry outage (10+ minutes). - Public aliases for new service variants are automatically available within 1 minute of registration. - Gateway metrics show route hit distribution across static and dynamic routes. - Zero request routing failures due to merged routes.

Owner Assignment: Gateway Team

Dashboard Impact: Routes tab distinguishes static routes from discovered dynamic routes; displays route availability status.

Go/No-Go Gate: - Merge algorithm handles ≥100 routes without latency increase. - Last-known-good enables 100% uptime on registry outage. - Public alias routes are served immediately after provider registration.

Phase v0.1.3: Dashboard Validation Console & Capabilities Catalog UI (Weeks 7–9)

Objectives: - Build Capabilities Catalog view in Dashboard (read-only). - Implement Validation Console for workflow validation results. - Add per-step diagnostics with field-level error reporting. - Integrate validation UI with Orchestrator validation APIs.

Key Deliverables:

  1. Capabilities Catalog View
  2. List all active service instances with capabilities
  3. Display: service name, version, instance ID, endpoint, lease expiry
  4. Filters: by provider, capability ID, service name
  5. Detail modal: click capability to view full metadata and JSON schema
  6. Health status per instance: active/expiring soon/stale

  7. Refresh button to manually invalidate discovery cache

  8. Validation Console View

  9. Display validation results for uploaded/published workflows
  10. Tabs: by workflow or by capability
  11. Per-workflow view: contract status, provider status, overall validation summary
  12. Per-step breakdown: step ID, step type, capability, validation status, diagnostics
  13. Inline error/warning messages with YAML line number references
  14. Manual re-validation trigger with progress indicator

  15. Export validation report as JSON

  16. Field-Level Diagnostics

  17. Integrate validation diagnostics with workflow YAML editor
  18. Gutter icons: red (error), yellow (warning), blue (info)
  19. Hover tooltip: diagnostic message and code
  20. Line highlighting in editor

  21. Breadcrumb navigation: workflow → step → field → diagnostic

  22. Provider Validation Integration

  23. Orchestrator calls provider validation endpoint during workflow upload/publish
  24. Results aggregated: contract layer + provider layer + diagnostics
  25. Timeout handling: if provider unreachable, show status provider_skipped with warning

Exit Criteria: - Capabilities Catalog loads ≤1 second with 500+ capabilities. - Validation Console displays results matching Orchestrator validation logic. - Field-level diagnostics are accurate and helpful (survey feedback ≥4/5). - Validation re-trigger latency <5 seconds. - Zero UI crashes or rendering issues with large workflow definitions (100+ steps).

Owner Assignment: Dashboard Team

Dashboard Impact: Three new tabs: Capabilities (Catalog), Validation, Diagnostics. Workflow upload/publish flow includes validation check inline.

Go/No-Go Gate: - Validation Console accuracy: 100% match with Orchestrator validation logic. - Dashboard load with 1000+ capabilities: ≤2 seconds. - Field-level diagnostics correctly map to workflow YAML lines.

Phase v0.1.4: Operational UX—Route Map, Trigger Health, Incidents (Weeks 10–12)

Objectives: - Build interactive Route Map for visual route-to-capability mapping. - Implement Trigger Health view for endpoint health and uptime tracking. - Create Incident Actions UI for operator-driven incident response. - Add health timeline and metrics drill-down.

Key Deliverables:

  1. Route Map View
  2. For ≤50 routes: Interactive force-directed graph
  3. For >50 routes: Searchable list fallback with filters
  4. Click node: show details (metadata, metrics, recent errors)
  5. Filter by: provider, service, route path, status

  6. Zoom and pan controls; minimap for navigation

  7. Trigger Health Dashboard

  8. Table view: trigger name, last execution, success rate, P95 latency
  9. Timeline: health status over past 24 hours
  10. Drill-down: click trigger to show recent execution history

  11. Health indicators: Healthy (≥95% success), Degraded (90–95%), Unhealthy (<90%)

  12. Incident Actions UI

  13. Show recent incidents (last 24 hours) tied to triggers/routes
  14. Action buttons: Acknowledge, Investigate, Re-run trigger, Scale provider, View logs
  15. Status: open, acknowledged, resolved, ignored

  16. Incident severity: critical, warning, info

  17. Health Timeline & Metrics Drill-Down

  18. 24-hour health status graph per route (success/error rate)
  19. Click time point: show relevant logs and metrics for that window
  20. Comparison: compare health across multiple routes/providers
  21. Export: download health report as CSV

Exit Criteria: - Route Map renders ≤2 seconds for 50 routes. - Trigger Health data updates every 30 seconds (no refresh lag). - Incident Actions execute correctly (ack, investigate, re-run, scale, view logs). - Health timeline accurate to within 1 minute. - Zero UI crashes with 1000+ incidents.

Owner Assignment: Dashboard Team, Platform Engineering (metrics/logs integration)

Dashboard Impact: Three new tabs: Route Map, Trigger Health, Incidents. Home dashboard shows incident count and health summary.

Go/No-Go Gate: - Route Map graph renders in ≤2s for 50 routes; list fallback for >50 routes. - Trigger Health data freshness: ≤30 second latency from event to UI. - Incident Actions execute within 5 seconds.

Phase v0.1.5: Observability, Runbooks & Hardening (Weeks 13–15)

Objectives: - Build comprehensive Grafana dashboards for Registry and Provider metrics. - Define alert rules for common failure scenarios. - Document operator runbooks for incident response. - Add provider health checks and proactive monitoring.

Key Deliverables:

  1. Grafana Dashboards
  2. Registry Health Dashboard (active instances, heartbeat latency, re-registration rate, evictions)
  3. Provider Health Dashboard (per-provider metrics, validation latency, discovery cache hit rate)

  4. Route & Trigger Dashboard (route availability, trigger execution success, error distribution)

  5. Alert Rules (Prometheus)

  6. Registry availability: alert if no heartbeat responses for 2 minutes
  7. Provider connection: alert if re-registration rate >5/minute
  8. Discovery cache staleness: alert if last-known-good is >1 hour old
  9. Validation latency: alert if P95 validation time >10 seconds

  10. Route unavailability: alert if route unreachable for >5 minutes

  11. Operator Runbooks

  12. Registry service crashes: restart registry pod, verify provider re-registration completes
  13. Provider repeated re-registration: check provider logs, verify registry connectivity
  14. Discovery cache staleness: manually invalidate cache, verify registry health
  15. Validation timeouts: increase timeout threshold, scale provider instances
  16. Route conflicts during merge: inspect route metadata, manually override priority

  17. Incident spike: acknowledge all incidents, investigate registry metrics, consider provider scaling

  18. Provider Health Checks

  19. Periodic availability checks: registry pings provider health endpoint
  20. Response time SLA monitoring: alert if provider response time > threshold
  21. Capability contract validation: registry periodically re-validates provider's capabilities

  22. Lease renewal proactivity: alert if provider fails to heartbeat 2+ times consecutively

  23. Documentation

  24. Architecture decision rationale (link to ADRs)
  25. Registry API reference with curl examples
  26. Provider SDK tutorial (Python and Go)
  27. Troubleshooting guide for common issues
  28. Upgrade path from v0.1.1 to v0.1.5

Exit Criteria: - All dashboards render correctly with realistic data. - Alert rules trigger appropriately on simulated failures. - Runbooks provide clear steps to resolve each scenario in ≤10 minutes. - Documentation is complete and tested (>90% coverage of APIs and SDKs). - No critical gaps in observability (metrics exist for all critical paths).

Owner Assignment: Platform Engineering (Observability), Documentation Team

Dashboard Impact: Metrics view with drill-down to Grafana dashboards; Logs view with structured log search; Runbooks tab with incident playbooks.

Go/No-Go Gate: - All dashboards render without errors. - Alert rules have <1% false positive rate on 24-hour smoke test. - Runbooks successfully guide operator resolution in <10 minutes each.

Part 3: Architecture Questions for Decision-Making

Q1: Per-Consumer Discovery Cache TTL

Recommendation: Option B (configurable per consumer)

Rationale: Different consumers have different staleness tolerances. Gateway can tolerate slightly stale routes if it reduces registry load. Orchestrator may need fresher discovery for validation. Dashboard can use longer TTL since it's interactive. Configurability allows tuning without code changes.

Decision: Implement per-consumer TTL configuration via environment variables or ConfigMap.

Q2: Provider Validation Blocking Strategy

Recommendation: Option A (synchronous with reasonable timeout)

Rationale: Users expect immediate feedback during workflow authoring. Timeout (5-10s) prevents indefinite blocking. Strict/non-strict modes allow flexibility for legacy workflows.

Decision: Implement synchronous with 5s timeout; non-strict mode allows warning-only fallback.

Q3: Provider Instance Selection

Recommendation: Option B for v0.1.1, migrate to Option D in v0.1.2

Rationale: Round-robin and load-aware add complexity not needed for v0.1.1. First available is simple and works. Priority hint allows provider to specify preference.

Decision: Implement first-available in v0.1.1; add priority field to capability metadata for v0.1.2.

Q4: Heartbeat Interval & Lease TTL

Recommendation: Option B (30s heartbeat, 90s TTL)

Rationale: 30s heartbeat provides reasonable freshness (provider down detected in ~90s accounting for failure threshold). 90s TTL prevents short-lived instances from being evicted prematurely.

Decision: Set defaults to 30s heartbeat, 90s TTL; make configurable per deployment.

Q5: Step Versioning Requirement

Recommendation: Option B (optional, backward compatible)

Rationale: Backward compatibility is critical. Default to "1.0" for workflows without explicit versions. Encourages adoption without forcing migration.

Decision: Implement optional with safe defaults; document versioning best practices.

Q6: Gateway Route Conflict Resolution

Recommendation: Option B (automatic with logging)

Rationale: Automatic resolution avoids manual bottlenecks. Deterministic priority ensures predictable behavior. Logging provides visibility for operators.

Decision: Implement automatic resolution with detailed conflict logging; add dashboard alert for high conflict rate.

Q7: Observability Baseline

Recommendation: Option C (phased)

Rationale: v0.1.1 focus is core functionality. Comprehensive observability can follow in v0.1.3+ without blocking MVP.

Decision: Implement structured logging + basic Prometheus metrics in v0.1.1; add traces and dashboards in v0.1.5.

Part 4: Risk Assessment & Mitigation

Risk 1: Registry Becomes Single Point of Failure

  • Mitigation: Last-known-good caching, 2+ replicas, automatic failover, monitoring
  • Owner: Platform Engineering

Risk 2: Provider Thundering Herd on Registry Restart

  • Mitigation: Exponential backoff + jitter, test with 100 concurrent, rate limiting
  • Owner: Platform Engineering + Framework Team

Risk 3: Validation Latency Impact on Workflow Authoring

  • Mitigation: 5s timeout, non-strict mode, progressive validation, provider scaling
  • Owner: Dashboard Team + Orchestrator Team

Risk 4: Discovery Cache Staleness Causes Route Serving Errors

  • Mitigation: Last-known-good snapshot, staleness monitoring, manual invalidation, recovery testing
  • Owner: Gateway Team

Risk 5: mTLS Certificate Expiry Breaks Provider Communication

  • Mitigation: Certificate expiry monitoring, cert-manager automation, rotation testing
  • Owner: Platform Engineering + Security Team

Risk 6: Step Versioning Confusion

  • Mitigation: Documentation with examples, dashboard recommendations, helpful error messages
  • Owner: Dashboard Team + Documentation Team

Part 5: Timeline & Ownership

Phased Delivery

Phase Version Duration Owner
1 v0.1.1 3 weeks Platform Eng + Core Engine
2 v0.1.2 3 weeks Gateway Team
3 v0.1.3 3 weeks Dashboard Team
4 v0.1.4 3 weeks Dashboard Team + Platform Eng
5 v0.1.5 3 weeks Platform Eng + Documentation

Total Effort: ~64 engineer-weeks

Part 6: Scope Isolation RFC (Informational Track)

Note: This section documents an exploratory track for future consideration. It is not blocking v0.1.1 delivery.

Problem

Current loop and parallel step implementations expose parent context to child iterations, allowing unintended mutations. Long-running workflows can accumulate state pollution.

Proposed Solution

  1. Loop Scope Isolation: Each loop iteration runs with isolated context snapshot; explicit export_context_keys declares which variables propagate to parent.
  2. Parallel Scope Isolation: Each parallel branch runs with isolated context snapshot; outputs_map specifies aggregation strategy.

Implementation Timeline

  • v0.1.2: RFC proposal and community feedback
  • v0.1.3+: Implement scope isolation as opt-in feature flag
  • v0.2.0: Make scope isolation default

Part 7: Success Metrics & Post-Launch Evaluation

Phase Completion Metrics

Phase Metric Target
v0.1.1 Registry availability >99.5%
v0.1.1 Provider re-registration time <2 min
v0.1.2 Route merge correctness 100% accuracy
v0.1.3 Validation latency (contract) <10ms
v0.1.3 Validation latency (provider P95) <5s
v0.1.4 Route Map render time <2s
v0.1.5 Alert rule accuracy <1% false positive

Post-Launch Evaluation (Week 16+)

  • Provider re-registration success rate: ≥99%
  • Registry uptime: >99.9%
  • Discovery cache hit rate: ≥95%
  • Validation p99 latency: <10s
  • Dashboard UX survey: ≥4/5 satisfaction
  • MTTD provider failures: <2 min
  • MTTR via runbooks: <10 min per incident

Conclusion

This implementation plan provides a phased, pragmatic roadmap for MeshFlows Capability Registry v0.1.x. By front-loading architecture decisions, establishing clear phase objectives, and investing in observability from day one, the team can deliver a robust, self-healing service discovery system.

Key Success Factors: 1. Strict adherence to go/no-go gates at each phase boundary 2. Early validation of core assumptions 3. Comprehensive observability and runbook documentation 4. Close collaboration between teams 5. Regular stakeholder sync-ups

Next Steps: 1. ADR review and approval 2. Resource allocation and team assignment 3. Detailed sprint planning for v0.1.1 4. Environment setup and CI/CD pipeline preparation 5. Kickoff sprint planning; begin implementation

Document Prepared By: MeshFlows Engineering Leadership
Review Status: Ready for Engineering Kickoff
Last Updated: April 2026