Capability Registry and Extensibility (v0.1.x)¶

This document describes a pragmatic v0.1.x design to make MeshFlows more modular:

services can self-register their capabilities
services can re-register after registry restarts or reconnect events
gateway and orchestrator can discover new service variants (for example JMS next to RabbitMQ)
workflow steps can declare a version in YAML

The goal is to reduce mandatory engine changes for new integrations.

Goals¶

Add runtime capability discovery without breaking current static behavior.
Support automatic re-registration of service instances.
Enable protocol variants through capability contracts, not hardcoded service names.
Introduce step-level versioning in workflow YAML.
Keep compatibility with existing workflows and edge routes.

Non-Goals (v0.1.x)¶

Full dynamic loading of brand-new orchestrator step classes.
Cross-cluster federation of registries.
Runtime hot-swap of orchestration semantics per step type.

Current Baseline¶

Gateway routes are currently loaded from runtime configuration.
Orchestrator supports a fixed set of step types.
Trigger loops are implemented explicitly (for example RabbitMQ loop).

This proposal adds a capability registry as an extra control-plane layer, while keeping the existing behavior as fallback.

Architecture¶

Components¶

Capability Registry service:
stores service instances and capabilities
exposes APIs for register/heartbeat/discovery
applies lease TTL and stale-instance eviction
Capability-aware Gateway (consumer):
merges static edge routes with discovered routes
keeps last-known-good route set when registry is unavailable
Capability-aware Orchestrator (consumer):
resolves generic service-backed steps via discovered capabilities
keeps static execution paths as fallback
Provider services (producers):
register capabilities at startup
send periodic heartbeats
re-register automatically after reconnect or registry restart

Registry Storage Model¶

Suggested top-level entities:

service_instance
instance_id (stable for process lifetime)
service_name
service_version
endpoint (cluster URL)
started_at
generation (monotonic, assigned by registry)
lease_expires_at
capability
capability_id (for example messaging.publish)
capability_version (semver)
contract_schema_version
provider (for example rabbitmq, jms)
metadata (constraints, auth, payload hints)

Service Registration Lifecycle¶

Register¶

On startup, each provider service calls:

POST /v1/services/register

Payload example:

{
  "instance_id": "egress-jms-7c9f5d8c4f-9x2kq",
  "service_name": "egress-jms",
  "service_version": "0.1.3",
  "endpoint": "http://egress-jms:8080",
  "capabilities": [
    {
      "capability_id": "messaging.publish",
      "capability_version": "1.0.0",
      "contract_schema_version": "1.0",
      "provider": "jms",
      "metadata": {
        "supports_transactions": true,
        "max_payload_bytes": 1048576
      }
    }
  ]
}

Response example:

{
  "accepted": true,
  "lease_ttl_seconds": 30,
  "heartbeat_interval_seconds": 10,
  "generation": 42,
  "registry_epoch": "2026-04-27T08:15:00Z"
}

Heartbeat¶

Service refreshes its lease using:

PUT /v1/services/{instance_id}/heartbeat

Payload example:

{
  "generation": 42,
  "health": "ok"
}

If registry returns 404 or 410, service must perform full register again.

Re-Registration Rules¶

Providers must re-register when one of these signals occurs:

Heartbeat returns 404 or 410.
Heartbeat returns generation mismatch.
Registry endpoint was unavailable and becomes reachable again.
Service capabilities changed at runtime.

Recommended provider algorithm:

Start -> register immediately.
If register fails, retry with exponential backoff + jitter.
Send heartbeat on fixed interval (heartbeat_interval_seconds).
On heartbeat failure threshold, switch to reconnect mode.
In reconnect mode, try register again (not heartbeat).

Registry Restart Behavior¶

After registry restart, in-memory state can be empty (depending on storage mode). Because providers heartbeat continuously, missing entries are detected quickly and providers re-register themselves without manual action.

Discovery APIs (Gateway/Orchestrator)¶

GET /v1/discovery/capabilities
filter by capability_id, provider, min_version
GET /v1/discovery/routes
optional generated public route hints
GET /v1/discovery/health
registry operational status

Consumers should cache responses with short TTL and keep a last-known-good snapshot.

Workflow YAML: Step Versioning¶

To support gradual evolution, each service-backed step can include:

step_version: version of the step contract expected by the workflow
capability_version: minimum capability contract required from provider

Example: Generic Service-Backed Step¶

name: message-publish-demo
invocation:
  type: ["http", "schedule"]
steps:
  - id: publish_order
    type: service_call
    step_version: "1.1"
    capability:
      id: messaging.publish
      provider: jms
      capability_version: ">=1.0 <2.0"
    request:
      destination: "orders.events"
      payload_from: previous
      headers:
        content-type: application/json

Compatibility Rules¶

If step_version is omitted, default to "1.0".
If provider does not satisfy capability_version, orchestrator fails fast with clear error.
Unknown optional attributes are ignored with warning in v0.1.x.
Unknown required attributes return validation error.

Validation Strategy: Contract First, Provider Second¶

For workflow validation, use a layered model:

Central contract validation (required)
Provider semantic validation (optional but recommended)
Provider runtime enforcement (always required)

Why This Model¶

Central validation gives fast and deterministic feedback for API, CI, and dashboard editor.
Provider validation adds deep protocol-specific checks (for example JMS destination policies).
Runtime enforcement remains the final safety net against drift.

Layer 1: Central Contract Validation (Orchestrator)¶

Validate step shape and compatibility without calling provider services:

required fields and data types
step_version parsing and defaults
capability.id, provider, and capability_version syntax
cross-field rules in generic step contract

This layer should be available even if registry/provider services are temporarily unavailable.

Layer 2: Provider Semantic Validation (Capability Provider)¶

If capability metadata indicates validator support, orchestrator calls provider validation endpoint.

Capability metadata hint example:

{
  "capability_id": "messaging.publish",
  "provider": "jms",
  "metadata": {
    "validate_supported": true,
    "validate_endpoint": "/v1/capabilities/messaging.publish/validate"
  }
}

Proposed API:

POST /v1/capabilities/{capability_id}/validate

Request example:

{
  "workflow_name": "message-publish-demo",
  "step_id": "publish_order",
  "step_version": "1.1",
  "capability_version": ">=1.0 <2.0",
  "step": {
    "type": "service_call",
    "request": {
      "destination": "orders.events",
      "headers": {"content-type": "application/json"}
    }
  }
}

Response example:

{
  "valid": true,
  "diagnostics": [
    {
      "severity": "warning",
      "code": "JMS_HEADER_NON_STANDARD",
      "message": "Header 'content-type' is mapped to provider-specific property.",
      "path": "request.headers.content-type"
    }
  ]
}

Layer 3: Runtime Enforcement (Provider)¶

Even after successful validation, provider must re-check critical invariants at execution time:

destination existence/permissions
payload/size constraints
auth and connection constraints

Upload/Validate Decision Policy¶

Recommended policy in v0.1.x:

Contract validation failure: always block workflow publish/reload.
Provider validation failure:
strict mode (WORKFLOW_VALIDATE_PROVIDER_STRICT=true): block
non-strict mode: allow with warning status
Provider unreachable during validation:
strict mode: block
non-strict mode: allow with warning + provider_status=skipped

Validation Result Schema (Editor + API)¶

Persist both layers in one result object for dashboard/editor:

{
  "workflow": "message-publish-demo",
  "contract_status": "passed",
  "provider_status": "warning",
  "steps": [
    {
      "step_id": "publish_order",
      "contract": {"status": "passed", "diagnostics": []},
      "provider": {
        "status": "warning",
        "diagnostics": [
          {
            "severity": "warning",
            "code": "JMS_HEADER_NON_STANDARD",
            "message": "Header 'content-type' is mapped to provider-specific property.",
            "path": "request.headers.content-type"
          }
        ]
      }
    }
  ]
}

This allows the workflow editor to:

render forms from the contract schema
show inline diagnostics per field path
distinguish contract errors from provider warnings

Gateway Route Extensibility¶

For endpoint flexibility (for example exposing api/loops/v1), gateway can merge:

static configured routes
registry-derived route aliases per capability

Route conflicts are resolved by deterministic priority:

explicit static route
explicit dynamic route with higher priority
other dynamic route

Orchestrator Scope Model (Recommended Increment)¶

To match strict scope boundaries:

each loop/parallel scope runs with a local context snapshot
local mutations stay local unless explicitly exported
parent scope receives only declared outputs

Proposed export model:

outputs_map for parallel scopes
explicit export_context_keys for loop scopes

This can be added incrementally without breaking existing workflows by keeping legacy mode as default and introducing strict mode via feature flag.

Security Model¶

Registry APIs require service identity (mTLS or signed service JWT).
Each capability registration is authorized by service role.
Discovery endpoints for internal consumers only (cluster-private).
Registry payloads must be schema-validated and size-limited.

Rollout Plan (v0.1.x)¶

v0.1.1:
introduce registry service + register/heartbeat APIs
provider auto re-registration logic
v0.1.2:
gateway discovery client + static merge + last-known-good cache
v0.1.3:
orchestrator generic service_call step
step versioning fields in YAML validation
v0.1.4:
strict scope mode (opt-in)

Failure Handling¶

Registry unavailable:
providers keep retrying registration
gateway/orchestrator continue on cached discovery snapshot
Provider disappears:
lease expires and instance is evicted
orchestrator selects another compatible provider or fails fast
Capability downgrade:
compatibility check blocks execution if contract not met

Operational Metrics¶

Registry metrics to expose:

active_instances
active_capabilities
register_success_total / register_fail_total
heartbeat_success_total / heartbeat_fail_total
stale_evictions_total
discovery_latency_ms

Provider metrics to expose:

registry_register_attempts_total
registry_register_success_total
registry_heartbeat_failures_total
registry_reregister_total

Summary¶

This v0.1.x design keeps current MeshFlows behavior stable while introducing:

dynamic, self-healing service discovery
explicit compatibility via step/capability versioning
a path toward protocol variants (RabbitMQ, JMS, etc.) without immediate core rewrites