Skip to content

Secrets Management

MeshFlows uses two different secret layers. Pick the right one before you store a value.

Layer Purpose Typical names How to manage
Runtime secrets Values that workflows / connections resolve via $secret:name weatherstack-access-key, rabbitmq-password, minio-access-key Dashboard Admin → Secrets, Storage API, optional CI upload
Platform (K8s) secrets Pod env / mounts for Engine services meshflows-identity-auth, meshflows-storage-auth, image pull Forgejo environment secrets → deploy.yml reconcile

Developers deploying connection credentials: follow the full howto → Deploy MeshFlows Runtime Secrets.

This page covers platform / CI secrets and points at runtime sync that bridges the two layers.

Secret Types (platform view)

  • Platform secrets — JWT keys, admin bootstrap credentials, encryption keys, service tokens.
  • Connection / runtime secrets — API keys, OAuth client secrets, passwords referenced as $secret:… in connections (stored in Storage, not as arbitrary K8s Secret keys).
  • Runtime service tokens — reload / invocation / internal auth between microservices (usually K8s Secrets wired into pods).

Never put plaintext credentials in workflow or connection YAML in Git.

Runtime secrets (summary)

  1. Create the value in the target environment’s Storage store (Dashboard or API).
  2. Reference it in flows/connections/*.yaml as $secret:<exact-name>.
  3. Deploy uploads connection YAML unchanged; the Orchestrator resolves secrets at reload/run.
  4. Engine deploy auto-syncs RabbitMQ and MinIO credentials into runtime names rabbitmq-user / rabbitmq-password and minio-access-key / minio-secret-key.

Full steps, verification, and common mistakes: Deploy MeshFlows Runtime Secrets.

CI Path: Forgejo Environment Secrets

Deploy jobs in deploy.yml bind to Forgejo environments (lab, dev, tst, acc, prd). Environment-scoped secrets override repository defaults.

Before each engine rollout, the workflow step Ensure required Engine auth secrets reconciles Forgejo secrets into Kubernetes:

Forgejo secret K8s secret (key)
IDENTITY_JWT_SECRET meshflows-identity-auth (jwt_secret)
IDENTITY_ENCRYPTION_KEY meshflows-identity-auth (encryption_key)
ORCH_RELOAD_TOKEN meshflows-orchestrator-auth (reload_token)
RABBITMQ_USER / RABBITMQ_PASSWORD / RABBITMQ_INTERNAL_TOKEN meshflows-rabbitmq-auth
STORAGE_SERVICE_READ_TOKEN / WRITE / ADMIN meshflows-storage-auth
SES_SERVICE_TOKEN meshflows-ses-auth (service_token)
CAPABILITY_REGISTRY_TOKEN meshflows-capability-registry-auth (token)
SCHEDULER_RELOAD_TOKEN meshflows-scheduler-auth
ORCHESTRATOR_POSTGRES_PASSWORD meshflows-orchestrator-postgres

Reconcile rules:

  • Empty Forgejo value → keep existing Kubernetes secret value.
  • Neither Forgejo nor K8s value → CI generates a fallback (logged as hint — set explicitly for production).
  • After rotation: update Forgejo secret → redeploy with the same release_id → dependent pods restart when tokens change.

Full table and mandatory vs optional secrets: .github/workflows/README.md.

Optional demo/CI runtime seeds (SECRET_SFTP_DEMO_*, SECRET_ACME_PROD_*, …) are rendered into engine/config/secrets.yaml during workflows deploy — not the same as the K8s table above. Mapping Forgejo → runtime name, and the fact that upload is always upsert (no insert-only): runtime secrets howto.

Setup checklist (Forgejo-only, no kubectl)

  1. Create environments: Settings → Environments → lab, dev, tst, acc, prd.
  2. Set repo-level shared secrets (REGISTRY_*, WORKFLOW_DISPATCH_TOKEN, WIKI_PUSH_TOKEN).
  3. Override per-env secrets on prd/acc (JWT, storage, Postgres passwords).
  4. Deploy via Actions or wait for auto-deploy after green build.
  5. Verify in deploy logs (secret reconcile + post-deploy verify steps).

Optional: kubeconfig helper scripts

For offline bootstrap or legacy tooling (not used by deploy.yml):

# Print base64 kubeconfig for manual paste into Forgejo Actions secrets
bash engine/scripts/generate-cluster-secret.sh --env DEV --kube ~/.kube/config

# Print curl/gh command for a specific forge host
bash engine/scripts/set-cluster-secret.sh \
  --host https://code.meshflows.org --repo meshflowsgpl/meshflows --env DEV \
  --file ~/.kube/config

Forgejo deploy runners use host kubectl; CLUSTER_<ENV> secrets are optional for normal CI.

Deploy runtime secrets via API

Runtime (connection) secrets can be created from a pipeline without opening the dashboard.

Mode Query / body Existing name
Exclusive (safe create) overwrite=false 409 Conflict, value unchanged
Overwrite (rotate / upsert) overwrite=true (default) updated

Dashboard (session + meshflows:runtime:secrets:write):

POST /api/runtime/secrets/{name}
Content-Type: application/json

{"value": "…", "overwrite": false}

Storage (cluster-internal admin token, used by deploy.yml):

POST /internal/secrets/{name}?overwrite=false
Authorization: Bearer <STORAGE_ADMIN_TOKEN>
Content-Type: application/json

{"value": "…"}

Bulk YAML upload accepts the same flag: POST /internal/upload/secrets?overwrite=false. On conflict the whole document is rejected and nothing is written. prune=true is incompatible with exclusive mode.

Default remains upsert so dashboard save and existing CI secret sync keep working.

Dummy sample after a default deploy

Every engine deploy (dev / linux-cluster via deploy.yml) seeds two non-credential examples so you can see runtime secrets without bringing real keys:

Name File Mode Redeploy
example-dummy-overwrite engine/config/secrets.yaml upsert (overwrite=true) value reset to dummy-not-a-real-credential
example-dummy-exclusive engine/config/secrets.example-dummy-exclusive.yaml exclusive (overwrite=false) created once; later deploys log 409 and keep the existing value

After deploy: Dashboard → Admin → Secrets. View the dummy names (needs meshflows:runtime:secrets:read). Deploy logs show Created exclusive dummy… or already present (not overwritten).

Change the exclusive dummy in the dashboard, redeploy, and confirm the dashboard value is unchanged — that is the #131 exclusive path. The overwrite dummy returns to the placeholder on every upload of secrets.yaml.

Local Development

Copy engine/.env.example to engine/.env and replace all placeholder values. Never commit .env.

Manual Kubernetes Pattern

For bootstrap or offline clusters without CI:

kubectl create secret generic meshflows-identity-auth \
  --from-literal=jwt_secret=<value> \
  --from-literal=encryption_key=<value> \
  --namespace meshflows-dev

Offline seed files (/etc/meshflows/secrets.<env>.env + apply-cluster-secrets.sh) are optional bootstrap aids — not required when using Forgejo deploy.

  • Store platform secrets in Kubernetes / Forgejo environment secrets — not in Git.
  • Store connection credentials as runtime secrets; reference names only in connections.
  • Rotate secrets on schedule and after incidents.
  • Use unique credentials per production environment (prd, acc).

Validation

  • Ensure pods receive expected env vars after platform secret changes.
  • Confirm traces/logs mask sensitive headers and values.
  • Test credential-dependent workflows after each rotation (connection Test in Dashboard).
  • Call admin endpoints with X-Reload-Token after changing ORCH_RELOAD_TOKEN.