Skip to content

Deploy MeshFlows Runtime Secrets

Use this howto when a workflow or connection needs credentials (API keys, passwords, OAuth client secrets). Keep values out of Git; commit only $secret:… references.

For Forgejo / Kubernetes platform secrets (JWT, image pull, Postgres, service tokens), see Secrets Management (Engine).

Runtime secret vs $secret:name

Concept What it is Where it lives
Runtime secret Named credential value (weatherstack-access-key → real key) Storage runtime secret store (per environment / namespace)
$secret:name Placeholder in connection (or similar) YAML Git under flows/connections/ — name only, never the value
K8s Secret Cluster object for pods (auth tokens, MinIO, RabbitMQ broker user, …) Namespace meshflows-<env> — not what $secret: resolves

At load/run time the Orchestrator loads secrets from Storage and replaces $secret:foo with the stored value. Unresolved refs fail connection use / service_call steps.

flowchart LR
  Git["flows/connections/*.yaml\naccess_key: $secret:my-key"]
  Store["Storage runtime secrets\nname=my-key"]
  Orch["Orchestrator reload"]
  Egress["Egress provider"]
  Git -->|"upload connection YAML"| Store
  Store -->|"load secrets + resolve"| Orch
  Orch -->|"resolved fields"| Egress

Create or update a secret

Secrets are per environment (lab / dev / tst / acc / prd → namespace meshflows-<env>). Creating a secret in Dashboard dev does not put it on prd.

Dashboard (preferred for app credentials)

  1. Open the Dashboard for the target environment.
  2. Go to Admin → Secrets (needs meshflows:runtime:secrets:read; create/update needs meshflows:runtime:secrets:write).
  3. Enter name (must match the $secret:… suffix exactly, e.g. weatherstack-access-key) and value.
  4. Save. The UI reloads the Orchestrator when you have admin rights so connections pick up the new value.
  5. If reload was skipped, use Reload runtime (platform admin) or wait for the next deploy reload.

Do not paste real values into Git, tickets, or wiki pages.

API (Dashboard proxy)

Authenticated Dashboard session with write scope:

POST /api/runtime/secrets/{name}
Content-Type: application/json

{"value": "<your-secret-value>"}

List (metadata only): GET /api/runtime/secrets
Get one value: GET /api/runtime/secrets/{name}
Delete: DELETE /api/runtime/secrets/{name}

Storage internal API (ops / CI)

Trusted callers with Storage admin bearer token (cluster secret meshflows-storage-auth / admin_token):

POST /internal/secrets/{name}
Authorization: Bearer <storage-admin-token>
Content-Type: application/json

{"value": "<your-secret-value>"}

Same pattern as deploy sync for RabbitMQ / MinIO (see below). There is no separate end-user CLI; use Dashboard, curl to the API, or CI upload.

Reference secrets in connection YAML

Commit the connection under flows/connections/ with placeholders only:

# flows/connections/weatherstack.yaml
name: weatherstack
type: weatherstack
access_key: "$secret:weatherstack-access-key"

HTTP header example:

name: external-api
type: http
base_url: https://api.example.com
default_headers:
  X-API-Key: $secret:api-key

OAuth client credentials:

name: salesforce_api
type: http
base_url: https://your-instance.salesforce.com
oauth:
  token_url: https://login.salesforce.com/services/oauth2/token
  client_id: "$secret:sf-client-id"
  client_secret: "$secret:sf-client-secret"
  scope: "api"

Rules:

  • The string after $secret: is the runtime secret name.
  • Connection YAML is uploaded by deploy as written — deploy does not expand $secret: inside connection files.
  • Real values must already exist in the Storage secret store for that namespace (Dashboard, API, CI secrets upload, or auto-sync).

More connection patterns: Connections & Credentials.

How deploy picks secrets up (and what stays out of Git)

What Git must never contain

  • Plaintext passwords, API keys, private keys, OAuth client secrets
  • Rendered secrets.yaml with real values

Safe in Git: $secret:name (and non-sensitive connection fields such as hostnames when they are not secret).

Automatic sync (Engine deploy)

On Engine deploy, CI copies selected Kubernetes credentials into the Storage runtime store so connections can use $secret:…:

Runtime secret name Source K8s secret
rabbitmq-user / rabbitmq-password meshflows-rabbitmq-auth
minio-access-key / minio-secret-key meshflows-orchestrator-minio

After broker or MinIO password rotation, re-run Engine deploy (or update the matching runtime secrets manually) so Storage stays in sync. See RabbitMQ.

Optional CI upload: engine/config/secrets.yaml

Workflows deploy (deploy-workflows in .github/workflows/deploy.yml) may upload:

  1. Render engine/config/secrets.yaml with engine/scripts/render-upload-substitutions.py
  2. POST /internal/upload/secrets to Storage
  3. Reload Orchestrator

In that file, $secret:NAME / $var:NAME are filled from Forgejo environment variables:

  • Secret NAME → env SECRET_<NORMALIZED> or SECRET_<NORMALIZED>_<ENV> (e.g. SECRET_SFTP_DEMO_PASSWORD_DEV)
  • Var NAME → VAR_<NORMALIZED> / VAR_<NORMALIZED>_<ENV>

Normalization: non-alphanumeric characters become _, then uppercase (mijneigen_secret → MIJNEIGEN_SECRET). Env-specific keys win over the generic key when both exist.

Only names wired in the workflow env: block are available. Missing substitutions fail the render step. Prefer Dashboard for one-off app secrets; use Forgejo + secrets.yaml for demo/CI-seeded credentials that must land on every deploy.

Namespaces: meshflows-lab, meshflows-dev, meshflows-tst, meshflows-acc, meshflows-prd (prefix from MESHFLOWS_NAMESPACE).

Example: Forgejo MY_SECRET → runtime mijneigen_secret

The YAML key is the runtime secret name (what $secret:… in connections resolves). The $secret:… / $var:… placeholder only selects which Forgejo/env value is rendered at deploy time — the names do not have to match.

  1. Create a Forgejo Actions secret (repo or environment dev-workflows / prd-workflows / …), e.g. MY_SECRET.
  2. Wire it into deploy-workflows env: in .github/workflows/deploy.yml under the lookup name the render script expects:
# deploy-workflows env: (illustrative)
SECRET_MIJNEIGEN_SECRET: ${{ secrets.MY_SECRET }}
# optional per-env override:
# SECRET_MIJNEIGEN_SECRET_DEV: ${{ secrets.MY_SECRET }}
  1. Map it in engine/config/secrets.yaml:
secrets:
  mijneigen_secret:          # ← runtime name used by connections as $secret:mijneigen_secret
    value: $secret:mijneigen_secret   # ← looks up SECRET_MIJNEIGEN_SECRET[_<ENV>]

Alternative when you want the placeholder to mirror the Forgejo name:

secrets:
  mijneigen_secret:
    value: $secret:my_secret   # looks up SECRET_MY_SECRET → wire env SECRET_MY_SECRET: ${{ secrets.MY_SECRET }}
  1. Deploy workflows (or full deploy). CI renders the file and uploads to Storage; Orchestrator reload picks it up.
  2. In connection YAML use only the runtime name: api_key: "$secret:mijneigen_secret".

Do not commit real values. Keep Forgejo secret values out of tickets and wiki pages.

Insert-only / no-overwrite (not supported today)

There is no create-if-not-exists flag on runtime secrets.

Path Behaviour today
Dashboard Admin → Secrets save Always upsert (creates or overwrites)
POST /api/runtime/secrets/{name} Always upsert
POST /internal/secrets/{name} Always upsert (CI RabbitMQ/MinIO sync uses this)
POST /internal/upload/secrets (secrets.yaml) Always upsert every listed name; optional ?prune=true also deletes secrets not in the file

Platform K8s reconcile (“empty Forgejo value → keep existing K8s key”) does not apply to Storage runtime secrets.

Safest current practice when you must not overwrite an existing value:

  1. Prefer Dashboard once for app credentials that operators own — create the secret manually and do not list that name in engine/config/secrets.yaml (otherwise every workflows deploy overwrites it with the Forgejo value).
  2. CI insert-only pattern (custom job / one-off ops script), not built into MeshFlows:
# Pseudocode — Storage admin token; never echo the value
# 1) GET /internal/secrets/{name}  →  if HTTP 200: skip
# 2) if HTTP 404: POST /internal/secrets/{name}  with {"value":"..."}

Same idea via Dashboard proxy: GET /api/runtime/secrets/{name} then POST only when missing. There is no official overwrite=false query/body on these endpoints.

Warnings:

  • Putting a name in secrets.yaml means every successful workflows deploy replaces the Storage value for that name with the rendered Forgejo value.
  • ?prune=true on secrets upload is destructive for names absent from the YAML (deploy currently uploads without prune; do not enable prune casually).
  • RabbitMQ / MinIO Engine sync always overwrites rabbitmq-* / minio-* runtime secrets from K8s — treat those names as platform-owned.
  • Missing Forgejo/env substitution fails the render step (deploy aborts); it does not “skip and keep old”.

Platform secrets are a different pipeline

Forgejo secrets such as IDENTITY_JWT_SECRET, ORCH_RELOAD_TOKEN, STORAGE_SERVICE_* are reconciled into Kubernetes Secrets for pods. They are not automatically exposed as $secret: names unless a dedicated sync step (like RabbitMQ/MinIO) or secrets.yaml upload maps them. Details: Engine Secrets Management.

Verify

  1. Dashboard → Secrets — name exists in the correct environment (value hidden until View).
  2. Dashboard → Connections — open the connection → Test (POST /api/connections/{name}/test). Failures often mean missing/unresolved $secret: or wrong value.
  3. Run a workflow that uses the connection (service_call with connection:). Check run / traces for secret-resolution or auth errors (values should stay masked).
  4. After creating secrets via API without UI reload: trigger Orchestrator reload (POST /api/runtime/reload from Dashboard, or /admin/reload with reload token).

Common mistakes

Mistake Fix
Commit plaintext credentials in flows/connections/ Use $secret:name only; store the value in Dashboard/Storage
Secret created in dev, workflow runs in prd Create/update the secret in each target environment
Name typo ($secret:api_key vs secret api-key) Names must match exactly (allowed: letters, digits, _, ., -)
Updated secret but still old behaviour Reload Orchestrator / use Dashboard save+reload
Expect $secret: in connection YAML to be filled by GitHub/Forgejo env Connection upload does not substitute; only engine/config/secrets.yaml render does
Forgejo secret set but not in deploy-workflows env: Render cannot see it — add SECRET_<NORMALIZED>: ${{ secrets.… }} to the job env
Manual Dashboard value overwritten on next deploy Name was also in secrets.yaml — remove it from CI upload or accept Forgejo as source of truth
Expect CI upload to skip existing secrets Not supported — upsert always; use Dashboard-only or GET-then-POST insert-only script
Confuse K8s Secret meshflows-rabbitmq-auth with runtime $secret:rabbitmq-password Broker user is K8s; connection resolution uses Storage — Engine deploy syncs them
Delete a secret still referenced by a connection Check references / force-delete only when intentional; update or remove $secret: refs first
Missing RBAC Need meshflows:runtime:secrets:read / :write (see IAM role templates)