Skip to content

Weekly TLS + Forgejo runner health

Preventive monitoring for the failure chain: wrong ACME Gateway parentRef → stuck Challenges → expired meshflows-tls → dead code.meshflows.org TLS → Forgejo runners offline (x509) + ImagePullBackOff.

This is check-only by default. It does not weekly re-issue certificates (that burns Let's Encrypt rate limits). cert-manager renews automatically ~day 60 of a 90-day cert.

Workflow

File .github/workflows/weekly-tls-runner-health.yml
Schedule Monday 06:15 UTC (15 6 * * 1)
Runner deploy-lab (kubectl / cluster access required)
Manual Actions → Weekly TLS + Runner Health → workflow_dispatch

Script: engine/scripts/check-tls-runner-health.sh

What it checks

  1. ClusterIssuer / Issuer ACME gatewayHTTPRoute.parentRefs — Gateway name must be meshflows-{lab\|dev\|tst}, never bare meshflows.
  2. Certificate meshflows-tls in meshflows-lab / dev / tst: Ready=True and secret notAfter more than 21 days away.
  3. Challenges Pending longer than 2 days → fail (would have caught the multi-week stuck HTTP-01).
  4. Pods in meshflows-ci Running (informational; Running ≠ Forgejo-online).
  5. Forgejo Actions runners online for labels meshflows, deploy-lab, deploy-dev, deploy-tst via API.

Secrets

Secret Purpose
WORKFLOW_DISPATCH_TOKEN PAT (write:repository; write:issue if creating issues) to list runners / optional issue
CLEANUP_ACTION_RUNS_TOKEN Fallback PAT if the above is empty

Do not use the ephemeral job FORGEJO_TOKEN for the runners API. Never log token values.

Optional dispatch input create_issue_on_fail=true opens a Forgejo issue with the check report.

Chicken-egg (TLS already dead)

If code.meshflows.org TLS is already broken, self-hosted runners cannot poll Forgejo and the scheduled workflow will not start. Then run on the control plane:

ssh -o BatchMode=yes k8s-master 'kubectl get certificate,challenge -A'
# or clone/checkout and:
bash engine/scripts/check-tls-runner-health.sh

Runbook (safe remediations)

  1. Wrong parentRef meshflows — re-apply ClusterIssuers from git (templates already use env Gateways):
bash engine/scripts/apply-cluster-issuers.sh <acme-email> lab dev tst

Or Actions → infra.yml → components=clusterissuers-only. Redeploy envs so namespaced Issuers match engine/deploy/k8s/meshflows-letsencrypt-issuer.yaml (meshflows-__MESHFLOWS_ENV__).

  1. Stuck Pending challenges — only after solvers point at meshflows-<env>:
kubectl delete challenge -n meshflows-<env> --all
  1. Expired / IncorrectIssuer — use existing reconcile (no weekly force re-issue):
bash engine/scripts/reconcile-tls-certificate.sh meshflows-<env> letsencrypt-prod
  1. Runners offline after TLS restore — usually self-heal once polling works; else redeploy-ci-runners.yml.

  2. http-to-https catch-all — ACME paths must win via longest-path match on /.well-known/acme-challenge/...; edge-router proxies ACME without redirect. See Lab HTTPS.