Weekly TLS + Forgejo runner health¶
Preventive monitoring for the failure chain: wrong ACME Gateway parentRef → stuck Challenges → expired meshflows-tls → dead code.meshflows.org TLS → Forgejo runners offline (x509) + ImagePullBackOff.
This is check-only by default. It does not weekly re-issue certificates (that burns Let's Encrypt rate limits). cert-manager renews automatically ~day 60 of a 90-day cert.
Workflow¶
| File | .github/workflows/weekly-tls-runner-health.yml |
| Schedule | Monday 06:15 UTC (15 6 * * 1) |
| Runner | deploy-lab (kubectl / cluster access required) |
| Manual | Actions → Weekly TLS + Runner Health → workflow_dispatch |
Script: engine/scripts/check-tls-runner-health.sh
What it checks¶
- ClusterIssuer / Issuer ACME
gatewayHTTPRoute.parentRefs— Gateway name must bemeshflows-{lab\|dev\|tst}, never baremeshflows. - Certificate
meshflows-tlsinmeshflows-lab/dev/tst:Ready=Trueand secretnotAftermore than 21 days away. - Challenges Pending longer than 2 days → fail (would have caught the multi-week stuck HTTP-01).
- Pods in
meshflows-ciRunning (informational; Running ≠ Forgejo-online). - Forgejo Actions runners online for labels
meshflows,deploy-lab,deploy-dev,deploy-tstvia API.
Secrets¶
| Secret | Purpose |
|---|---|
WORKFLOW_DISPATCH_TOKEN |
PAT (write:repository; write:issue if creating issues) to list runners / optional issue |
CLEANUP_ACTION_RUNS_TOKEN |
Fallback PAT if the above is empty |
Do not use the ephemeral job FORGEJO_TOKEN for the runners API. Never log token values.
Optional dispatch input create_issue_on_fail=true opens a Forgejo issue with the check report.
Chicken-egg (TLS already dead)¶
If code.meshflows.org TLS is already broken, self-hosted runners cannot poll Forgejo and the scheduled workflow will not start. Then run on the control plane:
ssh -o BatchMode=yes k8s-master 'kubectl get certificate,challenge -A'
# or clone/checkout and:
bash engine/scripts/check-tls-runner-health.sh
Runbook (safe remediations)¶
- Wrong parentRef
meshflows— re-apply ClusterIssuers from git (templates already use env Gateways):
bash engine/scripts/apply-cluster-issuers.sh <acme-email> lab dev tst
Or Actions → infra.yml → components=clusterissuers-only. Redeploy envs so namespaced Issuers match engine/deploy/k8s/meshflows-letsencrypt-issuer.yaml (meshflows-__MESHFLOWS_ENV__).
- Stuck Pending challenges — only after solvers point at
meshflows-<env>:
kubectl delete challenge -n meshflows-<env> --all
- Expired / IncorrectIssuer — use existing reconcile (no weekly force re-issue):
bash engine/scripts/reconcile-tls-certificate.sh meshflows-<env> letsencrypt-prod
-
Runners offline after TLS restore — usually self-heal once polling works; else
redeploy-ci-runners.yml. -
http-to-httpscatch-all — ACME paths must win via longest-path match on/.well-known/acme-challenge/...; edge-router proxies ACME without redirect. See Lab HTTPS.
Related¶
- Lab HTTPS Setup
- CI/CD Pipeline
engine/scripts/apply-cluster-issuers.shengine/scripts/reconcile-tls-certificate.sh