Background jobs and leader election
The platform API can run as several replicas. Periodic work must still run
once per schedule cluster-wide, and long-running work (deploy rollouts)
must survive an API restart. This page lists every timer and background task
in platform/api/src and how each one is made exactly-once.
Inventory
| Work | Before | Now | Schedule |
|---|---|---|---|
Preview decay (lib/preview-decay.ts) |
setInterval + startup setTimeout in every replica |
Leader loop | PREVIEW_DECAY_INTERVAL_SECONDS (default 900), TTL PREVIEW_TTL_HOURS (default 72) |
Deploy rollout (POST /deploy) |
fire-and-forget setTimeout, lost on restart |
Step-logged operation (lib/deploy-op.ts, #23) under the deploy lock; started on the accepting replica |
per request |
| Argo health polling (rollout step) | setTimeout sleep loop inside the rollout |
Unchanged loop, now the wait-healthy step of the deploy operation |
5 s × 36 |
Operation resume (operation-resume) |
— | Leader loop: resumes non-terminal operations whose lock is free | every 60 s and right after gaining leadership |
Outbox dispatch (outbox-dispatch) |
— | Leader loop | every 5 s |
Operation cleanup (operation-cleanup) |
— | Leader loop: prunes expired idempotency keys, webhook deliveries, outbox rows | hourly |
| Deploy recovery | — | Leader loop: fails orphaned rollouts with neither a job nor an operation | every 15 s and right after gaining leadership |
| Token cleanup | — | Leader loop | hourly |
Telemetry consumer (telemetry-consumer, #49) |
synchronous Mongo/Loki writes inside POST /sdk/logs, /sdk/api-metrics, /sdk/heartbeat |
Leader loop: Redis Streams consumer group forwarding the queued events (see telemetry-ingestion.md) | every 1 s |
Alert evaluation (POST /alerts/evaluate) |
request-driven; caller supplies the metric values | Unchanged (no timer) | — |
withTimeout (lib/with-timeout.ts) |
per-request timeout helper | Unchanged (not background work) | — |
Webhook preview deploys (routes/webhooks.ts) |
inline in the webhook request | Unchanged (not background work) | — |
Alert evaluation stays request-driven: there is no metric source inside the API to evaluate on a schedule yet. A scheduled evaluator belongs with the alerting work (monitors/SLOs) and would register as another leader job.
Leader election
lib/leader.ts implements leader election on a coordination.k8s.io/v1
Lease named platform-api-leader in the API's own namespace (platform).
- Lease duration 10 s, renew/retry every 2 s (
LEADER_LEASE_DURATION_SECONDS,LEADER_RETRY_PERIOD_SECONDS). - Updates are compare-and-swap on
resourceVersion, so two candidates can never both win. - A follower takes over once the holder's record has not changed for a full lease duration, measured on its own clock (clock skew between nodes does not matter). Worst case after a hard kill: about 12 s.
- A leader that cannot renew steps down after 6 s (before any peer may take
over), and
isLeader()turns false as soon as its last renew is stale. - On SIGTERM/SIGINT the leader releases the Lease; a peer takes over at its next retry (about 2 s).
- Outside Kubernetes, or with
LEADER_ELECTION=disabled, the single process is always the leader.
API: isLeader(), onLeadershipChange(fn), getLeadership().start()/stop().
Jobs registered with registerLeaderJob() (lib/scheduler.ts) start their
timer on leadership gain and clear it on loss; runs of the same job never
overlap.
RBAC
The platform-api Role in the platform namespace grants:
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["create"] # create cannot be limited by name
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
resourceNames: ["platform-api-leader"]
verbs: ["get", "update"]
No cluster-wide Lease access is granted. The manifests are
platformctl/internal/manifests/data/platform-api-rbac.yaml and
k8s/platform/api-rbac.yaml; they must stay identical (a test enforces this).
Durable deploy rollouts
POST /deploy takes the per-environment deploy lock
(lock:project:<id>:deploy:<envId>), saves the Deployment and creates a
deployment.deploy operation (operations + operation_steps, see
lib/operations.ts). The accepting replica runs it while holding the lock
(auto-renewed) and records each step: apply, wait-healthy, finalize.
If that replica dies, its lock lapses and the leader's operation-resume job:
- re-runs the step that was in flight (every step is idempotent: Argo app ensure+sync, preview apply, ingress ensure), or fails the deploy as superseded when a newer deployment of the same environment exists;
- once a step was interrupted
maxAttempts(2) times, fails the operation and the Deployment (metadata.errorexplains the interruption).
The operation worker has no leader mechanism of its own: its three jobs
(operation-resume, outbox-dispatch, operation-cleanup) are ordinary
leader jobs on the platform-api-leader Lease.
Deployments that are still pending/building/deploying after 10 minutes
with neither a background_jobs row nor a deploy operation (rollouts started
by an older API version) are marked failed by deploy-recovery. A Deployment
therefore never stays "in progress" after a restart.
background_jobs remains the generic durable-job table (claim with a
conditional UPDATE, 60 s heartbeated lock, max_attempts); a legacy
type=deploy row is handed over to a deploy operation. Finished rows are
pruned after 30 days by token-cleanup.
CronJob alternative
Leader loops are the default because they cost nothing extra on a single server. A Kubernetes CronJob spins up a pod per run (image pull, memory spike), so it suits infrequent jobs only (hourly or slower). To move a job to a CronJob:
- Set
JOBS_EXTERNAL=<name>[,<name>]on the API Deployment; the in-process timer for those jobs is then not started. - Run
node dist/jobs/<name>.jsfrom the API image in a CronJob withconcurrencyPolicy: Forbidand the sameenvFrom: platform-env.
Available entry points: dist/jobs/preview-decay.js, dist/jobs/token-cleanup.js.
Deploy recovery is not available as a CronJob: resuming a rollout needs a
long-lived process.
apiVersion: batch/v1
kind: CronJob
metadata:
name: platform-api-cron-token-cleanup
namespace: platform
spec:
schedule: "17 * * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 1
template:
spec:
restartPolicy: Never
automountServiceAccountToken: false
securityContext: { runAsNonRoot: true, runAsUser: 1000, seccompProfile: { type: RuntimeDefault } }
containers:
- name: job
image: <platform-api image>
command: ["node", "dist/jobs/token-cleanup.js"]
envFrom: [{ secretRef: { name: platform-env } }]
securityContext: { allowPrivilegeEscalation: false, capabilities: { drop: ["ALL"] } }
resources:
requests: { cpu: 50m, memory: 96Mi }
limits: { cpu: 500m, memory: 256Mi }
Metrics
GET /api/metrics/jobs (devops/admin) returns the serving replica's counters
in Prometheus text format:
platform_job_runs_total{job,result}—resultissuccess,failureorskipped; jobs:preview-decay,deploy-recovery,token-cleanup,operation-resume,outbox-dispatch,operation-cleanup.platform_job_last_success_timestamp_seconds{job}platform_api_leader— 1 on the replica holding the Lease.
Counters are per process; collect every API pod and sum across them.
Verifying with two replicas
- Scale the API to 2 replicas and set
PREVIEW_DECAY_INTERVAL_SECONDS=60. kubectl -n platform get lease platform-api-leader -o yamlshows one holder.kubectl -n platform logs -l app=platform-api -f | grep '\[decay\] run started'prints one line per minute in total.- Delete the holder pod (
kubectl delete pod --grace-period=0 --force ...); the other pod logsacquired leadershipwithin about 12 s. - Start a deploy, then delete the pod that accepted it; the Deployment either
finishes (resumed by the leader) or turns
failed— never staysdeploying.