Jobs, Retries and Concurrency¶

Every Terraform or OpenTofu run CAPTF does is one Kubernetes Job, started by the controller, watched to the end and counted. This chapter follows a Job from the decision to run it to the bookkeeping after it finishes, and explains the machinery that keeps those Jobs from colliding: the names that make creation idempotent, the retry rules, the deadlines, the leases, the cache-lag checks, the state lock and leader election. It is the model behind the “nothing is happening” questions that Slow Jobs and Failing Jobs answer one symptom at a time, and it goes deeper than The Reconcile Lifecycle, which it extends.

The pages:

Two more pages sit alongside them:

• Requeue intervals and schedules.
• Nothing is happening: from a wait reason to the action.

This page covers the life of a Job, the operations and the flags.

The life of a Job¶

%%{init: {"themeVariables": {"fontSize": "13px"}, "flowchart": {"nodeSpacing": 28, "rankSpacing": 34, "padding": 10}}}%%
flowchart TD
    A[Reconcile: bookkeeping,<br/>no active Job] --> B[DecideOp picks an operation]
    B --> C{Backoff,<br/>approval or gate?}
    C -- wait --> W[Requeue]
    C -- run --> D[Take the run lease,<br/>and the cluster lease]
    D --> E{Lease free?}
    E -- no --> W
    E -- yes --> F[Create the Job,<br/>then its per-run Secret]
    F --> G[Job runs]
    G --> H[Bookkeeping: read the pod,<br/>delete the per-run Secret]
    H --> I[Set conditions and lastRun,<br/>release the leases, prune]
    I --> A

Four rules shape the diagram:

• At most one Job per object. The reconcile does nothing else while a Job of the object runs, except to watch it. The run lease makes that hold across managers and across crashes.
• Creation is idempotent. Names are deterministic, so a retry after a crash or a stale read finds the Job it already created.
• The controller owns retries. A Job has backoffLimit: 0, a pod restartPolicy: Never and no TTL; every retry is a new Job, named with a new attempt number, started only after the backoff.
• Results are read once. After a Job finishes, one pass reads its pod, deletes its per-run Secret, records the outcome on the object and marks the Job as bookkept. Later passes read the Job’s annotations, not its pod.

The operations¶

Which of these wins when several are possible is the priority order. Every op is a Job of the same shape, differing in the runner’s --op.

Manager flags and where the rest lives¶

The manager has no flags for backoff or for the failure limit. The retry timing is compiled in, and the limit is the per-object spec.jobs.failedJobsHistoryLimit. The flags that matter here:

The per-object knobs are in spec.jobs: activeDeadlineSeconds, lockTimeoutSeconds, successfulJobsHistoryLimit and failedJobsHistoryLimit. See Tuning Jobs. The full flag list is Manager Flags.