Production Readiness¶

This page is a go-live checklist for the CAPTF manager and the namespaces it serves. Each item says what to check and why, and links to the page that has the detail. It also says plainly what CAPTF does not do for you, because several things a production cluster needs are the operator’s to provide.

The checklist¶

Manager availability¶

• Replicas. The shipped Deployment runs one replica, with the leader election flag set. One replica is a valid production setting: while it restarts, running Jobs continue, and the manager picks them up again. No PodDisruptionBudget or anti-affinity is shipped; add them if you scale up.
• Leader election. The binary’s --leader-elect defaults to false; the shipped manifest passes it. If you build your own manifest, enable it before you run more than one replica, or two managers reconcile the same objects. See Configuration: leader election.
• Failover time. The manager does not release the election lease when it stops, so a replacement waits for the lease to expire: up to 15 seconds with the defaults (lease 15s, renew 10s, retry 2s). A failover loses nothing: Jobs are Kubernetes objects, and the names, the leases and the cache-lag checks make the resumed work idempotent. The webhooks run on every replica, so writes keep working while no manager leads. See Leader election and failover.
• The webhook is on the write path. All webhooks fail closed. If every manager pod is down, creating or updating a Terraform* object fails, including Cluster API’s own writes. See Webhook Unavailable.

Sizing¶

• Manager resources. The shipped requests are 10m CPU and 64Mi of memory, with limits of 500m and 256Mi. They suit a small installation. The manager holds informers for Jobs and managed Secrets, so memory grows with the number of objects, Jobs and Secrets. Watch the manager’s working set and raise the limit before it reaches it.
• Concurrency. --terraformcluster-concurrency and its machine, pool and template counterparts default to 10 each. They cap reconciles in flight per kind, not Jobs. A reconcile starts a Job and returns, so no flag caps the number of Jobs running at once; the bounds are one Job per object and your namespace quotas. Raise a concurrency flag when objects queue behind each other. See Configuration: per-kind concurrency.
• Resync. --sync-period (10 minutes) is a safety net, not a schedule. Drift runs every --drift-default-interval (30 minutes) unless an object sets its own. A large fleet with the default drift interval starts a burst of Jobs every half hour. The jitter spreads it by up to a tenth of the interval; set longer intervals if that is too much. See Requeue intervals and schedules.
• Job pods. The runner’s default request is 250m CPU and 512Mi, with a 2Gi memory limit and no CPU limit. A large plan needs more: set spec.jobs.resources. See Tuning Jobs.

Webhook certificates¶

The validating webhooks are served with a certificate cert-manager issues from a self-signed Issuer and injects into the webhook configuration. The certificate Secret is mounted into the manager pod, and the volume is not optional.

• cert-manager must be installed (cert-manager.io/v1). Without it the certificate Secret is never created and the manager pod stays in ContainerCreating.
• Watch the certificate. Check that the Certificate is Ready and its renewal works; an expired certificate stops all writes to Terraform* objects.
• Bring your own. --webhook-cert-dir, --webhook-cert-name and --webhook-key-name point the server at another certificate source if you do not use cert-manager. You then own CA injection too.

See Installation and Webhook Unavailable.

Required environment¶

The manager needs POD_NAMESPACE and SERVICE_ACCOUNT_NAME, which the shipped Deployment sets from the downward API. From them it computes its own username. The TerraformMachine webhook lets only that user set spec.providerID.

CAPTF_MANAGER_IMAGE supplies the runner image unless --runner-image is set.

Secrets at rest¶

• Enable etcd encryption at rest (an EncryptionConfiguration, or your platform’s equivalent) on the management cluster, and protect etcd snapshots as you would the state. See No encryption at rest of its own.
• State backups. --state-backups (default 5) keeps that many copies per object in the same namespace. They protect against a bad apply or a deleted state Secret, not against losing the namespace or the cluster. Plan external backups; see Disaster Recovery.
• Size. A state near 1 MiB per Secret, or rendered inputs near 1,000,000 bytes, need attention before they fail: see Size Limits.

Alerts and metrics¶

The Prometheus component is opt-in and not part of the release manifest. Build it from a checkout of the tag you installed, and apply it again after each upgrade. It adds a ServiceMonitor and a PrometheusRule named captf-alerts with eleven alerts. CAPTF ships rules, not routing: connect them to your Alertmanager.

Route CAPTFDestroyStuck and CAPTFStateUnreadable to someone who can act: both mean infrastructure the controller can no longer manage safely. The ServiceMonitor scrapes over HTTPS with the Prometheus ServiceAccount prometheus-k8s in monitoring; edit the shipped RBAC if yours differs. See Observability.

Three metrics have no alert and are worth a dashboard: captf_jobs_active (Jobs in flight), captf_ready (objects by readiness) and captf_identity_denied_total (denied identity use). Also watch captf_lease_waits_total for contention and captf_state_backups_total for backups being taken. See Metrics.

Approver and writer split¶

Gates also bind only the cluster: machines and pools are never gated, and a change of the cluster’s exports re-applies pools without approval. See Limits.

Network policy¶

• The manager. config/network-policy is an opt-in component. It allows inbound TCP 9443 (webhook), 8443 (metrics) and 9440 (probes) and outbound DNS, 6443 and 443; everything else is denied. It needs a CNI that enforces NetworkPolicy. Check that your API server’s address and port are covered by those rules, since the policy allows 6443 and 443 to any destination.
• Job pods. The component does not cover them. A sample (job-egress-sample.yaml, not applied) is a starting point to copy into each tenant namespace. Its cloud-provider egress is deliberately left out: add what your modules need, which depends on the cloud and the registry. Jobs hold cloud credentials, so egress is the control that limits where they can go. See Network exposure.

Runner Jobs¶

• Pod Security. The Job pod defaults satisfy the baseline profile. restricted needs an image that runs as non-root, set through jobs.podSecurityContext, because runAsNonRoot is not defaulted. Label the tenant namespace with the level you enforce and test a Job in it. See Pod security.
• Job policy defaults. Defaults are a one-hour deadline, a five-minute lock timeout, three retained successful and three failed Jobs per op, and the resources above. Set cluster-wide values in spec.defaults.jobs on the TerraformCluster so that machines and pools inherit them. See Tuning Jobs and Deadlines and lock timeouts.
• ResourceQuota. Quotas can block what CAPTF creates, and no quota-specific condition, event or metric exists. A rejected Job create is returned as a reconcile error and gives its leases back. A pod the quota refuses stays attached to a Job that never starts, which would show up as CAPTFJobQueueSlow. Per Job, the namespace holds a Job and a pod, a per-run Secret, the state, durable-inputs and backup Secrets, and a few Leases; each namespace also holds a captf-runner ServiceAccount and RoleBinding. Size object-count quotas for retained Jobs: finished Jobs stay until pruned. See Reconcile Errors.

The first week¶