Runbooks

Each runbook below covers one symptom: what it looks like, why it happens, how to check, how to fix it, and how to confirm the fix worked. Alerts and condition reasons are cross-references, not a substitute for reading the page: start from whichever alert fired or condition you see, but read the whole runbook before you act.

• Failing Jobs — a TerraformCluster, TerraformMachine or TerraformMachinePool whose Jobs keep failing. CAPTFJobFailing, CAPTFNoRecentSuccess; ApplyJobSucceeded=False and DriftJobSucceeded=False.
• Stuck Destroy — a destroy Job that cannot succeed, and how to remove the object’s finalizer safely. CAPTFDestroyStuck; ApplyJobSucceeded=False/DestroyFailed.
• Unreadable State — a state Secret CAPTF cannot parse. CAPTFStateUnreadable; StateReadable=False/StateCorrupt, StateInconsistent or StateEncrypted.
• State Restore — restoring a Terraform or OpenTofu state from a CAPTF-managed backup. StateReadable=False/StateLost; RestoreJobSucceeded.
• Stale State Lock — clearing a state lock left behind by a killed or evicted runner. CAPTFForceUnlocks; StateReadable=False/StateLocked.
• Size Limits — a state or rendered inputs approaching the Secret size limit. CAPTFStateNearSecretLimit, CAPTFInputsNearLimit; ApplyJobSucceeded=False/InputsTooLarge.
• Slow Jobs — Jobs that take a long time to run, or a long time to start. CAPTFJobSlow, CAPTFJobQueueSlow.
• Reconcile Errors — the controller itself failing to reconcile. CAPTFReconcileErrors.
• Identities and Credentials — an object that cannot resolve or mirror its TerraformClusterIdentity. IdentityAllowed=False, CredentialsMirrored=False.
• Webhook Unavailable — writes to a Terraform* object failing because the admission webhook cannot be reached.
• clusterctl move — moving a Cluster’s Terraform* objects with clusterctl move: the procedure, what does and does not come along, and cleaning up what a move leaves behind.

Every runbook above applies to TerraformCluster, TerraformMachine and TerraformMachinePool alike unless it says otherwise.

See also

• Observability — the metrics and alerts these runbooks are reached from.
• Conditions reference — every condition type and reason named above.