Runbooks¶
If you already have a condition and a reason, Troubleshooting by Condition looks it up directly.
Each runbook below covers one symptom: what it looks like, why it happens, how to check, how to fix it, and how to confirm the fix worked. Alerts and condition reasons are cross-references, not a substitute for reading the page: start from whichever alert fired or condition you see, but read the whole runbook before you act.
| Runbook | Covers | Alerts and conditions |
|---|---|---|
| Failing Jobs | A TerraformCluster, TerraformMachine or TerraformMachinePool whose Jobs keep failing. | CAPTFJobFailing, CAPTFNoRecentSuccess; ApplyJobSucceeded=False and DriftJobSucceeded=False |
| Stuck Destroy | A destroy Job that cannot succeed, and how to remove the object’s finalizer safely. | CAPTFDestroyStuck; ApplyJobSucceeded=False/DestroyFailed |
| Unreadable State | A state Secret CAPTF cannot parse. | CAPTFStateUnreadable; StateReadable=False/StateCorrupt, StateInconsistent or StateEncrypted |
| State Restore | Restoring a Terraform or OpenTofu state from a CAPTF-managed backup. | StateReadable=False/StateLost; RestoreJobSucceeded |
| Stale State Lock | Clearing a state lock left behind by a killed or evicted runner. | CAPTFForceUnlocks; StateReadable=False/StateLocked |
| Size Limits | A state or rendered inputs approaching the Secret size limit. | CAPTFStateNearSecretLimit, CAPTFInputsNearLimit; ApplyJobSucceeded=False/InputsTooLarge |
| Slow Jobs | Jobs that take a long time to run, or a long time to start. | CAPTFJobSlow, CAPTFJobQueueSlow |
| Reconcile Errors | The controller itself failing to reconcile. | CAPTFReconcileErrors |
| Identities and Credentials | An object that cannot resolve or mirror its TerraformClusterIdentity. | IdentityAllowed=False, CredentialsMirrored=False |
| Webhook Unavailable | Writes to a Terraform* object failing because the admission webhook cannot be reached. | None |
| Total State Loss and Import | The state is gone with no backup: rebuild it from a workstation, or abandon and recreate the object with import blocks. | StateReadable=False/StateLost |
| clusterctl move | Moving a Cluster’s Terraform* objects with clusterctl move: the procedure, what does and does not come along, and cleaning up what a move leaves behind. | None |
For how deletion works and a flowchart from a stuck object to the right runbook, see Deletion and Teardown. For why an object has no Job and is waiting, see Nothing Is Happening.
Every runbook above applies to TerraformCluster, TerraformMachine and TerraformMachinePool alike unless it says otherwise.