Runbooks
Each runbook below covers one symptom: what it looks like, why it happens, how to check, how to fix it, and how to confirm the fix worked. Alerts and condition reasons are cross-references, not a substitute for reading the page: start from whichever alert fired or condition you see, but read the whole runbook before you act.
- Failing Jobs — a
TerraformCluster,TerraformMachineorTerraformMachinePoolwhose Jobs keep failing.CAPTFJobFailing,CAPTFNoRecentSuccess;ApplyJobSucceeded=FalseandDriftJobSucceeded=False. - Stuck Destroy — a
destroyJob that cannot succeed, and how to remove the object’s finalizer safely.CAPTFDestroyStuck;ApplyJobSucceeded=False/DestroyFailed. - Unreadable State — a state Secret CAPTF cannot
parse.
CAPTFStateUnreadable;StateReadable=False/StateCorrupt,StateInconsistentorStateEncrypted. - State Restore — restoring a Terraform or OpenTofu
state from a CAPTF-managed backup.
StateReadable=False/StateLost;RestoreJobSucceeded. - Stale State Lock — clearing a state lock left behind by
a killed or evicted runner.
CAPTFForceUnlocks;StateReadable=False/StateLocked. - Size Limits — a state or rendered inputs approaching
the Secret size limit.
CAPTFStateNearSecretLimit,CAPTFInputsNearLimit;ApplyJobSucceeded=False/InputsTooLarge. - Slow Jobs — Jobs that take a long time to run, or a long
time to start.
CAPTFJobSlow,CAPTFJobQueueSlow. - Reconcile Errors — the controller itself failing
to reconcile.
CAPTFReconcileErrors. - Identities and Credentials — an object
that cannot resolve or mirror its
TerraformClusterIdentity.IdentityAllowed=False,CredentialsMirrored=False. - Webhook Unavailable — writes to a
Terraform*object failing because the admission webhook cannot be reached. - clusterctl move — moving a Cluster’s
Terraform*objects withclusterctl move: the procedure, what does and does not come along, and cleaning up what a move leaves behind.
Every runbook above applies to TerraformCluster, TerraformMachine and
TerraformMachinePool alike unless it says otherwise.
See also
- Observability — the metrics and alerts these runbooks are reached from.
- Conditions reference — every condition type and reason named above.