Skip to content

My Object Will Not Delete

An object with a deletionTimestamp that does not go away is waiting on one of a short list of things. Read the object first:

kubectl get <kind> -n <ns> <name> -o yaml

Look at metadata.finalizers (CAPTF’s must be the one holding it), the Deleting condition’s message, DeletionBlocked, StateReadable and ApplyJobSucceeded, then follow the chart.

%%{init: {"themeVariables": {"fontSize": "13px"}, "flowchart": {"nodeSpacing": 28, "rankSpacing": 34, "padding": 10}}}%%
flowchart TD
    S[Object has a deletionTimestamp<br/>and the finalizer stays] --> P{Paused=True?}
    P -- yes --> P1[Unpause the Cluster or remove<br/>the paused annotation]
    P -- no --> D{DeletionBlocked=True?}
    D -- yes --> D1[Delete the machines and pools<br/>it counts, then wait]
    D -- no --> J{A Job is running?}
    J -- yes --> J1[Wait: the destroy starts after it]
    J -- no --> W{ApplyJobSucceeded<br/>is a lease wait?}
    W -- yes --> W1[Wait or inspect the lease holder]
    W -- no --> R{StateReadable<br/>False?}
    R -- "StateLocked" --> R1[Stale lock runbook]
    R -- "Lost, Corrupt,<br/>Encrypted, Inconsistent" --> R2[Restore a backup,<br/>or abandon]
    R -- no --> C{Deleting message says<br/>it waits for credentials?}
    C -- yes --> C1[Fix the named condition,<br/>or abandon]
    C -- no --> F{ApplyJobSucceeded<br/>False?}
    F -- "IdentityNotAllowed" --> F1[Allow the namespace again,<br/>or abandon]
    F -- "DestroyFailed" --> F2[Fix the failure,<br/>or abandon]
    F -- "JobDeadlineExceeded" --> F3[Raise the deadline]
    F -- no --> Z[No condition explains it:<br/>check the manager]

Symptom to action

What you see Meaning Action
Paused=True, or the Cluster has spec.paused: true; Deleting says Deletion waits until the object is unpaused A paused object runs only bookkeeping, so it never destroys; clusterctl move relies on this Unpause. See Order
kubectl delete terraformmachine is refused: delete the Machine <name> instead A live Machine references it Delete the Machine; see Order
DeletionBlocked=True/DependentsExist on a TerraformCluster Machines or pools with its cluster label still exist List them with -l cluster.x-k8s.io/cluster-name=<name>; they delete through their Machines. If one is stuck, work on that object first
A Job is running (status.activeJob) The destroy waits for it Wait; see Slow Jobs
ApplyJobSucceeded=Unknown/WaitingForRunLease, WaitingForClusterOperation or WaitingForMachineOperations The destroy waits for a lease Wait; see Leases
StateReadable=False/StateLost, StateCorrupt, StateEncrypted or StateInconsistent Held on the state Restore or abandon. See Unreadable State
StateReadable=False/StateLocked A foreign holder has the state lock; the destroy Job will wait and fail Stale State Lock
Deleting message The destroy Job waits for its credentials: … Credentials cannot be prepared, often in a terminating namespace Fix the named condition (identities), or abandon. See Terminating namespaces
ApplyJobSucceeded=False/IdentityNotAllowed The identity no longer allows the namespace, or is gone Allow the namespace again, or abandon
ApplyJobSucceeded=False/DestroyFailed with a Job The destroy failed; it retries with backoff forever Failing Jobs, then Stuck Destroy
DestroyFailed, message The durable inputs Secret is missing The destroy cannot be rendered Restore the Secret, or abandon
ApplyJobSucceeded=False/JobDeadlineExceeded The destroy ran out of time Raise activeDeadlineSeconds; see Tuning Jobs
ApplyJobSucceeded=False/ImagePullFailed or ImageInvalid The pinned image cannot run Failing Jobs
Nothing explains it The manager is not reconciling Reconcile Errors, Webhook Unavailable

A message that names the abandon annotation

It tells you the controller sees no way to proceed alone. The abandon page lists exactly which cases the annotation releases, and an object whose state reads and whose destroy can start is destroyed anyway.

Abandoning or stripping leaves cloud resources running

A destroy that cannot be completed and an object that must go anyway end the same way: back up what you need, clean up the cloud resources, then either abandon or strip the finalizer.

See also