Events¶
The controller records an event when something happens to an object, once per transition or occurrence rather than on every reconcile. Read them with:
Most Warning events repeat what a condition already says, at the moment it changed, and name the Job involved. Use them for the timeline: what happened first. The Events reference is the generated list; this page adds what to do.
Events expire; conditions do not
Kubernetes keeps events for a limited time (one hour by default), so the condition is the record to rely on. Messages never carry credentials, variable values, output values or raw stderr, and are cut at 512 bytes.
Warning events¶
| Reason | Emitted on | When | What to do | See |
|---|---|---|---|---|
ConditionChanged | TerraformCluster, TerraformMachine, TerraformMachinePool | Any other owned condition changed. It is a Warning when the condition enters its bad state, Normal otherwise. The note reads <Type>: <Status>/<Reason>: <message>. | Find the type and reason in the conditions table. | Conditions |
DestructivePlanBlocked | TerraformCluster, TerraformMachinePool | An apply stopped before a plan that deletes or replaces resources: a cluster apply, or a pool apply of a change of the cluster’s exports. Once per blocked Job, in place of JobFailed. | Review the plan summary in the condition and approve the hash it names, or change the inputs. A blocked pool change is held while the pool keeps applying its last exports. | Destructive-plan guard |
DigestUnknown | TerraformCluster, TerraformMachine, TerraformMachinePool | No image digest is pinned, so an operation runs the spec’s image reference, or a Job succeeded without a readable digest. | Usually clears after the next successful apply. Pin the image by digest in the spec if you need it fixed. | Job inputs |
DriftDetected | TerraformCluster, TerraformMachine, TerraformMachinePool | A drift check found a difference. Once per finding. | Decide whether to accept it or remediate. | Drift |
ForceUnlocked | TerraformCluster, TerraformMachine, TerraformMachinePool | A Job was started with a stale state lock to force-unlock: its holder pod no longer exists. | Nothing, unless it repeats. Frequent unlocks mean runners are being killed: check evictions and memory limits. | The state lock |
IdentityNotAllowed | TerraformCluster, TerraformMachine, TerraformMachinePool | IdentityAllowed entered False, whatever the reason: IdentityNotFound, NamespaceNotAllowed or SecretNotFound. Again when the reason or message changes. | Read the condition’s reason and fix it. | Identity runbook |
IdentitySecretNotFound | TerraformClusterIdentity | The identity’s credentials Secret went missing. | Recreate the Secret at spec.secretRef. | Identity runbook |
ImageInspectFailed | TerraformMachineTemplate | The registry could not be read for the template’s capacity labels. | Fix the image reference or the registry credentials. | Templates |
InfrastructureAbandoned | TerraformCluster, TerraformMachine, TerraformMachinePool | A deletion was released by captf.io/abandon-infrastructure without a destroy. The note names the cause. | Clean up the cloud resources: they keep running and are untracked. | Held deletions |
InstanceUnhealthy | TerraformCluster, TerraformMachine, TerraformMachinePool | InfrastructureHealthy became False for an unhealthy, degraded, stopped or terminated instance. | Look at the instance. Remediation can replace it. | Machine remediation |
JobDeadlineExceeded | TerraformCluster, TerraformMachine, TerraformMachinePool | A Job hit activeDeadlineSeconds. Once per Job. | Raise the deadline or find what hangs. | Deadlines |
JobFailed | TerraformCluster, TerraformMachine, TerraformMachinePool | A Job failed, or an apply or destroy could not start (ApplyJobSucceeded is False without a Job). Once per Job. | Read the condition and the Job’s logs. | Failing Jobs |
JobInterrupted | TerraformCluster, TerraformMachine, TerraformMachinePool | A Job was stopped from outside: a drain, an eviction or a deletion. It retries without backoff. | Nothing, unless it repeats: find what keeps stopping the pod. | Retries |
OutputsInvalid | TerraformCluster, TerraformMachine, TerraformMachinePool | OutputsValid entered False: OutputsMissing, OutputsInvalid, FailureDomainMismatch or ProviderIDChanged. Again when the reason or message changes. | Fix the output named in the condition. | Module contract |
PlanChanged | TerraformCluster | An approved apply planned other changes and stopped before applying them. Once per such Job. | Review the new plan and approve it. | Manual plan approval |
RemediationRequested | TerraformMachine | The owner Machine was annotated with cluster.x-k8s.io/remediate-machine. | Expected with a MachineHealthCheck. Check why the instance is unhealthy. | Machine remediation |
ReplicasManagedExternally | TerraformMachinePool | Valid autoscaler annotations, but another controller owns spec.replicas. | Pick one owner for the replicas. | Machine pools |
StateLocked | TerraformCluster, TerraformMachine, TerraformMachinePool | The state lock is held by something other than the object’s runner. | Release it with force-unlock once the holder is gone. | Stale lock |
StateLost | TerraformCluster, TerraformMachine, TerraformMachinePool | A provisioned object’s state is gone or carries no inputs hash. | Restore a backup. | Unreadable state |
StateRestoreFailed | TerraformCluster, TerraformMachine, TerraformMachinePool | A restore Job failed. It is not retried for the same serial. | Read the Job’s logs, then retry with a fresh annotation. | Other manual actions |
StateUnreadable | TerraformCluster, TerraformMachine, TerraformMachinePool | The state could not be read: corrupt, encrypted or inconsistent. | See the StateReadable reason. | Unreadable state |
StuckJobDeleted | TerraformCluster, TerraformMachine, TerraformMachinePool | A Job that could never start, because its per-run Secret was missing and no pod started, was deleted. It starts again. | Nothing, unless it repeats: check quota and API errors in the manager’s log. | Naming and adoption |
StepFailed | TerraformCluster, TerraformMachine, TerraformMachinePool (runner) | A runtime step failed. The note carries the runner’s curated summary, never raw stderr. Needs --runner-events. | Read the Job’s logs for the full output. | Failing Jobs |
RunFinished | TerraformCluster, TerraformMachine, TerraformMachinePool (runner) | The run ended. A Warning when it did not succeed: failed, interrupted, blocked before a destructive plan, or stopped because the approved plan changed. Needs --runner-events. | As for StepFailed. | Failing Jobs |
Informational events¶
These are Normal and need no action. They are the record of what the controller did.
| Reason | Emitted on | Meaning | See |
|---|---|---|---|
JobCreated | TerraformCluster, TerraformMachine, TerraformMachinePool | A Job started: op, attempt, image and why. | Jobs |
JobSucceeded | TerraformCluster, TerraformMachine, TerraformMachinePool | An apply, destroy, refresh or drift Job succeeded. | Jobs |
WaitingForRunLease | TerraformCluster, TerraformMachine, TerraformMachinePool | An operation waits for the run lease. Once per wait. | Leases |
WaitingForClusterOperation | TerraformMachine, TerraformMachinePool | An operation waits for its cluster’s. | Leases |
WaitingForMachineOperations | TerraformCluster | An operation waits for its machines’ and pools’. | Leases |
DestructivePlanApprovalConsumed | TerraformCluster | The approved destructive apply succeeded and the annotation was removed. | Destructive-plan guard |
PlanReady | TerraformCluster | A plan Job planned a change under Manual: counts, hash and the approve command. | Manual plan approval |
PlanApproved | TerraformCluster | The apply of an approved plan started. | Manual plan approval |
PlanApplied | TerraformCluster | The approved plan was applied and the annotation removed. | Manual plan approval |
DeletionStarted | TerraformCluster, TerraformMachine, TerraformMachinePool | The first reconcile with a deletion timestamp. | Deletion and Teardown |
Destroyed | TerraformCluster, TerraformMachine, TerraformMachinePool | The destroy succeeded and cleanup ran. | Cleanup |
FinalizerRemoved | TerraformCluster, TerraformMachine, TerraformMachinePool | The finalizer was removed: after a destroy, with no state, or without an owner. | Order and finalizers |
Paused | TerraformCluster, TerraformMachine, TerraformMachinePool | The Paused condition became True. | Order and finalizers |
Resumed | TerraformCluster, TerraformMachine, TerraformMachinePool | The Paused condition became False again. | Lifecycle |
Provisioned | TerraformCluster, TerraformMachine, TerraformMachinePool | status.initialization.provisioned latched true. | Lifecycle |
ProviderIDSet | TerraformMachine, TerraformMachinePool | spec.providerID was written. | Machine role |
ControlPlaneEndpointSet | TerraformCluster | spec.controlPlaneEndpoint was written from the module output. | Cluster role |
FailureDomainsChanged | TerraformCluster | status.failureDomains changed. | Cluster role |
InputsChanged | TerraformCluster, TerraformMachine, TerraformMachinePool | The inputs hash differs from the state’s and an apply starts. | Choosing the operation |
DigestPinned | TerraformCluster, TerraformMachine, TerraformMachinePool | An image digest was recorded, or re-pinned after an apply. | Job inputs |
StateAdopted | TerraformCluster, TerraformMachine, TerraformMachinePool | The state a successful apply wrote was adopted with its inputs hash. | Terraform State |
StateBackedUp | TerraformCluster, TerraformMachine, TerraformMachinePool | A new state serial was copied into a backup. | Backups |
StateRestored | TerraformCluster, TerraformMachine, TerraformMachinePool | A restore Job pushed a backup and the annotation was removed. | State restore |
DriftResolved | TerraformCluster, TerraformMachine, TerraformMachinePool | DriftDetected went from True to False. | Drift |
DriftRemediationStarted | TerraformCluster, TerraformMachine, TerraformMachinePool | An apply remediating drift started. | Drift |
InstanceHealthy | TerraformCluster, TerraformMachine, TerraformMachinePool | InfrastructureHealthy became True. | Drift and health |
RemediationWithdrawn | TerraformMachine | The instance read healthy again and the remediation annotation was removed. | Machine remediation |
ReplicasWrittenBack | TerraformMachinePool | An autoscaled pool’s observed replicas were written to MachinePool.spec.replicas. | Machine pools |
IdentitySecretFound | TerraformClusterIdentity | The credentials Secret appeared. | Identities |
MirrorCreated | TerraformCluster, TerraformMachine, TerraformMachinePool | The namespace’s credential mirror was created for this object. | Identities |
OwnerReferencesRepaired | TerraformCluster, TerraformMachine, TerraformMachinePool | Secrets of this object (state, backups, durable inputs, plan key, its mirror entry) were owned by it again after a restore, or because a chunk had no owner reference. | State adoption |
MirrorRemoved | TerraformCluster, TerraformMachine, TerraformMachinePool | The mirror was deleted: its last user went, or the identity no longer allows the namespace. | Cleanup |
CapacityResolved | TerraformMachineTemplate | A template’s capacity or node info changed from its image labels. | Templates |
RunStarted | TerraformCluster, TerraformMachine, TerraformMachinePool (runner) | The runtime is ready and the first step is about to run. | Job Environment |
StepStarted | TerraformCluster, TerraformMachine, TerraformMachinePool (runner) | A runtime step started. | Job Environment |
StepSucceeded | TerraformCluster, TerraformMachine, TerraformMachinePool (runner) | A runtime step finished. | Job Environment |
PlanSummary | TerraformCluster, TerraformMachine, TerraformMachinePool (runner) | A plan the runner parsed: counts only. | Job Environment |
ResourcesChanged | TerraformCluster, TerraformMachine, TerraformMachinePool (runner) | What an apply or destroy step changed: counts only. | Job Environment |