Metrics¶

The manager exposes captf_* series that describe Jobs, reconcile decisions, state, drift and health. The shipped alerts are written on them, and the examples below build on the same series. Jump to: Jobs, reconcile decisions, state and inputs, drift and health, locks, leases and approvals, identity and images, build info, controller-runtime series.

Where they are served¶

The metrics are on the manager’s diagnostics endpoint, HTTPS on :8443 (--diagnostics-address), at /metrics. Requests are authenticated with a TokenReview and authorized with a SubjectAccessReview for get on the non-resource URL /metrics; a request without a token gets 401. --insecure-diagnostics serves plain HTTP with no checks and is for local development only. See Observability and Manager flags.

The opt-in config/prometheus component ships a Service (captf-controller-manager-metrics, port 8443, named metrics), a ServiceMonitor that scrapes it over HTTPS with the Prometheus ServiceAccount’s token, and the RBAC that token needs. It is not part of the release manifest; see Enabling the Prometheus component.

Conventions¶

• Metrics are declared with k8s.io/component-base/metrics at stability level ALPHA, so names and labels can change between releases. On the wire, each HELP string starts with [ALPHA].
• Durations are in seconds, sizes in bytes and timestamps in Unix seconds.
• Labels are bounded enums: kind, op, result, reason, step, action and error_kind. Only the eight per-object gauges carry namespace and name, and CAPTF removes a gauge’s series when its object is deleted, so they do not accumulate.
• kind is the CAPTF kind: TerraformCluster, TerraformMachine or TerraformMachinePool. op is the operation: apply, destroy, refresh, drift, plan or restore.
• Failure counters count transitions, not reconciles. They go up once when an object or Job newly enters a bad state, so use increase() to ask whether something newly broke.

Jobs¶

Every runner Job is observed when it completes. A Job is one op for one object; see Jobs.

Label values:

Notes on how each series is built:

• op="plan" is a plan Job under applyPolicy: Manual; it applies nothing.
• blocked and plan_changed are not failures. Each is reported by its own condition and event, and CAPTFJobFailing does not count them.
• captf_job_duration_seconds buckets run from 15 seconds to 2 hours, and captf_job_step_duration_seconds from 1 second to 2 hours.
• captf_job_queue_seconds has buckets from 5 seconds to 30 minutes, with 300 seconds, the CAPTFJobQueueSlow threshold, as a boundary. CAPTF does not record it when the pod reports no start.
• captf_job_attempts is 1 plus the failed Jobs of the op since it last succeeded. Interrupted Jobs do not count.
• captf_resources_changed_total comes from the runtime’s final summary line.

# p95 apply duration by kind over the last day, for Jobs that ran their course
histogram_quantile(0.95, sum by (le, kind) (
  rate(captf_job_duration_seconds_bucket{op="apply",result=~"succeeded|failed"}[1d])))

# Failures by error kind and failing step, last hour
sum by (kind, op, error_kind, step) (increase(captf_job_errors_total[1h])) > 0

# Slowest runner steps (p90)
topk(5, histogram_quantile(0.9, sum by (le, kind, op, step) (
  rate(captf_job_step_duration_seconds_bucket[6h]))))

# Resources destroyed by applies in the last 24 hours
sum by (kind) (increase(captf_resources_changed_total{op="apply",action="destroy"}[24h]))

Reconcile decisions¶

op is none when the reconcile decided to run nothing. reason says why: for example NoState, InputsChanged, DriftDue, DriftRemediation or Deleting start a Job, while UpToDate, JobActive, LastApplyFailedBackoff or DeletingBackoff start nothing.

# Why applies started in the last day
sum by (kind, reason) (increase(captf_reconcile_op_decisions_total{op="apply"}[1d]))

State and inputs¶

The per-object gauges hold the last value the manager read. See State.

Label values:

The Kubernetes backend holds at most 1 MiB per Secret, and no Job starts when the inputs exceed 1000000 bytes. CAPTFStateNearSecretLimit and CAPTFInputsNearLimit warn at 900 KiB and 900000 bytes.

# The ten largest states, as a share of one Secret's 1 MiB
topk(10, captf_state_bytes / (1024 * 1024))

# Objects with unreadable state in the last 15 minutes, by reason
sum by (kind, reason) (increase(captf_state_read_errors_total[15m])) > 0

Drift and health¶

See Drift and health. The gauges use the condition value: 1 for True, 0 for False and -1 for Unknown.

captf_last_success_timestamp_seconds exports drift and refresh only while that op is scheduled: not while the object is deleting or paused, and only with a drift interval or health checks set. A series that goes missing is therefore not a failure by itself.

# Objects that are not Ready
captf_ready == 0

# Objects whose drift check or health refresh is overdue, in hours
(time() - captf_last_success_timestamp_seconds{op=~"drift|refresh"}) / 3600 > 6

Locks, leases and approvals¶

Label values:

See Leases and Approvals.

# Where operations wait, by reason
sum by (kind, reason) (increase(captf_lease_waits_total[1h]))

Identity and images¶

# Identity refusals in the last day
sum by (reason) (increase(captf_identity_denied_total[1d]))

Build info¶

captf_build_info

Controller-runtime and other series¶

The same endpoint serves series that CAPTF does not define:

CAPTF merges its registry with the controller-runtime defaults and the component-base registry, and drops component-base’s own go_* and process_* families because controller-runtime already serves them.

# Reconcile errors per second by controller
sum by (controller) (rate(controller_runtime_reconcile_errors_total{controller=~"terraform.*"}[10m]))