Azure¶

The Azure modules run a Kubernetes cluster on Azure virtual machines. The cluster role creates one resource group per cluster, holding the API server’s Standard load balancer, the nodes’ network security groups, application security groups and managed identities with their role assignments. The machine role creates one Linux VM per Machine, and the machinepool role one virtual machine scale set per MachinePool. The source is captf-io/azure-modules. The modules are pre-release; read the status note in Cloud Modules first.

Images¶

The images carry hashicorp/azurerm 5.7.0, pinned exactly.

Prerequisites¶

• Network. A virtual network with a subnet for the nodes, and optionally a second subnet in the same network for workers, in the subscription the identity works in. Nodes get no public IP, so the subnets need an egress path, such as a NAT gateway or a firewall, to the image registries and Azure’s endpoints. With the default internal API endpoint, the management cluster must reach the control-plane subnet (through peering or a VPN, for example).
• Resource providers registered in the subscription: Microsoft.Compute, Microsoft.Network, Microsoft.ManagedIdentity, Microsoft.Authorization and Microsoft.Insights (autoscale settings). The modules never register them.
• Permissions of the identity’s service principal:
  • Contributor on the subscription: the cluster role creates a resource group, and joins the subnets, which must be in the same subscription. Narrow Contributor, and the network’s resource group needs Network Contributor.
  • Role Based Access Control Administrator on the subscription, best with a condition that limits it to assigning Contributor, Network Contributor and AcrPull: the cluster role grants the node identities.
  • Managed Identity Operator on any node identity you bring, so the machine and pool roles can attach it.
• Node images. A Linux image built for Cluster API, with kubeadm, the kubelet, a container runtime, cloud-init, curl and iptables. The CAPZ reference images work: /communityGalleries/ClusterAPI-f72ceb4f-5159-4c26-a0fe-2ea738f0d019/images/capi-ubun2-2404/versions/{semver}.
• In the workload cluster: cloud-provider-azure (the cloud controller manager and cloud-node-manager), with --configure-cloud-routes=false for an overlay CNI, and the Azure Disk CSI driver for persistent volumes. Both read /etc/kubernetes/azure.json, which the modules write on every node from cloud-config bootstrap data.

Identity Secret¶

The Secret holds the azurerm provider’s environment variables:

For a client certificate, set ARM_CLIENT_CERTIFICATE_PATH to a file under /var/run/captf/credentials/ and ARM_CLIENT_CERTIFICATE_PASSWORD instead of ARM_CLIENT_SECRET. From examples/identity.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: azure
  namespace: captf-system
type: Opaque
stringData:
  ARM_TENANT_ID: 00000000-0000-0000-0000-000000000000
  ARM_SUBSCRIPTION_ID: 00000000-0000-0000-0000-000000000000
  ARM_CLIENT_ID: 00000000-0000-0000-0000-000000000000
  ARM_CLIENT_SECRET: replace-me
  ARM_USE_CLI: "false"
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: TerraformClusterIdentity
metadata:
  name: azure
spec:
  secretRef:
    name: azure
    namespace: captf-system
  allowedNamespaces:
    list:
    - team-a

See Identities and Credentials for how the Secret reaches the Jobs.

Quick start¶

1. Create the identity from the repository’s example:

   export NAMESPACE=team-a AZURE_TENANT_ID=... AZURE_SUBSCRIPTION_ID=... \
     AZURE_CLIENT_ID=... AZURE_CLIENT_SECRET=...
   clusterctl generate yaml --from examples/identity.yaml | kubectl apply -f -
   

2. Generate the cluster from examples/cluster-kubeadm.yaml: a KubeadmControlPlane of three, a MachineDeployment of two and a MachinePool of two. Quote the image ID so the shell keeps the braces:

   export CLUSTER_NAME=demo KUBERNETES_VERSION=v1.34.1
   export AZURE_SUBNET_ID=/subscriptions/<id>/resourceGroups/network/providers/Microsoft.Network/virtualNetworks/hub/subnets/nodes
   export AZURE_SSH_PUBLIC_KEY="$(cat ~/.ssh/id_ed25519.pub)"
   export AZURE_IMAGE_ID='/communityGalleries/ClusterAPI-f72ceb4f-5159-4c26-a0fe-2ea738f0d019/images/capi-ubun2-2404/versions/{semver}'
   clusterctl generate yaml --from examples/cluster-kubeadm.yaml | kubectl apply -n "$NAMESPACE" -f -
   

3. Once the first control-plane node answers, install a CNI (for example Calico with VXLAN) and cloud-provider-azure in the workload cluster. cloud-provider-azure sets each Node’s provider ID; only then do the Machines get their Nodes and the workers join.

The KubeadmConfigs in the example name each Node after its VM (nodeRegistration.name: '{{ ds.meta_data["local_hostname"] }}'), which cloud-provider-azure needs to find the VM.

API endpoint¶

• Internal (the default): a Standard load balancer frontend in the control-plane subnet, zone-redundant where the region has zones, with a dynamic address unless api_load_balancer_private_ip sets one.
• Public (api_load_balancer_public = true): a static Standard public IP. Include the management cluster’s egress and the nodes’ NAT gateway addresses in api_allowed_cidrs: the load balancer keeps the client’s source address, so the control plane’s security group sees them.
• Probes. HTTPS GET /readyz every 5 seconds with kubeadm; TCP with distribution = "rke2", which may disable anonymous access, and on the supervisor port 9345.
• Hairpin. An Azure internal load balancer drops a flow from a backend to its own frontend when it maps the flow back to that backend. The machine role therefore installs a small service on control-plane nodes (cloud-config bootstrap data only): while the node’s own API server answers /readyz, an iptables rule sends the node’s traffic for the frontend to it directly; otherwise the rule is gone and the load balancer takes the traffic to another control-plane node, since this node’s probe is down too. With RKE2 it covers port 9345 as well. Turn it off with api_server_hairpin_workaround = false; with Ignition, add the equivalent to your bootstrap configuration.
• The endpoint guard records api_load_balancer_public, the endpoint port, the control-plane subnet and the private address the frontend got, and fails any later plan that changes one (see Shared Behavior). Making the current dynamic address static is allowed, since it moves nothing.

Exports¶

The cluster role’s exports carry the schema captf.io/azure-cluster/v1.

Nothing in the exports is secret: the cloud provider authenticates with the nodes’ managed identities.

Tags¶

Azure tag names cannot contain /, so captf.io/<key> becomes captf.io_<key> (see Shared Behavior). Tag names are case-insensitive, so additional_tags rejects any key that starts with captf.io_ or captf.io/ in any case, and takes at most 44 tags: Azure allows 50 per resource. Values longer than 256 characters keep 247, then - and 8 hex characters of their sha256.

Not taggable: role assignments, security rules, load balancer backend pools, probes and rules, NIC associations, the OS disk Azure creates with a VM, and the instances, NICs and OS disks Azure creates for a scale set. All of them live in the cluster’s resource group, which is tagged.

The evidence for each is in DESIGN.md.