Required configuration

The cluster configuration file can be generated by using clusterctl generate cluster command. This command actually uses a template file and replaces the values surrounded by ${} with environment variables. You have to set all required environment variables in advance. The following sections explain some more details about what should be configured.

Note: You can also use template files by manually replacing values in copies of the template file.

clusterctl generate cluster capi-quickstart \
  --kubernetes-version v1.21.3 \
  --control-plane-machine-count=1 \
  --worker-machine-count=1 \
  > capi-quickstart.yaml

Note: additional template files are provided, offering capabilities beyond the default template file. These can be utilized via the clusterctl --flavor parameter. Additional environment variables are often required by these templates. See clusterctl documentation for further details about flavors.

In order to fetch the configuration parameters via the terminal, please install cmk and jq

Cluster Level Configurations

These configurations are passed while defining the CloudStackCluster and apply to the entire cluster.

Failure Domains

The Cluster API Provider offers high-availability clusters deployed across multiple Failure Domains, each of which is a diverse set of infrastructure that doesn’t share required dependencies (such as power source, networking provider) with other failure domains. In the face of a dependency failure, only cluster members deployed on the failed failure domain will themselves fail. The remaining cluster members will continue operation, and the cluster manager will likely restore capacity by allocating replacement cluster members in the surviving failure domains.

CloudStack implements this concept with Zones. The Cluster API Provider CloudStack enables end-users to specify the CloudStack Zone for each Failure Domain, as well as the particular network of this Zone to use.

The templates provided with ClusterAPI Provider CloudStack only define a single failure domain, mapped to a specific CloudStack Zone and a specific network on that CloudStack Zone. The below parameters allow the end user to define these. No additional parameters are required to use these templates.

ClusterAPI Provider CloudStack additionally supports optional attributes beyond these, allowing each Failure Domain to use a distinct Domain/Account or CloudStack management endpoint. Note that these additional attributes are not utilized in the provided templates.

Advanced users can define cluster templates manually that declare additional Failure Domains, and can utilize the additional failure domain attributes supported by ClusterAPI Provider CloudStack. See the failure domain API definition for more details.

Zone

The Zone must be declared via an environment variable CLOUDSTACK_ZONE_NAME and is a mandatory parameter. As of now, only advanced zones without security groups is supported.

The list of zones can be fetched using the cmk cli as follows :

cmk list zones listall=true | jq '.zone[] | {name, id}'

Network

The network must be declared as an environment variable CLOUDSTACK_NETWORK_NAME and is a mandatory parameter. As of now, only isolated and shared networks are supported.

If the specified network does not exist, a new isolated network will be created. The newly created network will have a default egress firewall policy that allows all TCP, UDP and ICMP traffic from the cluster to the outside world.

The list of networks for the specific zone can be fetched using the cmk cli as follows :

cmk list networks listall=true zoneid=<zoneid> | jq '.network[] | {name, id, type}'

CloudStack Endpoint Credentials Secret (optional for provided templates when used with provided getting-started process)

A reference to a Kubernetes Secret containing a YAML object containing credentials for accessing a particular CloudStack management endpoint. The YAML object is of the form:

         api-url: <cloudstackApiUrl>
         api-key: <cloudstackApiKey>
         secret-key: <cloudstackSecretKey>
         verify-ssl: true|false

Optional environment Variables CLOUDSTACK_FD1_SECRET_NAME and CLOUDSTACK_FD1_SECRET_NAMESPACE allow the end-user to override the template’s default settings, utilizing a differently named secret.

CloudStack Failure Domain Name (optional for provided templates)

When using multiple Failure Domains each requires a distinct name. The provided templates do not configure multiple Failure Domains, and only utilize a default name. This name can be changed by defining environment variable CLOUDSTACK_FD1_NAME. Doing so has no effect on the operation of clusters defined with the templates. This option is included mainly to convey the need and mechanism for naming failure domains when multiple failure domains are defined via custom-authored templates.

Cluster Endpoint

The endpoint of the workload cluster that will be provisioned. It can either be an IP or an FQDN, resolvable from the management cluster.

If on an isolated network, and the endpoint is an IP, it must be an IP in the Public IP range. The necessary Firewall and LoadBalancing rules will be automatically created on Apache CloudStack for the specified IP.

If on a shared network, and the endpoint is an IP, it must belong to the shared network range and not allocated to any other resource on CloudStack.

The Endpoint is exposed in two parts, as the CLUSTER_ENDPOINT_IP and CLUSTER_ENDPOINT_PORT environment variables. CLUSTER_ENDPOINT_PORT is optional, and defaults to 6443.

The list of Public IPs for the specific zone can be fetched using the cmk cli as follows :

cmk list publicipaddresses listall=true zoneid=<zone-id> forvirtualnetwork=true allocatedonly=false | jq '.publicipaddress[] | select(.state == "Free" or .state == "Reserved") | .ipaddress'

Machine Level Configurations

These configurations are passed while defining the CloudStackMachine. They can differ based on the MachineSet mapped to it.

Service Offerings

The service offerings for the Control Plane Nodes are specified by the CLOUDSTACK_CONTROL_PLANE_MACHINE_OFFERING environment variable. It must have a minimum of 2GB RAM and 2 vCPU

The service offerings for the Worker Nodes are specified by the CLOUDSTACK_WORKER_MACHINE_OFFERING environment variable.

The list of Service offerings for the specific zone can be fetched using the cmk cli as follows :

cmk list serviceofferings listall=true zoneid=<zone-id> cpunumber=2 memory=2048 | jq '.serviceoffering[] | {name, id}'

Virtual Machine Template

We currently depend on an up-to-date version of cloud-init otherwise the operating system choice is yours. The kubeadm bootstrap provider we’re using also depends on some pre-installed software like a container runtime, kubelet, kubeadm, etc. For an examples how to build such an image take a look at CloudStack CAPI Images.

Prebuilt images can be found here

The image can be referenced by exposing it as an environment variable CLOUDSTACK_TEMPLATE_NAME.

The list of Templates for the specific zone can be fetched using the cmk cli as follows :

cmk list templates zoneid=<zone-id> templatefilter=executable | jq '.template[] | {name, id}'

Optional Configurations

Integration with CloudStack CKS

CAPC cluster can be integrated with CloudStack CKS to provide a comprehensive view of the cluster resources on CloudStack. The integration can be enabled by setting the CLOUDSTACK_SYNC_WITH_ACS environment variable to true before generating the cluster or by setting CloudStackCluster.spec.syncWithACS to true in the cluster definition yaml.

Project

CAPC cluster can be deployed within a specific project in CloudStack. The project name can be specified by adding the CloudStackCluster.spec.project field in the yaml specification.

The list of projects can be fetched using the cmk cli as follows :

cmk list projects listall=true | jq '.project[] | {name, id}'

## Cluster Level Configurations

These configurations are passed while defining the `CloudStackCluster` and apply to the entire cluster.

### Account

The account in which the CAPC cluster resources are to be created. Please note that the credentials of the user passed to CAPC via the
`CLOUDSTACK_B64ENCODED_SECRET` must have access the the specified account.

The account can be specified by adding the `CloudStackCluster.spec.account` field in the yaml specification

The list of accounts can be fetched using the cmk cli as follows :

cmk list accounts listall=true | jq ‘.account[] | {name, id}’

> Please note that if the optional configurations of account and domainid are passed,
> the corresponding account must have access to the specified resources on CloudStack such as the
> Network, Public IP, VM Template, Service Offering, SSH Key, Affinity Group, etc

### Domain

The domain / subdomain in which the CAPC cluster resources are to be created. Please note that the credentials of the user passed to CAPC via the
`CLOUDSTACK_B64ENCODED_SECRET` must have access the the specified domain

The domain can be specified by adding the `CloudStackCluster.spec.domain` field in the yaml specification

The list of domains can be fetched using the cmk cli as follows :

cmk list domains listall=true | jq ‘.domain[] | {name, id, path}’

> Please note that if the optional configurations of account and domainid are passed,
> the corresponding account must have access to the specified resources on CloudStack such as the
> Network, Public IP, VM Template, Service Offering, SSH Key, Affinity Group, etc

## Machine Level Configurations

These configurations are passed while defining the `CloudStackMachine`. They can differ based on the MachineSet mapped.

### SSH KeyPair

The SSH key pair must be an existing key registered in Apache CloudStack and is exposed as the environment variable `CLOUDSTACK_SSH_KEY_NAME`.

The ssh keypair can be specified by adding the `CloudStackMachine.spec.sshKey` field in the yaml specification

The list of SSH Keypairs can be fetched using the cmk cli as follows :

cmk list sshkeypairs listall=true | jq ‘.sshkeypair[] | {name, id}’

If the user wishes to pass a public key not registered in CloudStack directly to the node, it can be done by adding the `KubeadmConfigTemplate.spec.template.spec.users`
spec in the cluster definition yaml. Eg:

```yaml
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: ${CLUSTER_NAME}-md-0
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            provider-id: "cloudstack:///'{{ ds.meta_data.instance_id }}'"
          name: '{{ local_hostname }}'
      users:
        - name: ${OS_USERID}
          sshAuthorizedKeys:
            - ${SSH_KEY_MATERIAL}
          sudo: ALL=(ALL) NOPASSWD:ALL

To learn how to configure the required network access in order to SSH into the node, please see here

Affinity Groups

The nodes in the MachineDeployment mapped to a corresponding CloudStackMachine can have a specific host affinity or be assigned to affinity groups. The affinity can either be specified pro (host affinity) or anti (host anti affinity) in the CloudStackMachine.spec.affinity field in the yaml specification and the required affinity groups will be created in CloudStack If existing affinity groups in CloudStack wish to be used, the group IDs can be passed as a list in the CloudStackMachine.spec.affinitygroupids field in the yaml specification

The list of existing affinity groups can be fetched using the cmk cli as follows :

cmk list affinitygroups listall=true | jq '.affinitygroup[] | {name, id}'

VM Details

These are arbitrary key value pairs which are passed as VM details while deploying the nodes.

The VM details can be specified by adding the CloudStackMachine.spec.details field in the yaml specification

Log level

TODO / Maybe add feature ?

Timeout settings

TODO / Add feature

Apache CloudStack Credentials

1. Generate the API and Secret Key for your Apache CloudStack instance either via the UI (Accounts > User > API Key) or the getUserKeys API.

2. Create a file named cloud-config, substituting in your own environment’s values

   [Global]
   api-url = <cloudstackApiUrl>
   api-key = <cloudstackApiKey>
   secret-key = <cloudstackSecretKey>
   

3. Run the following command to save the above Apache CloudStack connection info into an environment variable :

   export CLOUDSTACK_B64ENCODED_SECRET=$(base64 -w0 -i cloud-config)
   

4. Once exported, the management cluster can be initialized with Apache CloudStack as the underlying infrastructure provider

   clusterctl init --infrastructure cloudstack