Add fully-baked bootstrapping images to TCE-based cluster creations

[joshrosso]

Feature Request

When creating a cluster (standalone or otherwise) TCE/TKG first creates a bootstrap cluster. This cluster is created using a kind image that comes pre-baked with images such as kube-apiserver, kube-controller-manager, and etcd. To bootstrap a cluster, we also need to pull the following images.

projects.registry.vmware.com/tkg/cert-manager-controller
projects.registry.vmware.com/tkg/cluster-api/cluster-api-aws-controller
projects.registry.vmware.com/tkg/cluster-api/cluster-api-vsphere-controller
projects.registry.vmware.com/tkg/cluster-api/cluster-api-docker-controller
projects.registry.vmware.com/tkg/cluster-api/kubeadm-bootstrap-controller
projects.registry.vmware.com/tkg/cluster-api/kubeadm-control-plane-controller
projects.registry.vmware.com/tkg/cluster-api/kube-rbac-proxy
projects.registry.vmware.com/tkg/cert-manager-cainjector
projects.registry.vmware.com/tkg/cert-manager-webhook

This can add non-trivial time to the bootstrapping process. Rather than downloading those images every bootstrap, we should pre-bake bootstrapping images such that they are ready to go for cluster creation.

Describe alternatives you've considered

Doing a partial bake to not include luster-api-${PROVIDER}-controller. The proposed appraoch does unecessarily add images fo providers that will not bein use. However, for simplicity, I believe the extra size of these controller is worth it for the trade-off of 1 universal image.

[joshrosso]

I have implemented this functionality in a kind fork. To build the image, you can:

1. Clone the fork

2. cd into the fork's directory

3. Run `make build

4. Add a YAML file containing any additional images.

   preload:
     images:
       - "projects.registry.vmware.com/tkg/cert-manager-controller:v0.16.1_vmware.1"
       - "projects.registry.vmware.com/tkg/cluster-api/cluster-api-aws-controller:v0.6.4_vmware.1"
       - "projects.registry.vmware.com/tkg/cluster-api/kubeadm-bootstrap-controller:v0.3.14_vmware.2"
       - "projects.registry.vmware.com/tkg/cluster-api/kubeadm-control-plane-controller:v0.3.14_vmware.2"
       - "projects.registry.vmware.com/tkg/cluster-api/kube-rbac-proxy:v0.4.1_vmware.2"
       - "projects.registry.vmware.com/tkg/cert-manager-cainjector:v0.16.1_vmware.1"
       - "projects.registry.vmware.com/tkg/cert-manager-webhook:v0.16.1_vmware.1"

5. Building the image by running

   bin/kind build node-image \
     --kube-root=/home/josh/d/kubernetes \
     --preload-file hack/preload.yaml

   • replace --kube-root with the location of the kubernetes/kubernetes repo
   • replace --preload-file with the location of the aforementioned YAML file

[joshrosso]

We are involved in an upstream discussion around how to solve this problem in kind so that we do not need to build from a fork.

kubernetes-sigs/kind#2160

[jorgemoralespou]

@joshrosso Is there information about the size that the specific layers for each provider add? It would be good to have that information as part of the decision taking of having one single universal image (which I initially prefer as well).