Building windows custom machine image for creating Tanzu Kubernetes Grid workload clusters

Feb 27, 2022 Kubernetes TKG Windows Bring Your Own Image BYOI Workload Clusters
Share on:

In this post, we will delve into the process of building windows custom machine images which can later be used to create Tanzu Kubernetes Grid workload clusters. The instructions assume that you have already reviewed the pre-requisites.

High level the process:

Download Windows ISO

We will use a windows evaluation ISO from Microsoft Eval Center. It is neither recommended nor supported approach. It is used here for demo purposes only.

A recent (newer than April 2021) Windows Server 2019 ISO image. Download through your Microsoft Developer Network (MSDN) or Volume Licensing (VL) account. The use of evaluation media is not supported or recommended.

Download VMware Tools

Download the latest version from VMware Tools.

1vmware_tools_iso_url='https://packages.vmware.com/tools/releases/latest/windows/VMware-tools-windows-11.3.5-18557794.iso'
2wget -nv ${vmware_tools_iso_url}

Install govc

Install and setup govc

1govc_url="https://github.com/vmware/govmomi/releases/download/v0.24.0/govc_linux_amd64.gz"
2wget -nv ${govc_url}
3gunzip govc_linux_amd64.gz
4chmod +x govc_linux_amd64
5sudo mv govc_linux_amd64 /usr/local/bin/govc

Configure Environment Variables

Setting these up is helpful as these values are used multiple times throughout the instructions

 1export VMWARE_TOOLS_ISO_NAME='VMware-tools-windows-11.3.5-18557794.iso'
 2export WINDOWS_ISO_NAME='17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso'
 3export GOVC_URL=10.190.99.201
 4export GOVC_USERNAME=administrator@vsphere.local # vCenter username
 5export GOVC_PASSWORD=VMware\!23 # vCenter password
 6export GOVC_DATACENTER=Datacenter
 7export GOVC_CLUSTER=Cluster
 8export GOVC_DATASTORE=vsanDatastore
 9export GOVC_INSECURE=1
10export GOVC_NETWORK='VM Network'

You can confirm if govc is setup correctly by using govc about.

Upload Windows ISO and VMware tools ISO to the datastore

This operation can take several minutes to complete as it uploads the Windows and VMware Tool ISOs to the datastore

1govc datastore.mkdir iso
2govc datastore.upload ${HOME}/${WINDOWS_ISO_NAME} iso/${WINDOWS_ISO_NAME}
3govc datastore.upload ${HOME}/${VMWARE_TOOLS_ISO_NAME} iso/${VMWARE_TOOLS_ISO_NAME}

Deploy image-builder-resource-kit K8s deployment to management clusters

In this step we will apply the yaml below to our management clusters. This in turn creates a namespace, service and a deployment which hosts the binaries needed to Kubernetify your windows images. I really like this method of serving binaries as this reduces the effort that a user would have to spend getting these from multiple resources, verifying compatibility etc.

On my local system I have saved this yaml as builder.yaml

 1apiVersion: v1
 2kind: Namespace
 3metadata:
 4 name: imagebuilder
 5---
 6apiVersion: v1
 7kind: Service
 8metadata:
 9 name: imagebuilder-wrs
10 namespace: imagebuilder
11spec:
12 selector:
13   app: image-builder-resource-kit
14 type: NodePort
15 ports:
16 - port: 3000
17   targetPort: 3000
18   nodePort: 30008
19---
20apiVersion: apps/v1
21kind: Deployment
22metadata:
23 name: image-builder-resource-kit
24 namespace: imagebuilder
25spec:
26 selector:
27   matchLabels:
28     app: image-builder-resource-kit
29 template:
30   metadata:
31     labels:
32       app: image-builder-resource-kit
33   spec:
34     nodeSelector:
35       kubernetes.io/os: linux
36     containers:
37     - name: windows-imagebuilder-resourcekit
38       image: projects.registry.vmware.com/tkg/windows-resource-bundle:v1.22.5_vmware.1-tkg.1
39       imagePullPolicy: Always
40       ports:
41         - containerPort: 3000

To deploy the objects that we discussed above, switch to the management cluster context

1# Switch context to management cluster
2kubectl config use-context mgmt-e-admin@mgmt-e
3
4kubectl apply -f builder.yaml

Sample output

 1kubectl config use-context mgmt-e-admin@mgmt-e
 2Switched to context "mgmt-e-admin@mgmt-e".
 3
 4kubectl apply -f builder.yaml
 5namespace/imagebuilder created
 6service/imagebuilder-wrs created
 7deployment.apps/image-builder-resource-kit created
 8
 9# Verify that image-builder-resource-kit deployment is running
10kubectl get all -n imagebuilder
11NAME                                              READY   STATUS    RESTARTS   AGE
12pod/image-builder-resource-kit-7f5cbc8cc9-6tc72   1/1     Running   0          86s
13
14NAME                       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
15service/imagebuilder-wrs   NodePort   100.71.222.48   <none>        3000:30008/TCP   86s
16
17NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
18deployment.apps/image-builder-resource-kit   1/1     1            1           87s
19
20NAME                                                    DESIRED   CURRENT   READY   AGE
21replicaset.apps/image-builder-resource-kit-7f5cbc8cc9   1         1         1       87s

Set NODE_IP, this is used for validation and later used when configuring windows.json

1# Used to configure windows.json later
2export NODE_IP=$(kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}' --selector='!node-role.kubernetes.io/master' | awk '{print $1}')

You can also verify if the deployment is ready to serve the needed binaries

 1curl $NODE_IP:30008
 2
 3{
 4    "containerd": {
 5        "version": "v1.5.9+vmware.1",
 6        "path": "files/containerd/cri-containerd-v1.5.9+vmware.1.windows-amd64.tar",
 7        "sha256": "0aa0df1c3d6c545b360e75fa4d20f604e1bb2ccd05b1651dc92941fdb4b49587"
 8    },
 9    "antrea-windows": {
10        "version": "v1.2.3+vmware.4-advanced",
11        "path": "files/antrea-windows/antrea-windows-advanced.zip",
12        "sha256": "5ced2e68f7ebaf79d56e12371cdb4734c3fa5300c1536630b095079a83b2c7c8"
13    },
14    "kubelet": {
15        "version": "v1.22.5+vmware.1",
16        "path": "files/kubernetes/kubelet.exe",
17        "sha256": "d938ebc7232ba4535aa69be5227848c43daf63000b506626aba0e57580b24f70"
18    },
19    "kubeadm": {
20        "version": "v1.22.5+vmware.1",
21        "path": "files/kubernetes/kubeadm.exe",
22        "sha256": "17e06499a8bc395a5539062a281acdbcbcc72e2cbf999aff8942e8767c582379"
23    },
24    "kubectl": {
25        "version": "v1.22.5+vmware.1",
26        "path": "files/kubernetes/kubectl.exe",
27        "sha256": "742eff490ca60536f0135652a78f2be90be8736eaff81a91bee64066773b0fd6"
28    },
29    "kube-proxy": {
30        "version": "v1.22.5+vmware.1",
31        "path": "files/kubernetes/kube-proxy.exe",
32        "sha256": "3d55bb854aee5eb149cc936bf07cf9a5a013b6317debdce4eae149e30b2cb3c0"
33    }
34}

Configure windows.json

If you have been following the steps in this post than the only thing that needs update in the json file below is the property unattend_timezone. Please choose a value that suits your setup.

 1cat <<EOF > $HOME/windows.json
 2{
 3  "unattend_timezone": "UTC",
 4  "windows_updates_categories": "CriticalUpdates SecurityUpdates UpdateRollups",
 5  "kubernetes_semver": "v1.22.5",
 6  "cluster": "${GOVC_CLUSTER}",
 7  "template": "",
 8  "password": "${GOVC_PASSWORD}",
 9  "folder": "",
10  "runtime": "containerd",
11  "username": "${GOVC_USERNAME}",
12  "datastore": "${GOVC_DATASTORE}",
13  "datacenter": "${GOVC_DATACENTER}",
14  "convert_to_template": "true",
15  "vmtools_iso_path": "[${GOVC_DATASTORE}] iso/${VMWARE_TOOLS_ISO_NAME}",
16  "insecure_connection": "true",
17  "disable_hypervisor": "false",
18  "network": "${GOVC_NETWORK}",
19  "linked_clone": "false",
20  "os_iso_path": "[${GOVC_DATASTORE}] iso/${WINDOWS_ISO_NAME}",
21  "resource_pool": "${GOVC_RESOURCE_POOL}",
22  "vcenter_server": "${GOVC_URL}",
23  "create_snapshot": "false",
24  "netbios_host_name_compatibility": "false",
25  "kubernetes_base_url": "http://$NODE_IP:30008/files/kubernetes/",
26  "containerd_url": "http://$NODE_IP:30008/files/containerd/cri-containerd-v1.5.9+vmware.1.windows-amd64.tar",
27  "containerd_sha256_windows": "0aa0df1c3d6c545b360e75fa4d20f604e1bb2ccd05b1651dc92941fdb4b49587",
28  "pause_image": "mcr.microsoft.com/oss/kubernetes/pause:3.5",
29  "prepull": "false",
30  "additional_prepull_images": "mcr.microsoft.com/windows/servercore:ltsc2019",
31  "additional_download_files": "",
32  "additional_executables": "true",
33  "additional_executables_destination_path": "c:/k/antrea/",
34  "additional_executables_list": "http://$NODE_IP:30008/files/antrea-windows/antrea-windows-advanced.zip",
35  "load_additional_components": "true"
36}
37EOF

Configure autounattend.xml

Download and Update autounattend.xml

1curl https://vdc-download.vmware.com/sampleExchange/v1/downloads/7719 --output autounattend.xml

If you are using evaluation copy remove all occurrences of section ProductKey from autounattend.xml. If this is not done the windows boot will get stuck with the message on the UI No Image Available

Build windows custom machine image

1docker run -it --rm --mount type=bind,source=$(pwd)/windows.json,target=/windows.json \
2--mount type=bind,source=$(pwd)/autounattend.xml,target=/home/imagebuilder/packer/ova/windows/windows-2019/autounattend.xml \
3-e PACKER_VAR_FILES="/windows.json" \
4-e IB_OVFTOOL=1 \
5-e IB_OVFTOOL_ARGS='--skipManifestCheck' \
6-e PACKER_FLAGS='-force -on-error=ask' \
7-e PACKER_LOG=1 \
8-t projects-stg.registry.vmware.com/tkg/image-builder:v0.1.11_vmware.3 \
9build-node-ova-vsphere-windows-2019