kind - local Kubernetes cluster using Docker container nodes

Page content

My environment

  • Pop!_OS
  • $ uname -a
    Linux system76 5.11.0-7620-generic #21~1624379747~20.10~3abeff8-Ubuntu SMP Wed Jun 23 02:23:59 UTC  x86_64 x86_64 x86_64 GNU/Linux
    

Set up pre-requirements for kind

Install latest Go

cd /usr/local/src
sudo curl -L -O https://golang.org/dl/go1.16.5.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.16.5.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
/etc/profile
PATH=/usr/local/go/bin:$PATH

Rootless Docker

Refer to my post.

Install kubectl

https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management

sudo apt install -y apt-transport-https ca-certificates curl
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubectl

# check 
sudo kind create cluster

Install kind

https://kind.sigs.k8s.io/docs/user/quick-start/

cd /usr/local/src
sudo curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.11.1/kind-linux-amd64
sudo chmod +x ./kind
sudo mv ./kind /usr/bin
(as root user)

Troubleshooting 1. cgroup v2

$ kind create cluster
ERROR: failed to create cluster: running kind with rootless provider requires cgroup v2, see https://kind.sigs.k8s.io/docs/user/rootless/

There was no /etc/default/grub in Pop!_OS…

Check whether the system supports cgroup v2.

$ grep cgroup /proc/filesystems
nodev	cgroup
nodev	cgroup2

OK, cgroup2 is supported.

https://github.com/opencontainers/runc/blob/master/docs/cgroup-v2.md

$ sudo apt install -y dbus-user-session
$ systemctl --user start dbus

https://github.com/opencontainers/runc/blob/master/docs/cgroup-v2.md#am-i-using-cgroup-v2

Am I using cgroup v2? Yes if /sys/fs/cgroup/cgroup.controllers is present.

Still no changes…

https://www.kernel.org/doc/html/v5.5/admin-guide/cgroup-v2.html

$ df -h
Filesystem             Size  Used Avail Use% Mounted on
tmpfs                  1.6G  2.0M  1.6G   1% /run
/dev/mapper/data-root  450G  408G   19G  96% /
tmpfs                  7.8G  459M  7.3G   6% /dev/shm
tmpfs                  5.0M  4.0K  5.0M   1% /run/lock
tmpfs                  4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/nvme0n1p1         511M  347M  165M  68% /boot/efi
/dev/nvme0n1p2         4.0G  2.3G  1.8G  56% /recovery
tmpfs                  1.6G  160K  1.6G   1% /run/user/1000

cgroup is mounted on /sys/fs/cgroup

https://wiki.archlinux.org/title/cgroups

/etc/sysctl.conf
systemd.unified_cgroup_hierarchy=1
$ sudo sysctl -p
sysctl: cannot stat /proc/sys/systemd/unified_cgroup_hierarchy: No such file or directory

/boot/efi/EFI/Pop_OS-ee80fe8b-8d46-4f70-816c-b9619e87c840/cmdline

# From
# initrd=\EFI\Pop_OS-ee80fe8b-8d46-4f70-816c-b9619e87c840\initrd.img root=UUID={{ My_UUID }} ro quiet loglevel=0 systemd.show_status=false splash
# To
initrd=\EFI\Pop_OS-ee80fe8b-8d46-4f70-816c-b9619e87c840\initrd.img root=UUID={{ My_UUID }} ro quiet loglevel=0 systemd.show_status=false systemd.unified_cgroup_hierarchy=1 splash

restart

# cat /proc/cmdline 
initrd=\EFI\Pop_OS-ee80fe8b-8d46-4f70-816c-b9619e87c840\initrd.img root=UUID={{ My_UUID }} ro quiet loglevel=0 systemd.show_status=false splash

No change…

$ bootctl status

I reboot the PC, but no change…

Pop!_OS manage the kernel’s command-lines parameters with (kernelstub)[https://github.com/isantop/kernelstub].

$ sudo vim /etc/kernelstub/configuration

Add user.kernel_options.systemd.unified_cgroup_hierarchy=1 in the json file, and run kernelstub

Reboot the PC and /proc/cmdline Changed!

/sys/fs/cgroup/cgroup.controllers exists!! Now, my laptop uses cgroup v2.

Troubleshooting 2. rootless kind

I’ve enabled cgroup v2, but kind fails..

$ kind create cluster
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

Follow https://kind.sigs.k8s.io/docs/user/rootless/#host-requirements.

$ cat /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=yes
$ sudo systemctl daemon-reload
$ cat /etc/modules-load.d/iptables.conf 
iptable_nat
ip6table_nat

Done!! Check kind works.

$ kind create cluster
Creating cluster "kind" ...
 βœ“ Ensuring node image (kindest/node:v1.21.1) πŸ–Ό 
 βœ“ Preparing nodes πŸ“¦  
 βœ“ Writing configuration πŸ“œ 
 βœ“ Starting control-plane πŸ•ΉοΈ 
 βœ“ Installing CNI πŸ”Œ 
 βœ“ Installing StorageClass πŸ’Ύ 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

Delete the cluster

$ kind delete cluster

kind cluster from configuration file

Three nodes (two workers) cluster config

kind-example-config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
$ kind create cluster --config kind-example-config.yml 
Creating cluster "kind" ...
 βœ“ Ensuring node image (kindest/node:v1.21.1) πŸ–Ό
 βœ“ Preparing nodes πŸ“¦ πŸ“¦ πŸ“¦  
 βœ“ Writing configuration πŸ“œ 
 βœ“ Starting control-plane πŸ•ΉοΈ 
 βœ“ Installing CNI πŸ”Œ 
 βœ“ Installing StorageClass πŸ’Ύ 
 βœ“ Joining worker nodes 🚜 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community πŸ™‚

$ docker ps -a
CONTAINER ID   IMAGE                  COMMAND                  CREATED              STATUS              PORTS                       NAMES
c898ef1dcb6f   kindest/node:v1.21.1   "/usr/local/bin/entr…"   About a minute ago   Up About a minute                               kind-worker2
a1a775dd026f   kindest/node:v1.21.1   "/usr/local/bin/entr…"   About a minute ago   Up About a minute   127.0.0.1:35205->6443/tcp   kind-control-plane
94d8f85eaeea   kindest/node:v1.21.1   "/usr/local/bin/entr…"   About a minute ago   Up About a minute                               kind-worker