Node Allocatable in AKS

Kubernetes’ Node Allocatable feature allows the cluster to reserve the resources of node for system daemons of OS and Kubernetes itself. For example, when I ran kubectl describe node for a node in my AKS cluster, I got the following capacity related output. The size of this node was Standard DS2 v2 which had 2 CPU cores and 7GB memory.

Capacity:
  attachable-volumes-azure-disk:  8
  cpu:                            2
  ephemeral-storage:              101445900Ki
  hugepages-1Gi:                  0
  hugepages-2Mi:                  0
  memory:                         7113660Ki
  pods:                           110
Allocatable:
  attachable-volumes-azure-disk:  8
  cpu:                            1900m
  ephemeral-storage:              93492541286
  hugepages-1Gi:                  0
  hugepages-2Mi:                  0
  memory:                         4668348Ki
  pods:                           110

From the output, out of 2 cores and 7GB memory, there were 1900 millicores and 66% memory (4668348/7113660) allocatable to pods. The details about how these numbers were calculated are described in this document. In short, the configuration of node allocatable in AKS is as follows:

CPU

CPU cores on host1248163264
Kube-reserved (millicores)60100140180260420740

Memory

  • eviction-hard: 750Mi, which is the default configuration of the upstream aks-engine.
  • kube-reserved: regressive rate
    • 25% of the first 4 GB of memory
    • 20% of the next 4 GB of memory (up to 8 GB)
    • 10% of the next 8 GB of memory (up to 16 GB)
    • 6% of the next 112 GB of memory (up to 128 GB)
    • 2% of any memory above 128 GB

However, the above memory reservation doesn’t seem to applicable to Windows nodes. The following was the output when I ran the kubectl describe node for a Windows node. More memory was reserved for Windows nodes.

Capacity:
  attachable-volumes-azure-disk:  8
  cpu:                            2
  ephemeral-storage:              133703676Ki
  memory:                         7339572Ki
  pods:                           30
Allocatable:
  attachable-volumes-azure-disk:  8
  cpu:                            1900m
  ephemeral-storage:              133703676Ki
  memory:                         3565108Ki
  pods:                           30

The document doesn’t have any information regarding Windows node. GKE reserves approximately 1.5 times more resources on Windows Server nodes. Not sure if it’s the same for AKS. I’ve opened an issue to ask for further information.

This post provides a good comparison of reserved resources for 3 major cloud offerings: AKS, GKE and EKS.

CKAD Certified

Early this week, I took the 2nd try of the CKAD exam and passed it. I scored 89% this time despite the situation where I ran out of time and couldn’t finish the last question completely. Now I am CKAD certified!

I had my first try of the exam in Aug last year without many preparations. As CNCF gives you a free 2nd chance if the 1st one is not successful, I wanted to take the 1st exam as an opportunity to test my Kubernetes knowledge, to get familiar with the test environment, and to sense how difficult it is. I scored 64% of the 1st exam which is 2% short of the passing score.

Late last year I was busy on my work and until recently I got some time to properly prepare for and take the exam again. I spent about 2 weeks to polish my skills with kubectl and other tools, and used the resources in the following two github repos heavily.

Here are several my tips regarding the exam which I hope could help those who are preparing for the certificate.

  • The only tools that you can use in the exam environment are kubectl, vim and tmux. So be very familiar with them.
  • In the exam, you are allowed to open another browser tab to connect to https://kubernetes.io/docs, but you may not have enough time to read the docs in detail. I relied on kubectl explain more than checking the docs.
  • As the exam environment runs in Chrome, a big screen definitely helps.
  • Most importantly, a lot of practices. The history showed that I tapped k/kubectl for 1372 times and vim for 429 times in 2 weeks before the exam.

Install Minikube on Ubuntu Server 17.10

I have some experiences with Docker and containers, but never played with Kubernetes before. I started to explore Kubernetes recently as I may need a container orchestration solution in the coming projects. Kubernetes is supported by Azure AKS. Even Docker has announced their support of it. Looks like it is going to be the major container orchestration solution in the market for the coming years.

I started with deploying a local Kubernetes cluster with Minikube on a Ubuntu 17.10 server on Azure. Kubernetes has a document on its site which is about installing the Minikube. But it is very brief. So in this post, I will try to document the step by step procedure both for the future reference of myself and for others who are new to Kubernetes.

Install a Hypervisor

To install Minikube, the first step is to install a hypervisor on the server. On Linux, both VirtualBox and KVM are supported hypervisors. I chose to install KVM and followed the guidance here. The following are steps.

  • Make sure VT-x or AMD-v virtualization is enabled. In Azure, if the VM is based on vCPUs, the virtualization is enabled. To double check, run command egrep -c '(vmx|svm)' /proc/cpuinfo, if the output is 1, the virtualization is enabled.
  • Install the KVM packages with the following command:
sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils
  • Use the following command to add the current user to the libvert group, and then logout and login to make it work. Note, in the guidance the group name is libvirtd, but on Ubuntu 17.10, the name has changed to libvert.
sudo adduser `id -un` libvirt
  • Test if your install has been successful with the following command:
virsh list --all
  • Install virt-manager so that we have a UI to manage VMs
sudo apt-get install virt-manager

Install kubectl

Follow the instruction here to install kubectl. The following are the commands:

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Install Minikube

Follow the instruction on the release notes of Minikube to install it. I used the following command:

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.25.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

When you finish this step, according to the official document, the installation of Minikube has been completed. But before you can use it, there are several other components which needs to be installed as well.

Install Docker, Docker-Machine, and KVM driver

Minikube can run on natively on the Ubuntu server without a virtual machine. To do so, Docker needs to be installed on the server. Docker-CE has a different way to be installed and Docker has a document for it.

Docker Machine can be installed with the following commands:

curl -L https://github.com/docker/machine/releases/download/v0.13.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && \
sudo install /tmp/docker-machine /usr/local/bin/docker-machine

Finally, we need to install a VM driver for the docker machine. Kubernetes team ships a KVM2 driver which is supposed to replace the KVM driver created by others. However, I failed to make the Minikube work with the KVM2 driver. There is a bug report for this issue and hope the Kubernetes team will fix it soon.

So I installed the KVM driver with the following command:

curl -LO https://github.com/dhiltgen/docker-machine-kvm/releases/download/v0.10.0/docker-machine-driver-kvm-ubuntu16.04
sudo cp docker-machine-driver-kvm-ubuntu16.04 /usr/local/bin/docker-machine-driver-kvm
sudo chmod +x /usr/local/bin/docker-machine-driver-kvm

Test if Minikube Works

With the completion of all the above steps, we can test the Minikube now.

minikube start --vm-driver kvm

It will create a vm named as minikube in KVM and configure a local Kubernetes cluster based on it. With kubectl, you should be able to see the cluster info and node info.

kubectl cluster-info
kubectl get nodes

With that, you can start to explore Kubernetes.