The third part of this blog post series deals with master nodes and its components.
Among others we will have a look at the characteristics of an API server in an HA setup.
Systemd service definition
Since kubelet is already installed on all kubernetes nodes we continue with the service definition on the master nodes.
For the communication with the API server the local port 8080 is used on the master nodes:
Here we are facing a problem.
Kubelet needs the API server in order to register the node status in the cluster.
But the API server is started by kubelet.
This means kubelet does not work correctly until it reaches its local API server.
So kubelet runs an endless loop while starting in order to reach the API server.
The API server needs a server certificate like the etcd nodes.
This certificate has to be accepted by clients that access via the load balancer.
Therefore, it needs IP addresses and FQDN’s of both API servers and the load balancer.
In addition, the DNS entries for the API Url (kubernetes and kubernetes.default) will be inserted.
In contrary to all other certificates the one for the API server is created only on one master node and will be copied to the second one later on.
The reason for this is the creation and validation of the API token which is created with the key of the API server.
A bearer token which is required for authorization against the kubernetes API is provided for each application within a container by using a file.
This token is created by the primary kube controller manager with the API server key.
When an application sends a request to the API it is “load balanced” to one of the master nodes.
But the master node can only accept the request if the tokens’ key is correct.
By copying the certificate as well as the keys both master nodes are now able to validate encrypted connections with the token.
The certificate of the Root CA has to be added to the one of the API server just as it was done with the etcd peer certificate.
Afterwards, the files (apiserver.crt, apiserver.key, ca.crt) are copied to the same position with identical rights on the master02 node.
Now the YAML file that describes the API server can be created on both master nodes.
Here we use an image of quay.io.
The options used for the pod show some specific features.
service-cluster-ip-range represents the net which is used cluster-wide for the containers.
The first IP, 10.3.0.1, is the IP of the kubernetes.default service.
The second IP, 10.3.0.2, will be used for the cluster DNS service.
It is part of the kubelet service definition already.
The IP of the loadbalancer is used as advertise-address.
The reason is an error in the kube proxies and kubelet.
Due to this error only the first entry of a list of API server Url’s would be used.
Using the loadbalancer’s IP as advertise-address causes kube proxy to use static IPtables rules with this IP.
So the packages will be distributed by the loadbalancer.
Kube controller manager
The kube controller manager pod is created with the option service-account-private-key-file to forward the key of the API server.
Kube proxy pod of the master nodes
The kube proxy pod of the master nodes uses the local network interface (127.0.0.1) of the API servers for internal communication.
Finally, the kube scheduler pod is defined.
There are no specialities here.
Putting the master nodes into operation
kubelet is put into operation by simply using the following command:
This may last several minutes because at this stage the master nodes download the images.
Afterwards, the list of containers is similar to the following one (output shortened):
But the log files still contain errors.
The containers are registered in the namespacekube-system.
The namespace does not exist by default, but has to be created on only one of the master nodes.
kubelet and the API server work closely together.
This requires a precise arrangement of services and pods.
The HA setup comes along with additional specialities that influence the worker nodes.
The usage of the loadbalancer’s IP in the option advertise-address of the API server pods circumvents an error in kubernetes version 1.2.2.
Additional pods are based on the API server and can be easily set up.
Once the namespace kube-system is created (after the start of the API server) the master nodes are ready to receive requests from the worker nodes.
Karsten Peskova is a qualified civil engineer and has held a variety of different jobs since joining the software industry many years ago. He enjoys working directly with our customers, but also solving technical problems of all kinds.