Kubernetes Distributions: Tutorial & Explanation

Kubernetes is the de facto standard for container orchestration today. From simple test deployments on a Raspberry Pi, to highly distributed systems running across multiple regions, you can find Kubernetes just about everywhere. That’s because Kubernetes is extensible, reliable, scalable, and feature-rich. It supports many use cases including load balancing, failover, and automatic scheduling.

However, getting started with Kubernetes can result in questions with no single answer. There are multiple Kubernetes distributions with their own sets of pros and cons, and so hoping for one right option isn’t recommended. For example, minikube makes it easy to test and develop but isn’t the right choice for enterprise workloads at scale. Take a look at just how many choices there are in the Kubernetes distribution landscape:

An overview of the Kubernetes distribution landscape. (Source CNCF)

This article will introduce some of the most popular Kubernetes distributions, look at the main Kubernetes components, and help you understand how to pick the right one for your use case.

Kubernetes started gaining popularity around 2017-18, but the concept existed long before that. Google used Borg from 2003-2004 as a cluster management system to run thousands of applications and jobs. The birth of Kubernetes can be attributed to Google making Borg an open-source solution.

With the release of Kubernetes v1.0 in 2015, Google and the Linux Foundation formed the Cloud Native Computing Foundation (CNCF). With this, many leading companies like Redhat, IBM, Microsoft, and Amazon joined the Kubernetes Ecosystem. The timeline below indicates a rough series of events from 2003 to 2022.

Timeline of Kubernetes Events

As Kubernetes gained popularity, many companies started providing add-on services, features, and capabilities to help their customers achieve their business goals more quickly. Each of these offerings is known as a distribution; a modified version of the upstream Kubernetes distribution or bundled with additional tools and features. The current list maintained by CNCF includes dozens of distributions (see the full list here) though many are specialized and not relevant to most users.

The table below summarizes the types of distributions that are explained in more detail in this article.

Regardless of the Kubernetes distribution, every Kubernetes cluster has two main components: the control plane and worker node.

Control plane

The control plane is the brain, making important decisions within and about the cluster and responding to instructions. There are four main components within the control plane. It is highly recommended that the control plane components run on the same server or node, and these nodes should not run any workloads or user containers on them. The following table summarizes their use.

Additionally, there is a cloud controller manager, which is similar to the controller manager. The main difference is the cloud controller manager allows you to link to the cloud-specific APIs. Kubernetes clusters that run on-premises may not have a cloud controller manager.

Worker node

The worker node is responsible for running containers and has three main components, as listed in the table below.

These components are made available on GitHub in a Kubernetes open-source project. One of the ways to install, configure, and set up is to download these source codes and compile them. This method works across all infrastructures, servers, and virtual machines in the cloud or on-premises.

The downside of this approach is that it is complex and time-consuming. If you fail at some stage or want to update, you will have to rebuild it. Following this method of setting up a cluster is called vanilla Kubernetes or a vanilla distribution.

Fortunately, complex vanilla Kubernetes installs aren’t the only option. Multiple vendors have created distributions of their own that streamline and optimize Kubernetes deployments.

In addition to setting up and configuring the cluster, there are other features available related to security, observability or DevOps that developers or operations teams can leverage. Kubernetes with tools that make the installation easier and come with other add-ons are known as a Kubernetes distribution.

The different Kubernetes distributions can be grouped into four key categories:

• Opinionated distributions build the vendor’s opinions and best practices related to installation, configuration, and operation into the distribution.
• Upstream and open-source distributions are simpler open-source Kubernetes distributions that make installation easier without add-ons. Some of these distributions use upstream Kubernetes and others do not.
• Cloud-managed distributions are offered by cloud platforms like AWS, GCP, and Azure. With these distributions, the provider manages the control plane and users focus on development and productivity.
• Single-node distributions have control plane and worker nodes on the same server. These Kubernetes distributions are often used as learning environments and can be beneficial for use cases like running an application in Raspberry Pi, edge, and IoT devices.

The following table summarizes the different Kubernetes distributions.

VMware Tanzu

Tanzu is a portfolio of products by VMware that allows administrators and developers familiar with the VMware architecture to quickly provision, manage and run Kubernetes clusters. Tanzu Kubernetes Grid is an upstream Kubernetes-compatible distribution that integrates with the VMware ecosystem with 24x7 support from the VMware Global support services.

Here are four key Tanzu offerings from VMware:

• Tanzu Kubernetes Grid (TKG): TKG is a multi-cloud Kubernetes service by VMware to run on VSphere or public cloud virtual machines.
• Tanzu Kubernetes Grid Integrated (TKGi): TKGi is a production-grade solution based on Pivotal Kubernetes Service (PKS), and it integrates BOSH and Ops manager. TKGi is helpful if you want to create a multi-tenant Kubernetes Environment.
• Tanzu Mission Control (TMC): TMC is a service related to VMware Cloud that lets you attach clusters from multi-cloud and allows you to monitor and manage them from a single console.
• Tanzu Kubernetes Grid Service (TKGS): TKGS is Kubernetes with vSphere integration which means that administrators can provision Kubernetes clusters using the vSphere platform and the associated vSphere services.

Red Hat OpenShift

RedHat has two distributions of Kubernetes, one of them being OpenShift, an enterprise-ready distribution with a robust SLA and paid professional support. While OpenShift gets new Kubernetes features later than upstream Kubernetes, there is a strong focus on stability, security, and integration with other Red Hat technologies that help the developer community to be more productive.

Redhat Enterprise Linux is the base operating system, and CRI-O is the container runtime environment. OpenShift has a multi-cloud architecture and works well with AWS (Red Hat OpenShift Service on AWS) and Microsoft Azure (Azure Red Hat OpenShift). `oc` is the administrative tool for Openshift similar to `kubectl` that Kubernetes uses.

OpenShift features a simple and flexible installer that deploys the cluster on any infrastructure and comes with an update service.

The free community distribution of OpenShift is OKD or Origin. OKD uses Fedora as the underlying operating system. It is an excellent option if you want to try or have a local cluster on your machine using the same APIs as OpenShift.

Canonical Kubernetes

Charmed Kubernetes is an enterprise-grade, upstream Kubernetes by Canonical which offers cluster lifecycle management and is capable of multi-cloud deployment. Canonical offers managed Kubernetes, where Canonical engineers will build and operate the cluster 24/7 on bare metal, OpenStack, or any public clouds. Canonical is the publisher of Ubuntu, a popular Linux distribution. In addition to the distribution, Canonical also offers Enterprise support for a cost. Regarding the container runtime environment, Charmed Kubernetes supports Containerd, Kata containers and CRI-O.

Canonical offers a single-node Kubernetes distribution known as MicroK8. It is a lightweight Kubernetes distribution that can run on your laptop or any edge of IoT devices. Installation of MicroK8s is straightforward, with a single command to install followed by three commands to add your user to the microk8s group, as shown below.

sudo snap install microk8s --classic --channel=1.25
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube

Rancher Kubernetes

Rancher is a multi-cloud Kubernetes management platform that allows you to deploy clusters on private cloud, public cloud or vSphere and provides centralized authentication and observability. For IT operators, this means that they can use Rancher as a single pane of glass to manage on-premises clusters as well as those hosted on clouds like Azure Kubernetes Service (AKS), Elastic Kubernetes Service (EKS) and Google Kubernetes Engine (GKE).

Rancher also provides enterprises with a hosted Kubernetes service, a premium service where Rancher takes care of the installation, upgrades, and operations of the control plane with an SLA of 99.9%. Customers can add support for tools like Grafana, Prometheus, Istio, and Fluentd by paying additional subscription costs.

Additionally, Rancher has a product called Rancher Kubernetes Engine (RKE). This CNCF-certified Kubernetes distribution runs within Docker containers and K3s, a lightweight Kubernetes distribution with very low resource requirements.

Cloud-managed Kubernetes

Kubernetes has become so popular in the last few years, major cloud providers have come up with their own cloud-managed Kubernetes distributions. The three most popular options are:

• AWS Elastic Kubernetes Service (EKS)
• Azure Kubernetes Service (AKS)
• CGP Google Kubernetes Engine (GKE)

These managed Kubernetes services integrate with the cloud provider's services and technologies. Cloud providers deploy, run and maintain the control plane and let you shift your focus to the workload.

EKS charges a few cents per hour for the control plane, whereas AKS and GKE offer it for free. While most of these services have similar offerings and technologies built around them, there are minor differences, and the choice of managed Kubernetes distribution depends on which cloud is preferred. They all have solutions for multi-cloud and hybrid-cloud deployments like Anthos from Google, Arc from Azure, and EKS Anywhere from AWS.

With so many distributions available, how does anyone choose a particular distribution? The following five questions can guide you towards the best solution:

1. Do you want your cluster on-premises or on the cloud?
2. Are you looking to manage and operate the cluster yourself?
3. Do you have requirements where you need to deploy your application(s) in multiple clouds or multiple regions?
4. Do you currently use AWS, Azure, or GCP?
5. Do you plan to use multiple Kubernetes Clusters?

There are more than 120 distributions of CNCF-certified Kubernetes available, and any of them *could* be an excellent choice. What matters is the available tools and services surrounding the distribution and if the maintainers offer enterprise support to the level you require.

Consider your staff's skill set and align technology decisions accordingly. Another metric is the popularity based on the number of posts, issues and discussions in forums like Stack Overflow, Discord, and Reddit.

Kubernetes has erased the line between cloud and on-premise. Public cloud providers are offering their services within the data center using Google Anthos or Azure Arc, and on-premises providers are establishing their presence on the cloud like Microsoft Azure, Red Hat OpenShift, and Red Hat OpenShift Service on AWS.

With so many distributions available, there is a lot to consider when choosing a Kubernetes distribution. The concepts and questions covered in this article should help you make an informed choice.