Kubernetes on VMware: Tutorial & Best Practices

VMware has been a staple in the world of server virtualization for years. In 2015, VMware recognized the rising popularity of containerization and believed that containers and virtual machines (VMs) work better together. VMware’s first attempt at commercial container technology was Project Bonneville. Then, in 2019, they introduced the Tanzu Portfolio, which enabled building, managing, and running applications using Kubernetes on VMware

This article will take a closer look at Kubernetes on VMware, including its history, Tanzu offerings, how to deploy a cluster with Tanzu Community Edition, and key best practices.

Before project Bonneville, in 2009, a small team at VMware was working on project B29. The focus was to make the application portable by ensuring the developers focused on writing the code while B29 handled the dependencies. The concept was similar to containers as we know them now. Project B29 was named Cloudfoundry and became Pivotal Software, a joint venture by VMware, EMC, and General Electric in 2013.

VMware's Project Bonneville was based on the notion that containers and virtual machines are just containers and vice versa. The idea was to download, isolate and start each Docker container in a virtual machine within the vSphere environment, combined with VMware's instant clone technology to provide developers and admins security, isolation, and transparency.

In 2017, Pivotal Software and VMware launched Pivotal Container Service (PKS), a production-ready Kubernetes on vSphere. The next big step that VMware took was in 2018 by acquiring Heptio, a company that provided professional services for companies that run Kubernetes. Finally, in 2019 VMware acquired Pivotal Software, which led to the Tanzu Application platform, the current flagship Kubernetes offering from VMware.

Before we dive into the topic of Kubernetes on VMware, it is essential to understand three concepts:

• VMware Cloud Foundation (VCF): VCF is a complete infrastructure stack for hybrid cloud containing vSphere for compute virtualization, NSX-T for network virtualization, and vSAN for storage virtualization.
• VSphere: vSphere is the compute virtualization layer of VCF. One must depend on other networking and storage solutions if they do not go with the VCF.
• Tanzu: VMware Tanzu is a brand that covers a range of products related to modern application development.

Tanzu is a VMware brand name for anything related to Kubernetes or containers and their associated services. It consists of several products, as described in the table below.

VMWare has slowly and steadily made a significant investment in Kubernetes. vSphere 7.0 was a game changer, and it is one of the few platforms where you can run Kubernetes workloads, containers, and virtual machines side by side.

What makes VMware stand out among the hundreds of Kubernetes distributions and platforms?

• Running virtual machines alongside Kubernetes and managing them is one of the most vital points. There are other solutions like Kubevirt, which converts your virtual machine into a pod, meaning that the virtual machine is within your Kubernetes cluster. This gives administrators, operators, and developers a stable and consistent platform.
• The Kubernetes setup on VMware is simple and does not require much effort to integrate the platform with the underlying computing, storage, and networking.
• Administration and Operations become relatively straightforward with the availability of GUI-based management dashboards.
• VMware brings a comprehensive set of enterprise-grade tools to cover all aspects of Kubernetes, from application development to security and monitoring. Having such integrations makes day two operations more accessible.
• VMWare Kubernetes can run on Azure and AWS; the experience is identical to running it on-premises. It allows for the use and integration of cloud technologies and services.

Tanzu Kubernetes Grid, or TKG, is the distribution of Kubernetes by VMware. TKG comes in two primary flavors. One is more suitable to run on multi-cloud environments and on-premises, while the other works on VSphere 7.0 or above.

Tanzu Kubernetes Grid multi-cloud (TKGm)

Below is an overview of the key characteristics of TKGm.

• TKGm provides a consistent experience running a Kubernetes cluster in the cloud and on-premises.
• With TKGm, the first component that one has to create is the management cluster. As the name suggests, a management cluster is a Kubernetes cluster responsible for managing other Kubernetes clusters.
• The management cluster hosts a Kubernetes project called Cluster API, which allows cluster creation and management to be declarative using Kubernetes-style APIs.
• Cluster APIs can create multiple Kubernetes clusters known as workload clusters.
• Based on the requirements, you can have multiple workload Kubernetes clusters per management cluster — or multiple management clusters — each of which can have one Kubernetes cluster.
• All the workload Kubernetes clusters share certain Kubernetes Services like container registry (Harbor), Observability tools (Prometheus and Grafana), Ingress control (Contour), Networking (Calico), and others.
• The management cluster, with its workload clusters and the shared Kubernetes services collectively, is known as Tanzu Kubernetes Grid Instance.

An overview of Tanzu Kubernetes Grid multi-cloud. (Source)

Tanzu Kubernetes Grid Service (TKGs)

Below is an overview of the key characteristics of TKGs.

• Tanzu Kubernetes Grid service works with vSphere 7.0 or above.
• VMware completely overhauled the vSphere 7.0 architecture to ensure that one can deploy virtual machines using Kubernetes constructs.
• Each vSphere cluster will have a Supervisor Cluster, and the relationship between a vSphere cluster and a Supervisor Cluster is always 1:1.
• A Supervisor Cluster is a Tanzu running on vSphere that relies on ESXi as its compute layer. In other words, the Supervisor Cluster is a Kubernetes control plane inside the hypervisor that enables running container workloads in ESXi.
• Once a Supervisor Cluster is enabled, you can create a supervisor namespace, called vSphere Namespaces. This namespace is not the same as a Kubernetes namespace.
• With the supervisor namespace created, you can create a Tanzu Kubernetes cluster, which acts as your workload cluster. The workload Kubernetes cluster, like every other Kubernetes cluster, has control plane nodes and worker nodes.
• You can also have virtual machines and vSphere pods in the same supervisor namespace.
• vSphere pods differ from Kubernetes pods since they are created directly on top of the ESXi host. To run vSphere pods, you don't need a Tanzu Kubernetes cluster, but the Supervisor cluster is required.
• VSphere Pods rely on NSX-T networking and cannot be deployed with vSphere networking.

An overview of Tanzu Kubernetes Grid service. (Source)

Tanzu Community Edition

VMware Tanzu Community Edition is a free community-supported distribution of Tanzu. It is open-source and extremely easy to install and run on your laptop or any virtual machine in any public cloud. Tanzu Community Edition features a pluggable architecture, where you can start small and add new capabilities and tooling as you grow. The image below shows the tooling ecosystem available with this distribution but is not limited to these.

An overview of Tanzu Kubernetes Community Edition tooling. (Source)

How to install Tanzu Community Edition

Tanzu Community Edition requires Tanzu CLI, which provides a user interface along with the command line interface. The community edition provides two methods to provision a Kubernetes cluster.

1. Standalone clusters
   • The easiest way to provision a Kubernetes cluster
   • Suitable for development and test environments
   • Not scalable
2. Workload clusters managed by a management cluster
   • As the first step, one has to create a management cluster, and then the management cluster provisions multiple Kubernetes workload clusters as required
   • Ideal for production environments
   • Scalable with a dependency on the underlying hardware

Prerequisites to deploy a standalone Tanzu cluster

• macOS or Linux (This tutorial focuses on Linux). For Windows, you will have to download a zip file from here (https://tanzucommunityedition.io/download/)
• Homebrew
• Docker

Steps to deploy a standalone Tanzu cluster

tanzu standalone-cluster create  -f configuration.yaml

The configuration.yaml file is one of the previous configuration files you might have; otherwise, this file will be generated when you use the UI method to install. Alternatively, you can also use the UI to install, which is a much easier process and is the recommended method.

tanzu standalone-cluster create --ui

Running this command will open an installer browser, as shown in the picture below, and, depending on your selected option, will ask for further configuration and credential details. After you enter details like how many nodes you would like to have, the size of these nodes, etc., the cluster provisioning will start.

Tanzu Kubernetes Community Edition installation. (Source)

Alternative options to run Kubernetes on VMware

There are other options to run Kubernetes on virtual machines with VMware. Some of the options are to run Vanilla Kubernetes, Canonical’s Charmed Kubernetes, or Rancher Kubernetes on vSphere virtual machines. The downside of this approach is that you might not get Kubernetes support from VMware. Typically VMware only supports vSphere in these types of deployments.

Integrating with the VMware tooling ecosystem can also be tricky. Experts recommend using VMware Tanzu and associated products if you run containers on VMware infrastructure unless you have a compelling reason or a preexisting investment with any other distributions.

For any cloud-native application, observability plays an important role. Monitoring containerized applications within Kubernetes poses various challenges due to the transient nature of the containers and the number of resources and metrics that require monitoring.

Like dynamic containers, application monitoring has to be dynamic. Concerning security, it is important to protect your workloads from malicious actors and hackers.

You can do that by:

• Monitoring the network utilization
• Monitoring for suspicious activity
• Monitoring for failed and unsuccessful logins
• Monitoring for vulnerabilities and risks

When it comes to security monitoring, logs are one of the most critical data that is used to detect anomalies. Centralized logging is a highly recommended practice when it comes to any enterprise. The central repository also forms the source for any auditing tools or processes.

Additionally, without cost management, there is a high chance of the costs exceeding the budget due to how containers and Kubernetes work. It is critical to have proper cost management tools in place.

The sections below review popular recommended tools for different Kubernetes on VMware use cases.

Log Aggregation: Fluentd

Fluentd is an open-source data collector, while the Fluent Bit is a lightweight agent that acts as a data forwarder for Fluentd. Fluentd can aggregate all your logging data and then push it to tools like Elasticsearch for analytics. Kibana acts as the visualization interface.

This combination of Elasticsearch, Fluentd, and Kibana is the EFK stack. Alternatively, there is also an ELK stack, which replaces Fluentd with Logstash.

Monitoring: Prometheus and Grafana / Aria Operations

Prometheus and Grafana are the most preferred open-source tool combinations when monitoring Kubernetes clusters. Together they can give detailed insights on performance bottlenecks, metrics, the overall health of the Kubernetes cluster, network usage, and help with across-the-board observability. Prometheus is excellent at monitoring multidimensional data, including time-series data. Grafana is an open-source metrics dashboard to display data.

As part of its Tanzu portfolio, VMware offers VMware Aria Operations, which also works with non-Kubernetes environments. It has over 250 integrations, and you can integrate your existing diverse monitoring tools with VMware Aria to get a single pane of glass view.

Security: VMware Carbon Black

VMware Carbon Black is a container image scanning tool with a central dashboard for vulnerability monitoring. VMware Carbon Black integrates seamlessly into CI/CD pipelines and helps to shift security left. It also allows container deployments using specific white-listed registries and repositories.

Cost Optimization: Kubecost

Kubecost is one of the best solutions to monitor your Kubernetes cost and optimize it. For every company, it is important to balance the cost and the performance, and with the multi-cloud and hybrid nature of Kubernetes, cost management becomes very tricky.

Kubecost resolves this issue by giving you a single view of your cost across multiple clusters. The cost allocation feature within Kubecost allows you to assign various costs — like the cost of the server, licenses required, etc. — to derive the accurate cost of running a Kubernetes cluster.

VMware Kubernetes is a good choice if you already use the VMware ecosystem or want to run Kubernetes in a hybrid environment. VMware has a complicated portfolio for Kubernetes, but that works to VMware's advantage as users with diverse use cases find VMware to be the solution to their problems. If you are new to VMware, take the time to understand their portfolio of products, and choose them according to your needs and budget.