Deep dive into the Kubernetes DevOps tool ecosystem which help you provision and administrate clusters, run Kubernetes locally, run and debug apps, and more
🎉 Kubecost 2.0 is here! Learn more about the massive new feature additions and predictive learning

The Guide to Kubernetes DevOps Tools

It’s fascinating to think that merely 10-15 years ago, all we needed to deploy an application were a set of Bash scripts. Then, containerization took hold, and Docker containers running on Kubernetes orchestrator became standard. In this article, we will aim to provide an overview of the suite of DevOps tools available for Kubernetes and its ecosystem. For the sake of brevity, we will divide Kubernetes tooling into feature groups and take a closer look at each one.

Feature groups

Roughly speaking, all of the tools present within the Kubernetes ecosystem can be divided into the following groups:

  • cluster provisioning, tools used to create Kubernetes clusters
  • cluster administration, tools used to control and instruct Kubernetes to run things
  • application provisioning, tools used to provision Kubernetes applications onto the clusters
  • local Kubernetes distributions, tools used to deploy local development environments and stripped-down Kubernetes clusters without the need to pay for cloud resources
  • application debugging, tools used to develop and debug applications directly on Kubernetes clusters

The most famous representatives of each category can be seen in the table below

Feature group Tools Short description
Cluster provisioning IaC tools (Terraform, Pulumi) Used to create infrastructure from code and can be used to create and provision Kubernetes clusters
Kubespray Deploys clusters to multiple cloud providers, leverages Ansible to provision resources and is suited to create production setups
Kops Integrates with a small number of supported cloud providers, is self-sufficient and can create production-ready HA clusters out of the box
Kubeadm Better suited to cluster configuration rather than provisioning, and can be used for creating dev environments or experimenting with Kubernetes
Local Kubernetes kind Running Kubernetes clusters inside Docker containers
k3s/k3d k3s is a lightweight and optimized Kubernetes distribution that’s run locally within virtual machines or Docker (using k3d wrapper)
minikube Deploys a single-node Kubernetes cluster locally in Docker, VM or on bare-metal
microK8S Highly-available multi-node deployment of Kubernetes, supporting different cloud providers
Cluster administration kube* CLI tools Set of CLI tools, that are used to interact with clusters and make cluster operator’s life easier
Lens GUI application providing a wide variety of Kubernetes cluster controls
k9s Terminal UI application designed to make it easier to navigate, observe and manage Kubernetes clusters
Kubecost GUI + API application to visualize and allocate resource consumption on your cluster. Assists with properly sizing resource requests to optimize workload efficiency.
App deployment Helm Package manager for Kubernetes applications, allows for app sharing and provides features for release engineering
Kustomize Configuration manager and template renderer, it provides a way to traverse Kubernetes manifests based on layers of yaml artifacts
GitOps operators (FluxCD, ArgoCD) Solutions to deliver applications to Kubernetes clusters in a declarative way, using Git repos as the only source of truth
App debugging squash Interconnects apps running on Kubernetes and local IDE, without modification to code or deployments
telepresence A two-way proxy connecting a local application to any dependencies in remote Kubernetes clusters
Bridge to Kubernetes VSCode extension allowing for remote debugging of apps running on Kubernetes
CloudCode IntelliJ IntelliJ extension with the same goal as Bridge to Kubernetes, but more enterprise focussed

Comprehensive Kubernetes cost monitoring & optimization

Cluster provisioning tools

Selecting the right tools for the provisioning of Kubernetes clusters can often lead to more questions than answers. To help you, the following questions and considerations should prove useful during your decision making:

  • Is your infrastructure described as code (IaC), and does it use managed services? If so, then the likes of Terraform and Pulumi are your best choice as they avoid the need to use additional tools. IaC tools alone are not sufficient if you require the use of standard clusters, as you will need to manage the entire installation and configuration process yourself
  • Are you creating production-ready clusters? Then kubespray or kops would be a good fit; otherwise, the simplicity of kubeadm is the way to go.
  • Do you intend to create standard clusters across multiple cloud providers? Then again, kubespray is a good offering.

Evaluating each tool against your organization’s existing infrastructure and future roadmap is always a good practice. This will save your organization and its engineers from needless migration between tools and dran-out adaptation periods.

Local Kubernetes distributions

As an alternative to creating a fully-fledged Kubernetes cluster in the cloud, and for simple tasks such as local development or testing, it might make more sense to run Kubernetes locally, for example on a laptop.

All of the tools we’ve listed above are relatively similar, and are capable of delivering a lightweight version of Kubernetes. For example, creating a cluster via Docker with kind is as simple as running:

kind create cluster

With minikube:

minikube start

You may choose from plenty of configuration options too, such as the Kubernetes version, its runtime, add-ons, or even the virtualization driver used for cluster node creation. The key point remains: a few commands can get you up and running with a fully operational Kubernetes cluster.

Cluster administration tools

Having a running cluster means nothing without the ability to interact with it. To this end we can take two routes: terminal or graphical interface.

kubectl is the official command-line tool and allows us to interact with the Kubernetes APIs that are provided by the cluster control plane. Any operation, ranging from describing a payload to deploying an application can be performed via Kubectl.

In our example below, we easily return a list of pods running in a ‘demo’ namespace and obtain a set of logs from a pod:

kubectl get pods -n demo
kubectl logs my-pod -n demo

kubectx and kubens on the other hand, allows for the pre-selection of a cluster and a namespace where commands can be run respectively. This is mostly a cosmetic improvement, but one that saves the excessive typing of commands.

For instance, in our previous example above we used commands that referenced a namespace - this can be omitted when using kubens:

kubens demo
# all further commands will now apply to demo namespace
kubectl get pods
kubectl logs my-pod

k9s is a significantly different tool - a terminal UI application that provides a way to look in-depth at a selected cluster. It has built-in dashboards and metric aggregation that allows us to track both cluster and applications health. It’s also able to perform simple day-to-day tasks such as listing running pods or getting the logs from a payload.

k9s

List of pods and its resource footprint (source)

Lens IDE is a powerful, standalone GUI application that can control and explore Kubernetes clusters graphically. Whether running commands against a cluster, deploying an application via Helm, or just exploring Ingress configurations, Lens does it all visually.

k9s

List of pods and details of a particular pod (source)

No matter how different these tools may appear at first glance, they are all based upon one source of truth, the official Kubernetes APIs. Choosing one tool over another might be a matter of practicality, use case limitations (for example, usage of CLIs in CI/CD pipelines) or just personal preference.

Kubecost is a tool designed to assist teams in monitoring and managing costs and capacity within Kubernetes environments. The tool integrates with your cluster to enable efficient tracking, management, and reduction of expenses. It offers real-time cost visibility and insights, aiding in the reduction of cloud costs for Kubernetes users.

K8s clusters handling 10B daily API calls use Kubecost

The features encompass flexible cost breakdowns, unified monitoring of Kubernetes and external spending, personalized optimization suggestions, customizable alerts and governance. Kubecost can be installed via a Helm Chart on your infrastructure within minutes (free for individual clusters), but is also available as a Software as a Service (SaaS) for those who prefer a cloud-hosted solution. Kubecost also founded the OpenCost open-source project worth noting.

kubecost-overview

Application deployment

Deploying applications into a Kubernetes cluster is typically achieved via manifests - files written in YAML that contain the instructions sent to the Kubernetes API. Maintaining these files, especially at scale and when working with multiple app versions and environments is tricky and requires good organization. This is where tools such as Helm and Kustomize can come to the rescue.

Kustomize is a declarative templating engine that uses the concept of a base manifest and overlays, which are specific to the environment or configuration. Kustomize is simple and Kubernetes native, but its inability to handle operations at scale (multiple applications, multiple overlays and so on) make it unsuitable for larger organizations.

Helm is a more feature-rich tool that manages and packages manifests. Helm uses charts, which are a combination of templates and value files. Charts can be stored centrally into a chart museum and this provides a way to easily use and share them. Helm also handles the full lifecycle of an application. It’s a mature tool with plenty of features and good documentation.

It’s worth mentioning at this point a set of practices called GitOps. These provide a means to configure clusters and deploy applications using declarative definitions stored in something called a Git repo. Two representative tools that can implement GitOps are Flux and Argo CD. There are many similarities between them (reconciliation mode, secrets management, alerting and monitoring) but also quite a few differences (deployment model, support for Helm and Kustomize, web UI).

Application debugging

Lastly, we will examine some tools which are particularly useful when debugging applications running on Kubernetes. Some of these tools are IDE extensions, such as Bridge to Kubernetes, and CloudCode, where others are standalone applications, such as squash and telepresence. These inject agents into Kubernetes clusters and capture what’s happening via either network proxies or process thread debugging. Choosing between them is really just a case of convenience, functionality or personal choice.

Conclusion

During this article, we looked at some of the various tools available to aid the deployment of both Kubernetes clusters and the applications within. We provided a few practical demonstrations and even a little advice on which tools might be suitable for your specific use-cases.

If you found this article useful, and we hope that you did, we’d like to draw your attention to others that have been written as part of an overall guide to Kubernetes DevOps Tools. By following the links below, you can find articles that dive further into some of the concepts and topics covered during this piece. This includes several IaC tools and their differences (Pulumi vs Terraform), GitOps operators (ArgoCD) and tools used for the provisioning of Kubernetes (kubeadm, kubespray). See you next time!

Learn how to manage K8s costs via the Kubecost APIs

Chapters

Continue reading this series