Learn how to manage multiple Kubernetes clusters as a single entity with Kubernetes Cluster Federation to improve scalability, availability, and resilience

Kubernetes Federation: Tutorial & Examples

Kubernetes has become a universal container orchestration solution for application deployments of all sizes. The features and functionality of Kubernetes have matured to the point that it is now used to manage some of the largest deployment workloads in the world. These workloads can be spread over multiple clusters, data centers, and geographical locations.

However, these multi-cluster designs often include a variety of cloud providers, data centers, regions, and zones where the applications are deployed. Deploying your applications or services on clusters with such diverse capabilities can be quite cumbersome.

This is where Kubernetes Cluster Federation (aka KubeFed) comes in: It lets you manage multiple Kubernetes clusters as if they were one. In this article, we will look at what federation is and how you can use it to manage your multi-cluster Kubernetes deployments.

Concept Summary
What is Kubernetes Cluster Federation?
  • Federation is an essential concept in Kubernetes that enables the loose coupling of multiple independent Kubernetes clusters.
  • It’s possible to aggregate the resources and services of multiple clusters into a single, logical entity while keeping each cluster isolated and independent.
Difference between multi-cluster and federated Kubernetes clusters
  • A multi-cluster deployment simply deploys multiple clusters across one or more data centers.
  • Kubernetes Cluster Federation enables multiple Kubernetes clusters to be linked together, allowing them to share resources and manage workloads among them.
Kubernetes Cluster Federation architecture
  • A host cluster manages deployments and propagates changes to participating clusters.
  • A standard API exists to manage cluster-wide operations.
  • DNS entries are managed by federated configurations.
Setting up Kubernetes Cluster Federation
  • KubeFed
  • Deploying federated resources
Cloud solution similar to Kubernetes Cluster Federation
  • GCP Anthos

Comprehensive Kubernetes cost monitoring & optimization

What is Kubernetes Cluster Federation?

Kubernetes Cluster Federation is an essential concept in Kubernetes that enables the loose coupling of multiple independent Kubernetes clusters. Federation allows you to aggregate the resources and services of multiple clusters into a single, logical entity while keeping each cluster isolated and separate. This can be useful in many situations, such as:

  • Deploying a global service that needs to span multiple data centers or regions
  • Providing a consistent experience for users across multiple disparate clusters
  • Aggregating the resources of multiple clusters for reporting or monitoring purposes

Federation enables users to manage a group of Kubernetes clusters as a single entity by replicating and sharing resources across multiple clusters. Kubernetes Cluster Federation provides numerous benefits, including:

  • Reduced operational overhead for managing multiple Kubernetes clusters
  • Improved application uptime by replicating applications across multiple regions or data centers
  • Easy management of multi-cluster deployments with a centralized control plane
  • The ability to scale horizontally by adding new regions or data centers without affecting running applications

The difference between multi-cluster and federated Kubernetes clusters

Kubernetes multi-clusters

In computing, a cluster refers to a group of computers connected to work as a single system. A Kubernetes cluster is a group of nodes used to run containerized applications. Kubernetes multi-cluster deployments allow you to distribute the application workload across clusters.

A multi-cluster Kubernetes deployment can be used to achieve several goals, including improved scalability, availability, and resiliency. By spanning multiple clusters, a Kubernetes multi-cluster deployment can provide greater resilience in the face of hardware failures or network outages. A Kubernetes multi-cluster deployment can also offer increased performance and scalability through distributed load balancing across multiple clusters.

Federated Kubernetes clusters

A federated cluster is a multi-cluster deployment in which a central control plane manages a group of child clusters. The child clusters can be located anywhere — on-premises, in different data centers, or with various cloud providers. The federated Kubernetes approach allows administrators to manage all clusters from a single control point.

Federated Kubernetes is best suited for organizations that need to manage a large number of Kubernetes clusters across multiple data centers or regions. This makes it easy to keep your Kubernetes clusters consistent and reduces the time and effort required to manage them.

Setting up Kubernetes Cluster Federation

To demonstrate Kubernetes Cluster Federation, we will use Amazon Elastic Kubernetes Service (EKS) as a Kubernetes cluster.

Deployment Architecture

Here’s what you will deploy to set up Kubernetes Cluster Federation on AWS EKS clusters.

  • Host machine (bastion host / local system) to deploy the resources
  • EKS clusters in three different AWS regions:
    • Host Cluster (us-east-2)
    • Federated Cluster 1 (eu-west-1)
    • Federated Cluster 2 (eu-central-1)

Prerequisites

To follow this tutorial, you’ll need:

Creating EKS clusters

To set up EKS clusters, create three manifest files: host-cluster.yaml, federated-cluster-1.yaml, and federated-cluster-2.yaml.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: host-cluster
  region: us-east-2

nodeGroups:
  - name: ng-1
    instanceType: t3.small
    desiredCapacity: 1
    volumeSize: 8

host-cluster.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: federated-cluster-1
  region: eu-west-1

nodeGroups:
  - name: ng-1
    instanceType: t3.small
    desiredCapacity: 1
    volumeSize: 8

federated-cluster-1.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: federated-cluster-2
  region: eu-central-1

nodeGroups:
  - name: ng-1
    instanceType: t3.small
    desiredCapacity: 1
    volumeSize: 8

federated-cluster-2.yaml

To apply the configuration, run these commands in your terminal window:

> eksctl create cluster -f federated-cluster-1.yaml

> eksctl create cluster -f federated-cluster-2.yaml

> eksctl create cluster -f host-cluster.yaml

A quick tip: To speed up the cluster deployments, use multi-tabs/windows on Terminal so that you can deploy all three clusters simultaneously.

K8s clusters handling 10B daily API calls use Kubecost

Once the clusters are ready, you should see an output similar to that shown below.

[✔]  EKS cluster "federated-cluster-1" in "eu-west-1" region is ready

[✔]  EKS cluster "federated-cluster-2" in "eu-central-1" region is ready

[✔]  EKS cluster "host-cluster" in "us-east-2" region is ready

To verify the clusters and get context names, use this command:

> kubectl config get-contexts --no-headers=true -o name

iamadmin@federated-cluster-1.eu-west-1.eksctl.io
iamadmin@federated-cluster-2.eu-central-1.eksctl.io
iamadmin@host-cluster.us-east-2.eksctl.io

Now, let’s rename cluster context names for better readability.

To rename a context, use the command syntax:

> kubectl config rename-context old-name new-name
> kubectl config rename-context iamadmin@federated-cluster-1.eu-west-1.eksctl.io federated-cluster-1

> kubectl config rename-context iamadmin@federated-cluster-2.eu-central-1.eksctl.io federated-cluster-2

> kubectl config rename-context iamadmin@host-cluster.us-east-2.eksctl.io host-cluster

Verify new context names:

> kubectl config get-contexts --no-headers=true -o name

Now, we can use these three clusters and interact with them using kubectl. First, we must update the current context to the host cluster’s access. To update the current context, execute this command:

> kubectl config use-context host-cluster

Switched to context "host-cluster".

KubeFed

KubeFed, short for “Kubernetes Cluster Federation,” is a tool for managing and operating federated Kubernetes clusters. KubeFed provides a command-line interface and a set of Kubernetes manifest files for managing federated clusters. It offers several useful features, such as automatic workload scheduling and management across multiple clusters.

To install kubefed using the helm charts in the host cluster, execute:

> helm repo add kubefed-charts https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts

> helm repo update

Next, to install a KubeFed chart archive to the cluster, execute:

> helm --namespace kube-federation-system upgrade -i kubefed kubefed-charts/kubefed --create-namespace

The command above will install the KubeFed chart into the kube-federation-system namespace.

To view all the resources added by KubeFed in the host cluster, execute:

> kubectl get all -n kube-federation-system

Registering multiple clusters

While registering multiple clusters under the host cluster, you have the option to include the host cluster as part of the federation. The resources will be deployed on the host cluster and other federated clusters by adding a host cluster as part of the federation.

The example below adds the host cluster as a part of the federated clusters.

Next, you need to add EKS clusters to the host cluster’s federation control plane using kubefedctl.

> kubefedctl join host-cluster --cluster-context host-cluster --host-cluster-context host-cluster --v=2

> kubefedctl join federated-cluster-1 --cluster-context federated-cluster-1 --host-cluster-context host-cluster --v=2

> kubefedctl join federated-cluster-2 --cluster-context federated-cluster-2 --host-cluster-context host-cluster --v=2

To get all federated clusters, execute:

> kubectl get kubefedclusters -n kube-federation-system

NAME                  AGE   READY   KUBERNETES-VERSION
federated-cluster-1   68s   True    v1.23.13-eks-fb459a0
federated-cluster-2   60s   True    v1.23.13-eks-fb459a0
host-cluster          77s   True    v1.23.13-eks-fb459a0

Deploying a demo deployment using federated deployment

We will first create a namespace that will be deployed to all the federated clusters. All resources deployed in this specific namespace will be distributed among all federated clusters.

First, you need to create a namespace on the KubeFed host cluster.

apiVersion: v1
kind: Namespace
metadata:
  name: federated-ns

ns.yaml

To create the namespace, execute:

> kubectl apply -f ns.yaml

namespace/federated-ns created

You can label each cluster with a standard tag to share resources across all clusters. While deploying federated resources, you can specify this label to distribute the resources among clusters.

> kubectl label kubefedclusters -n kube-federation-system host-cluster federation-enabled=true

> kubectl label kubefedclusters -n kube-federation-system federated-cluster-1 federation-enabled=true

> kubectl label kubefedclusters -n kube-federation-system federated-cluster-2 federation-enabled=true

To share a namespace resource (in this case, the federated-ns namespace) with other clusters, you need to create a FederatedNamespace object:

apiVersion: types.kubefed.io/v1beta1
kind: FederatedNamespace
metadata:
  name: federated-ns
  namespace: federated-ns
spec:
  placement:
    clusterSelector:
      matchLabels:
        federation-enabled: "true"

federated-ns.yaml

> kubectl apply -f federated-ns.yaml

To get the federated namespace from each cluster, execute:

> for context in $(kubectl config get-contexts --no-headers=true -o name)
do
        echo "Getting namespaces  in context $context"
        kubectl get ns federated-ns --context=$context
done

Getting namespaces  in context federated-cluster-1
NAME           STATUS   AGE
federated-ns   Active   23s
Getting namespaces  in context federated-cluster-2
NAME           STATUS   AGE
federated-ns   Active   23s
Getting namespaces  in context host-cluster
NAME           STATUS   AGE
federated-ns   Active   12m

Now that the namespace has been federated, you can begin federating deployments. To test, start with a sample app deployment and a load balancer service:

apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: sample-app
  namespace: federated-ns
spec:
  placement:
    clusterSelector:
      matchLabels: {}
  template:
    spec:
      selector:
        matchLabels:
          app: sample-app
      template:
        metadata:
          labels:
            app: sample-app
        spec:
          containers:
          - image: k8s.gcr.io/echoserver:1.10
            name: sample-app
            ports:
            - containerPort: 8080
              name: http
            - containerPort: 8443
              name: https

federated-deployment.yaml

> kubectl apply -f federated-deployment.yaml

Next, verify federated deployment resource status:

> kubectl get federateddeployment sample-app -n federated-ns -o json | jq .status.clusters


[
  {
    "name": "federated-cluster-2"
  },
  {
    "name": "host-cluster"
  },
  {
    "name": "federated-cluster-1"
  }
]

The deployment is federated to all the clusters. Now, let’s take a look at the pods:

> for context in $(kubectl config get-contexts --no-headers=true -o name)
do
        echo "Getting pods  in context $context"
        kubectl get pods -n federated-ns --context=$context
done

Getting pods  in context federated-cluster-1
NAME                         READY   STATUS    RESTARTS   AGE
sample-app-dc484855d-wl5br   1/1     Running   0          7m49s
Getting pods  in context federated-cluster-2
NAME                         READY   STATUS    RESTARTS   AGE
sample-app-dc484855d-4n6g7   1/1     Running   0          7m50s
Getting pods  in context host-cluster
NAME                         READY   STATUS    RESTARTS   AGE
sample-app-dc484855d-g86sk   1/1     Running   0          7m51s

To deploy a load balancer service for deployment, apply a federated service manifest:

apiVersion: types.kubefed.io/v1beta1
kind: FederatedService
metadata:
  name: sample-app-svc-lb
  namespace: federated-ns
spec:
  placement:
    clusterSelector:
      matchLabels: {}
  template:
    spec:
      ports:
      - name: http
        port: 8080
        protocol: TCP
        targetPort: 8080
      - name: https
        port: 8443
        protocol: TCP
        targetPort: 8443
      selector:
        app: sample-app
      type: LoadBalancer

federated-svc.yaml

> kubectl apply -f federated-svc.yaml

federatedservice.types.kubefed.io/sample-app-svc-lb created

To verify load balancer service in all the clusters, execute this command:

> for context in $(kubectl config get-contexts --no-headers=true -o name)
do
  echo "Getting service in context $context"
  kubectl get svc sample-app-svc-lb -n federated-ns --context=$context
done

Getting service in context federated-cluster-1
NAME                TYPE           CLUSTER-IP      EXTERNAL-IP                                                              PORT(S)                         AGE
sample-app-svc-lb   LoadBalancer   10.100.154.92   a8d553a33d1f44da1bdea1feb066fe90-942868406.eu-west-1.elb.amazonaws.com   8080:32027/TCP,8443:31651/TCP   9s

Getting service in context federated-cluster-2
NAME                TYPE           CLUSTER-IP      EXTERNAL-IP                                                                  PORT(S)                         AGE
sample-app-svc-lb   LoadBalancer   10.100.174.82   aea26b82f526e49b6b70697733f4b613-1748865926.eu-central-1.elb.amazonaws.com   8080:31325/TCP,8443:31678/TCP   11s

Getting service in context host-cluster
NAME                TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                         AGE
sample-app-svc-lb   LoadBalancer   10.100.115.143   a78df2cd114fd470dac51f322c2edd5c-1497053958.us-east-2.elb.amazonaws.com   8080:30211/TCP,8443:30215/TCP   11s

You have now successfully federated a namespace, deployment, and service across three Kubernetes clusters.

Cleanup

To delete all the resources created above, execute:

> kubectl delete -f federated-svc.yaml

> kubectl delete -f federated-deployment.yaml

> kubectl delete -f federated-ns.yaml

> kubectl delete -f ns.yaml

> eksctl delete cluster -f federated-cluster-2.yaml

> eksctl delete cluster -f federated-cluster-1.yaml

> eksctl delete cluster -f host-cluster.yaml
Learn how to manage K8s costs via the Kubecost APIs

Cloud solution similar to Kubernetes Cluster Federation

Google Cloud Anthos

Anthos is a Google Cloud product that provides a consistently managed Kubernetes experience across multiple environments, so you can run your applications confidently on-premises or in the cloud (Google or otherwise). It also includes a fully managed service mesh with built-in visibility to monitor, trace, and control traffic flowing to and from your services without installing or managing any third-party software.

Anthos also allows you to use the same CI/CD tools and processes across your entire fleet of workloads, regardless of where they’re running. This makes it easy to adopt cloud-native practices if you still need to start using them, and makes it easier for teams to work together across different environments.

With Anthos, you get all these benefits without sacrificing freedom of choice or portability. You can use the Anthos pluggable architecture to swap out critical components with alternatives that better fit your needs. For example, if you want to use a different monitoring solution or container registry, you can do so without rearchitecting your entire application. This lets you take advantage of the best features of each cloud without being locked into any one provider.

Conclusion

Kubernetes Cluster Federation is a powerful concept that enables the loose coupling of multiple independent Kubernetes clusters. Federation allows you to aggregate the resources and services of multiple clusters into a single, logical entity while keeping each cluster isolated and independent.

Many tools are available to help you manage Kubernetes Cluster Federation, including Lens, Kubecost, and Prometheus Thanos. Each tool has unique features and benefits that can be leveraged to manage Kubernetes Cluster Federation efficiently.

When working with federated Kubernetes clusters, it is essential to consider the cost implications of running multiple clusters. Kubecost is a valuable tool for managing costs within a Kubernetes infrastructure.

By understanding the basics of Kubernetes Cluster Federation and leveraging the right tools for the job, you can manage your federated Kubernetes clusters efficiently and effectively.

Comprehensive Kubernetes cost monitoring & optimization

Continue reading this series