Learn about Kubernetes secrets management and how to use the Kubernetes External Secrets operator to store your Kubernetes secrets more safely
🎉 Kubecost 2.0 is here! Learn more about the massive new feature additions and predictive learning

Kubernetes External Secrets: How to Store and Use Sensitive Data in Kubernetes Cluster

Kubernetes secrets are sensitive pieces of information that should be kept secure, such as passwords, API keys, and tokens. Without secrets, it is nearly impossible to use Kubernetes for most environments and workloads.

Backend applications might need access to the database for processing and storing data, frontend applications might require API keys to integrate with third-party APIs, and the Kubernetes cluster might need access to pull container images from private container registries.

While Kubernetes has built-in Secrets for storing and using sensitive data. However, it's often insufficient because they are stored unencrypted in the Kubernetes data store. This article will show you how to securely install and use Kubernetes External Secrets for handling sensitive data in a Kubernetes cluster.

Secrets Management In Kubernetes

Secrets management in Kubernetes provides secure ways to manage the creation, storage, rotation, and removal of digital credentials while reducing human involvement and minimizing potential sources of error.

Kubernetes Secrets Management Overview
Concept Summary
Kubernetes Secrets
  • Secrets are Kubernetes objects that contain sensitive data such as credentials, API keys, etc.
  • Kubernetes Secrets are the default construct for managing sensitive data in Kubernetes.
Kubernetes External Secrets
  • Kubernetes External Secrets store sensitive data outside the Kubernetes cluster.
  • External providers such as HashiCorp Vault and AWS Secrets Manager handle entire secret lifecycle management.
  • Kubernetes cluster objects such as pods can reference these externally stored secrets.
External Secrets Operator (ESO)
  • External Secrets Operator (ESO) is a Kubernetes Operator that interacts with external providers.
  • ESO uses APIs these external providers provide and fetches the secrets stored in external backends.
  • ESO is compatible with several secrets providers such as AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, etc.

Kubernetes Secrets

Kubernetes Secrets are built-in objects that store and manage secrets in a Kubernetes cluster. Secrets are created and stored as a Kubernetes Secret object and used by pods or other objects in a cluster.

Kubernetes Secrets provide an easy and declarative way to create and manage secrets in a Kubernetes cluster.

However, there are a few downsides to using this built-in secrets management mechanism. Specifically, Kubernetes Secrets have several downsides such as:

  • Stored as Base64 encoded objects so anyone with Cluster access can decode the secrets.
  • Created either by kubectl CLI or in YAML manifests, making them insecure to integrate with version control systems.
  • Difficult to manage and synchronize when managing multiple environments.
  • No default mechanism to rotate and update the secrets

Kubernetes External Secrets

Kubernetes External Secrets store information outside the Kubernetes cluster while still allowing Kubernetes resources to use them. They are stored in an external service that Kubernetes interacts with to read and write secrets.

External secrets have several advantages over Kubernetes Secrets:

  • Sensitive data is managed by external services outside the Kubernetes cluster and is less likely to be compromised as both systems need to be compromised to access sensitive data.
  • Kubernetes External Secrets allow you to use different storage backends for your secrets, such as HashiCorp Vault, AWS Secrets Manager, Google Secrets Manager, or Azure Key Vault.
  • Most importantly, you can take advantage of the security features of these services, such as encryption of data-at-rest and in-transit and scheduling for secrets’ rotation.

External Secrets Operator (ESO)

To use Kubernetes External Secrets, you must configure an external secrets backend and create a Kubernetes Secret object that points to the external backend. Kubernetes will then interact with the secret backend to read and write the secrets.

A diagram detailing how the Kubernetes External Secrets Operator works.

A diagram detailing how the Kubernetes External Secrets Operator works.

The External Secrets Operator makes using external secret management systems easier with Kubernetes. The External Secrets Operator will read the required information from the external API and inject it into a Kubernetes Secret for you. With this operator, you can easily incorporate secrets from providers like AWS Secrets Manager, HashiCorp Vault, and many more.

Comprehensive Kubernetes cost monitoring & optimization

How to Manage Kubernetes External

Secrets with AWS Secrets Manager

To demonstrate Kubernetes External Secrets, we will use Amazon Elastic Kubernetes Service (EKS) as a Kubernetes Cluster and Amazon Secrets Manager as an external secret store.

To follow this tutorial, you’ll need:

Deploying AWS EKS Cluster

To deploy an AWS EKS cluster, we will use eksctl CLI. To create the cluster, create a cluster.yaml manifest file with the below configuration.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: eks-external-secrets
  region: us-east-1

nodeGroups:
  - name: ng-1
    instanceType: t3.small
    desiredCapacity: 1
    volumeSize: 8

To apply the configuration, run this command in your terminal:

$ eksctl create cluster -f cluster.yaml

This will create an EKS cluster with one node group containing a single node in the us-east-1 AWS region. Once the cluster is ready, you should see output similar to the output below.

2022-09-05 18:47:47 [âś”]  EKS cluster "eks-external-secrets" in "us-east-1" region is ready.

Now, we can use this cluster and interact with it using kubectl. First, we must update the kubeconfig file with newly created cluster access. To update the kubeconfig, execute the below command.

$ aws eks --region us-east-1 update-kubeconfig --name eks-external-secrets

Test cluster access by running:

$ kubectl get pods

No resources found in default namespace.

Deploying External Secrets Operator

Install Helm to deploy the External Secrets Operator in the cluster. Once installed, verify the Helm version.

$ helm version --short

Next, install the External Secrets Operator from the Helm chart repository external-secrets by executing the below commands:

$ helm repo add external-secrets https://charts.external-secrets.io

$ helm install external-secrets \
  external-secrets/external-secrets \
    --namespace external-secrets \
    --create-namespace \
    --set installCRDs=true

Once installation is complete, you will see the below message:

external-secrets has been deployed successfully!

Configuring IAM roles for service accounts (IRSA) and Secrets Manager

With IAM roles for IRSA, you can map AWS IAM roles to Kubernetes Service Accounts. This feature allows you to manage AWS credentials for your applications running on EKS without managing static credentials directly.

To securely use External Secrets, we will use IRSA to provide AWS credentials. This way, the External Secrets pods will have access only to the secrets they need in our AWS Secrets Manager secret.

To use IRSA, the first step is to create an IAM OIDC Provider for your cluster (if one doesn't already exist). After you associate the provider with your EKS cluster, you can create an IRSA service account that External Secrets will use.

To enable IAM OIDC Provider for your cluster, run the following command:

$ eksctl utils associate-iam-oidc-provider --cluster=eks-external-secrets --approve

Now, we will create a secret in AWS Secrets Manager. Create a secret with the name secret-api-key and store the username and key as secret values.

$ SECRET_ARN=$(aws secretsmanager create-secret --name secret-api-key \
    --secret-string "{\"username\":\"admin\",\"key\":\"SeCure@ApIKey\"}" \
    --region us-east-1 | jq -r .ARN)

Next, create an IAM policy that will grant the permissions to interact with AWS Secrets Manager. For additional security, the following policy only allows for the description and retrieval of a single specified secret.

$ IAM_POLICY_ARN=$(aws iam create-policy --policy-name eks-external-secrets-reader --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue"
      ],
      "Resource": ["'${SECRET_ARN}'"]
    }
  ]
}' | jq -r .Policy.Arn)

Finally, create an IRSA service account for the External Secrets Operator to authenticate and fetch the required secrets from AWS Secrets Manager.

Here we are creating this IRSA service account in the default namespace.

$ eksctl create iamserviceaccount \
    --name external-secrets-irsa \
    --namespace default \
    --cluster eks-external-secrets \
    --role-name "external-secrets-irsa-role" \
    --attach-policy-arn $IAM_POLICY_ARN \
    --approve \
    --override-existing-serviceaccounts

To verify the successful creation of the Kubernetes Service Account, execute the following command:

$ kubectl get sa

NAME                    SECRETS   AGE
default                 1         39m
external-secrets-irsa   1         3s
🎉 Kubecost Cloud is here! 🎉 Try our new multi-tenant SaaS Solution today!

Configuring External Secrets Operator

To work with External Secrets, we must create two custom Kubernetes resources. ExternalSecret and SecretStore.

ExternalSecret describes what data should be retrieved, processed, and stored as a Kubernetes Secret object.

SecretStore specifies how to access and interact with the external APIs. In this example, it’s AWS Secrets Manager.

First, create two manifest files, external-secret.yaml and secret-store.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: eks-external-secret
spec:
  refreshInterval: 10m
  secretStoreRef:
    name: eks-secret-store
    kind: SecretStore
  target:
    name: eks-secret
  data:
  - secretKey: username
    remoteRef:
      key: secret-api-key
      property: username
  - secretKey: key
    remoteRef:
      key: secret-api-key
      property: key

external-secret.yaml

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: eks-secret-store
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-irsa

secret-store.yaml

Next, create these resources by applying these YAML manifests. These custom Kubernetes resources are applied in the default namespace because an IRSA service account was also created in the default namespace.

$ kubectl apply -f secret-store.yaml

$ kubectl apply -f external-secret.yaml

Verify ExternalSecret deployment status.

$ kubectl get externalsecrets.external-secrets.io

NAME                  STORE              REFRESH INTERVAL   STATUS         READY
eks-external-secret   eks-secret-store   10m                 SecretSynced   True

You should see Status as SecretSynced.

After you've created the ExternalSecret resource, you'll be able to see the new Kubernetes Secret that has been synchronized with the Secrets Manager store. Execute the following command:

$ kubectl describe secret eks-secret

Name:         eks-secret
Namespace:    default
Labels:       <none>
Annotations:  reconcile.external-secrets.io/data-hash: a946eefe36be05541b6060cc5c7b467b

Type:  Opaque

Data
====
key:       13 bytes
username:  5 bytes

You have now created a Kubernetes Secret that can be used within your pod specification for use by your applications.

Consuming Secret in Pod

By syncing your AWS Secrets Manager secret to a Kubernetes Secret, External Secrets allows you to use and consume the secret in your Pod specification.

We will deploy a simple busybox pod in the default namespace and use the secret via pod environment variables.

Create a manifest file external-secrets-demo-pod.yaml with the following specifications.

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox:1.35.0
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
    env:
      - name: API_USERNAME
        valueFrom:
          secretKeyRef:
            name: eks-secret
            key: username
      - name: API_KEY
        valueFrom:
          secretKeyRef:
            name: eks-secret
            key: key
  restartPolicy: Always

To deploy the pod, run the following command:

$ kubectl apply -f external-secrets-demo-pod.yaml

Once the pod is in a running state, run the following commands to get the container’s shell.

$ kubectl exec -it busybox -- sh

From the container’s shell, you can use echo to print the environment variables to view the secrets.

/ # echo $API_USERNAME
admin
/ # echo $API_KEY
SeCure@ApIKey

Comparing Kubernetes Secrets and External Secrets

Kubernetes External Secrets vs. Kubernetes Secrets
Feature Kubernetes Secrets Kubernetes External Secrets
Availability Kubernetes’s default secret management Need to install as Custom Resource Definition, and custom Operator
Secret Lifecycle Management Difficult to manage secrets at a large scale and across multiple environments Secrets are stored externally and inherit all the lifecycle features of external service
Security Secrets are not stored as encrypted objects; any entity having cluster access can decode the secrets data Secrets are managed outside of the cluster and stored encrypted
Version Control System Kubernetes Secrets are not secure to include in the Version Control Systems Most external secret management tools provide built-in version history. YAML manifests can be committed to version control systems as they only contain a reference to the secrets

Deploying Kubernetes External Secrets in a Large-scale Production Environment

Deploying and managing secrets in a large-scale production environment requires awareness of and proactiveness against risks like data breaches. There are a few things to consider when deploying secrets in a production environment to ensure the security of your data and protect your systems from unauthorized access.

To mitigate the risk of secrets exposure, best practices include:

  • Encrypt all sensitive data at rest and in transit
  • Rotate secrets regularly
  • Use strong passwords and passphrases for accessing external secrets backends
  • Restrict access to secrets to only those who need it
  • Monitor for unauthorized access using audit logs

Conclusion

In this article, we looked at the need for storing secrets securely in Kubernetes and explored External Secrets. While Kubernetes Secrets are a great way to store secrets within the cluster, they have security and lifecycle management limitations.

Kubernetes External Secrets address these shortcomings by storing secrets externally and providing an in-built secret rotation, robust lifecycle management, and multiple authentication mechanisms. We also reviewed the advantages of using External Secrets over standard Kubernetes Secrets.

If you are looking for a secure and robust way to manage secrets in your Kubernetes cluster, External Secrets are a great option to consider.

Comprehensive Kubernetes cost monitoring & optimization

Continue reading this series