Integrate AWS Secrets Manager with EKS Cluster
Streamlining Secret Management: Integrating AWS Secrets Manager with Your EKS Cluster
Introduction:
In today's rapidly evolving technological landscape, security is paramount. As organizations increasingly migrate their applications to the cloud, ensuring the security of sensitive information such as passwords, API keys, and other credentials becomes a critical concern. Amazon Web Services (AWS) offers a robust solution for managing these secrets securely through AWS Secrets Manager. When combined with Amazon Elastic Kubernetes Service (EKS), AWS Secrets Manager provides a seamless and highly secure method for managing secrets within Kubernetes clusters.
In this blog post, we'll explore how to integrate AWS Secrets Manager with an EKS cluster. We'll delve into the capabilities of both services and demonstrate how they can work together to enhance the security posture of your Kubernetes applications.
Overview of Services:
Amazon Elastic Kubernetes Service (EKS):
Amazon EKS is a managed Kubernetes service that simplifies the deployment, management, and scaling of containerized applications using Kubernetes on AWS. It provides a fully managed Kubernetes control plane and integrates with other AWS services to provide a seamless experience for deploying and managing containerized applications.
Key features of Amazon EKS include:
Managed Kubernetes control plane: AWS manages the control plane, ensuring high availability and scalability.
Integration with AWS services: EKS seamlessly integrates with various AWS services such as Elastic Load Balancing, IAM for authentication and authorization, CloudWatch for monitoring, and more.
Simplified operations: EKS automates tasks such as patching, updating, and scaling the Kubernetes infrastructure, allowing developers to focus on building and deploying applications.
AWS Secrets Manager:
AWS Secrets Manager is a fully managed service that helps you securely store, manage, and retrieve sensitive information such as API keys, passwords, and other credentials. It provides centralized management of secrets, eliminating the need to hardcode credentials in your applications or manage them manually.
Key features of AWS Secrets Manager include:
Secure storage: Secrets Manager encrypts your secrets using AWS Key Management Service (KMS) and provides fine-grained access control using AWS Identity and Access Management (IAM).
Automatic rotation: Secrets Manager can automatically rotate credentials for supported services, ensuring that your secrets are regularly updated without manual intervention.
Integration with AWS services: Secrets Manager seamlessly integrates with other AWS services such as Amazon RDS, Amazon Redshift, and now, Amazon EKS, allowing you to securely access secrets from your applications running on these services.
Step-by-Step Guide: Integrating AWS Secrets Manager with Amazon EKS
This guide demonstrates how to integrate AWS Secrets Manager with Amazon EKS for securely managing application secrets. By following these steps, you'll set up communication between your Kubernetes workloads and AWS Secrets Manager, ensuring seamless access to sensitive information.
Step 1: Set Environment Variables
Begin by defining environment variables to encapsulate your EKS cluster details. Confirm your current cluster context and set variables for the cluster name and region.
First, verify the current context of your cluster by executing the following command:
kubectl config current-context
Now define the cluster name & cluster region environment variables for the EKS cluster. Replace the <your-region> with your actual AWS region in which you are working.
export CLUSTER_NAME=$(aws eks describe-cluster --region <your-region> --name your-cluster-name --query "cluster.name" --output text)
export CLUSTER_REGION=$(aws eks describe-cluster --name ${CLUSTER_NAME} --region your-region --query "cluster.arn" --output text | cut -d: -f4)
# To validate the variables have been set properly, run the following commands
echo $CLUSTER_REGION
echo $CLUSTER_NAME
Step 2: Create a Secret in AWS Secrets Manager
Use the AWS CLI to create a sample secret in AWS Secrets Manager, eliminating the need to hard-code sensitive information in your application.
# The below command will also store the Secret’s ARN in a variable for later use.
SECRET_ARN=$(aws secretsmanager create-secret --name eksSecret --secret-string '{"username":"ekstutorial", "password":"eksisBest!"}' --region "$CLUSTER_REGION" --query ARN)
echo $SECRET_ARN
# The expected output must be an ARN (Amazon Resource Names) like below:
# "arn:aws:secretsmanager:ap-south-1:123456789:secret:eksSecret-KuuyrT"
Step 3: Create IAM Policy for Accessing the Secret
In this section, we'll create an IAM policy to grant permissions for accessing the secret stored in AWS Secrets Manager. Also, we'll use the $SECRET_ARN to specify which secret this IAM policy should apply to. This will ensure that only specified Secret can be accessed by EKS cluster.
POLICY_ARN=$(aws --region "$CLUSTER_REGION" --query Policy.Arn --output text iam create-policy --policy-name ekstutorial-secretsmanager-policy --policy-document '{
"Version": "2012-10-17",
"Statement": [ {
"Effect": "Allow",
"Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
"Resource": ['$SECRET_ARN']
} ]
}')
echo $POLICY_ARN
Step 4: Create IAM Role and Associate with Kubernetes Service Account
Utilize IAM Roles for Service Accounts (IRSA) to map Kubernetes service accounts to AWS IAM roles, enabling fine-grained permission management. By using eksctl, we'll create and associate an AWS IAM Role with a specific Kubernetes service account within the EKS cluster.
eksctl create iamserviceaccount --name ekstutorial-secretmanager-sa --region="$CLUSTER_REGION" --cluster "$CLUSTER_NAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts
# To ensure the "ekstutorial-secretmanager-sa" service account is correctly set up in the "default" namespace in your cluster. Run below command.
kubectl get sa ekstutorial-secretmanager-sa -o yaml
This will look like below:
NOTE:
You must have an OpenID Connect (OIDC) endpoint associated with your cluster before you running above commands.
With the Secret Store CSI driver, we will apply IAM permissions at the application pod level, not the CSI driver pods. This ensures that only the specific application pods that are leveraging the IRSA associated Kubernetes service account will have permission to access the secret stored in AWS Secrets Manager.
Step 5: Install AWS Secrets and Configuration Provider and Secrets Store CSI Driver
Install the AWS Secrets and Configuration Provider (ASCP) and Secrets Store CSI Driver using Helm, facilitating secure communication between AWS Secrets Manager and your Kubernetes cluster. This will set up a secure bridge between AWS Secrets Manager and your Kubernetes cluster.
It also enables your cluster to access secrets stored in AWS Secrets Manager without requiring complex application-code changes. The ASCP and Secrets Store CSI Driver will each be installed as DaemonSets to ensure a copy of the driver and provider are running on each node in the cluster.
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm install -n <secret-namespace> csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver
helm install -n <secret-namespace> secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws
# To verify the installation has completed successfully:
helm list -n <secret-namespace>
Replace the <secret-namespace> with the namespace on which you want to deploy your CSI driver and ASCP.
To verify the Secrets Store CSI Driver has started, run the following command:
kubectl --namespace=<secret-namespace> get pods -l "app=secrets-store-csi-driver"
Step 6: Create ASCP SecretProviderClass Resource
Define the SecretProviderClass Kubernetes object, specifying the secrets to fetch from AWS Secrets Manager and how to mount them into your pods.
This helps in the seamless integration of secrets within your Kubernetes workloads. ASCP SecretProviderClass Resource acts as a set of instructions for the AWS Secrets and Configuration Provider (ASCP), specifying which secrets to fetch from AWS Secrets Manager and how to mount them into your pods.
Note: The SecretProviderClass must be deployed in the same namespace as the workload that references it. To learn more, see SecretProviderClass documentation.
Create a Kubernetes manifest as below and give it a name as ekstutorial-spc.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: eks-tutorial-aws-secrets
namespace: default
spec:
provider: aws
parameters:
objects: |
- objectName: "eksSecret"
objectType: "secretsmanager"
To apply the YAML manifest, run the following command.
kubectl apply -f ekstutorial-spc.yaml
To verify the SecretProviderClass was created successfully, run the following command:
kubectl describe secretproviderclass eks-tutorial-aws-secrets
Step 7: Deploy Sample Workload
Finally, deploy a sample workload that bridges your application and AWS Secrets Manager. To complete the end-to-end process of securely managing and accessing secrets within your Kubernetes environment, mount the secret as a file on the workload's filesystem.
In the pod template, you will specify the Secrets Store CSI as the volume driver and then a path to mount your secret, just like you would a traditional volume mount.
In this example, we will mount the secret in the /mnt/secrets-store
location. So create a Kubernetes manifest called ekstutorial-app.yaml
and paste the following contents into it:
apiVersion: v1
kind: Pod
metadata:
name: TestSecretPod
namespace: default
spec:
serviceAccountName: ekstutorial-secretmanager-sa
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "eks-tutorial-aws-secrets"
containers:
- image: public.ecr.aws/docker/library/busybox:1.36
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
name: busybox
volumeMounts:
- name: secrets-store-inline
mountPath: "/mnt/secrets-store"
readOnly: true
restartPolicy: Always
Apply the YAML manifest.
kubectl apply -f ekstutorial-app.yaml
Step 8: Test the Secrets manager and EKS integration
In the end, we'll utilize Kubectl to enter the newly deployed pod and verify our ability to access the mounted secret.
kubectl exec -it $(kubectl get pods | awk '/TestSecretPod/{print $1}' | head -1) -- cat /mnt/secrets-store/eksSecret; echo
This command is expected to display the previously created secret:
{"username":"ekstutorial", "password":"eksisBest!"}
Conclusion
Upon completing this tutorial, you'll have successfully established an integration between AWS Secrets Manager and your Amazon EKS cluster. This integration streamlines the management of your application secrets, enabling effortless consumption by your EKS workloads without intricate code modifications. Enhancing security, governance, and portability of your applications is achieved with minimal overhead. Furthermore, this example can be effortlessly replicated for managing various types of secrets, including database credentials, API keys, and more.
For further insights into setting up and managing Amazon EKS for your workloads, delve into "Navigating Amazon EKS."