Use Case Scenario

Problem: In most Kubernetes applications, secrets are hardcoded or statically injected, requiring pod restarts when secrets like database passwords are rotated. This creates downtime and maintenance overhead.

Solution: Integrate AKS with Azure Key Vault using the CSI driver and enable automatic secret rotation. Secrets will be mounted into pods and updated dynamically without restarting the pod, ensuring zero-downtime secret updates.

Architecture Overview

The architecture includes:

• AKS Cluster
• Azure Key Vault
• Workload Identity (OIDC) for secure identity management
• CSI Secrets Store Driver for mounting secrets
• Auto-Rotation of secrets via polling

Step-by-Step Implementation

To create a AKS cluster using CLI please follow this blog: AKS Cluster Setup Using Azure CLI with OIDC & Azure Key Vault Integration

1. Enable OIDC and Workload Identity on exiting AKS cluster

az aks update \
  --name <cluster-name> \
  --resource-group <rg> \
  --enable-oidc-issuer \
  --enable-workload-identity

az aks update \
  --name <cluster-name> \
  --resource-group <rg> \
  --enable-oidc-issuer \
  --enable-workload-identity

To enable Azure Key Vault CSI driver after the cluster is created:

az aks enable-addons \
  --addons azure-keyvault-secrets-provider \
  --name <cluster-name> \
  --resource-group <rg>
  

az aks enable-addons \
  --addons azure-keyvault-secrets-provider \
  --name <cluster-name> \
  --resource-group <rg>
  

You can verify through azure portal under your kuberenets cluster dashboard “Security Configuration” tab

Verify that each node in your cluster’s node pool has a Secrets Store CSI Driver pod and a Secrets Store Provider Azure pod running

kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)' -o wide

kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)' -o wide

1.2 Keyvault creation and configuration

Create a key vault with Azure role-based access control (Azure RBAC).

az keyvault create -n my-demo-k8s-key-vault -g keyvault-demo -l eastus --enable-rbac-authorization

az keyvault create -n my-demo-k8s-key-vault -g keyvault-demo -l eastus --enable-rbac-authorization

2. Create a Managed Identity

Please export following values on your terminal, make sure you have added your subscription id..etc

export SUBSCRIPTION_ID=fe4a1fdb-6a1c-4a6d-a6b0-dbb12f6a00f8
export RESOURCE_GROUP=keyvault-demo
export UAMI=azurekeyvaultsecretsprovider-keyvault-demo-cluster
export KEYVAULT_NAME=my-demo-k8s-key-vault
export CLUSTER_NAME=keyvault-demo-cluster

az account set --subscription $SUBSCRIPTION_ID

export SUBSCRIPTION_ID=fe4a1fdb-6a1c-4a6d-a6b0-dbb12f6a00f8
export RESOURCE_GROUP=keyvault-demo
export UAMI=azurekeyvaultsecretsprovider-keyvault-demo-cluster
export KEYVAULT_NAME=my-demo-k8s-key-vault
export CLUSTER_NAME=keyvault-demo-cluster

az account set --subscription $SUBSCRIPTION_ID

To Create a managed identity, following azure cli command

az identity create --name $UAMI --resource-group $RESOURCE_GROUP

export USER_ASSIGNED_CLIENT_ID="$(az identity show -g $RESOURCE_GROUP --name $UAMI --query 'clientId' -o tsv)"

export IDENTITY_TENANT=$(az aks show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --query identity.tenantId -o tsv)

az identity create --name $UAMI --resource-group $RESOURCE_GROUP

export USER_ASSIGNED_CLIENT_ID="$(az identity show -g $RESOURCE_GROUP --name $UAMI --query 'clientId' -o tsv)"

export IDENTITY_TENANT=$(az aks show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --query identity.tenantId -o tsv)

Create a role assignment that grants the workload ID access the key vault

export KEYVAULT_SCOPE=$(az keyvault show --name $KEYVAULT_NAME --query id -o tsv)

az role assignment create --role "Key Vault Administrator" --assignee $USER_ASSIGNED_CLIENT_ID --scope $KEYVAULT_SCOPE

export KEYVAULT_SCOPE=$(az keyvault show --name $KEYVAULT_NAME --query id -o tsv)

az role assignment create --role "Key Vault Administrator" --assignee $USER_ASSIGNED_CLIENT_ID --scope $KEYVAULT_SCOPE

Get the AKS cluster OIDC Issuer URL

export AKS_OIDC_ISSUER="$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv)"

echo $AKS_OIDC_ISSUER

export AKS_OIDC_ISSUER="$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv)"

echo $AKS_OIDC_ISSUER

3. Annotate Kubernetes ServiceAccount

export SERVICE_ACCOUNT_NAME="workload-identity-sa"
export SERVICE_ACCOUNT_NAMESPACE="default" 

export SERVICE_ACCOUNT_NAME="workload-identity-sa"
export SERVICE_ACCOUNT_NAMESPACE="default" 

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${SERVICE_ACCOUNT_NAME}
  namespace: ${SERVICE_ACCOUNT_NAMESPACE}
  annotations:
    azure.workload.identity/client-id: "${USER_ASSIGNED_CLIENT_ID}"
EOF

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${SERVICE_ACCOUNT_NAME}
  namespace: ${SERVICE_ACCOUNT_NAMESPACE}
  annotations:
    azure.workload.identity/client-id: "${USER_ASSIGNED_CLIENT_ID}"
EOF

Or If above syntax confusing, SA direct yaml syntax given below fill details accordingly

apiVersion: v1
kind: ServiceAccount
metadata:
  name: workload-identity-sa
  annotations:
    azure.workload.identity/client-id: <your-client-id>
    namespace: default

apiVersion: v1
kind: ServiceAccount
metadata:
  name: workload-identity-sa
  annotations:
    azure.workload.identity/client-id: <your-client-id>
    namespace: default

Setup Federation

export FEDERATED_IDENTITY_NAME="aksfederatedidentity" 

az identity federated-credential create --name $FEDERATED_IDENTITY_NAME --identity-name $UAMI --resource-group $RESOURCE_GROUP --issuer ${AKS_OIDC_ISSUER} --subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}

export FEDERATED_IDENTITY_NAME="aksfederatedidentity" 

az identity federated-credential create --name $FEDERATED_IDENTITY_NAME --identity-name $UAMI --resource-group $RESOURCE_GROUP --issuer ${AKS_OIDC_ISSUER} --subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}

4. Create SecretProviderClass

cat <<EOF | kubectl apply -f -
# This is a SecretProviderClass example using workload identity to access your key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-wi # needs to be unique per namespace
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "${USER_ASSIGNED_CLIENT_ID}" # Setting this to use workload identity
    keyvaultName: ${KEYVAULT_NAME}       # Set to the name of your key vault
    cloudName: ""                         # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: DB-Password         # Set to the name of your secret
          objectType: secret              # object types: secret, key, or cert
          objectVersion: ""              
    tenantId: "${IDENTITY_TENANT}"        # The tenant ID of the key vault
EOF

cat <<EOF | kubectl apply -f -
# This is a SecretProviderClass example using workload identity to access your key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-wi # needs to be unique per namespace
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "${USER_ASSIGNED_CLIENT_ID}" # Setting this to use workload identity
    keyvaultName: ${KEYVAULT_NAME}       # Set to the name of your key vault
    cloudName: ""                         # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: DB-Password         # Set to the name of your secret
          objectType: secret              # object types: secret, key, or cert
          objectVersion: ""              
    tenantId: "${IDENTITY_TENANT}"        # The tenant ID of the key vault
EOF

Or use below direct yaml secretproviderclass and fill details accordingly

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-wi
spec:
  provider: azure
  parameters:
    keyvaultName: <your-kv-name>
    tenantId: <your-tenant-id>
    clientID: <your-client-id>
    objects: |
      array:
        - objectName: DB-Password
          objectType: secret
          objectVersion: ""

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-wi
spec:
  provider: azure
  parameters:
    keyvaultName: <your-kv-name>
    tenantId: <your-tenant-id>
    clientID: <your-client-id>
    objects: |
      array:
        - objectName: DB-Password
          objectType: secret
          objectVersion: ""

5. Deploy a Workload (e.g., BusyBox Test Pod)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox-secrets
  template:
    metadata:
      labels:
        app: busybox-secrets
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: workload-identity-sa
      containers:
        - name: busybox
          image: registry.k8s.io/e2e-test-images/busybox:1.29-4
          command: ["/bin/sleep", "10000"]
          volumeMounts:
            - name: secrets-store-vol
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store-vol
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: azure-kvname-wi
              rotationPollInterval: "30s"

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox-secrets
  template:
    metadata:
      labels:
        app: busybox-secrets
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: workload-identity-sa
      containers:
        - name: busybox
          image: registry.k8s.io/e2e-test-images/busybox:1.29-4
          command: ["/bin/sleep", "10000"]
          volumeMounts:
            - name: secrets-store-vol
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store-vol
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: azure-kvname-wi
              rotationPollInterval: "30s"

Now you can exec into the pod and verify the secret are mounted into the container. You can run following command to verify the same

kubectl exec <pod> -- ls /mnt/secrets-store/
kubectl exec <pod> -- cat /mnt/secrets-store/DB-Password

kubectl exec <pod> -- ls /mnt/secrets-store/
kubectl exec <pod> -- cat /mnt/secrets-store/DB-Password

⟳ Enable Auto-Rotation of Secrets

Run the following command to enable the rotation feature:

az aks addon update \
  --resource-group <rg> \
  --name <cluster-name> \
  --addon azure-keyvault-secrets-provider \
  --enable-secret-rotation \
  --rotation-poll-interval 30s

az aks addon update \
  --resource-group <rg> \
  --name <cluster-name> \
  --addon azure-keyvault-secrets-provider \
  --enable-secret-rotation \
  --rotation-poll-interval 30s

Verify it with:

kubectl -n kube-system describe ds aks-secrets-store-csi-driver

kubectl -n kube-system describe ds aks-secrets-store-csi-driver

Ensure the arguments include:

• --enable-secret-rotation=true
• --rotation-poll-interval=30s

To work Azure Key Vault auto rotation in deployment, Make sure following settings are there in first

1) Under SecretProviderClass Leave objectVersion blank

2) Add to volume attributes on deployment file: rotationPollInterval

volumeAttributes:
  secretProviderClass: azure-kvname-wi
  rotationPollInterval: "30s"

volumeAttributes:
  secretProviderClass: azure-kvname-wi
  rotationPollInterval: "30s"

Testing the Rotation

1. Manually update the secret version in Azure Key Vault.
2. Wait for rotationPollInterval.
3. Check the mounted file inside the pod:

kubectl exec <pod> -- ls /mnt/secrets-store/
kubectl exec <pod> -- cat /mnt/secrets-store/DB-Password

kubectl exec <pod> -- ls /mnt/secrets-store/
kubectl exec <pod> -- cat /mnt/secrets-store/DB-Password

4. Observe that the new value is reflected without restarting the pod.

Best Practices

• Avoid subPath in volume mounts (breaks rotation).
• Ensure your application reads secrets from file, not env vars.
• Consider implementing file watchers for dynamic config reloads.
• Monitor CSI driver logs: kubectl logs ds/aks-secrets-store-csi-driver -n kube-system

Conclusion

With Azure Key Vault, CSI driver, and workload identity, you can achieve secure, automated, and zero-downtime secret management for your AKS workloads. Auto-rotation ensures credentials like DB passwords can be updated without disrupting live applications.

Feel free to follow me on LinkedIn and share your thoughts. For a complete YAML reference or demo repo, connect with me or leave a comment!

The post Secure Secret Management in AKS with Azure Key Vault CSI Driver and Auto-Rotation Enabled appeared first on abdulrahmanuk.com.

Thankfully, Kubernetes provides powerful tools to enforce resource limits at the namespace level, allowing teams to share the cluster fairly. In this blog post, we’ll explore two important tools for namespace-level resource management:

• Resource Quotas
• Limit Ranges

Let’s dive in and see how they work, with practical examples to help you get started.

What Are Resource Quotas in Kubernetes?

Resource Quotas let administrators cap the total amount of resources that a specific namespace can use. Think of it as setting a budget for pods, CPU, memory, and other compute resources.

Why Use Resource Quotas?

• Define clear usage boundaries per namespace.
• Monitor and track resource usage in real-time.
• Automatically block workloads that try to exceed their limits.

How to Set a Resource Quota

To apply a Resource Quota, create a YAML file and apply it to the relevant namespace.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: resource-quota-example
  namespace: example-namespace
spec:
  hard:
    pods: "1"
    requests.cpu: "2"
    requests.memory: "5Gi"
    limits.cpu: "4"
    limits.memory: "10Gi"

Then apply it using:

kubectl apply -f resource-quota.yaml

This will limit the example-namespace to 1 pod, 2 CPU requests, 5Gi of memory requests, and caps CPU and memory usage at 4 CPUs and 10Gi, respectively.

Verifying the Resource Quota

Check if the quota has been successfully applied:

kubectl get resourcequota resource-quota-example -n example-namespace

Or to get detailed info:

kubectl describe ns example-namespace

Sample output:

Resource         Used  Hard
limits.cpu       0     4
limits.memory    0     10Gi
pods             0     1
requests.cpu     0     2
requests.memory  0     5Gi

Testing Quota Enforcement

Let’s try deploying more pods than allowed to see the quota in action.

Deployment with 2 replicas (but the quota allows only 1):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: quota-test
  namespace: example-namespace
  labels:
    app: deployment-label
spec:
  replicas: 2
  selector:
    matchLabels:
      app: deployment-label
  template:
    metadata:
      labels:
        app: deployment-label
    spec:
      containers:
      - name: nginx-deploy
        image: nginx:latest
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 100m
            memory: 100Mi

Apply the deployment:

kubectl apply -f deploy.yaml

Check the pod status:

kubectl get all -n example-namespace

ou’ll see something like:

Warning  FailedCreate  pods "quota-test-xxxx" is forbidden: exceeded quota: resource-quota-example...

The second pod fails to start—exactly what we want. The quota is working as intended!

What Are Limit Ranges in Kubernetes?

While Resource Quotas set overall namespace limits, Limit Ranges control the default and maximum/minimum resource limits for each pod, container, or PVC. This ensures no single pod or container hogs cluster resources.

Why Use Limit Ranges?

• Enforce consistent resource consumption across deployments.
• Prevent resource abuse by setting sensible defaults.
• Ensure all pods have a minimum allocation for healthy operation.

How to Set a Limit Range

Create a limit-range.yaml file:

apiVersion: v1
kind: LimitRange
metadata:
  name: limit-range-example
  namespace: example-namespace
spec:
  limits:
  - type: Pod
    max:
      cpu: "2"
      memory: "4Gi"
    min:
      cpu: "200m"
      memory: "256Mi"
    maxLimitRequestRatio:
      cpu: "4"
      memory: "8"
  - type: Container
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "250m"
      memory: "256Mi"
    max:
      cpu: "1"
      memory: "1Gi"
    min:
      cpu: "100m"
      memory: "128Mi"
    maxLimitRequestRatio:
      cpu: "2"
      memory: "4"
  - type: PersistentVolumeClaim
    max:
      storage: "10Gi"
    min:
      storage: "1Gi"
    default:
      storage: "5Gi"
    defaultRequest:
      storage: "2Gi"
    maxLimitRequestRatio:
      storage: "2"

Apply it:

kubectl apply -f limit-range.yaml

Limit Ranges apply defaults automatically to pods if you forget to specify them!

Verifying the Limit Range

Use the following commands:

kubectl describe limitrange limit-range-example -n example-namespace
kubectl describe ns example-namespace

You’ll see something like:

Type                   Resource  Min    Max   Default Request  Default Limit  Max Limit/Request Ratio
Pod                    cpu       200m   2     -                -              4
Pod                    memory    256Mi  4Gi   -                -              8
Container              cpu       100m   1     250m             500m           2
Container              memory    128Mi  1Gi   256Mi            512Mi          4
PersistentVolumeClaim  storage   1Gi    10Gi  2Gi              5Gi            2

Testing the Limit Range in Action

Create a deployment without resource specs and watch the defaults kick in:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: limit-test
  namespace: example-namespace
  labels:
    app: deployment-label
spec:
  replicas: 1
  selector:
    matchLabels:
      app: deployment-label
  template:
    metadata:
      labels:
        app: deployment-label
    spec:
      containers:
      - name: nginx-deploy
        image: nginx:latest

Apply it:

kubectl apply -f deploy.yaml

Now describe the pod and you’ll notice the resource requests and limits are automatically applied based on the LimitRange!

Final Thoughts

Namespace-level resource control in Kubernetes is not just a best practice—it’s a necessity for scalable and stable clusters.

Use Resource Quotas to enforce overall limits per namespace.
Use Limit Ranges to set smart defaults and enforce per-container or per-pod limits.

By combining both, you ensure fair resource distribution, avoid noisy neighbor problems, and maintain consistent workload performance across your Kubernetes environment.

Got questions about resource management in Kubernetes?
Drop a comment below or connect with me for more DevOps tips and Kubernetes best practices.

Happy Clustering!

The post Kubernetes Resource Limits Simplified – From Beginner to Production Ready appeared first on abdulrahmanuk.com.

Step 1: Create an Azure Resource Group

az group create --name keyvault-demo --location eastus

This creates a logical container for your AKS resources.

Step 2: Create the AKS Cluster with Workload Identity and Azure Key Vault Integration

az aks create \
  --name keyvault-demo-cluster \
  --resource-group keyvault-demo \
  --node-count 1 \
  --enable-addons azure-keyvault-secrets-provider \
  --enable-oidc-issuer \
  --enable-workload-identity

Explanation of Flags:

• --enable-addons azure-keyvault-secrets-provider: Installs the CSI driver and Azure Key Vault provider addon.
• --enable-oidc-issuer: Enables the OIDC issuer URL for secure authentication with federated identity.
• --enable-workload-identity: Activates Azure Workload Identity (replacement for AAD Pod Identity).

Step 3: Get AKS Credentials for kubectl

az aks get-credentials \
  --resource-group keyvault-demo \
  --name keyvault-demo-cluster

This updates your local kubeconfig so you can interact with the new cluster.

Step 4: Verify CSI Driver and Azure Provider Pods

Make sure everything is running correctly:

kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)' -o wide

You should see pods like:

• secrets-store-csi-driver-*
• secrets-store-provider-azure-*

Bonus: Why Use Azure Key Vault with AKS?

• Centralized Secrets Management
• Automatic Secret Rotation
• No Secret Mounting in Code
• Secure Identity Binding with Workload Identity

This setup is cloud-native, secure, and production-ready.

The post AKS Cluster Setup Using Azure CLI with OIDC & Azure Key Vault Integration appeared first on abdulrahmanuk.com.