Azure Kubernetes Service (AKS) and Managed Identities

-

In this blog I will be exploring the use of Azure Manged Identities in Azure Kubernetes Service (AKS).  We will then discuss how we can use managed identities according to security best practice. We will look at how we configure the managed identities for the AKS cluster so it can in turn manage other Azure resources.  We will explore how we can configure managed identities for our services/applications that are running on AKS so pods can reach out to other Azure services.

Managed Identities

Why are we using managed identities? The alternative is to use Service Principal accounts (SPNs). The issues with SPNs is you have a client secret which you have to manage and keep secure. Your cluster apps and services will need to access the SPNs you have created so this means potentially saving it in a few places so it is available to CI/CD pipelines. The secret attached to an SPN rotates so you need to ensure it is valid to ensure your cluster and services continue to run.

Managed identities essentially are using SPNs under the hood but they make the management simpler. Managed identities manage key rotation which occurs every 46 days. Instead of constantly having a account with a client ID and secret to access something services reach out to managed identities to request a token when they need it. 

Managed identities like SPNs can also be scoped across subscriptions,  In a hub and spoke configuration you may have some shared resources in a hub like your Azure Container Registry (ACR) where the cluster in the spoke configuration needs to pull down images from the ACR in the hub.  

In the scenario we will talk about we will be configuring the AKS cluster to use managed identities and we will use managed identities for the services we deploy to our AKS cluster. To see how to create an AKS cluster that uses managed identities read this Microsoft guide

AKS Identities

There are a couple of identities for AKS;

Control plane - Used by AKS control plane components to manage cluster resources including ingress load balancers and AKS managed public IPs, and Cluster auto-scaler operations

Kubelet - AKS Cluster name-agentpool

To view the details of these identities to extract IDs you can use the following Azure cli commands

All AKS details

 az aks show -g <rg-name> -n <cluster-name>

AKS Identity - control pane

 az aks show -g <rg-name> -n <cluster-name> --query "identity"

Kubelet Identity

 az aks show -g <rg-name> -n <cluster-name> --query "identityProfile"

Below are some of the common roles and scopes I have assigned to AKS clusters, notice how we limit the scope and roles to provide least privilege access;

Allow AKS to manage the Load Balance
Scope: AKS Subnet where the LB is deployed
Role: "Network Contributor"
Identity: AKS Identity

Allow AKS to access the Azure container registry
Scope: Azure Container registry
Role: "acrpull"
Identity: Kubelet Identity

Allow AKS to manage DNS records
Scope: DNS zone
Role: "DNS Zone Contributor""
Identity: Kubelet Identity

Ideally you should be including this config in your Infrastructure as Code (IaC) configuration. An example of how I would do this in Terraform for the two different identity type

resource "azurerm_role_assignment" "aks_network_role" {
  scope                = azurerm_subnet.subnet.id
  role_definition_name = "Network Contributor"
  principal_id         = azurerm_kubernetes_cluster.k8s.identity[0].principal_id
}

resource "azurerm_role_assignment" "managed_id_role" {
  scope                = data.azurerm_resource_group.k8s-rg.id
  role_definition_name = "Managed Identity Operator"
  principal_id         = azurerm_kubernetes_cluster.k8s.kubelet_identity[0].object_id
  depends_on           = [data.azurerm_resource_group.k8s-rg]
}

AAD Pod identity

The next challenge is how do the Pods you are running in AKS reach out to other Azure services.  The answer to this is AAD Pod Identity. AAD Pod identity is a service that you run on your AKS cluster which provides a way for pods to access Azure resources using Azure Active Directory and the managed identities we configure for our roles. Review the identity best practices here.

You have a resource group for the AKS service and one for the resources like the Virtual Machine Scale set (VMSS) that hosts the AKS node pool. The resource group below relates to the resource group containing the AKS VMSS.

The AAD Pod Identity role assignment wiki provides the roles we need to add to allow AAD Pod Identity access to assign permissions for the resources in the VMSS.  There are two additional roles we need to assign to the kubelet identity during the initial AAD Pod identity setup.  Again, you can add this to your AKS IaC config so it is in place .

Scope: AKS RG containing VMSS
Role: "Managed Identity Operator"
Identity: Kubelet Identity

Scope: AKS RG containing VMSS
Role: "Virtual Machine Contributor"
Identity: Kubelet Identity

Install AAD Pod Identity on AKS

To install AAD Pod Identity on to your cluster go to this guide here .  The Helm commands are;

helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
helm install aad-pod-identity aad-pod-identity/aad-pod-identity

Create Managed Identities for our application

The next task is to create managed identities for you application.  To make this secure you should make new managed identities per service/application and give these the least privileges to carry out the operations they need with a limited scope.

Scopes 

We can set a scope, what can this identity access as with all security best practice lock this down to as small a scope as possible.  We can scope to subscription, resource group and resource.

Roles

Each service has specific built-in roles, to review what roles can be assigned to the various Azure resources use this built-in roles reference. We should pick the role that best meets the need of the service/application without providing unnecessary elevated privileges.

Here are a couple of examples;

Storage

We have a application that needs to read files from a storage account. We would not want to give this account the Storage Blob Data Contributor role.  Instead we would create an Identity with the Storage Blod Data Reader Role and scope this down to the storage account it need access to .

Service Bus

We have a application that needs to send messages to a Service Bus queue. We can limit the managed identity down to Azure Service Bus Data Sender this allows for send access to Azure Service Bus resources. We can scope this down to the resource group containing the Service Bus resource.

Enabling AAD Pod Identity on our Services

Now we have the AAD Pod Identity service installed on our cluster and we have created some managed identities for our application we need to configure our pods so they can use this service. To allow our pods to make use of the AAD Pod Identity service we have to apply some configuration when we deploy each service.  You can include this in your Helm charts so it is a standard part of the deployment of each service.  This walkthrough shows you an example. 

To complete the config you will need the Managed Identity Resource ID and the client ID. The resource ID should be in this format.

/subscriptions/<subscription_id>/resourcegroups/<cluster-rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<managed-id-name>


We first create an Azure Identity

cat <<EOF | kubectl apply -f -
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
name: ${IDENTITY_NAME}
spec:
type: 0
resourceID: ${IDENTITY_RESOURCE_ID}
clientID: ${IDENTITY_CLIENT_ID}
EOF

We then create an Azure Identity Binding 

cat <<EOF | kubectl apply -f -
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: ${IDENTITY_NAME}-binding
spec:
  azureIdentity: ${IDENTITY_NAME}
  selector: ${IDENTITY_NAME}
EOF

When we deploy our service we then reference the identity binding from the step above

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: demo
  labels:
    aadpodidbinding: $IDENTITY_NAME
spec:
  containers:
  - name: demo
    image: mcr.microsoft.com/oss/azure/aad-pod-identity/demo:v1.7.1
    args:
      - --subscriptionid=${SUBSCRIPTION_ID}
      - --clientid=${IDENTITY_CLIENT_ID}
      - --resourcegroup=${IDENTITY_RESOURCE_GROUP}
    env:
      - name: MY_POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
      - name: MY_POD_NAMESPACE
        valueFrom:
          fieldRef:
            fieldPath: metadata.namespace
      - name: MY_POD_IP
        valueFrom:
          fieldRef:
            fieldPath: status.podIP
  nodeSelector:
    kubernetes.io/os: linux
EOF


The walkthrough guide provides some steps to confirm AAD Pod Identity is working from the pod as expected.

We have covered some of the basics of managed identities, the common roles we may need to assign to the cluster.  How we can use managed identities AAD pod identity to assign granular permissions to our applications running on AKS.  I hope this article has provided a good overview of the role of managed identities in AKS deployments.


Related Articles:

Comments

Post a Comment

Popular posts from this blog

Working with WSL and AKS

-
Image
Introduction I find I am working with  Azure Kubernetes Service (AKS) more and more recently and I thought I would share a few tips and snippets of code I have found useful. For reasons I will explain shortly I have started using the Windows Subsystem for Linux (WSL) as my main way of managing AKS.  If you are starting out with AKS I hope you will find some of these hints and tips useful. AKS is a fully managed Kubernetes service from Azure, if you want to find out more about AKS check out the material and video from Microsoft here  .  For the purpose of this document I am assuming you are familiar with AKS and have at least started to play around with it.   Commands There are some key commands you will need when working with AKS and kubectl you can find some of these on the first link below.  While working with AKS you will  be using other tools like Docker for creating and managing your container images.,  I have provided some key starte...
Read more

Next-gen Cloud Operations

-
Image
Moving from a traditional Operations Role to a Cloud/DevOps Engineer Role If you are in an infrastructure or operations role and want to make the move to a new role perhaps Cloud/DevOps engineer or an SRE role ,  maybe your company is making changes which means your role is evolving then this article may help to identify some of the skills you need to make this move.   As you read about some of these tools and skills you need to learn you will see many that you may feel fit with a developer role not a operations engineer but keep reading and embrace the code.  Start Small Version Control Cloud Platform Infrastructure as code  CI/CD Pipelines Code Editing   Find a Buddy Embrace new tools Conclusion It is worth noting that you have a lots of valuable and important knowledge you will bring with you to any new role. All the areas we cared about before, as shown in the operabilty diagram below, we still care about now, you may be managing similar solutions an...
Read more