So we started using IAM roles for service accounts in our Kubernetes cluster

At Track.Health we run our platform on an AWS EKS cluster. Up until now, we were using AWS credential files within our micro-services to access other AWS services(S3 for example).

Knowing it is a bad practice to keep credentials with the source code, I explored a few alternatives. Vaults were a good option to rotate keys and inject credentials as part of the build process.

However, today, I stumbled upon an eksctl command that lets you create a service account with a linked IAM role. The EKS cluster comes with an OpenID Connect (OIDC) identity provider which you can enable with eksctl after which you can create a service account backed by an IAM role.

The OIDC federation gives you the ability to assume an IAM role with STS(Secure Token Service).

More information on the same can be found here.

This post highlights the changes I did to get one of our micro-services that requires full access to S3 working with an IAM role backed service account.

EKSCTL setup

eksctl create iamserviceaccount \
--name s3fullaccess \
--namespace ns-utils \
--cluster  sandbox \
--attach-policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess \
--approve \
--override-existing-serviceaccounts \
--region ap-southeast-2

Let us verify the service account created on Kubernetes;

~$ kubectl describe sa s3fullaccess -n ns-utils
Name:                s3fullaccess
Namespace:           ns-utils
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>
Image pull secrets:  <none>
Mountable secrets:   s3fullaccess-token-bmp2z
Tokens:              s3fullaccess-token-bmp2z
Events:              <none>

What we really care about here is the annotation on the service account. This is used by an admission controller within AWS EKS that mutates pods that use this service account with two environment variables. We will see what these environment variables are in just a bit.

I specifically wanted to create this iamserviceaccount on the ns-utils namespace as that is where my micro-service runs. The policy specifies that I need S3 full access. As I have to pass the cluster name to the eksctl command, I scripted it as follows so that I can put it in our CI/CD pipeline.

CLUSTER_NAME=`eksctl get cluster --region ap-southeast-2 -o json | jq -r '.[0].name'`
eksctl create iamserviceaccount \
--name s3fullaccess \
--namespace ns-utils \
--cluster  $CLUSTER_NAME \
--attach-policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess \
--approve \
--override-existing-serviceaccounts \
--region ap-southeast-2

I installed the jq command-line utility to pass the json response and get the cluster name from the output. This was then passed into the — cluster option when creating the iamserviceaccount.

You can also use the script found here which does not depend on your cluster being created with eksctl to do the same.

With that out of the way, I proceeded to do the changes on my Kubernetes deployment yaml for the micro-service. To maintain brevity, I am only showing you the changes I had to do specifically to get my deployment working with OIDC.

spec:
  serviceAccountName: s3fullaccess
  securityContext:
    fsGroup: 65534

The serviceAccountName matches the one I provided when I created the iamserviceaccount with the eksctl command-line utility.

Hold up! What is up with the fsGroup you may ask.

To explain why, let us go into my pod and inspect.

$ env | grep AWS
AWS_ROLE_ARN=arn:aws:iam::780795068345:role/eksctl-sandbox-addon-iamserviceaccount-Role1-1Y053HXQ9Y5SY
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

An admission controller deployed on AWS EKS injects these environment variables when I run my pod with a service account that is backed by an IAM role.

Let us look at the AWS_WEB_IDENTITY_TOKEN_FILE path;

$ ls -ltr /var/run/secrets/eks.amazonaws.com/serviceaccount/token
lrwxrwxrwx 1 root root 12 May 18 15:07 /var/run/secrets/eks.amazonaws.com/serviceaccount/token -> ..data/token

As it points to a symbolic link, let us explore the actual file.

$ ls -ltr /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token
-rw-r----- 1 root nogroup 1022 May 18 15:07 /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token

The file belongs to the nogroup. The group id of the nogroup can be found in /etc/group;

$ cat /etc/group | grep nogroup
nogroup:x:65534:

This is where the 65534 value comes from which is specified as the fsGroup in my Kubernetes deployment file. Without this, your application will not be able to see the web token file.

Code related changes

Let us see what changes are needed to build up the credentials provider to access the AWS S3 service.

As we use Java, this is specific to the AWS Java SDK (Not the latest version).

Make sure your pom.xml has the following two dependencies;

<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-java-sdk-core</artifactId>
  <version>${aws.sdk.version}</version>
</dependency>
<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-java-sdk-sts</artifactId>
  <version>${aws.sdk.version}</version>
</dependency>

My current aws.sdk.version is 1.11.606.

Moving on to building the credentials provider(I am using Spring Boot so the configuration is exposed as a Spring bean);

As our cluster is running in the Sydney region, the region is passed in as ap-southeast-2.

The WebIdentityTokenCredentialsProvider deals with reading the environment variables that are injected to the POD and assuming the role with STS. We had to include the aws-java-sdk-sts as the WebIdentityTokenCredentialsProvider internally uses STSProfileCredentialsServiceProvider which has the following in its code;

try {
STS_CREDENTIALS_SERVICE = (ProfileCredentialsService)Class.forName("com.amazonaws.services.securitytoken.internal.STSProfileCredentialsService").newInstance();
} catch (ClassNotFoundException var1) {
throw new SdkClientException("To use assume role profiles the aws-java-sdk-sts module must be on the class path.", var1);
}

As STS uses short lived tokens, the SDK handles refreshing the token when it expires.

The whole process was quite seamless and now we can finally get rid of the AWS credentials files we were using before to access AWS services.

If there are any questions/clarifications needed, do not hesitate to hit me up with a comment.

Appreciate the read everyone. Have a good day ahead!

Follow me on https://dimashup.com for more posts.