fbpx

Bootstrap Security in Kubernetes Deployments


Kubernetes is one of the most popular and most used container orchestration tools. Kubernetes Workloads are the actual applications that are executed like a simple nginx server or maybe a cron job. Kubernetes Deployments is the most commonly used workload as it can be easily updated, scaled, and managed.

The recently released Kubernetes Hardening Guide is an excellent resource that provides proper guidance on how to effectively secure Kubernetes. The information presented in the guide clearly shows that securing and hardening Kubernetes is not just the job of the Kubernetes administrator but also of the developers who are deploying their workload on the clusters.

In this blog, I’ll discuss how developers deploying Kubernetes workloads like Deployments can bootstrap security by applying some of the guidelines provided by the ‘Kubernetes Hardening guide’.

This will be a practical hands-on guide where I shall take a simple Dockerfile and then incrementally add the security best practices to create a template Deployment manifest file which can then be reused by developers in a hurry.

Pre-Requisites

  • Docker is required, as we’ll be building from the ground up.
  • A single-node Kubernetes cluster like minikube should be sufficient to follow along with this guide, along with the kubectl utility. You can use the official minikube documentation to set it up in your environment.

I am using a standalone cluster created by Docker Desktop tied to WSL2 as the backend.

This guide will assume that you have a running cluster that is accessible through the kubectl utility as shown below.

Securing Deployments

Securing the Kubernetes workloads can effectively be compartmentalized into “Build time” and “Runtime” security. In order to run with the examples, we’ll make use of this simple Spring Boot HelloWorld application and deploy it in Kubernetes with build time and runtime security applied.

https://github.com/salecharohit/bootstrapsecurityinkubernetesdeployment

So before starting off, let’s clone this repository, build the docker container, and run the application locally.

git clone git@github.com:salecharohit/bootstrapsecurityinkubernetesdeployment.git
cd springbootmaven
docker build . -f Dockerfile.basic -t springbootmaven
docker run --name springboot -d -p 8080:8080 springbootmaven
curl http://localhost:8080

Expected Response:

Hello World From Spring Boot Build Using Maven on Alpine OS!

Build Time Security

Build time security focuses more on how the underlying containers can be built with a reduced footprint and are programmed to be executed with the least possible privileges.

We’ll discuss both these approaches with a problem solution approach

Attack Surface Reduction

When building applications in a container, the primary objective is to have the app run consistently and independently regardless of the environment, whether it be a data center, cloud, or even on-premise. However, when building these apps, there is one unwritten rule: it should be a standalone application without many dependencies.

Let’s take the example of our SpringBoot application. The only dependency for our application to run is that it needs a JVM or Java runtime. Anything else baked into the container is practically useless.

As an example, in our SpringBoot container, which is built on Alpine OS, we don’t have any specific need to have the requirement for the apk package manager to be installed.

docker exec -it springboot /bin/sh
apk add curl

So let’s try to remove the apk binary and rebuild or docker image.

We’ll make use of the Dockerfile.asr at this time to rebuild our docker container which is shared below:

FROM maven:3.8.1-openjdk-17-slim AS MAVEN_BUILD
WORKDIR /build/
COPY pom.xml /build/
COPY src /build/src/
RUN mvn package

FROM openjdk:17-alpine

RUN rm -f /sbin/apk && 
    rm -rf /etc/apk && 
    rm -rf /lib/apk && 
    rm -rf /usr/share/apk && 
    rm -rf rm -rf /var/lib/apk

COPY --from=MAVEN_BUILD /build/target/springbootmaven.jar /springbootmaven.jar
EXPOSE 8080
CMD java -jar /springbootmaven.jar

Let’s rebuild and re-rerun:

# First let's stop the previously running container
docker stop springboot 
# Next let's re-build and re-run
docker build . -f Dockerfile.asr -t springbootmaven 
docker run --name springboot -p 8080:8080 springbootmaven 
docker run --name springboot -d -p 8080:8080 springbootmaven 
curl http://localhost:8080

Now let’s try to run the apk add curl command again.

docker exec -it springboot /bin/sh 
apk add curl

So we successfully got rid of the apk dependency and yet have our application running successfully!

Below are some good scripts that have been written specifically for hardening Alpine OS. Pick and choose depending on your programming language and harden your base alpine image accordingly.

On the flip-side you can also have a look at the distroless container created by Google, which is also very highly recommended.

Switching User Context

One might argue that if an attacker gains an RCE inside the container, they might not be able to install packages like curl,wget, etc. to establish their persistence.

However, we are still running as a “root” user, and technically it is still possible to install apk back.

Let’s re-run our docker container and check the privileges with which it is currently running.

docker exec -it springboot /bin/sh 
whoami 
ping rohitsalecha.com

Hence, it is important that we run our container not as root, but as a user with limited privileges.

Dockerfile.lpr shows the addition of a few more commands that add a user and group called “boot” and assign it a working directory (which is its home directory). I’ve also assigned numerical values to the user and group, which we’ll discuss in detail in the Pod Security Context Section.

FROM maven:3.8.1-openjdk-17-slim AS MAVEN_BUILD
WORKDIR /build/
COPY pom.xml /build/
COPY src /build/src/
RUN mvn package

FROM openjdk:17-alpine

# Removing apk package manager
RUN rm -f /sbin/apk && 
    rm -rf /etc/apk && 
    rm -rf /lib/apk && 
    rm -rf /usr/share/apk && 
    rm -rf rm -rf /var/lib/apk

# Adding a user and group called "boot"
RUN addgroup boot -g 1337 &&  
    adduser -D -h /home/boot -u 1337 -s /bin/ash boot -G boot

# Changing the context that shall run the below commands with User "boot" instead of root
USER boot
WORKDIR /home/boot

# By default even in a non-root context, Docker copies the file as root. Hence its best practice to chown
# the files being copied as the user. https://stackoverflow.com/a/44766666/1679541
COPY --chown=boot:boot --from=MAVEN_BUILD /build/target/springbootmaven.jar /home/boot/springbootmaven.jar
EXPOSE 8080
CMD java -jar /home/boot/springbootmaven.jar

Lets rebuild and re-rerun:

# First let's stop the previously running container
docker stop springboot 
# Next let's re-build and re-run
docker build . -f Dockerfile.lpr -t springbootmaven docker run --name springboot -d -p 8080:8080 springbootmaven curl http://localhost:8080

Now let’s try to run the whoami command and check which privileges with which container are now running.

docker exec -it springboot /bin/sh 
whoami 
ping rohitsalecha.com

Runtime Security

Now we’ve got a good level of confidence in the build-time security wherein we’ve learned to remove the packages and also update the user context to run the container with limited privileges. These security features are applied when we are building the docker container; however, we also need to focus on the security posture of the container when it is running in the Kubernetes environment, which we’ll explore below.

Before we start with securing our Kubernetes deployment, let’s run our application on our Kubernetes cluster by first pushing our docker container to hub.docker.com. You can use this guide to get started for the same.

docker build . -f Dockerfile.lpr -t springbootmaven
docker tag springbootmaven salecharohit/springbootmaven
docker push salecharohit/springbootmaven
docker run -d -p 8080:8080 --name springboot salecharohit/springbootmaven
curl http://localhost:8080

Now that our docker image is ready let’s apply our kubernetes-basic.yaml file that will deploy this application and also a service that will help us connect to it.

# Create Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: boot
---
# Create SpringBoot Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: springbootmaven
  template:
    metadata:
      labels:
        app: springbootmaven
    spec:
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
---
# Create Service for SpringBoot Deployment
apiVersion: v1
kind: Service
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  ports:
  - name: "http"
    port: 8080
    targetPort: 8080
  selector:
    app: springbootmaven

Next, let’s deploy our Kubernetes manifests using the below commands:

kubectl apply -f kubernetes-basic.yaml 
kubectl get deploy -n boot
# Run a temporary container that will only curl our bootservice
kubectl run -it testpod --image=radial/busyboxplus:curl --restart=Never --rm -- curl http://springbootmaven.boot.svc.cluster.local:8080 

Expected Output: 
Hello World From Spring Boot Build Using Maven on Alpine OS!pod "testpod" deleted

Service Account Tokens

If a Pod needs to communicate with the Kubernetes API-Server, it needs Service Account Tokens for authentication.

By default, every pod gets assigned a service account token, which is mounted on /var/run/secrets/kubernetes.io/serviceaccount/token. Let’s view this in practice by deploying our SpringBoot app.

kubectl get pods -n boot
kubectl exec -it springbootmaven-7d7c5c8597-mndv9 -n boot -- /bin/sh
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -k -H "Authorization:Bearer $TOKEN" https://kubernetes.docker.internal:6443/version

An RCE vulnerability on your application can leak this access token to the attacker which they can abuse to read-write resources in the same namespace or even have a global read permissions.

The resolution for this issue is two-fold depending upon the situation:

  1. Pods do not need any access to the API-Server.
  2. Pods need access to the API-Server.

Pods That Do Not Need Access to the API-Server

This situation is fairly simple to solve by simply adding two lines to the Kubernetes manifest file as shown below:

       serviceAccountName: ""
      automountServiceAccountToken: false

The complete deployment file kubernetes-nosa.yaml is as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: springbootmaven
  template:
    metadata:
      labels:
        app: springbootmaven
    spec:
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
      serviceAccountName: ""
      automountServiceAccountToken: false  

Let’s check if the service account token is now mounted or not.

# Ensure our previous deploy is deleted. 
kubectl delete ns boot

# Apply with no service account token
kubectl apply -f kubernetes-nosa.yaml
kubectl get pods -n boot
kubectl exec -it springbootmaven-5568b9874f-8nml8 -n boot -- /bin/sh
cat /var/run/secrets/kubernetes.io/serviceaccount/token

As can be seen above, the default service account token is no longer mounted.

Pods That Need Access to the API-Server

In this situation, we need to create a ServiceAccount, Role, and RoleBinding that map the ServiceAccount to the Role. The below Kubernetes manifest:

  • Creates a ServiceAccount called bootserviceaccount to a specific namespace, i.e. boot.
  • Creates a Role called bootservicerole with only privileges to view running pods.
  • Creates a RoleBinding called bootservicerolebinding.
  • Mounts the ServiceAccount, thus creates using the following lines in the Deployment.
    ---
          spec:
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
      serviceAccountName: bootserviceaccount
    ---

This shall allow to only read pods in the “boot” namespace.

The complete deployment file kubernetes-withsa.yaml is as follows:

# Create Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: boot
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bootserviceaccount
  namespace: boot
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: bootservicerole
  namespace: boot
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: bootservicerolebinding
  namespace: boot
subjects:
  - kind: ServiceAccount
    name: bootserviceaccount
    namespace: boot
roleRef:
  kind: Role
  name: bootservicerole
  apiGroup: rbac.authorization.k8s.io
---
# Create SpringBoot Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: springbootmaven
  template:
    metadata:
      labels:
        app: springbootmaven
    spec:
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
      serviceAccountName: bootserviceaccount
---
# Create Service for SpringBoot Deployment
apiVersion: v1
kind: Service
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  ports:
  - name: "http"
    port: 8080
    targetPort: 8080
  selector:
    app: springbootmaven

Let’s apply and check if our application is running fine.

# Ensure our previous deploy is deleted. 
kubectl delete ns boot 
kubectl apply -f kubernetes-withsa.yaml 
kubectl run -it testpod --image=radial/busyboxplus:curl --restart=Never --rm -- curl http://springbootmaven.boot.svc.cluster.local:8080

Pod Security Contexts

Though we’ve configured our base docker image to run with non-root privileges, there are still few more configurations that need to be added as security best practices. These are:

  1. Restricting the capabilities of the container and the pod
  2. Disabling Privilege Escalation
  3. Configuring the container to run with a specific uid/gid created earlier in our Dockerfile.lpr

In the Kubernetes manifest files, there are two types of “SecurityContexts” defined.

  • Running at Pod-Level, which will be applied to all containers running in this pod
      ---
      securityContext:
        fsGroup: 1337
        runAsNonRoot: true
        runAsUser: 1337
      containers:
      ---

  • Running at Container-level
      ---
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          runAsUser: 1337
          capabilities:
            drop: ["SETUID", "SETGID"]
      serviceAccountName: ""
      automountServiceAccountToken: false
      ---

The complete deployment file kubernetes-ps.yaml embedded with the PodSecurity contexts is below.

# Create Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: boot
---
# Create SpringBoot Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: springbootmaven
  template:
    metadata:
      labels:
        app: springbootmaven
    spec:
      securityContext:
        fsGroup: 1337
        runAsNonRoot: true
        runAsUser: 1337
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          runAsUser: 1337
          capabilities:
            drop: ["SETUID", "SETGID"]
      serviceAccountName: ""
      automountServiceAccountToken: false
---
# Create Service for SpringBoot Deployment
apiVersion: v1
kind: Service
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  ports:
  - name: "http"
    port: 8080
    targetPort: 8080
  selector:
    app: springbootmaven

Let’s apply and test if our application is running.

# Ensure our previous apply is deleted
kubectl delete ns boot 
kubectl apply -f kubernetes-ps.yaml 
kubectl run -it testpod --image=radial/busyboxplus:curl --restart=Never --rm -- curl http://springbootmaven.boot.svc.cluster.local:8080 
kubectl get pods -n boot 
kubectl exec -it springbootmaven-56c64ff85-mqz2z -n boot -- /bin/sh 
whoami 
id 
ping google.com

You can drop more capabilities as per your requirements. The complete list of capabilities can be found here.

Features like AppArmor or SecComp require additional configurations of the control plane components.  Therefore, I’ve limited my discussion to out-of-the-box features that can be easily activated and ensure a good level of security assurance.

Immutable File-Systems

Applications running in a containerized environment seldom write data, as that practically goes against the logic of having an immutable system. However, at times, it may be needed for caching or temporary swapping/processing of files. Hence, to provide this functionality to the developer, we can mount an emptyDir as an ephemeral volume which is lost once the container is killed.

With this in place, we can also add another security context attribute called “readOnlyRootFilesystem” and set it as true, since the application running inside the container no longer needs to write anywhere on the file-system other than the ‘tmp’ directory.

The above requirements can be configured as shown below.

 ---
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
        securityContext:
          readOnlyRootFilesystem: true
        volumeMounts:
        - mountPath: /tmp
          name: tmp
      volumes:
      - emptyDir: {}
        name: tmp
  ---

The complete deployment file kubernetes-rofs.yaml is as follows:

# Create Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: boot
---
# Create SpringBoot Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: springbootmaven
  template:
    metadata:
      labels:
        app: springbootmaven
    spec:
      securityContext:
        fsGroup: 1337
        runAsNonRoot: true
        runAsUser: 1337
      containers:
      - image: salecharohit/springbootmaven
        name: springbootmaven
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          privileged: false
          runAsUser: 1337
          capabilities:
            drop: ["SETUID", "SETGID"]
        volumeMounts:
        - mountPath: /tmp
          name: tmp
      serviceAccountName: ""
      automountServiceAccountToken: false
      volumes:
      - emptyDir: {}
        name: tmp

---
# Create Service for SpringBoot Deployment
apiVersion: v1
kind: Service
metadata:
  labels:
    app: springbootmaven
  name: springbootmaven
  namespace: boot
spec:
  ports:
  - name: "http"
    port: 8080
    targetPort: 8080
  selector:
    app: springbootmaven

Let’s apply and test if our application is running.

# Ensure our previous apply is deleted
kubectl delete ns boot 
kubectl apply -f kubernetes-rofs.yaml 
kubectl run -it testpod --image=radial/busyboxplus:curl --restart=Never --rm -- curl http://springbootmaven.boot.svc.cluster.local:8080 
kubectl get pods -n boot 
kubectl exec -it springbootmaven-56c64ff85-mqz2z -n boot -- /bin/sh
pwd
touch test.txt

Conclusion

We’ve learned what different controls we can embed in our containerized application, and also looked at how to enable run-time protection mechanisms that can at least make things difficult for an external attacker to gain a foothold into our containerized systems.

The kubernetes-rofs.yaml can serve as a good template for developers to containerize their applications with default security features enabled while deploying in a Kubernetes environment.

Of course, the Dockerfile needs to be created for the specific applications but for that purpose I’ve collected a few of them here.



Source link

Leave a Reply

Shopping cart

0
image/svg+xml

No products in the cart.

Continue Shopping